fix: cleanup log output #345

mkellerman · 2024-07-06T02:44:03Z

Fix for #272

Removal of unneeded lines from autosens output
Removal of empty lines throughout console output
Added the context of the logs (ie: prepare/autosens.js, Middleware)

Reference to previous PR #331

New output:

OpenAPS.swift - autosense() - 305 DEV: Start autosens
JavaScriptWorker.swift - outputLogs() - 76 DEV: prepare/autosens.js: Using most recent,104,deviations since,Thu Jul 04 2024 19:37:44 GMT-0700 (Pacific Daylight Saving Time)
JavaScriptWorker.swift - outputLogs() - 76 DEV: prepare/autosens.js: 47% of non-meal deviations positive (>50% = resistance)
JavaScriptWorker.swift - outputLogs() - 76 DEV: prepare/autosens.js: 41% of non-meal deviations negative (>50% = sensitivity)
JavaScriptWorker.swift - outputLogs() - 76 DEV: prepare/autosens.js: RMS deviation: 6.68
JavaScriptWorker.swift - outputLogs() - 76 DEV: prepare/autosens.js: Sensitivity normal.
JavaScriptWorker.swift - outputLogs() - 76 DEV: prepare/autosens.js: ISF adjusted from 54 to 54
JavaScriptWorker.swift - outputLogs() - 76 DEV: prepare/autosens.js: Using most recent,288,deviations since,Thu Jul 04 2024 19:37:44 GMT-0700 (Pacific Daylight Saving Time)
JavaScriptWorker.swift - outputLogs() - 76 DEV: prepare/autosens.js: 39% of non-meal deviations positive (>50% = resistance)
JavaScriptWorker.swift - outputLogs() - 76 DEV: prepare/autosens.js: 47% of non-meal deviations negative (>50% = sensitivity)
JavaScriptWorker.swift - outputLogs() - 76 DEV: prepare/autosens.js: RMS deviation: 6.29
JavaScriptWorker.swift - outputLogs() - 76 DEV: prepare/autosens.js: Sensitivity normal.
JavaScriptWorker.swift - outputLogs() - 76 DEV: prepare/autosens.js: ISF adjusted from 54 to 54
OpenAPS.swift - autosense() - 321 DEV: AUTOSENS: {
    "ratio": 1,
    "newisf": 54
}
OpenAPS.swift - determineBasal(currentTemp:clock:) - 21 DEV: Start determineBasal
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: Middleware reason: Nothing changed
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: Pumphistory is empty!
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: insulinFactor set to : 55
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: Using weighted TDD average: 1 U, instead of past 24 h (0 U), weight: 0.65
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: , Dynamic ratios log: Dynamic Settings disabled
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: Default Half Basal Target used: 8.9 mmol/L
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: Autosens ratio: 1;
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: Threshold set to 3.9
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: ISF unchanged: 3
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: CR:10
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: currenttemp:0 lastTempAge:28670558m, tempModulus:8m
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: SMB disabled (no enableSMB preferences active or no condition satisfied)
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: profile.sens:3, sens:3, CSF:5.4
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: Carb Impact:4.1mg/dL per 5m; CI Duration:0hours; remaining CI (1.5h peak):0mg/dL per 5m
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: UAM Impact:4.1mg/dL per 5m; UAM Duration:0.2hours
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: minPredBG: 260 minIOBPredBG: 260 minZTGuardBG: 237
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: avgPredBG:260 COB/Carbs:0/0
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: BG projected to remain above 5.5 for 240minutes
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: naive_eventualBG:,237,bgUndershoot:,-167.5,zeroTempDuration:,240,zeroTempEffect:,216,carbsReq:,-71
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: NaN (. insulinReq: 2.98 U)
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: NaN (. insulinForManualBolus: 3 U)
JavaScriptWorker.swift - outputLogs() - 76 DEV: Middleware: Setting neutral temp basal of 1U/hr
OpenAPS.swift - determineBasal(currentTemp:clock:) - 80 DEV: SUGGESTED: {
    "temp": "absolute",
    "bg": 237,
...

mkellerman · 2024-07-06T20:13:34Z

@marionbarker would you mind running your scripts to see the diff in log size? Thanks!

marionbarker · 2024-07-07T14:15:05Z

I applied this patch about a day and a half ago to latest Trio - dev (at the time).
Then I got busy.
Attached is a full 24-hour log.

iPhone SE running iOS 17.5.1 with rPi DASH simulator and using a test Nightscout site as a CGM (uploaded some significant glucose excursions).
My python Omnipod parser works fine with this file. (It automatically renames the log_prev files with the date)

20240706_log_prev.txt

wc reports: 66773 510135 5702930 20240706_log_prev.txt
- for those who don't speak unix: the numbers are lines, words, characters
So per hour: 2782 lines; 237k characters.

From this comment, the prior art was about 3.3k lines/hour, 288k characters/hour.

The graphic is the NS site used with this test phone.

marionbarker

Giving my approval based on the fact it works for providing record of Omnipod messages.

Someone else more familiar with these logs should examine the result for other content.

mkellerman · 2024-07-07T14:31:28Z

I recommend a Squash & Merge. My history is a lot of trial and error.

mountrcg · 2024-07-07T19:26:48Z

I will have a look at the logs tonight

mountrcg · 2024-07-10T05:10:33Z

looking at the log that Marion attached

there still are still a lot of lines for autosens

middleware actually only outputs the first green boxed line, the remaining is determine-basal output - you could call that Algorithm or oref

mountrcg · 2024-07-11T05:21:04Z

@mkellerman The content pf the logging is totally readable and fine with me. I suggest that you give the green marked part another context e.g. like oref, but not middleware.
I am not sure whether you did not want to achieve more congestions with autosens like putting it all in one long line as Dan suggested - or is this done in the second PR?

mkellerman · 2024-07-12T22:14:51Z

@mountrcg, i'm unable to reproduce the autosens logs like you. Mine get trimmed correctly. Maybe something to do with Timezone/Language formatting?

Regarding 'Middleware', this is autogenerated by how we call the JavaScriptWorker... I didn't name these individually.

mkellerman · 2024-07-12T22:49:16Z

@marionbarker @mountrcg posted a new fix that might resolve the issue.

marionbarker · 2024-07-18T19:26:21Z

Summary:

Log files and time stamps are provided for someone else to review.
The message parser works fine with this code applied.

Configuration:

Starting configuration (yesterday) had both this PR (345) and PR 353 installed on top of most recent dev, commit 9672da25
This morning rebuild to each phone with the PR 353 patch applied but without this PR (remove the logging changes)

Medtronic Log:

This log had PR 345 installed until 06:50 local time
Use search term is DEV: Trio Started:
20240718_MDT-log.txt

Omnipod DASH Log:

This log had PR 345 installed until 06:48 local time
Use search term is DEV: Trio Started:
20240718_OmniPod-DASH-log.txt

The Omnipod messages are not affected by the logging changes, the parser still works for code both with and without PR 346 applied.

mkellerman · 2024-07-19T00:15:11Z

Thanks @marionbarker ! I dont get any of those u(xxxx) in my logs, so it's really hard to debug this one on my own. But i did fix the regex to include this pattern. it should be fixed now.

marionbarker · 2024-07-19T12:57:47Z

Downloaded new patch and built to

Medtronic Phone at 2024-07-18 21:37 local
Pod Phone at 2024-07-18 21:40 local

Both phones are also running changes from PR 353.

Below are the log files (so since midnight local on 2024-07-19) acquired around 05:50 am

MDT:

2024-07-19-Trio-MDT-with-prs-345-353-log.txt

Pod:

The pod one had a failure of the rPi simulator at 2024-07-19T05:32 (nothing to do with Trio, it just happens sometimes)
Resumed the rPi a few minutes before issuing the log
Omnipod message parser works fine with the file
2024-07-19-Trio-DASH-with-prs-345-353-log.txt

mkellerman · 2024-07-19T17:15:22Z

MDT logs:

024-07-19T00:52:38-0700 [OpenAPS] OpenAPS.swift - makeProfiles(useAutotune:) - 376 - DEV: Start makeProfiles
2024-07-19T00:52:38-0700 [OpenAPS] OpenAPS.swift - autosense() - 305 - DEV: Start autosens
2024-07-19T00:52:39-0700 [Nightscout] TidepoolManager.swift - uploadDose() - 305 - DEV: Success synchronizing Dose data:
2024-07-19T00:52:40-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: Using most recent,35,deviations since,Thu Jul 18 2024 00:52:38 GMT-0700 (Pacific Daylight Time
2024-07-19T00:52:40-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: Adding,11,more zero deviations
2024-07-19T00:52:40-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: 29% of non-meal deviations positive (>50% = resistance
2024-07-19T00:52:40-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: 23% of non-meal deviations negative (>50% = sensitivity
2024-07-19T00:52:40-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: RMS deviation: 1.29
2024-07-19T00:52:40-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: Sensitivity normal.
2024-07-19T00:52:40-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: ISF adjusted from 70 to 70
2024-07-19T00:52:40-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: Using most recent,35,deviations since,Thu Jul 18 2024 00:52:39 GMT-0700 (Pacific Daylight Time
2024-07-19T00:52:40-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: Adding,11,more zero deviations
2024-07-19T00:52:40-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: 29% of non-meal deviations positive (>50% = resistance
2024-07-19T00:52:40-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: 23% of non-meal deviations negative (>50% = sensitivity
2024-07-19T00:52:40-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: RMS deviation: 1.29
2024-07-19T00:52:40-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: Sensitivity normal.
2024-07-19T00:52:40-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: ISF adjusted from 70 to 70
2024-07-19T00:52:40-0700 [OpenAPS] OpenAPS.swift - autosense() - 321 - DEV: AUTOSENS: {
    "ratio": 1,
    "newisf": 70
}
2024-07-19T00:52:40-0700 [OpenAPS] OpenAPS.swift - determineBasal(currentTemp:clock:) - 21 - DEV: Start determineBasal

DASH logs:

2024-07-19T00:26:28-0700 [OpenAPS] OpenAPS.swift - makeProfiles(useAutotune:) - 376 - DEV: Start makeProfiles
2024-07-19T00:26:28-0700 [OpenAPS] OpenAPS.swift - autosense() - 305 - DEV: Start autosens
2024-07-19T00:26:30-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: Using most recent,97,deviations since,Thu Jul 18 2024 00:26:28 GMT-0700 (Pacific Daylight Time
2024-07-19T00:26:30-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: 77% of non-meal deviations positive (>50% = resistance
2024-07-19T00:26:30-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: 17% of non-meal deviations negative (>50% = sensitivity
2024-07-19T00:26:30-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: RMS deviation: 3.77
2024-07-19T00:26:30-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: Insulin resistance detected:
2024-07-19T00:26:30-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: Ratio limited from 1.5019607843137255 to 1.2
2024-07-19T00:26:30-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: ISF adjusted from 72 to 60
2024-07-19T00:26:30-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: Using most recent,192,deviations since,Thu Jul 18 2024 00:26:29 GMT-0700 (Pacific Daylight Time
2024-07-19T00:26:30-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: 72% of non-meal deviations positive (>50% = resistance
2024-07-19T00:26:30-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: 22% of non-meal deviations negative (>50% = sensitivity
2024-07-19T00:26:30-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: RMS deviation: 4.12
2024-07-19T00:26:30-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: Insulin resistance detected:
2024-07-19T00:26:30-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: Ratio limited from 1.380392156862745 to 1.2
2024-07-19T00:26:30-0700 [OpenAPS] JavaScriptWorker.swift - outputLogs() - 78 - DEV: prepare/autosens.js: ISF adjusted from 72 to 60
2024-07-19T00:26:30-0700 [OpenAPS] OpenAPS.swift - autosense() - 321 - DEV: AUTOSENS: {
    "ratio": 1.2,
    "newisf": 60
}
2024-07-19T00:26:30-0700 [OpenAPS] OpenAPS.swift - determineBasal(currentTemp:clock:) - 21 - DEV: Start determineBasal

mkellerman · 2024-07-19T17:16:19Z

I think this does the expected behaviour! Thanks @marionbarker for all the testing!

marionbarker

I can attest that the Omnipod messages come through fine after this update and the parser interprets them.
Someone else needs to agree the logs are otherwise OK.
Both MDT and DASH logs are posted in the comments.

mkellerman added 7 commits June 15, 2024 23:41

fix: Aggregate Javascript logs

0468f57

fix: refactored to a more generic log parsing method

037656f

fix: formatting

e64189f

Fix: comments

01e0ffd

refactored to simpler formatter and cleanup autosens

85bf02a

fix: output unknown outputs as regular log lines

0415460

chore: rename and cleanup some of the variables

14f0b4a

mkellerman requested review from dnzxy, bjornoleh, MikePlante1, aug0211, AndreasStokholm, Sjoerd-Bo3 and t1dude as code owners July 6, 2024 02:44

marionbarker previously approved these changes Jul 7, 2024

View reviewed changes

fix: autosens logs to be evaluated line by line

dd38115

mkellerman dismissed marionbarker’s stale review via dd38115 July 12, 2024 22:48

Merge remote-tracking branch 'upstream/dev' into fix-autosens-logs

81cef4a

fix regex to include u(x)

a77b123

mkellerman requested a review from marionbarker July 19, 2024 17:16

marionbarker approved these changes Jul 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: cleanup log output #345

fix: cleanup log output #345

mkellerman commented Jul 6, 2024

mkellerman commented Jul 6, 2024

marionbarker commented Jul 7, 2024

marionbarker left a comment

mkellerman commented Jul 7, 2024

mountrcg commented Jul 7, 2024

mountrcg commented Jul 10, 2024

mountrcg commented Jul 11, 2024

mkellerman commented Jul 12, 2024

mkellerman commented Jul 12, 2024

marionbarker commented Jul 18, 2024

mkellerman commented Jul 19, 2024

marionbarker commented Jul 19, 2024

mkellerman commented Jul 19, 2024

mkellerman commented Jul 19, 2024

marionbarker left a comment

fix: cleanup log output #345

Are you sure you want to change the base?

fix: cleanup log output #345

Conversation

mkellerman commented Jul 6, 2024

mkellerman commented Jul 6, 2024

marionbarker commented Jul 7, 2024

marionbarker left a comment

Choose a reason for hiding this comment

mkellerman commented Jul 7, 2024

mountrcg commented Jul 7, 2024

mountrcg commented Jul 10, 2024

mountrcg commented Jul 11, 2024

mkellerman commented Jul 12, 2024

mkellerman commented Jul 12, 2024

marionbarker commented Jul 18, 2024

Summary:

Configuration:

Medtronic Log:

Omnipod DASH Log:

mkellerman commented Jul 19, 2024

marionbarker commented Jul 19, 2024

mkellerman commented Jul 19, 2024

mkellerman commented Jul 19, 2024

marionbarker left a comment

Choose a reason for hiding this comment