Fix issue "returned non-zero exit status 52: b" on MCW data site

[tschaffter]

Submissions:

• 9713417
• 9713423

The logs generated by the controller show:

"type": "PROGRESS NOTES", "patientId": "1192", "note_name": "dataset/mcw-phi-20210608/fhirStores/evaluation/fhir/Note/1192"}}']' in image 'curlimages/curl:7.73.0' returned     non-zero exit status 52: b'

Interestingly, I encountered this post that says that 52 is an error code if Apache Spark, which I use, associated to Out Of Memory.

However, the above logs is associated to a curl command. I found this comment online:

After some investigation, we found when
curl returns error code 52, it is because Apache server gets seg fault.
It seems not to be a problem with curl 7.9.5 either because I installed
another curl 7.9.5 on my own PC (linux) and I run it to call the same
web page and so far it works fine.

Did the tool crashed?

Other posts mention that this error is associated to "curl: (52) Empty reply from server".

Also:

When you get error 52 back, it literally means "Empty reply from server". When a client (like curl) speaks HTTP to a server, that server must send something back for it to be legitimate HTTP. Protocol-wise, there is no way to be compliant and not respond anything.

Error 52 means the server did just that. It didn't respond with a single byte of data (and then closed the connection). No server in good health should ever do this.

@gkowalski I'm wondering whether the tool crashed, which may have led to curl receiving an empty response. This is way having a look at the MCW logs for the tool could help to identify if the tool crashed.

[tschaffter]

The tool annotated 193 notes then died during the submission 9713417.

From 9713419_fail.csv.zip:

"
[Stage 191:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 186/186] X.X.X.X () {32 vars in 432 bytes} [Fri Aug  6 17:51:50 2021] POST /api/v1/textPersonNameAnnotations => generated 149 bytes in 1162 msecs (HTTP/1.1 200) 2 headers in 72 bytes (1 switches on core 0)"
"
[Stage 192:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 187/187] X.X.X.X () {32 vars in 432 bytes} [Fri Aug  6 17:51:52 2021] POST /api/v1/textPersonNameAnnotations => generated 562 bytes in 2165 msecs (HTTP/1.1 200) 2 headers in 72 bytes (1 switches on core 0)"
"
[Stage 193:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 188/188] X.X.X.X () {32 vars in 432 bytes} [Fri Aug  6 17:51:54 2021] POST /api/v1/textPersonNameAnnotations => generated 258 bytes in 1270 msecs (HTTP/1.1 200) 2 headers in 72 bytes (1 switches on core 0)"
"
[Stage 194:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 189/189] X.X.X.X () {32 vars in 432 bytes} [Fri Aug  6 17:51:56 2021] POST /api/v1/textPersonNameAnnotations => generated 470 bytes in 7631 msecs (HTTP/1.1 200) 2 headers in 72 bytes (1 switches on core 0)"
"
[Stage 195:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 190/190] X.X.X.X () {32 vars in 432 bytes} [Fri Aug  6 17:52:04 2021] POST /api/v1/textPersonNameAnnotations => generated 351 bytes in 6201 msecs (HTTP/1.1 200) 2 headers in 72 bytes (1 switches on core 0)"
"
[Stage 196:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 191/191] X.X.X.X () {32 vars in 432 bytes} [Fri Aug  6 17:52:11 2021] POST /api/v1/textPersonNameAnnotations => generated 663 bytes in 1790 msecs (HTTP/1.1 200) 2 headers in 72 bytes (1 switches on core 0)"
"
[Stage 197:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 192/192] X.X.X.X () {32 vars in 431 bytes} [Fri Aug  6 17:52:13 2021] POST /api/v1/textPersonNameAnnotations => generated 38 bytes in 1383 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)"
"
[Stage 198:==========================================>              (3 + 1) / 4]

[Stage 198:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 193/193] X.X.X.X () {32 vars in 433 bytes} [Fri Aug  6 17:52:15 2021] POST /api/v1/textPersonNameAnnotations => generated 38 bytes in 65091 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)"
SIGINT/SIGQUIT received...killing workers...
gateway "uWSGI http 1" has been buried (pid: 77)
Fri Aug  6 18:05:11 2021 - uWSGI worker 1 screams: UAAAAAAH my master disconnected: i will kill myself !!!

Second submission that failed:

From 9713424_fail.csv.zip:

"
[Stage 198:==========================================>              (3 + 1) / 4]

[Stage 198:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 193/193] X.X.X.X () {32 vars in 433 bytes} [Fri Aug  6 19:47:33 2021] POST /api/v1/textPersonNameAnnotations => generated 38 bytes in 63493 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)"
SIGINT/SIGQUIT received...killing workers...
gateway "uWSGI http 1" has been buried (pid: 77)
Fri Aug  6 19:50:12 2021 - uWSGI worker 1 screams: UAAAAAAH my master disconnected: i will kill myself !!!

[tschaffter]

Interestingly both submissions failed at/after the same request (193), which is before the end of the MCW dataset (424 notes). The tool logs don't show an error, just SIGINT/SIGQUIT received...killing workers....

[tschaffter]

@thomasyu888 Given the above information, are there cases where event outside of the tool - for example from the controller - that could kill the tool?

[thomasyu888]

@tschaffter Not that I know of, unless the tool reached max memory allotted to it by the docker run command.