Run is now failing

[melat0nin]

The past few days my runs have been failing, with the follow (not very helpful) output:

It may take some time...

1170 Zotero items are retrieved.
ZOTERO: Start formatting 1170 annotations/notes...
It may take some time depending on the number of annotations...
A complete message will show up once it's done!


ZOTERO: Formatting Zotero Items is completed!!


NOTE: 1170 Zotero annotations/notes (out of 1170) failed to format.
You can run `save_failed_items_to_json()` class method to save those items.

ZOTERO: Detail of failed items are saved into /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/zotero2readwise/failed_zotero_items.json


Readwise: Push 0 Zotero annotations/notes to Readwise...
It may take some time depending on the number of highlights...
A complete message will show up once it's done!

Traceback (most recent call last):
  File "run.py", line 64, in <module>
    zt2rw.run()
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/zotero2readwise/zt2rw.py", line 55, in run
    self.readwise.post_zotero_annotations_to_readwise(formatted_items)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/zotero2readwise/readwise.py", line 149, in post_zotero_annotations_to_readwise
    self.create_highlights(rw_highlights)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/zotero2readwise/readwise.py", line 72, in create_highlights
    raise Zotero2ReadwiseError(
zotero2readwise.exception.Zotero2ReadwiseError: Uploading to Readwise failed with following details:
POST request Status Code=400 (Bad Request)
Error log is saved to error_log_400_failed_post_request_to_readwise.json file.
Error: Process completed with exit code 1.

Any thoughts on what the issue is? Looks like the issue with with formatting, which suggests a problem with the Zotero API. But it's clearly connecting, if it's able to grab 1170 items.

[Mela]

I'm having the same problem. In the run's log I'm getting the message:

The following actions uses node12 which is deprecated and will be forced to run on node16: actions/setup-python@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/

[e-alizadeh]

@melat0nin It seems to me that the issue in your case lies in retrieving the data from Zotero server.
Can you follow the instructions here to generate a local file. It may give us extra information.

I probably need to improve the logging since the reason I believe Readwise fails is that no Zotero highlight was retrieved in the first place. But generating both error_log_400_failed_post_request_to_readwise.json and failed_zotero_items.json would be very helpful here.

Have you had any changes or updates in your Zotero configuration?

[e-alizadeh]

But one thing that I should do is to have better logs and output them in the automation!

[pcuellar]

@e-alizadeh thanks for the work on this automation. Will you be moving the workflow to node16 or should we stick to node12 while still active?

Update: I tried forcing node12 through env and wasn't able to. I wonder if it's not available anymore?

[e-alizadeh]

@pcuellar @Mela You can simply update line 20 of the automation.yml file from v2 to v4 to suppress that warning. You can check this commit.

The other change in the commit about pinpointing the version in line 15 is highly recommended.

[melat0nin]

@e-alizadeh I finally ran the script locally, and this is the full content of error_log_400_failed_post_request_to_readwise.json:

{"non_field_errors": ["This list may not be empty."]}

failed_zotero_items.json contains just a JSON object with my account metadata and all the items from my Zotero library, so I'm not sure there will be anything of use in there.

Any thoughts? (For the Github automation I have updated automation.yml from your previous comment, but that is obviously a different concern)

[e-alizadeh]

@melat0nin It's difficult to know exactly what's going on without knowing exactly which highlight or annotation causing it.
I've created an issue e-alizadeh/Zotero2Readwise#66 to add the raw highlights to the error logs.
Once this is implemented, we can better debug this issue. FYI, I won't be able to work on this until next week!

[thelivingphilosophy]

I had a similar problem to this a few weeks back. In the end it turned out that I had an empty note — I'd highlighted went to add a note and then not written anything. Once I deleted that it ran again. Not sure if this is the same issue but thought I'd share just in case

[Mela]

I had a similar problem to this a few weeks back. In the end it turned out that I had an empty note — I'd highlighted went to add a note and then not written anything. Once I deleted that it ran again. Not sure if this is the same issue but thought I'd share just in case

Ugh. Absolutely possible, that this is the problem in my case, too. Just: How did you find it? Is there an option to show all hightlights and notes in Zotero?

[melat0nin]

@e-alizadeh I've just realised that I should have mentioned that I'm using the Zotero 7 beta, which I changed to in the past few weeks (around the time the error started, though I can't be absolutely sure).

I note that the error I get seems to refer to all highlights (NOTE: 1170 Zotero annotations/notes (out of 1170) failed to format.). Might that imply some difference in formatting/JSON between Zotero 6 and 7?

[thelivingphilosophy]

Ugh. Absolutely possible, that this is the problem in my case, too. Just: How did you find it? Is there an option to show all hightlights and notes in Zotero?

Not that I know of @Mela. I'm actually having a similar issue again myself which is what brought me back to these issues. At that time I was just getting started with Zotero and I remember having that weird note. If you go into a pdf in Zotero there's a left sidebar (at least on the desktop) which shows you all of the annotations and highlights in that document

[Mela]

Just synced my repository and sync works again! Thank you @e-alizadeh!

[pcuellar]

Just reporting I started having this issue too. I recently moved to Zotero 7 Beta, maybe it does have something to do with the way things are formatte as @melat0nin mentioned