ERROR: Unable to perform a database vacuum: out of memory

[alonjr]

Hello,

I've been experimenting with onedrive to understand its behavior in various conditions so that I know what to expect once deployed.

In my Ubuntu 22.04 environment I am only looking into a one-way sync as a cloud backup for data on my server.
At some point in my experimentations running onedrive (in the command line, many are in dry-mode, and not yet via systemd) I started getting the message ...
An application configuration change has been detected where a --resync is required

Before trying a --resync I thought of clearing all state data (e.g. sqlite and .hash files) but that did not resolve the issue.

I then ran onedrive with a --resync, which prompted me to authorize with OneDrive service, after which I got the following error message ...

Application has not been successfully authorised. Please check your URI response entry and try again.
ERROR: Unable to perform a database vacuum: out of memory

I assume that there is no issue with the response URI because when I run onedrive with just the --reauth option I get an affirmative message ...
Application has been successfully authorised, however no additional command switches were provided.

Any troubleshooting tips would be greatly appreciate.

Cheers
Alon

[abraunegg]

@alonjr

In my Ubuntu 22.04 environment

OK .. but what application version (onedrive --version) did you install in your Crapbuntu Ubuntu environment? Why is it that Ubuntu users generally always miss providing this very important information point?

..
I am only looking into a one-way sync as a cloud backup for data on my server.
At some point in my experimentations running onedrive (in the command line, many are in dry-mode, and not yet via systemd) I started getting the message ...
An application configuration change has been detected where a --resync is required
Before trying a --resync I thought of clearing all state data (e.g. sqlite and .hash files) but that did not resolve the issue.

Please read the documentation more carefully.

When you change certain options you must perform a resync to ensure that the correct data is being synced. This performs the correct cleanup required.

I then ran onedrive with a --resync, which prompted me to authorize with OneDrive service, after which I got the following error message ...

Application has not been successfully authorised. Please check your URI response entry and try again.
ERROR: Unable to perform a database vacuum: out of memory

This probably comes down to a number of items:

1. You are using Ubuntu. As such you need to install the correct application version. As you are experimenting, if you are able to, dont use Ubuntu and use a better distribution.
2. You do not have enough system memory to process the command on exit. All database operations are run in memory and need to be written to disk on exit - so your system might not have enough free memory to perform this function. This is not something that I can solve for as this is a hardware issue.
3. You are using --dry-run and you performed a manual cleanup - so the application was unable to read and/or process what was required, resulting in the error generated.

Tips to solve:

1. Use the correct application version
2. Dont use Ubuntu if you have an option to use something else
3. Dont manually delete stuff - follow what the application is asking you
4. Authorise the application first without --dry-run - so that the authentication state is confirmed
5. Run your tests / configuration tests as per your requirement with --dry-run. If the application then asks you to perform a --resync - do so with --dry-run .... using: --resync --dry-run

[alonjr]

@abraunegg, thank you for your input. I sense that you have a strong feeling against ubuntu, I respect that.

I followed the install and configuration instructions carefully. I'm running onedrive v2.4.23-1+np1.
I don't see any issue with memory running out, therefore I was a little lost on that. But I will refrain from going nuclear with manual cleanup.

i'll report back if I find anything that could contribute to this project.

cheers
Alon

[maximilianpalm]

Hey @alonjr ,
same issue here. I first installed the onedrive package provided by ubuntu. Later I found out that the only version supported by Microsoft is the one installed from their description, so I installed that. Now I'm running onedrive v2.4.23-1+np1

First I tried this:

$ onedrive --verbose --resync
[...]
Are you sure you wish to proceed with --resync? [Y/N] Y

And got this: ERROR: Unable to perform a database vacuum: out of memory, with 26 GB RAM and ~30 GB Swap available. So I think there might be some room for improvement in the error handling and reporting.

Anyway, the following command worked for me:

$ onedrive --verbose --resync --synchronize
[...]
Sync with OneDrive is complete

PS: Why not just provide a decent package? Why is this still so complicated? Why is there no official GUI? Sorry, just ranting here...
PPS: I will probably say this every time I get a chance from now on: forcing the Windows naming conventions on anyone using OneDrive on any operating system is a big bummer for me.

[abraunegg]

@maximilianpalm

same issue here. I first installed the onedrive package provided by ubuntu.

Using the Ubuntu packages is 100% not supported as they contain bugs and issues that have been already fixed, this is why the OpenSuSE Repository exists.

This is called out in a number of places:
readme.md

install.md

and

Later I found out that the only version supported by Microsoft is the one installed from their description

Microsoft do not officially support Linux for OneDrive. The access of Microsoft OneDrive on Linux platforms is performed by the availability of the OneDrive API. This client is not written by or supported by Microsoft. It is an Open Source application.

$ onedrive --verbose --resync

OK ... if this is the command that generated the error - that should be easily reproducible, however, as per the usage.md document, you failed to include either --synchronize or --monitor on your command line ... to perform a sync operation one of those must be present.

Please read the documentation.

Now - as to the actual full error message you would have received:

onedrive --confdir '~/.config/onedrive-personal/' --verbose --resync

Using 'user' Config Dir: /home/alex/.config/onedrive-personal/
Configuration file successfully loaded

The use of --resync will remove your local 'onedrive' client state, thus no record will exist regarding your current 'sync status'
This has the potential to overwrite local versions of files with potentially older versions downloaded from OneDrive which can lead to data loss
If in-doubt, backup your local data first before proceeding with --resync

Are you sure you wish to proceed with --resync? [Y/N] y

Deleting the saved application sync status ...
Checking Application Version ...
Initializing the OneDrive API ...
Configuring Global Azure AD Endpoints
Using Curl defaults for all HTTP operations

--synchronize or --monitor switches missing from your command line input. Please add one (not both) of these switches to your command line or use 'onedrive --help' for further assistance.

No OneDrive sync will be performed without one of these two arguments being present.


ERROR: Unable to perform a database vacuum: out of memory

What is happening is that because you did not add the right switches in, the application did not initialise the DB, thus it cannot be vacuumed .... so an application exit bug that can be easily fixed.

PS: Why not just provide a decent package?

Please clarify what you mean by a decent package. A client that works, that is relatively bug-free, stable and maintained would indicate in my book a decent package. What does a 'decent' package mean to you?

Why is this still so complicated?

Please tell me what you find so complicated?

Why is there no official GUI?

There is .. but I am guessing because you are using Crapbuntu Ubuntu you are just expecting something to be there.

Please re-read the documentation - especially readme.md to find the details regarding the GUI for this client.

PPS: I will probably say this every time I get a chance from now on: forcing the Windows naming conventions on anyone using OneDrive on any operating system is a big bummer for me.

Please take this grievance away from here - this is not this projects imposition, but it is @microsoft who through the OneDrive API and the OneDrive platform have imposed these limitations (features) because of how they built the platform.

[luca-ing]

$ onedrive --verbose --resync

OK ... if this is the command that generated the error - that should be easily reproducible, however, as per the usage.md document, you failed to include either --synchronize or --monitor on your command line ... to perform a sync operation one of those must be present.

I fell into that same trap.

I wish there was an error message telling me that I'm missing a required option (to a newbie, --resync and --synchronize feel the same), and not something about the authentication not being successful (which, as far as I can tell, it actually was).

[abraunegg]

@luca-ing

I wish there was an error message telling me that I'm missing a required option

The client 100% outputs that sort of message - for example:

v2.4.25 Client Version

alex@debian-12-x64:~/onedrive-2.4.25-master$ ./onedrive --verbose --resync
Using 'user' Config Dir: /home/alex/.config/onedrive
Using 'system' Config Dir: /etc/onedrive
Configuration file successfully loaded

The use of --resync will remove your local 'onedrive' client state, thus no record will exist regarding your current 'sync status'
This has the potential to overwrite local versions of files with potentially older versions downloaded from OneDrive which can lead to data loss
If in-doubt, backup your local data first before proceeding with --resync

Are you sure you wish to proceed with --resync? [Y/N] y

config file has been updated, checking if --resync needed
Deleting the saved application sync status ...
Checking Application Version ...
Initializing the OneDrive API ...
Configuring Global Azure AD Endpoints
Using Curl defaults for all HTTP operations

--synchronize or --monitor switches missing from your command line input. Please add one (not both) of these switches to your command line or use 'onedrive --help' for further assistance.

No OneDrive sync will be performed without one of these two arguments being present.

alex@debian-12-x64:~/onedrive-2.4.25-master$ 

v2.5.x RC2|RC3

alex@debian-12-x64:~/onedrive-v.2.5.0-RC3-dev$ ./onedrive --verbose --resync
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application state data: /home/alex/.config/onedrive

The usage of --resync will delete your local 'onedrive' client state, thus no record of your current 'sync status' will exist.
This has the potential to overwrite local versions of files with perhaps older versions of documents downloaded from OneDrive, resulting in local data loss.
If in doubt, backup your local data before using --resync

Are you sure you wish to proceed with --resync? [Y/N] y

Deleting the saved application sync status ...
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
Application Version:  onedrive v2.5.0-rc2-25-g71bce90
Account Type:         personal
Default Drive ID:     66d53be8a5056eca
Default Root ID:      66D53BE8A5056ECA!101
Remaining Free Space: 4.90 GB (5261334937 bytes)
Sync Engine Initialised with new Onedrive API instance

Your command line input is missing either the '--sync' or '--monitor' switches. Please include one (but not both) of these switches in your command line, or refer to 'onedrive --help' for additional guidance.

It is important to note that you must include one of these two arguments in your command line for the application to perform a synchronisation with Microsoft OneDrive

alex@debian-12-x64:~/onedrive-v.2.5.0-RC3-dev$

If you are not at least getting the v2.4.25 output you are running the wrong client version ... ensure you are running the correct client version please.

[luca-ing]

[luca@godzilla accounts]$ onedrive --version
onedrive v2.4.25

[luca@godzilla accounts]$ onedrive -v --confdir ~/.config/onedrive/accounts/luca.ingianni\@vempio.de-2/ --resync

[...]

Application has been successfully authorised, however no additional command switches were provided.

Please use 'onedrive --help' for further assistance in regards to running this application.


ERROR: Unable to perform a database vacuum: out of memory

[luca@godzilla accounts]$

I recognise it now: I probably also had --dry-run in the command line that resulted in unsuccessful signup, which is slightly surprising, but documented.

Interestingly, the message in your output was different despite identical version number. What you showed me is far more helpful than what I saw, so thank you very much for improving it. Tis is a wonderfully helpful message. Had I seen this, I wouldn't have been confused.

Maybe you could add a similarly helpful message in the case of authentication combined with--dry-run, as this feels like a common situation for someone new easing their way into using your tool.

[abraunegg]

All messages should be the same irrespective of the use of --dry-run

[luca-ing]

Unfortunately, they are not:

[luca@godzilla accounts]$ onedrive -v --confdir ~/.config/onedrive/accounts/luca.ingianni\@vempio.de-3/ --resync --dry-run
Using 'user' Config Dir: /home/luca/.config/onedrive/accounts/[email protected]/
No user or system config file found, using application defaults

The use of --resync will remove your local 'onedrive' client state, thus no record will exist regarding your current 'sync status'
This has the potential to overwrite local versions of files with potentially older versions downloaded from OneDrive which can lead to data loss
If in-doubt, backup your local data first before proceeding with --resync

Are you sure you wish to proceed with --resync? [Y/N] y

DRY-RUN Configured. Output below shows what 'would' have occurred.
Deleting the saved application sync status ...
Checking Application Version ...
Initializing the OneDrive API ...
Configuring Global Azure AD Endpoints
Using Curl defaults for all HTTP operations
Authorize this app visiting:

[...]


Application has not been successfully authorised. Please check your URI response entry and try again.


ERROR: Unable to perform a database vacuum: out of memory

[luca@godzilla accounts]$ 

[abraunegg]

@luca-ing
Let me clarify my statement:

If you are authenticating the client for the first time - like you are - and using --dry-run the application will hit that error message Application has not been successfully authorised ... first because the client is not in an authenticated state. The application checks to see if the auth process completed successfully first before processing CLI options.

When the client is in an authenticated state, all messages should be the same irrespective of the use of --dry-run