mongoDB Unsupported OP_QUERY command

[jrobinedwards]

I have installed cygnus on Centos 9.
It has MongoDB 7.0.5 running on localhost

Cygnus-ngsi was installed using the documentation at this url:
https://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/installation_and_administration_guide/install_with_rpm.html

Cygnus version from the endpoint /v1/version is "1.9.0.b5547662c0223268bacb8f220911aa7e9bd24131"

I am trying to persist data in mongodb but the cygnus logs show this error:
time=2024-03-04T13:15:41.009Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=persistAggregation | msg=com.telefonica.iot.cygnus.sinks.NGSIMongoSink[413] : [mongo-sink] Persisting data at NGSIMongoSink. Database: sth_default, Collection: sth_/_0004A30B00220604_AirQualitySensor, Data: [Document{{recvTime=Mon Mar 04 13:12:31 UTC 2024, attrName=LAT, attrType=Float, attrValue=64.745788574}}, Document{{recvTime=Mon Mar 04 13:12:31 UTC 2024, attrName=LON, attrType=Float, attrValue=20.959072113}}, Document{{recvTime=Mon Mar 04 13:12:31 UTC 2024, attrName=N02, attrType=Float, attrValue=-0.001132657}}, Document{{recvTime=Mon Mar 04 13:12:31 UTC 2024, attrName=PM1, attrType=Float, attrValue=0}}, Document{{recvTime=Mon Mar 04 13:12:31 UTC 2024, attrName=PM10, attrType=Float, attrValue=0}}, Document{{recvTime=Mon Mar 04 13:12:31 UTC 2024, attrName=PM25, attrType=Float, attrValue=0}}]
time=2024-03-04T13:15:41.016Z | lvl=ERROR | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=processRollbackedBatches | msg=com.telefonica.iot.cygnus.sinks.NGSISink[398] : CygnusPersistenceError. -, Command failed with error 352: 'Unsupported OP_QUERY command: create. The client driver may require an upgrade. For more details see https://dochub.mongodb.org/core/legacy-opcode-removal' on server 127.0.0.1:27017. The full response is { "ok" : 0.0, "errmsg" : "Unsupported OP_QUERY command: create. The client driver may require an upgrade. For more details see https://dochub.mongodb.org/core/legacy-opcode-removal", "code" : 352, "codeName" : "UnsupportedOpQueryCommand" }. Stack trace: [com.telefonica.iot.cygnus.sinks.NGSIMongoSink.persistAggregation(NGSIMongoSink.java:421), com.telefonica.iot.cygnus.sinks.NGSIMongoSink.persistBatch(NGSIMongoSink.java:129), com.telefonica.iot.cygnus.sinks.NGSISink.processRollbackedBatches(NGSISink.java:390), com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:372), org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68), org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147), java.lang.Thread.run(Thread.java:750)]
time=2024-03-04T13:15:41.017Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=doRollbackAgain | msg=com.telefonica.iot.cygnus.sinks.NGSISink[464] : Rollbacking again (db955f28-da28-11ee-9c17-00163eee2746; cbnotif=2), this was retry #5

Does this mean that mongodb 7 is not supported or do the java drivers need to be updated? How can I get this data to persist in mongo?
Thank you!

[AlvaroVega]

It seems you are using a very old version of cygnus (1.9.0)
In order to use a modern version of mongoDB (v7) make sure you are using latests version of cygnus which uses java 17 and mongodb driver 3.12, which could works with mongodbv7 according with https://www.mongodb.com/docs/drivers/java/sync/current/compatibility/

[jrobinedwards]

It seems you are using a very old version of cygnus (1.9.0) In order to use a modern version of mongoDB (v7) make sure you are using latests version of cygnus which uses java 17 and mongodb driver 3.12, which could works with mongodbv7 according with https://www.mongodb.com/docs/drivers/java/sync/current/compatibility/

Ok thanks for letting me know @AlvaroVega This link is for the latest version of Cygnus: https://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/installation_and_administration_guide/install_with_rpm.html - perhaps someone needs to update this page.

According to this link it says to configure the repo using:
sudo wget -P /etc/yum.repos.d/ https://nexus.lab.fiware.org/repository/raw/public/repositories/el/7/x86_64/fiware-release.repo

Now my question is what is the latest version to install via RPM from the above repo - or is there another way to install it via RPM?

[AlvaroVega]

It seems you are using a very old version of cygnus (1.9.0) In order to use a modern version of mongoDB (v7) make sure you are using latests version of cygnus which uses java 17 and mongodb driver 3.12, which could works with mongodbv7 according with https://www.mongodb.com/docs/drivers/java/sync/current/compatibility/

Ok thanks for letting me know @AlvaroVega This link is for the latest version of Cygnus: https://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/installation_and_administration_guide/install_with_rpm.html - perhaps someone needs to update this page.

According to this link it says to configure the repo using: sudo wget -P /etc/yum.repos.d/ https://nexus.lab.fiware.org/repository/raw/public/repositories/el/7/x86_64/fiware-release.repo

Now my question is what is the latest version to install via RPM from the above repo - or is there another way to install it via RPM?

Unfortunately I do not know who is in charge of https://nexus.lab.fiware.org/ (which has too old versions of cygnus). Maybe @jason-fox or @mapedraza know someone to ask.

[fgalan]

It seems you are using a very old version of cygnus (1.9.0) In order to use a modern version of mongoDB (v7) make sure you are using latests version of cygnus which uses java 17 and mongodb driver 3.12, which could works with mongodbv7 according with https://www.mongodb.com/docs/drivers/java/sync/current/compatibility/

Ok thanks for letting me know @AlvaroVega This link is for the latest version of Cygnus: https://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/installation_and_administration_guide/install_with_rpm.html - perhaps someone needs to update this page.

According to this link it says to configure the repo using: sudo wget -P /etc/yum.repos.d/ https://nexus.lab.fiware.org/repository/raw/public/repositories/el/7/x86_64/fiware-release.repo

Now my question is what is the latest version to install via RPM from the above repo - or is there another way to install it via RPM?

We are no longer maintaining RPM-based deployments. Although some legacy artifacts still remain in the repository (such as the .spec files that we still use for keeping track of per-componente changelog) all our deployments are based now in Dockers.

[jrobinedwards]

@AlvaroVega and @fgalan for your feedback and pointing me to the right directions. This is kinda annoying as I spent quite a lot of time setting up cygnus on CentOS with RPM.

[jrobinedwards]

On another note - is this guide on building from sources correct and upto date?
https://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/installation_and_administration_guide/install_from_sources.html

[AlvaroVega]

On another note - is this guide on building from sources correct and upto date? https://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/installation_and_administration_guide/install_from_sources.html

That link is not updated, dependencies versions are old and some JDK option is also old, but the way of build (using maven) is right.

[fgalan]

@AlvaroVega and @fgalan for your feedback and pointing me to the right directions. This is kinda annoying as I spent quite a lot of time setting up cygnus on CentOS with RPM.

I hope that after the changes done in the PR #2354 nobody else fall into the same misunderstanding. Feel free to provide feedback in the PR if you want.

[fgalan]

After merging PR #2354 I think this issue can be closed.

@jrobinedwards what do you think?

[jrobinedwards]

After merging PR #2354 I think this issue can be closed.

@jrobinedwards what do you think?

Thanks @fgalan for removing the documentation for installing with RPM.

[jrobinedwards]

On another note - is this guide on building from sources correct and upto date? https://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/installation_and_administration_guide/install_from_sources.html

That link is not updated, dependencies versions are old and some JDK option is also old, but the way of build (using maven) is right.

Thank you for your reply @AlvaroVega @fgalan will someone please be able to update this too with the right dependencies and options.

[fgalan]

On another note - is this guide on building from sources correct and upto date? https://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/installation_and_administration_guide/install_from_sources.html

That link is not updated, dependencies versions are old and some JDK option is also old, but the way of build (using maven) is right.

Thank you for your reply @AlvaroVega @fgalan will someone please be able to update this too with the right dependencies and options.

@jrobinedwards could you tell us the particular text/link you refer in that document, please?

[jrobinedwards]

On another note - is this guide on building from sources correct and upto date? https://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/installation_and_administration_guide/install_from_sources.html

That link is not updated, dependencies versions are old and some JDK option is also old, but the way of build (using maven) is right.

Thank you for your reply @AlvaroVega @fgalan will someone please be able to update this too with the right dependencies and options.

@jrobinedwards could you tell us the particular text/link you refer in that document, please?

@fgalan I'm not sure what's outdated in this document :https://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/installation_and_administration_guide/install_from_sources.html
Perhaps @AlvaroVega will have a better clue as he mentioned previously that the "That link is not updated, dependencies versions are old and some JDK option is also old, but the way of build (using maven) is right."

[jrobinedwards]

So I followed the instructions here: https://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/installation_and_administration_guide/install_from_sources.html

I got Flume, JDK and Maven, versions and repo listed below:
Flume:
wget https://downloads.apache.org/flume/1.11.0/apache-flume-1.11.0-bin.tar.gz
flume-ng version
Flume 1.11.0
Source code repository: https://git.apache.org/repos/asf/flume.git
Revision: 1a15927e594fd0d05a59d804b90a9c31ec93f5e1
Compiled by rgoers on Sun Oct 16 14:44:15 MST 2022
From source with checksum bbbca682177262aac3a89defde369a37

JDK:
wget https://download.java.net/java/GA/jdk21.0.2/f2283984656d49d69e91c558476027ac/13/GPL/openjdk-21.0.2_linux-x64_bin.tar.gz

java -version
openjdk version "21.0.2" 2024-01-16
OpenJDK Runtime Environment (build 21.0.2+13-58)
OpenJDK 64-Bit Server VM (build 21.0.2+13-58, mixed mode, sharing)

Maven:
wget https://dlcdn.apache.org/maven/maven-3/3.9.6/binaries/apache-maven-3.9.6-bin.tar.gz

mvn -version
Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home: /opt/apache-maven-3.9.6
Java version: 21.0.2, vendor: Oracle Corporation, runtime: /opt/jdk-21.0.2
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.15.0-89-generic", arch: "amd64", family: "unix"

Cygnus
I cloned the cygnus repo and checked out the release:
git clone https://github.com/telefonicaid/fiware-cygnus.git
cd fiware-cygnus
git checkout release/3.6.0

Then ran compile.sh
cd cygnus-ngsi
sh ./compile.sh 3.6.0 /home/admin/apache-flume-1.11.0-bin

But got the following errors (oh, I also had to remove the -XX:MaxPermSize=128m from compile.sh) :

[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for com.telefonica.iot:cygnus-ngsi:jar:3.6.0
[WARNING] 'build.pluginManagement.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin org.apache.maven.plugins:maven-assembly-plugin @ line 117, column 25
[WARNING] Reporting configuration should be done in section, not in maven-site-plugin as reportPlugins parameter. @ line 162, column 36
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO]
[INFO] -------------------< com.telefonica.iot:cygnus-ngsi >-------------------
[INFO] Building cygnus-ngsi 3.6.0
[INFO] from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from central: https://repo.maven.apache.org/maven2/com/telefonica/iot/cygnus-common/3.6.0/cygnus-common-3.6.0.pom
[WARNING] The POM for com.telefonica.iot:cygnus-common:jar:3.6.0 is missing, no dependency information available
Downloading from central: https://repo.maven.apache.org/maven2/com/telefonica/iot/cygnus-common/3.6.0/cygnus-common-3.6.0.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.232 s
[INFO] Finished at: 2024-03-07T07:21:40Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project cygnus-ngsi: Could not resolve dependencies for project com.telefonica.iot:cygnus-ngsi:jar:3.6.0: The following artifacts could not be resolved: com.telefonica.iot:cygnus-common:jar:3.6.0 (absent): Could not find artifact com.telefonica.iot:cygnus-common:jar:3.6.0 in central (https://repo.maven.apache.org/maven2) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
cp: cannot stat 'target/cygnus-ngsi-3.6.0-jar-with-dependencies.jar': No such file or directory

Any ideas?

[AlvaroVega]

As I said you in #2353 (comment) I encorage you to use JDK 17.
Maybe just following steps done by Dockerfile #2353 (comment) you get success.

Using compile.sh scripts, you should execute compile.sh at the root level which executes first in cygnus-common and then in cygnus-ngsi.

[fgalan]

As I said you in #2353 (comment) I encorage you to use JDK 17. Maybe just following steps done by Dockerfile #2353 (comment) you get success.

Using compile.sh scripts, you should execute compile.sh at the root level which executes first in cygnus-common and then in cygnus-ngsi.

@jrobinedwards if at the end you go through the installation process following the steps in the Dockerfile as @AlvaroVega suggets, it would be great if you could do a pull request with suggested changes to improve the install_from_sources.md documentation based on your experience, please.

Thanks in advance! :)

[jrobinedwards]

@AlvaroVega @fgalan Thank you for your input. Last week, I did manage to get cygnus build from sources by following the dockerfile. I also manage to use flume.env to secure http-source via ssl. I will do a pull request and update it with my findings.

[jrobinedwards]

Hello @fgalan & @AlvaroVega
I have the updated documentation and I tried to do a pull request but I think I need to be a collaborator as I got this message when doing a pull request:
Pull request creation failed. Validation failed: must be a collaborator
I have the .md file if you want I can add it to the comments.

[fgalan]

You can do the PR if you:

1. Fork the fiware-cygnus repository
2. Do a modification in a branch in your forked repository
3. Do a pull request using that branch

However, if you have having problems with this we can do the PR on behalf of you if you provide as the .md file, as you suggest.

Thanks for your contribution!