Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update install documentation #2364

Merged
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,51 @@ Content:
* [Prerequisites](#section1)
* [Installing cygnus-ngsi](#section2)
* [Cloning `fiware-cygnus`](#section2.1)
* [Installing `cygnus-ngsi`](#section2.2)
* [Known issues](#section2.3)
* [Installing dependencies](#section3)
* [Installing `cygnus-common`](#section2.2)
* [Installing `cygnus-ngsi`](#section2.3)
* [Setup `cygnus`](#section2.4)
* [Setup Agents](#section2.5)
* [Running `cygnus-ngsi`](#section2.6)
* [SSL Support](#section3)
* [Prerequisites](#section3.1)
* [Setup](#section3.2)

Please note that the commands listed here are for Debian systems.

## <a name="section1"></a>Prerequisites
[`cygnus-common`](../../cygnus-common/installation_and_administration_guide/install_from_sources.md) must be installed. This includes Maven, `cygnus` user creation, `log4j` path creation, Apache Flume and `cygnus-flume-ng` script installation.
Install JDK 17

$ sudo apt-get -y install openjdk-17-jdk openjdk-17-jdk-headless openjdk-17-jre openjdk-17-jre-headless

Add $JAVA_HOME to $PATH

$ nano .bashrc
$ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
$ export PATH=$PATH:$JAVA_HOME

Install Maven

$ sudo apt-get -y install maven

Create `cygnus` user

$ adduser cygnus

Install Apache Flume

$ wget https://downloads.apache.org/flume/1.11.0/apache-flume-1.11.0-bin.tar.gz
$ tar xzf apache-flume-1.11.0-bin.tar.gz
$ mv apache-flume-1.11.0-bin /opt/apache-flume-1.11.0
$ mkdir -p /opt/apache-flume-1.11.0/plugins.d/cygnus
$ mkdir -p /opt/apache-flume-1.11.0/plugins.d/cygnus/lib
$ mkdir -p /opt/apache-flume-1.11.0/plugins.d/cygnus/libext
$ chown -R cygnus:cygnus /opt/apache-flume-1.11.0/

Add FLUME_HOME to $PATH

$ nano .bashrc
$ export FLUME_HOME=/opt/apache-flume-1.11.0
$ export PATH=$PATH:$FLUME_HOME/bin

[Top](#top)

Expand All @@ -25,40 +64,111 @@ Start by cloning the Github repository:

[Top](#top)

### <a name="section2.2"></a>Installing `cygnus-ngsi`
`cygnus-ngsi` can be built as a fat Java jar file containing all third-party dependencies (**recommended**):
### <a name="section2.2"></a>Installing `cygnus-common`

$ cd cygnus-ngsi
$ APACHE_MAVEN_HOME/bin/mvn clean compile exec:exec assembly:single
$ cp target/cygnus-ngsi-<x.y.z>-jar-with-dependencies.jar APACHE_FLUME_HOME/plugins.d/cygnus/lib
$ cd fiware-cygnus/cygnus-common
$ mvn -B -T8 clean compile exec:exec assembly:single
$ cp target/cygnus-common-3.6.0-SNAPSHOT-jar-with-dependencies.jar /opt/apache-flume-1.11.0/plugins.d/cygnus/libext/
$ mvn install:install-file -Dfile=/opt/apache-flume-1.11.0/plugins.d/cygnus/libext/cygnus-common-3.6.0-SNAPSHOT-jar-with-dependencies.jar -DgroupId=com.telefonica.iot -DartifactId=cygnus-common -Dversion=3.6.0-SNAPSHOT -Dpackaging=jar -DgeneratePom=false

[Top](#top)

Or as a thin Java jar file:
### <a name="section2.3"></a>Installing `cygnus-ngsi`

$ cd cygnus-ngsi
$ APACHE_MAVEN_HOME/bin/mvn exec:exec package
$ cp target/cygnus-<x.y.z>.jar APACHE_FLUME_HOME/plugins.d/cygnus/lib
$ cd fiware-cygnus/cygnus-ngsi
$ mvn -B -T8 clean compile exec:exec assembly:single
$ cp target/cygnus-ngsi-3.6.0-SNAPSHOT-jar-with-dependencies.jar /opt/apache-flume-1.11.0/plugins.d/cygnus/lib/

Finally, please find a `compile.sh` script containing all the commands shown in this section. It must be parameterized with the version of the current branch and the Apache Flume base path.

[Top](#top)

### <a name="section2.3"></a>Known issues
It may happen while compiling `cygnus-ngsi` the Maven JVM has not enough memory. This can be changed as detailed at the [Maven official documentation](https://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError):
### <a name="section2.4"></a>Setup Cygnus

$ cp fiware-cygnus/cygnus-common/target/classes/cygnus-flume-ng /opt/apache-flume-1.11.0/bin/
$ chmod +x /opt/apache-flume-1.11.0/bin/cygnus-flume-ng
$ cp fiware-cygnus/cygnus-common/conf/log4j2.properties.template /opt/apache-flume-1.11.0/conf/log4j2.properties
$ cp fiware-cygnus/cygnus-ngsi/conf/name_mappings.conf.template /opt/apache-flume-1.11.0/conf/name_mappings.conf
$ mkdir -p /var/log/cygnus
$ chown -R cygnus:cygnus /var/log/cygnus

Clean up

$ cd fiware-cygnus/cygnus-common
$ mvn -B -T8 clean
$ cd fiware-cygnus/cygnus-ngsi
$ mvn -B -T8 clean>

[Top](#top)

### <a name="section2.5"></a>Setup Agents
Setup Cygnus-Common and Cygnus-Ngsi agents

### <a name="section2.6"></a>Running `cygnus-ngsi`
Please make sure that you have ports 5080 (admin port) and 5050 (default instance port) open.<br/>

$ export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=128m"
$ /opt/apache-flume-1.11.0/bin/cygnus-flume-ng agent --conf {{PATH-TO-CONF-FOLDER}} -f {{PATH-TO-AGENT-NGSI-CONF}} -n {{AGENT-NAME}} -p 5080 -Dflume.root.logger=INFO,LOGFILE -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Dflume.monitoring.type= -Dflume.monitoring.port=41414

[Top](#top)

## <a name="section3"></a>Installing dependencies
These are the packages you will need to install under `APACHE_FLUME_HOME/plugins.d/cygnus/libext/` **if you did not included them in the cygnus-common jar**:

| Cygnus dependencies | Version | Required by / comments |
|---|---|---|
| mockito-all | 1.9.5 | Unit tests |
| junit | 4.11 | Unit tests |
| httpcore | 4.3.1 | Overwrites the one bundled in Apache Flume |
| log4j | 1.2.17 | Logging |
| slf4j-simple | 1.7.21 | Logging |
| cygnus-common | latest | |
## <a name="section3"></a>SSL Support
Apache Flume has SSL support, [mentioned here](https://flume.apache.org/FlumeUserGuide.html#ssl-tls-support)

### <a name="section3.1"></a>Prerequisites for this is that you should have:
SSL certificated installed (using Certbot & let's encrypt)<br/>
Before doing this make sure that port 80 and 443 are open and that you have a domain assoicated with the IP of the server you are installing SSL on.<br/>
Install Certbot with snap

$ sudo apt install snapd
$ sudo snap install core; sudo snap refresh core
$ sudo apt remove certbot
$ sudo snap install --classic certbot
$ sudo ln -s /snap/bin/certbot /usr/bin/certbot

Issue certificates

$ sudo certbot certonly --standalone -d DOMAINNAME

Java Keystore setup using the above certificates.<br/>
These commands can be put in a bash file and run as a deployment hook, so whenever the certificates renewed the Java Keystore can be updated.<br/>
Replace all values withing {{}}

$ openssl pkcs12 -export \
-in {{ABSOLUTE_PATH_TO_LETSENCRYPT_FULLCHAIN}} \
-inkey {{ABSOLUTE_PATH_TO_LETSENCRYPT_PRIVKEY}} \
-out /tmp/cert.p12 \
-name letsencrypt-keystore \
-passout pass:{{TEMPORARY_PASSWORD}} \
> /dev/null 2>&1

Import PKCS12 file into keystore

$ keytool -noprompt -importkeystore \
-srckeystore /tmp/cert.p12 \
-srcstoretype PKCS12 \
-srcstorepass {{TEMPORARY_PASSWORD}} \
-deststorepass {{KEYSTORE_PASSWORD}} \
-destkeypass {{PRIVATE_KEY_PASSWORD}} \
-destkeystore {{ABSOLUTE_PATH_TO_KEYSTORE_IN_CONF_FOLDER}} \
-alias letsencrypt-keystore \
> /dev/null 2>&1

Delete PKCS12 file

$ rm /tmp/cert.p12
$ chown cygnus:cygnus /opt/apache-flume-1.11.0/conf/letsencrypt-keystore.jks`

### <a name="section3.2"></a>Setup
Clone /conf/flume-env.sh.template to /conf/flume-env.sh and edit to add the following:

$ export JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.keyStore=/path/to/letsencrypt-keystore.jks"`
$ export JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.keyStorePassword=password"`
$ export FLUME_SSL_KEYSTORE_PATH=/path/to/etsencrypt-keystore.jks`
$ export FLUME_SSL_KEYSTORE_PASSWORD=password`

Edit agent_ngsi config and add the following lines according to <a href='https://flume.apache.org/releases/content/1.11.0/FlumeUserGuide.html#http-source'>Flume http-source</a>

$ cygnus-ngsi.sources.http-source.ssl = true`


[Top](#top)
Loading