Secure secrets in repositories

Repository Secrets

Secrets in a repository need to be hidden from unauthorized parties. This document outlines how Jervis accomplishes that. See also Travis CI documentation on the subject. The security in this project is loosely based on my research in respository-secrets.

Jervis provides secure field support through the securityIO class (see Jervis groovydoc). By default Jervis uses asymmetric encryption to secure the fields; specifically RSA public-key encryption. The option to make the encryption stronger is available. All decrypted strings are then used as environment variables in the build. This functionality is provided by the Mask Passwords plugin.

Secure Jenkins from unauthorized parties

Before bothering to secure secrets in a repository, Jenkins should be secured. In addition to following Jenkins best practices, I recommend the following additional precautions.

• The Job DSL runtime runs on the master. Therefore, only grant the ability to create jobs to very trusted parties because anybody writing Job DSL scripts is basically an admin through scripting. Because of this, the private key only exists on the Jenkins master when decrypting secure properties.
• Secure communication with Jenkins by hosting with TLS (i.e. HTTPS).
• Ensure Jenkins requires a username and password. Otherwise, it will just be a web-based shell and easily compromised.
• Enable CSRF mitigation support in Jenkins.
• The master should have 0 executors. If it has executors then it should be an EXCLUSIVE slave and all jobs on the system must have labels to execute on their appropriate agent.
• Build jobs should never execute on the master. If a user gets a hold of the master ${JENKINS_HOME}/secret.key file then they can effectively compromise all secrets and passwords on the system.
• Additionally, jobs executing on the master can disable security by modifying ${JENKINS_HOME}/config.xml.
• Encrypt all communication between the master and slave. Whether you use SSH from master to slave or the new secure JNLP3 support from slave to master. SSH is tried and true while JNLP3 is still new and hasn't really been tested thoroughly.
• Track your $JENKINS_HOME configuration with git and don't forget to separately backup $JENKINS_HOME/secret.key.

Secure field support

There are a few fields in the .jervis.yml YAML file that support security. Here's example YAML.

jenkins:
  secrets_id: "<Jenkins RSA key identifier>"
  secrets:
    - key: MY_ENVIRONMENT_VAR
      secret: <rsa encrypted string>
    - key: ANOTHER_ENVIRON_VAR
      secret: <rsa encypted string>

The secrets_id is a Jenkins credentials ID which will be explained further in this document.

Encrypted file support

Since the users can access passwords via secure properties (environment variables) then they can manage their own secure file support using symmetric encryption. They could simply store the symmetric key pass phrase as an environment variable.

Configuring Jenkins

A user can configure Jenkins with their own RSA private key to decrypt secrets. This means they can decide what key length they want and aren't limited by the key size other than what openssl supports. In Jervis, projects are generated in the format of <Jenkins folder>/<project name>. The Jenkins folder is where the credentials must be stored. This separates credentials on a per-folder basis.

Generate private and public RSA keys. The public key is used to encrypt secure properties and the private key is used within Jenkins to decrypt.

#generate private key (recommended to use 2048-bit or 4096-bit RSA keys)
openssl genrsa -out ./id_rsa 2048
openssl rsa -in ./id_rsa -pubout -outform pem -out ./id_rsa.pub

Add the RSA private key to Jenkins.

1. Go to the project folder in Jenkins.
2. Click on the Credentials link in the left hand menu.
3. Add a new domain and call it secrets.
4. Add a new include filter and set it to localhost or some other arbitrary value so that it can't be referenced in jobs.
5. Save the domain and enter the domain.
6. Click Add Credentials.
7. Add SSH Username with private key. The username doesn't matter so just set it to secret.
8. Enter the private key directly and paste in the RSA private key.
9. Click on the advanced button and set the ID to a unique string. This is called the Jenkins credentials ID.
10. For clarity, set the description to be the same as the credentials ID.

Once you've configured your key in Jenkins add the following to your Jervis YAML file.

jenkins:
  secrets_id: "<Jenkins credentials ID>"

Encrypt secrets

This can be easily be done by saving id_rsa.pub with the project and using generate_secret.sh

Now that you have the private key installed in Jenkins let's encrypt secure properties using the public key.

echo -n 'plaintext' | openssl rsautl -encrypt -inkey ./id_rsa.pub -pubin | openssl enc -base64 -A

In the example above, secretplaintext is the text one wishes to encrypt. The result will be cipher text.

Add to your Jervis YAML the cipher text and give the key an environment variable name. This secure property will then be accessible in build jobs as an environment variable.

jenkins:
  secrets_id: "<Jenkins credentials ID>"
  secrets:
    - key: "ENV_VAR"
      secret: "<cipher text of secretplaintext>"

Now, when you generate a Jenkins job it will be generated with secure properties added via the Masked Passwords plugin.

To see an example of this check out the Jervis Secrets Test repository.

Encrypt private key before uploading to Jenkins

The private key can be encrypted using the AES algorithm with the ssh-keygen program. Here's an example.

chmod 600 id_rsa
ssh-keygen -p -f id_rsa

When you add the credential to Jenkins just be sure to set the password to the credential.

Enforcing stronger RSA keys

As of Jervis 1.0 release, RSA private keys smaller than 2048 bits are no longer allowed. If a Job DSL script or pipeline attempts to load a private key smaller than 1024 then the Job DSL script or pipeline script will fail with a KeyPairDecodeException. While 2048 bit keys are the minimum allowed, 3096 bit RSA private keys are recommended.

Security recommendation

Users should rotate their secrets, rather than migrate, because their data was encrypted using a known broken RSA algorithm. Data encrypted with this weaker algorithm is not considered secure and will still be accessible in git history. Even if the git history is "modified" it may still exist in cloned copies. It is safer to assume the encrypted data was compromised.

Migrate secrets (not recommended)

To migrate to larger RSA keys do the following:

1. Generate a new private key (do not set a password, yet):

   ssh-keygen -b 3096 -f id_rsa_3096
   openssl rsa -in id_rsa_3096 -pubout -outform pem -out id_rsa_3096.pub
   

2. Decrypt your old secrets using the weaker 1024 bit RSA key (let's say it decrypts to plaintext).

   echo 'ciphertext' | openssl enc -base64 -A -d | openssl rsautl -decrypt -inkey id_rsa_1024
   

3. Encrypt your plain text secret using your new larger private key.

   echo -n 'plaintext' | openssl rsautl -encrypt -inkey ./id_rsa_3096.pub -pubin | openssl enc -base64 -A
   

4. Encrypt your private key with AES before uploading it to Jenkins.

   ssh-keygen -p -f id_rsa_3096