Add Ansible CD using github action to deploy project on staging vm

.github/workflows/cd.yml

@@ -0,0 +1,42 @@
+name: CD
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - 'master'
+
+jobs:
+  cd:
+    if: |
+      github.event_name == 'push' || (
+        github.event_name == 'workflow_dispatch' &&
+        contains(fromJSON(vars.PROJECT_ADMINS), github.actor)
+      )
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v3
+    - name: Generate .env for staging vm from github secrets
+      run: |
+        echo "${{secrets.PRODUCTION_DOT_ENV_FILE}}" > .env
+    - name: Generate google-cloud-storage.json to src from secrets
+      run: |
+        echo "${{secrets.PRODUCTION_GOOGLE_CLOUD_STORAGE_JSON}}" > src/google-cloud-storage.json
+    - name: Decode private key file for OpenSSH access over Ansible
+      run: |
+        echo "${{secrets.SSH_PRIVATE_KEY}}" | base64 --decode > "private.pem"
+        chmod 400 private.pem
+    - name: Run playbook for deployment
+      uses: dawidd6/action-ansible-playbook@v2
+      with:
+        playbook: deploy.yml
+        inventory: |
+          pycontw:
+            hosts:
+              staging:
+                ansible_host: "${{secrets.VM_DOMAIN_IP}}"
+                ansible_user: "${{secrets.VM_USERNAME}}"
+                # secret file generated from previous step
+                ansible_ssh_private_key_file: private.pem
+                ansible_python_interpreter: "${{secrets.VM_PYTHON_INTERPRETER}}"

README.md

@@ -53,4 +53,8 @@ We strongly recommend you configure your editor to match our coding styles. You
 
 ## Deployment
 
+### Release to Production
 For site administrators, please refer to [document/deploy_docker_prod.md](/document/deploy_docker_prod.md).
+
+### Continuous Deployment
+Currently, continuous deployment is only integrated on PyCon's staging server, please refer to [document/continuous_deployment.md](/document/continuous_deployment.md) for the setup.

deploy.yml

@@ -0,0 +1,40 @@
+---
+- name: Deploy project to staging machine
+  hosts: staging
+  # escalate privilege
+  become: true
+  become_user: dev
+  vars:
+    project_dir: /home/dev/web-projects/pycontw-2023-ansible
+
+  tasks:
+    - name: Dependencies check dor docker and docker-compose in remote server
+      community.general.python_requirements_info:
+        dependencies:
+          - docker
+          - docker-compose
+
+    - name: Create project directory (if not exist)
+      ansible.builtin.file:
+        path: "{{ project_dir }}"
+        state: directory
+
+    # Copy project files to remote server (.env is included)
+    - name: Copy project files to remote server
+      ansible.posix.synchronize:
+        src: ./
+        dest: "{{ project_dir }}"
+        delete: true
+
+    - name: Ensure docker network network-2023 exists
+      community.docker.docker_network:
+        name: network-2023
+
+    - name: Build and start service
+      community.docker.docker_compose:
+        project_src: "{{ project_dir }}"
+        files:
+          # use ansible-specific compose file
+          - docker-compose-ansible.yml
+        build: true
+        state: present

docker-compose-ansible.yml

@@ -0,0 +1,47 @@
+version: "3.5"
+services:
+  web:
+    build: .
+    container_name: pycontw-2023-ansible
+    image: pycontw-2023_web-ansible
+    hostname: pycontw-2023
+    entrypoint: ""
+    command:
+      # Hacky script for quick demonstration purpose
+      - bash
+      - -c
+      - |
+        set -o errexit -o nounset -o pipefail
+        python3 manage.py compilemessages
+        python3 manage.py migrate
+        python3 manage.py collectstatic --no-input
+
+        exec uwsgi --http-socket :8000 \
+          --master \
+          --hook-master-start "unix_signal:15 gracefully_kill_them_all" \
+          --static-map /static=assets \
+          --static-map /media=media \
+          --mount /prs=pycontw2016/wsgi.py \
+          --manage-script-name \
+          --offload-threads 2
+    restart: always
+    environment:
+      # Save us from having to type `--setting=pycontw2016.settings.production`
+      DJANGO_SETTINGS_MODULE: pycontw2016.settings.production.pycontw2023
+      SCRIPT_NAME: /prs
+      SECRET_KEY: ${SECRET_KEY}
+      DATABASE_URL: ${DATABASE_URL}
+      EMAIL_URL: ${EMAIL_URL}
+      DSN_URL: ${DSN_URL}
+      GTM_TRACK_ID: ${GTM_TRACK_ID}
+      SLACK_WEBHOOK_URL: ${SLACK_WEBHOOK_URL}
+
+    volumes:
+      - ${MEDIA_ROOT}:/usr/local/app/src/media
+    networks:
+      - network
+
+networks:
+  network:
+    external: true
+    name: network-2023

document/continuous_deployment.md

@@ -0,0 +1,79 @@
+# Continuous Deployment on Staging Server
+
+The following describes how to setup continuous deployment for staging server. This setup presumes the site administrators have site deployment practices based on the docker production deployment [document/deploy_docker_prod.md](/document/deploy_docker_prod.md).
+
+# Introduction of CI/CD, GitHub Actions, Ansible and related settings
+Continuous integration (CI)
+- Refers to the build and unit testing stages of the software release process. Every revision that is committed triggers an automated build and test.
+
+Continuous delivery / Continuous Deployment (CD)
+- Usually as the next step for Continuous Integration, the code revision is built and tested in the application is automatically released to the production environment.
+
+GitHub Actions
+- A CI/CD platform or service provided by GitHub. It provides public runners with limited compute minutes to run CI/CD workflows defined at `.github/workflows` directory. We can also provision custom Github Actions runner to perform CI/CD task.
+
+Ansible
+- An automation tool that utilize *playbook* and *inventory* to manage production servers (nodes), such as sending commands, file transfers, system maintenance without manually setup via SSH/Remote session for it.
+
+Github Settings for secrets and variables
+- CI/CD workflows for Github Actions often obtain sensitive information, credentials or variable. In project settings, Github provides secrets vault and variable holder to store these information in the secure manner and able to retrieve and use these values when the workflows run.
+
+## High level comparison of CI/CD in this project
+| Conventional - Release | Github Actions + Ansible - Release |
+|-----------|------------|
+| `ENV` values managed by site admin | `ENV` values are set in project settings (secrets) |
+| Site admin solely manage the production server | Site admin gives rights to github actions to deploy release to the production server |
+| Site admin knows every deployment steps for docker | Deployment steps are defined in Ansible playbook (so everyone can understand deployment steps) |
+| Site admin runs commands in a SSH session | Ansible runs the commands to server as defined by the playbook |
+| Only admins have the server IP and private key | Server IP and private key are securely kept at github settings |
+| Release is manual | Release automatically once code merges to `master` branch |
+| Things are executed by hands | Things are executed by Github Actions' runner |
+
+## Requirements for Staging Server
+The staging server should have the following installed:
+- Docker 17.09+ (since we use `--chown` flag in the COPY directive)
+- Docker Compose
+- python3.6+
+- [docker](https://pypi.org/project/docker/) SDK for python
+- [docker-compose](https://pypi.org/project/docker-compose/) SDK for python
+
+
+## Prerequisite for Site Administrators
+- Gather Container Environment Variables as mention in [document/deploy_docker_prod.md](/document/deploy_docker_prod.md).
+- Have a ssh user and secret file for accessing GCE instance (staging machine)
+    - Secret file will be further encoded by base64
+- Administrators github Ids
+    - For CD workflow authorization
+
+## Settings for Github Actions Workflow
+After aboves steps, we have to add collected information to Github actions setting.
+
+Under the hood, we github action and [Ansible](https://www.ansible.com/overview/how-ansible-works) for continuous deployment. Github action will hold necessary variables and secrets that allows Ansible to access the staging VM on your behalf.
+
+So kindly configure project's action setting as the following:
+
+| Level  | Type   | Name   | Value (example) | Remarks |
+|-----------|------------|---------------|----------|------------|
+| Repository | secrets | PRODUCTION_DOT_ENV_FILE | `DATABASE_URL=...`   |   multiline support      |
+| Repository | secrets | PRODUCTION_GOOGLE_CLOUD_STORAGE_JSON | `{ ...`   |   multiline support      |
+| Repository | secrets | VM_USERNAME | cd_user  | user name for ssh {user_name}@{vm_domain}  |
+| Repository | secrets | VM_DOMAIN_IP | staging.pycon.tw  | IP address or Domain that points to the staging server  |
+| Repository | secrets | VM_PYTHON_INTERPRETER | `/home/dev/.pyenv/shims/python`  | path to your python environment that has docker/docker-compose packages installed |
+| Repository | secrets | SSH_PRIVATE_KEY | `21xa312....`     |  base64 encoded of key-pair (`.pem` file) |
+| Repository | variables | PROJECT_ADMINS | `["github_user_1", "github_user_2"]` | For example `["josix"]` |
+
+Reference
+- [Create a secret for a repository](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository)
+- [Create a variable for a repository](https://docs.github.com/en/actions/learn-github-actions/variables#creating-configuration-variables-for-a-repository)
+- Create base64 encoded string for `key.pem`
+    - `base64 -i key.pem` (mac)
+    - `cat key.pem | base64` (linux)
+
+## CD Workflow Rules
+### Events that triggers the pipeline
+1. When the PR merges to `master`
+    - no authorization needed, as PRs normally reviewed before merge
+2. Manually [trigger](https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow#running-a-workflow) the CD workflow (By admins)
+    - only for Administrator specify in repository's variable called *PROJECTS_ADMINS*
+
+Why? CD workflow will directly access to the GCE instance, should prevent unwanted deployments from PRs or push. (As a deployment guardian)