flux-template - An opinionated cluster agnostic Flux GitOps template repository

A lot of Flux template repositories expect you have already decided what type of Kubernetes distribution you want to use (i.e. the same one they are using). It works when you're following their design but it can be a pain when you even have a slighly different setup. This repository proposes the opposite approach, you know what core applications you want to run on your cluster, but you're not yet sure about the cluster infrastructure.

hetzner-k3s-flux-template is a real-world example I made using this template.

Table of contents

• Overview
• Quickstart
• Considerations
• Contributing
• License

Overview

This Flux template was created with a holistic but flexible view in mind. The repository structure is straightforward, easy to understand and easy to use. Still, every aspect is explained below:

Stack

All core components are optional, you can choose to only use the repository structure as a starting point.

Name	Description
Flux	Automated Kubernetes cluster updates using GitOps
SOPS	Secret management
Cilium	eBPF-based Networking, Observability, Security
cert-manager	X.509 certificate management
ExternalDNS	Synchronize exposed Kubernetes Services and Ingresses with DNS providers
Traefik	Edge Router / Ingress
Capacitor	General purpose UI for Flux
kube-prometheus	Monitoring stack, including Prometheus, Grafana, and Alertmanager
Loki	Log aggregation
Promtail	Log discovery
Kured	Reboot Daemon
System Upgrade Controller	Automated node updates
Reloader	Auto-reload Kubernetes resources based on ConfigMap/Secret changes

Flux Kustomization Reconciliation Flowchart

flowchart TD;
    id1[apps] -->|dependsOn| id2[configs]
    %% https://github.com/mermaid-js/mermaid/issues/2977
    subgraph "`&nbsp**infrastructure**&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp`"
        id2[configs] -->|dependsOn| id3[controllers]
        %% https://github.com/mermaid-js/mermaid/issues/2977
        subgraph "`&nbsp**controllers**&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp`"
            id3[controllers] -->|dependsOn| id4[monitoring]
            id4[upgrade] -->|dependsOn| id5[monitoring]
            id5[monitoring] -->|dependsOn| id6[network]
        end
        id6[network] -->|dependsOn| id7[image-automations]
        id7[image-automations] -->|dependsOn| id8[sources]
        id8[sources] -->|dependsOn| id9[notifications]
        id9[notifications]
    end

Loading

A fully set up repository structure is as follows:

Top directories

• apps: user related deployments (e.g. webapps, game servers)
• cluster: Flux configuration
• infrastructure: common infrastructure components (e.g. monitoring, network)
• templates: yaml template files
• tools: workspace and template rendering

Flux is set up to only look at apps, cluster and infrastructure:

├── 📁 apps
│   └── kustomization.yaml
│  
├── 📁 cluster
│   ├── 📁 flux-system
│   ├── apps.yaml
│   └── infrastructure.yaml
|
└── 📁 infrastructure
    ├── 📁 configs
    ├── 📁 controllers
    ├── 📁 image-automations
    ├── 📁 notifications
    └── 📁 sources

Applications

The apps directory is very straightforward, you can structure it however you want as long as you mention the resource in the kustomization.yaml file.

Cluster

The cluster directory holds all the files necessary for Flux to work. The flux-system directory is generated after bootstrapping, the apps.yaml and infrastructure.yaml hold the Flux Kustomization definitions.

Infrastructure

The infrastructure directory is structured into individual files and 5 predetermined sub directories:

• configs: Kubernetes custom resources such as cert issuers and networks policies
• controllers: namespaces and Helm release definitions for Kubernetes controllers
• image-automations: Image reflector and automation controllers
• notifications: Notification Controllers
• sources: Source Controllers

The configs, image-automations, notifications directories have no resources by default. For brevity, they are omitted from the following view:

📁 infrastructure
├── 📁 controllers
│   ├── 📁 monitoring
|   ├── 📁 network
|   ├── 📁 upgrade
|   └── kustomization.yaml
│  
└── 📁 sources
    ├── 📁 bucket
    ├── 📁 git
    ├── 📁 oci
    ├── 📁 helmrepos
    └── kustomization.yaml

The controllers and sources directories have the following sub directories:

controllers

• monitoring: Monitoring controllers
• network: Network controllers
• upgrade: Upgrade controllers

sources

• bucket: Buckets
• git: GitRepositories
• oci: OCIRepositories
• helmrepos: HelmRepositories

Quickstart

Getting started is easy, the following guide uses GitHub as a Git server.

Prerequisites

• A (preferably clean) Kubernetes cluster that does not have Flux installed, can be any distribution. For local testing you can use something like minikube, kind, microk8s, etc.
• age for secret keys.
• sops for encrypting secrets.
• just for commands.
• Docker for a containerized work environment.

1. Create a new Git repository by using this one as a template.

2. Create an age key that will be used for encrypting cluster based secrets:

age-keygen -o flux.agekey

Add the age secret key string (identity) AGE-SECRET-KEY-1... found in the file to the default identities file ~/.config/sops/age/keys.txt.

3. Fill config.yaml with your age public key (recipient) and your controller values.

4. Add and encrypt the kubeconfig file:

You can choose to use a different age key to encrypt the kubeconfig (this key will not be pushed to the cluster). Make sure that the identity also exists in the default identities file (~/.config/sops/age/keys.txt).

sops -e -i --age age... --encrypted-regex '^client-key-data$' --input-type yaml --output-type yaml kubeconfig
config

5. Build workspace container:

just build

6. Run workspace container:

just tools

7. Render templates:

just render

8. Create a flux-system namespace and create the sops-age secret:

Create flux-system namespace:

kubectl create ns flux-system

Create sops-age secret:

cat flux.agekey | kubectl create secret generic sops-age \
   --namespace=flux-system \
   --from-file=age.agekey=/dev/stdin

9. Bootstrap Flux:

Generate a GitHub PAT that can create repositories by checking all permissions under repo.

Export values:

export GITHUB_TOKEN=<your-token>
export GITHUB_USER=<your-username>
export GITHUB_REPO=<your-repo>

Bootstrap:

flux bootstrap github \
 --components-extra=image-reflector-controller,image-automation-controller \
 --owner=$GITHUB_USER \
 --repository=$GITHUB_REPO \
 --branch=main \
 --path=cluster \
 --read-write-key \
 --personal

If you wish to create a public repository, add the --private=false flag.

10. Done!

You now have a GitOps compliant cluster fully managed through a Git repository, the sky's the limit 🚀!

Considerations

This looks interesting but complicated, how can I understand this template better?

I've tried to make everything as intuitive as possible. Looking at the individual files and cross-referencing values should help get a rough view of how things work. I also recommend you to carefully look over the Flux documentation whenever you don't understand the contents of a file.

Why not a monorepo approach?

A monorepo is great in theory, dev, staging and prod in one repository with the ability for overlays to minimize duplicated resource declaration. In practice it gets tangled quick, there are a lot of nuances between different environments that can result in more complexity than structure.

Why not use X for Y?

That's for YOU to decide, this is only meant as a foundation for your own cluster/templates. You can choose to not install any core component and roll with your own.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

License

MIT

Acknowledgements

This "solution" could not have been without the following OSS:

• kronform
• home-ops