diff --git a/docs/Tutorial/tutorial_repositoryserver.rst b/docs/Tutorial/tutorial_repositoryserver.rst index 1c74062c2f..4ffe3dc6a7 100644 --- a/docs/Tutorial/tutorial_repositoryserver.rst +++ b/docs/Tutorial/tutorial_repositoryserver.rst @@ -39,12 +39,28 @@ application is installed in the ``default`` namespace. .. code-block:: yaml $ cat <> /var/log/time.log; sleep 1; done; truncate /var/log/time.log --size 0; done"] + volumeMounts: + - name: data + mountPath: /var/log + volumes: + - name: data + persistentVolumeClaim: + claimName: time-log-pvc EOF Starting Kopia Repository Server @@ -210,21 +233,77 @@ command used to create repository as specified in section :ref:`Creating a Kopia userAccessSecretRef: name: repository-server-user-access namespace: kanister - username: kanisterUser + username: kanisteruser EOF -Once the Repository Server is created, you will see a repository server pod and a service created -in kanister namespace. +Once the Repository Server is created, you will see a repository server pod and a service exposing the +the kopia repository server created in kanister namespace. + +.. code-block:: bash -######## -(TODO: List pods and services) + $ kubectl get pods,svc -n kanister + NAME READY STATUS RESTARTS AGE + pod/kanister-kanister-operator-5b7dfbf97b-5j5p5 2/2 Running 0 33m + pod/repo-server-pod-4tjcw 1/1 Running 0 2m13s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/kanister-kanister-operator ClusterIP 10.96.197.93 443/TCP 33m + service/repo-server-service-rq2pq ClusterIP 10.96.127.153 51515/TCP 2m13s To see if the server started successfully, you can check the status of the server using following command -######### -(TODO: Please describe repository server resource) +.. code-block:: bash + + $ kubectl get repositoryservers.cr.kanister.io kopia-repo-server -n kanister -oyaml + apiVersion: cr.kanister.io/v1alpha1 + kind: RepositoryServer + metadata: + annotations: + kubectl.kubernetes.io/last-applied-configuration: | + {"apiVersion":"cr.kanister.io/v1alpha1","kind":"RepositoryServer","metadata":{"annotations":{},"name":"kopia-repo-server","namespace":"kanister"},"spec":{"repository":{"hostname":"timelog.app","passwordSecretRef":{"name":"repository-pass","namespace":"kanister"},"rootPath":"/test/repo-controller","username":"kansiterAdmin"},"server":{"adminSecretRef":{"name":"repository-server-admin","namespace":"kanister"},"tlsSecretRef":{"name":"repository-server-tls-cert","namespace":"kanister"},"userAccess":{"userAccessSecretRef":{"name":"repository-server-user-access","namespace":"kanister"},"username":"kanisteruser"}},"storage":{"credentialSecretRef":{"name":"s3-loc-creds","namespace":"kanister"},"secretRef":{"name":"s3-location","namespace":"kanister"}}}} + creationTimestamp: "2023-06-05T05:45:49Z" + generation: 1 + name: kopia-repo-server + namespace: kanister + resourceVersion: "41529" + uid: b4458c4f-b2d5-4dcd-99de-a0a4d32ed216 + spec: + repository: + hostname: timelog.app + passwordSecretRef: + name: repository-pass + namespace: kanister + rootPath: /test/repo-controller + username: kansiterAdmin + server: + adminSecretRef: + name: repository-server-admin + namespace: kanister + tlsSecretRef: + name: repository-server-tls-cert + namespace: kanister + userAccess: + userAccessSecretRef: + name: repository-server-user-access + namespace: kanister + username: kanisteruser + storage: + credentialSecretRef: + name: s3-loc-creds + namespace: kanister + secretRef: + name: s3-location + namespace: kanister + status: + progress: ServerReady + serverInfo: + podName: repo-server-pod-4tjcw + serviceName: repo-server-service-rq2pq + +``pod/repo-server-pod-4tjcw`` and ``service/repo-server-service-rq2pq`` populated in +``status.serverInfo`` field should be used by the client to connect to the server Invoking Kanister Actions ========================= @@ -247,10 +326,14 @@ Kanister function ``BackupDataUsingKopiaServer`` that uses kopia repository serv data to s3 storage. The action ``restore`` uses two kanister functions ``ScaleWorkload`` and ``RestoreDataUsingKopiaServer``. ``ScaleWorkload`` function scales down the timelog application before restoring the data. ``RestoreDataUsingKopiaServer`` restores data using kopia repository server -form s3 storage +form s3 storage. For more information of kanister function refer :doc:`Kanister's parameter templating `. +We are using output artifacts here to store the path of our data in s3 and snapshot ID that +that will be used as ``backupIdentifier`` while performing restore. To know more about +artifacts you can refer :ref:`tutorial`. + Blueprint --------- @@ -279,7 +362,6 @@ Blueprint pod: "{{ index .Deployment.Pods 0 }}" container: test-container includePath: /var/log - restore: inputArtifactNames: - timeLog @@ -297,7 +379,7 @@ Blueprint args: namespace: "{{ .Deployment.Namespace }}" pod: "{{ index .Deployment.Pods 0 }}" - image: ghcr.io/kanisterio/kanister-tools:0.89.0 + image: ghcr.io/kanisterio/kanister-tools:0.92.0 backupIdentifier: "{{ .ArtifactsIn.backupIdentifier.KeyValue.id }}" restorePath: /var/log - func: ScaleWorkload @@ -307,10 +389,8 @@ Blueprint name: "{{ .Deployment.Name }}" kind: Deployment replicas: 1 - EOF - Once we create a Blueprint, we can see its events by using the following command: .. code-block:: yaml @@ -328,13 +408,90 @@ multiple actions, each acting on a different Kubernetes object. The ActionSet we're about to create in this tutorial specifies the ``time-logger`` Deployment we created earlier and selects the ``backup`` action inside our Blueprint. + +Add some data in the time logger app +.. code-block:: bash + kubectl exec -it time-logger-6d89687cbb-bmdj8 -n default -it sh + sh-5.1# cd /var/log/ + sh-5.1# ls + time.log + sh-5.1# echo "hello world" >> test.log + sh-5.1# cat test.log + hello world + ActionSet ---------------- +--------- .. code-block:: bash # Create action set using the blueprint created in above step - $ kanctl create actionset --action backup --namespace kanister --blueprint time-log-bp --deployment time-logger/time-logger --repository-server=kopia-repo-server + $ kanctl create actionset --action backup --namespace kanister --blueprint time-log-bp --deployment default/time-logger --repository-server kanister/kopia-repo-server + actionset actionset backup-rlcnp created ``--repository-server`` flag is used to provide the reference to the repository server CR that we created -in step :ref:`Creating Repository Server custom resource`. The CR is made available to the kanister -functions using template parameters. +in step :ref:`Creating Repository Server custom resource`. Since the details related to kopia repository server and +the secrets are present in the CR, the blueprint will be able to read these details using +template parameters and will perform backup using kopia repository server + + +.. code-block:: bash + + Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Started Action 14s Kanister Controller Executing action backup + Normal Started Phase 14s Kanister Controller Executing phase backupToS3 + Normal Ended Phase 9s Kanister Controller Completed phase backupToS3 + Normal Update Complete 9s Kanister Controller Updated ActionSet 'backup-rlcnp' Status->complete + + +Lets delete the date from ``timelogger`` app. + +.. code-block:: bash + kubectl exec -it time-logger-6d89687cbb-bmdj8 -n default -it sh + sh-5.1# cd /var/log/ + sh-5.1# ls -lrt + total 12 + -rw-r--r-- 1 root root 12 Jun 5 06:22 test.log + -rw-r--r-- 1 root root 7308 Jun 5 06:26 time.log + sh-5.1# rm -rf test.log + sh-5.1# ls -lrt + total 8 + -rw-r--r-- 1 root root 7482 Jun 5 06:26 time.log + + +Lets perform restore now, by using ``restore`` action from the ``time-log-bp`` blueprint + +.. code-block:: bash + kanctl --namespace kanister create actionset --action restore --from "backup-rlcnp" --repository-server kanister/kopia-repo-server + actionset restore-backup-rlcnp-g5h65 create + +We can see if the restore is successful by describing the actionset + +.. code-block:: bash + $kubectl describe actionsets.cr.kanister.io restore-backup-rlcnp-g5h65 -n kanister + + Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Started Action 20s Kanister Controller Executing action restore + Normal Started Phase 20s Kanister Controller Executing phase shutdownPod + Normal Ended Phase 8s Kanister Controller Completed phase shutdownPod + Normal Started Phase 8s Kanister Controller Executing phase restoreFromS3 + Normal Ended Phase 4s Kanister Controller Completed phase restoreFromS3 + Normal Started Phase 4s Kanister Controller Executing phase bringupPod + Normal Ended Phase 3s Kanister Controller Completed phase bringupPod + Normal Update Complete 2s Kanister Controller Updated ActionSet 'restore-backup-rlcnp-g5h65' Status->complete + +Lets check if the data was restored successfully. We should see the ``time.log`` file that was removed +before performing restore + +.. code-block:: bash + kubectl exec -it time-logger-6d89687cbb-pv5x6 -n default -it sh + sh-5.1# ls -lrt /var/log + total 16 + -rw-r--r-- 1 root root 12 Jun 5 06:22 test.log + -rw-r--r-- 1 root root 9715 Jun 5 06:32 time.log + sh-5.1# cat /var/log/test.log + hello world + + diff --git a/docs/architecture.rst b/docs/architecture.rst index 1d6e4b3759..a371e68e3e 100644 --- a/docs/architecture.rst +++ b/docs/architecture.rst @@ -7,16 +7,42 @@ Architecture :local: -Kopia Repository Server Controller -================================== +Kopia Repository Server Workflow +======================= +Introducting Kopia +------------------ + +Kopia is a powerful, cross-platform tool for managing encrypted backups in the cloud. +It provides fast and secure backups, using compression, data deduplication, and client-side end-to-end encryption. +It supports a variety of backup storage targets, including object stores, which allows users to choose the storage provider that better addresses their needs. +In Kopia, these storage locations are called repositories. It is a lock-free system that allows concurrent multi-client operations including garbage collection. +To explore other features of Kopia, see its `documentation`_ Kopia Repository Server ----------------------- +A Kopia Repository Server allows Kopia clients proxy access to the backend storage location through it. +Kopia Repository. At any time, a repository server can only connect to a single repository. Due to this a separate instance of the server will be used for each repository. + +In Kanister, the server will comprise a Kubernetes Pod, Service. The pod runs the kopia repository +server process that will be used by kopia clients to perform backup and restore. Kopia clients would +only need a username/password and service name to connect to server without the need to know +the backend storage location. This provides enhanced security since only authorized users will be allowed +to access the kopia repository server. These authorized users need to be added to the server +before starting the server + Kopia Repository ---------------- +The backup storage location is called a "Repository" in Kopia. +Only a single repository can exist at a particular path in the backend storage location. Users opting to use separate repositories are recommended to use unique path prefixes for each repository. +For example, a repository for a namespace called monitoring on S3 storage bucket called test-bucket could be created at the location ``s3://test-bucket//repo/`` +Accessing the repository requires the storage location and credential information similar to a Kanister Profile CR and a unique password used by Kopia during encryption, along with a unique +path prefix mentioned above. + +To know more about the design of kopia repository server controller refer its `design documentation + The design of Kanister follows the operator pattern. This means Kanister defines its own resources and interacts with those resources through a controller. `This blog post