Strimzi Postgres-Debizium Example

Example to create strimzi cluster. Run kafka-connect with debezium plugin for PostgresSql and sink to ElasticSearch

Get Started

1. Clone repo and move inside that directory

   git clone https://github.com/ankur198/Strimzi-Postgres-Debizium-Example
   cd Strimzi-Postgres-Debizium-Example

2. Create Namespace

   kubectl create ns <insert-namespace-name-here>

   eg

   kubectl create ns kafka

3. Switch to namespace

   kubectl config set-context --current --namespace=<insert-namespace-name-here>

4. Create a k8s secret more info

   kubectl create secret docker-registry regcred \
       --docker-server=<your-registry-server> \
       --docker-username=<your-name> \
       --docker-password=<your-pword> \
       --docker-email=<your-email>

   where docker-server will be https://index.docker.io/v1/ in case of docker hub

5. Replace *.sample.yaml with your own file

6. Apply

   ⚠️ This sample is maintained for docker-desktop overlay only.

   kustomize build ./overlays/<overlay-name>/ | kubectl apply -f -

   or

   kubectl apply -k ./overlays/<overlay-name>

   eg

   kubectl apply -k ./overlays/docker-desktop

Validate

1. To get all topics

   kubectl exec cluster-kafka-0 -c kafka -n kafka -it -- \
       bin/kafka-topics.sh \
           --bootstrap-server localhost:9092 \
           --list

2. To consume topic (via CLI)

   kubectl exec cluster-kafka-0 -c kafka -n kafka -it -- \
       bin/kafka-console-consumer.sh \
           --bootstrap-server localhost:9092 \
           --topic <topic-name>

   or

   docker run --rm -it quay.io/strimzi/kafka:0.22.1-kafka-2.7.0 \
       bin/kafka-console-consumer.sh \
       --bootstrap-server <ip>:9094 \
       --topic <topic-name>

3. Create data in DB

   kubectl exec postgres -n db -it -- \
       psql -U postgres strimzi -c \
           "INSERT INTO public.users (name) VALUES ('new user');"

4. Update data in DB

   kubectl exec postgres -n db -it -- \
       psql -U postgres strimzi -c \
           "UPDATE public.users SET name='update user' WHERE name='new user';"

5. See data in DB

   kubectl exec postgres -n db -it -- \
       psql -U postgres strimzi -c \
           "SELECT * FROM public.users;"

See in Kibana

1. Goto https://localhost:5601 to open Kibana dashboard
2. Login with username=kibana and password=foo
3. Open discover and add index.

Logs should be available in discover tab

Resources

• Debezium PostgresSQL Source Connector | Confluent Documentations
• Build your own connector image
• Azure PGSQL CONFIGURATION