Kafka failed to run

[vpankaj97]

System - Windows 11
Docker on WSL2
Note. Inititated from inside WSL2 Ubuntu instance

Log:

2024-06-06 08:50:51 ===> User
2024-06-06 08:50:51 uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
2024-06-06 08:50:51 ===> Setting default values of environment variables if not already set.
2024-06-06 08:50:51 KAFKA_CONTROLLER_LISTENER_NAMES not set. Setting it to default value: "CONTROLLER"
2024-06-06 08:50:51 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR not set. Setting it to default value: "1"
2024-06-06 08:50:51 KAFKA_LOG_DIRS not set. Setting it to default value: "/tmp/kraft-combined-logs"
2024-06-06 08:50:51 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS not set. Setting it to default value: "0"
2024-06-06 08:50:51 KAFKA_CONTROLLER_QUORUM_VOTERS not set. Setting it to default value: "1@localhost:29093"
2024-06-06 08:50:51 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR not set. Setting it to default value: "1"
2024-06-06 08:50:51 KAFKA_NODE_ID not set. Setting it to default value: "1"
2024-06-06 08:50:51 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR not set. Setting it to default value: "1"
2024-06-06 08:50:51 KAFKA_REST_BOOTSTRAP_SERVERS not set. Setting it to default value: "localhost:29092"
2024-06-06 08:50:51 CLUSTER_ID not set. Setting it to default value: "4L6g3nShT-eMCtK--X86sw"
2024-06-06 08:50:51 KAFKA_REST_LISTENERS not set. Setting it to default value: "http://0.0.0.0:8082"
2024-06-06 08:50:51 KAFKA_REST_HOST_NAME not set. Setting it to default value: "rest-proxy"
2024-06-06 08:50:51 KAFKA_PROCESS_ROLES not set. Setting it to default value: "broker,controller"
2024-06-06 08:50:51 ===> Configuring ...
2024-06-06 08:50:51 Running in KRaft mode...
2024-06-06 08:50:51 ===> Running preflight checks ... 
2024-06-06 08:50:51 ===> Check if /var/lib/kafka/data is writable ...
2024-06-06 08:50:51 Error: error in checking operation "writable" on file "/var/lib/kafka/data": permission denied
2024-06-06 08:50:51 Usage:
2024-06-06 08:50:51   ub path <path-to-file> <operation> [flags]
2024-06-06 08:50:51 
2024-06-06 08:50:51 Flags:
2024-06-06 08:50:51   -h, --help   help for path
2024-06-06 08:50:51 
2024-06-06 08:50:51 error in executing the command: error in checking operation "writable" on file "/var/lib/kafka/data": permission denied

[vpankaj97]

Seems like it is related to usage of {HOME} in "volume" field for many services docker-compose file.

Issue:
If {HOME} = XYZ -> then it gets XYZ/data/kafka, which is wrong

Expected -> XYZ/insta-infra/data/kafka (as most will clone this repo)

Fix:
Replace {HOME} with "."

Example -> 

For
volumes:
      - "{HOME}/data/kafka:/var/lib/kafka/data"

We write:
volumes:
      - "./data/kafka:/var/lib/kafka/data"
     

[pflooky]

Thanks for raising this issue @vpankaj97! Initially, I thought of persisting data for services outside of the repo but was under the assumption that it would have write permissions.

I think it is a good idea to persist it in a relative pathway given it should have write access but to persist in a specific folder pattern that all services follow.
I.e. instead of ${HOME}/data/<service>, use ./data/<service>/persist

This will help try to keep the data/<service> folders a bit cleaner. It also allows us to add in the .gitignore to ignore the pattern data/*/persist.

[vpankaj97]

Makes sense.

Currently, I just made: ./data/<service>/engine as the default for RW access.
Edit. In my local system only

Though, I'm still facing issues, ex - MYSQL not working, Cassandra not working, Kafka broker not found.

[vpankaj97]

MYSQL error with conf - ./data/mysql/persist

2024-06-06 10:41:44 2024-06-06T05:11:44.523963Z 0 [System] [MY-015017] [Server] MySQL Server Initialization - start.
2024-06-06 10:41:44 2024-06-06T05:11:44.524634Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.4.0) initializing of server in progress as process 81
2024-06-06 10:41:44 2024-06-06T05:11:44.528895Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-06-06 10:41:44 2024-06-06T05:11:44.631904Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-06-06 10:41:45 2024-06-06T05:11:45.726461Z 0 [ERROR] [MY-000067] [Server] unknown variable 'default-authentication-plugin=mysql_native_password'.
2024-06-06 10:41:45 2024-06-06T05:11:45.726727Z 0 [ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable. You can remove all files that the server added to it.
2024-06-06 10:41:45 2024-06-06T05:11:45.726753Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-06-06 10:41:46 2024-06-06T05:11:46.809215Z 0 [System] [MY-015018] [Server] MySQL Server Initialization - end.
2024-06-06 10:41:44 2024-06-06 05:11:44+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.4.0-1.el9 started.
2024-06-06 10:41:44 2024-06-06 05:11:44+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-06-06 10:41:44 2024-06-06 05:11:44+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.4.0-1.el9 started.
2024-06-06 10:41:44 2024-06-06 05:11:44+00:00 [Note] [Entrypoint]: Initializing database files

[pflooky]

MYSQL error with conf - ./data/mysql/persist

2024-06-06 10:41:44 2024-06-06T05:11:44.523963Z 0 [System] [MY-015017] [Server] MySQL Server Initialization - start.
2024-06-06 10:41:44 2024-06-06T05:11:44.524634Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.4.0) initializing of server in progress as process 81
2024-06-06 10:41:44 2024-06-06T05:11:44.528895Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-06-06 10:41:44 2024-06-06T05:11:44.631904Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-06-06 10:41:45 2024-06-06T05:11:45.726461Z 0 [ERROR] [MY-000067] [Server] unknown variable 'default-authentication-plugin=mysql_native_password'.
2024-06-06 10:41:45 2024-06-06T05:11:45.726727Z 0 [ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable. You can remove all files that the server added to it.
2024-06-06 10:41:45 2024-06-06T05:11:45.726753Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-06-06 10:41:46 2024-06-06T05:11:46.809215Z 0 [System] [MY-015018] [Server] MySQL Server Initialization - end.
2024-06-06 10:41:44 2024-06-06 05:11:44+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.4.0-1.el9 started.
2024-06-06 10:41:44 2024-06-06 05:11:44+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-06-06 10:41:44 2024-06-06 05:11:44+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.4.0-1.el9 started.
2024-06-06 10:41:44 2024-06-06 05:11:44+00:00 [Note] [Entrypoint]: Initializing database files

Thanks for reporting this problem as well. Looks like in version 8.4.0 of MySQL, we need to use the command --mysql-native-password=ON to enable login. Let me add that in.

[pflooky]

MYSQL error with conf - ./data/mysql/persist

2024-06-06 10:41:44 2024-06-06T05:11:44.523963Z 0 [System] [MY-015017] [Server] MySQL Server Initialization - start.
2024-06-06 10:41:44 2024-06-06T05:11:44.524634Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.4.0) initializing of server in progress as process 81
2024-06-06 10:41:44 2024-06-06T05:11:44.528895Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-06-06 10:41:44 2024-06-06T05:11:44.631904Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-06-06 10:41:45 2024-06-06T05:11:45.726461Z 0 [ERROR] [MY-000067] [Server] unknown variable 'default-authentication-plugin=mysql_native_password'.
2024-06-06 10:41:45 2024-06-06T05:11:45.726727Z 0 [ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable. You can remove all files that the server added to it.
2024-06-06 10:41:45 2024-06-06T05:11:45.726753Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-06-06 10:41:46 2024-06-06T05:11:46.809215Z 0 [System] [MY-015018] [Server] MySQL Server Initialization - end.
2024-06-06 10:41:44 2024-06-06 05:11:44+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.4.0-1.el9 started.
2024-06-06 10:41:44 2024-06-06 05:11:44+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-06-06 10:41:44 2024-06-06 05:11:44+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.4.0-1.el9 started.
2024-06-06 10:41:44 2024-06-06 05:11:44+00:00 [Note] [Entrypoint]: Initializing database files

I tried in my laptop by changing to use ./data/mysql/persist and it was okay. Can you try this volume mount instead?
- "./data/mysql/persist:/var/lib/mysql:rw"

[pflooky]

I ran into this issue recently when running on GitHub Actions and it's due to the user on the host being different to the user in the container. I have now changed the default behaviour of insta-infra to not persist data so it should work without having to worry about folder permissions.

To persist data, you now need to include the flag -p when running to persist data (i.e. ./run.sh -p kafka).

I will close this issue now.