mirror of
https://github.com/apache/kafka.git
synced 2026-10-28 08:24:58 +08:00
KAFKA-21258 Move the Docker image usage guide from docker/examples/README.md to the website docs (#23819)
Move the Docker image usage guide (default configs, file input, environment variables, SASL and SSL modes) from docker/examples/README.md to docs/getting-started/docker.md so it shows on the website. The README keeps the docker compose examples and links to the docs page. Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
This commit is contained in:
Murali Basani authored and GitHub committed
1 parent
46dc4b51f7
commit
f45fc4bbd3
3 files changed
+94
-94
No files matched your search
+1
-1
@@ -180,7 +180,7 @@ docker buildx imagetools create --tag apache/kafka-native:3.8.0 apache/kafka-nat
|
||||
Using the image in a docker container
|
||||
-------------------------------------
|
||||
|
||||
Please check [this](./examples/README.md) for usage guide of the docker image.
|
||||
Please check [docs/getting-started/docker.md](../docs/getting-started/docker.md) for the usage guide of the docker image, and [examples/README.md](./examples/README.md) for Docker Compose examples.
|
||||
|
||||
Releasing the Docker Official Image
|
||||
-----------------------------------
|
||||
|
||||
@@ -1,100 +1,15 @@
|
||||
Kafka Docker Image Usage Guide
|
||||
==============================
|
||||
Kafka Docker Image Examples
|
||||
===========================
|
||||
|
||||
Introduction
|
||||
------------
|
||||
|
||||
This document contains usage guide as well as examples for Docker image.
|
||||
This document contains examples for the Docker image.
|
||||
Docker Compose files are provided in this directory for the example use cases.
|
||||
|
||||
Kafka server can be started using following ways:
|
||||
|
||||
- Default configs
|
||||
- File input
|
||||
- Environment variables
|
||||
|
||||
Installation Preparation
|
||||
------------
|
||||
|
||||
Note that the `Docker` version **must be >= 20.10.4**.
|
||||
|
||||
The prior Docker versions may cause permission errors when running the Kafka container, as they do not correctly set directory permissions when creating container paths like `/opt/kafka/config`.
|
||||
|
||||
If you are using the prior version, you may encounter the following error during container startup:
|
||||
```text
|
||||
===> User
|
||||
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
|
||||
===> Setting default values of environment variables if not already set.
|
||||
===> Configuring …
|
||||
Running in KRaft mode…
|
||||
/opt/kafka/config/ file not writable
|
||||
```
|
||||
|
||||
To avoid this, **please upgrade Docker to 20.10.4 or later**.
|
||||
|
||||
Running on default configs
|
||||
--------------------------
|
||||
|
||||
If no user provided configuration (file input or environment variables) is passed to the Docker container, the default KRaft configuration for single combined-mode node will be used.
|
||||
This default configuration is packaged in the Kafka tarball.
|
||||
|
||||
Use input file for providing configs
|
||||
------------------------------------
|
||||
|
||||
- This method requires users to provide path to a local folder which contains the Kafka property files and mount it to Docker container using Docker volume.
|
||||
- It replaces the default KRaft configuration file present in Docker container.
|
||||
- The Command `docker run --volume /path/to/property/folder:/mnt/shared/config -p 9092:9092 apache/kafka:latest` can be used to mount the folder containing the property files.
|
||||
- Property files will be only read by the Docker container.
|
||||
|
||||
Using Environment Variables
|
||||
---------------------------
|
||||
|
||||
When using the environment variables, you need to set all properties required to start the KRaft node.
|
||||
Therefore, the recommended way to use environment variables is via Docker Compose, which allows users to set all the properties that are needed.
|
||||
It is also possible to use the input file to have a common set of configurations, and then override specific node properties using the environment variables.
|
||||
|
||||
- Kafka property defined via environment variables will override the value of that property defined in the user provided property file.
|
||||
- If properties are provided via environment variables only, all required properties must be specified.
|
||||
- The following rules must be used to construct the environment variable key name:
|
||||
- Replace `.` with `_`
|
||||
- Replace `_` with `__` (double underscore)
|
||||
- Replace `-` with `___` (triple underscore)
|
||||
- Prefix the result with `KAFKA_`
|
||||
- Examples:
|
||||
- For `abc.def`, use `KAFKA_ABC_DEF`
|
||||
- For `abc-def`, use `KAFKA_ABC___DEF`
|
||||
- For `abc_def`, use `KAFKA_ABC__DEF`
|
||||
|
||||
- To provide configs to log4j property files, following points should be considered:
|
||||
- log4j properties provided via environment variables will be appended to the default properties file (log4j properties files bundled with Kafka).
|
||||
- `KAFKA_LOG4J_ROOT_LOGLEVEL` can be provided to set the value of `log4j.rootLogger` in `log4j2.yaml` and `tools-log4j2.yaml`.
|
||||
- log4j loggers can be added to `log4j2.yaml` by setting them in `KAFKA_LOG4J_LOGGERS` environment variable in a single comma separated string.
|
||||
- Example:
|
||||
- Assuming that `KAFKA_LOG4J_LOGGERS='property1=value1,property2=value2'` environment variable is provided to Docker container.
|
||||
- `log4j.logger.property1=value1` and `log4j.logger.property2=value2` will be added to the `log4j2.yaml` file inside Docker container.
|
||||
|
||||
Running in SASL mode
|
||||
--------------------
|
||||
|
||||
- SASL mode requires a JAAS configuration file to be mounted into the Docker container and referenced via the `KAFKA_OPTS` environment variable.
|
||||
- Mount the directory containing the JAAS config file to `/etc/kafka/secrets` in the Docker container.
|
||||
- Set `KAFKA_OPTS` to `-Djava.security.auth.login.config=/etc/kafka/secrets/<jaas_config_filename>`.
|
||||
- Set `KAFKA_SASL_ENABLED_MECHANISMS` to the desired SASL mechanism (e.g. `GSSAPI` , `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`).
|
||||
- Ensure `KAFKA_ADVERTISED_LISTENERS` contains a `SASL_PLAINTEXT://` or `SASL_SSL://` listener.
|
||||
- For inter-broker SASL communication, set `KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL` to the desired SASL *mechanism* (e.g. `PLAIN`, `SCRAM-SHA-256`). To choose between `SASL_PLAINTEXT` and `SASL_SSL` for inter-broker traffic, set `KAFKA_INTER_BROKER_LISTENER_NAME` to a listener whose security protocol is mapped accordingly in `KAFKA_LISTENER_SECURITY_PROTOCOL_MAP`.
|
||||
- The Docker image `configure` script will log a warning if `KAFKA_OPTS` does not contain the `java.security.auth.login.config` property when SASL listeners are detected.
|
||||
- For more details on SASL mechanisms and configuration, refer to the [Kafka SASL authentication documentation](https://kafka.apache.org/documentation/#security_sasl).
|
||||
|
||||
Running in SSL mode
|
||||
-------------------
|
||||
|
||||
- Recommended way to run in ssl mode is by mounting secrets on `/etc/kafka/secrets` in Docker container and providing configs following through environment variables (`KAFKA_SSL_KEYSTORE_FILENAME`, `KAFKA_SSL_KEYSTORE_CREDENTIALS`, `KAFKA_SSL_KEY_CREDENTIALS`, `KAFKA_SSL_TRUSTSTORE_FILENAME` and `KAFKA_SSL_TRUSTSTORE_CREDENTIALS`) to let the Docker image scripts extract passwords and populate correct paths in `server.properties`.
|
||||
- Please ensure appropriate `KAFKA_ADVERTISED_LISTENERS` are provided through environment variables to enable SSL mode in Kafka server, i.e. it should contain an `SSL` listener.
|
||||
- Alternatively property file input can be used to provide ssl properties.
|
||||
- Make sure you set location of truststore and keystore correctly when using file input. See example for file input in `docker-compose-files/single-node/file-input` for better clarity.
|
||||
- Note that advertised.listeners property needs to be provided along with SSL properties in file input and cannot be provided through environment variable separately.
|
||||
- In conclusion, ssl properties with advertised.listeners should be treated as a group and provided in file input or environment variables in it's entirety.
|
||||
- In case ssl properties are provided both through file input and environment variables, environment variable properties will override the file input properties, just as mentioned in the beginning of this section.
|
||||
For the usage guide of the Docker image (default configs, file input, environment variables, SASL and SSL modes),
|
||||
see the **Getting Started > Docker** page of the Kafka documentation on the website
|
||||
(source: [docs/getting-started/docker.md](../../docs/getting-started/docker.md)).
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
@@ -73,8 +73,93 @@ $ docker run -p 9092:9092 apache/kafka-native:4.4.0
|
||||
|
||||
### SASL and SSL
|
||||
|
||||
The native image supports SASL authentication since 4.4.0; earlier native images do not. Docker Compose examples for SASL and SSL, which work with both the JVM and the native image, are described in the [usage guide](https://github.com/apache/kafka/blob/trunk/docker/examples/README.md).
|
||||
The native image supports SASL authentication since 4.4.0; earlier native images do not. See [Running in SASL mode](#running-in-sasl-mode) and [Running in SSL mode](#running-in-ssl-mode) below; both apply to the JVM and the native image.
|
||||
|
||||
## Usage guide
|
||||
|
||||
Detailed instructions for using the Docker image are mentioned [here](https://github.com/apache/kafka/blob/trunk/docker/examples/README.md).
|
||||
The Kafka server in the Docker container can be configured in the following ways:
|
||||
|
||||
- Default configs
|
||||
- File input
|
||||
- Environment variables
|
||||
|
||||
### Installation preparation
|
||||
|
||||
Note that the Docker version **must be >= 20.10.4**.
|
||||
|
||||
Earlier Docker versions may cause permission errors when running the Kafka container, as they do not correctly set directory permissions when creating container paths like `/opt/kafka/config`.
|
||||
|
||||
With an earlier version, you may encounter the following error during container startup:
|
||||
|
||||
```text
|
||||
===> User
|
||||
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
|
||||
===> Setting default values of environment variables if not already set.
|
||||
===> Configuring …
|
||||
Running in KRaft mode…
|
||||
/opt/kafka/config/ file not writable
|
||||
```
|
||||
|
||||
To avoid this, **please upgrade Docker to 20.10.4 or later**.
|
||||
|
||||
### Running on default configs
|
||||
|
||||
If no user provided configuration (file input or environment variables) is passed to the Docker container, the default KRaft configuration for a single combined-mode node will be used. This default configuration is packaged in the Kafka tarball.
|
||||
|
||||
### Using an input file for providing configs
|
||||
|
||||
- This method requires users to provide the path to a local folder which contains the Kafka property files and mount it to the Docker container using a Docker volume.
|
||||
- It replaces the default KRaft configuration file present in the Docker container.
|
||||
- The following command can be used to mount the folder containing the property files:
|
||||
|
||||
```bash
|
||||
$ docker run --volume /path/to/property/folder:/mnt/shared/config -p 9092:9092 apache/kafka:latest
|
||||
```
|
||||
|
||||
- Property files will only be read by the Docker container.
|
||||
|
||||
### Using environment variables
|
||||
|
||||
When using environment variables, you need to set all properties required to start the KRaft node. Therefore, the recommended way to use environment variables is via Docker Compose, which allows users to set all the properties that are needed. It is also possible to use the input file to have a common set of configurations, and then override specific node properties using environment variables.
|
||||
|
||||
- A Kafka property defined via an environment variable will override the value of that property defined in the user provided property file.
|
||||
- If properties are provided via environment variables only, all required properties must be specified.
|
||||
- The following rules must be used to construct the environment variable key name:
|
||||
- Replace `.` with `_`
|
||||
- Replace `_` with `__` (double underscore)
|
||||
- Replace `-` with `___` (triple underscore)
|
||||
- Prefix the result with `KAFKA_`
|
||||
- Examples:
|
||||
- For `abc.def`, use `KAFKA_ABC_DEF`
|
||||
- For `abc-def`, use `KAFKA_ABC___DEF`
|
||||
- For `abc_def`, use `KAFKA_ABC__DEF`
|
||||
- To provide configs to log4j property files, the following points should be considered:
|
||||
- log4j properties provided via environment variables will be appended to the default properties file (log4j properties files bundled with Kafka).
|
||||
- `KAFKA_LOG4J_ROOT_LOGLEVEL` can be provided to set the value of `log4j.rootLogger` in `log4j2.yaml` and `tools-log4j2.yaml`.
|
||||
- log4j loggers can be added to `log4j2.yaml` by setting them in the `KAFKA_LOG4J_LOGGERS` environment variable as a single comma separated string.
|
||||
- Example: assuming that `KAFKA_LOG4J_LOGGERS='property1=value1,property2=value2'` is provided to the Docker container, `log4j.logger.property1=value1` and `log4j.logger.property2=value2` will be added to the `log4j2.yaml` file inside the Docker container.
|
||||
|
||||
### Running in SASL mode
|
||||
|
||||
- SASL mode requires a JAAS configuration file to be mounted into the Docker container and referenced via the `KAFKA_OPTS` environment variable.
|
||||
- Mount the directory containing the JAAS config file to `/etc/kafka/secrets` in the Docker container.
|
||||
- Set `KAFKA_OPTS` to `-Djava.security.auth.login.config=/etc/kafka/secrets/<jaas_config_filename>`.
|
||||
- Set `KAFKA_SASL_ENABLED_MECHANISMS` to the desired SASL mechanism (e.g. `GSSAPI`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`).
|
||||
- Ensure `KAFKA_ADVERTISED_LISTENERS` contains a `SASL_PLAINTEXT://` or `SASL_SSL://` listener.
|
||||
- For inter-broker SASL communication, set `KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL` to the desired SASL *mechanism* (e.g. `PLAIN`, `SCRAM-SHA-256`). To choose between `SASL_PLAINTEXT` and `SASL_SSL` for inter-broker traffic, set `KAFKA_INTER_BROKER_LISTENER_NAME` to a listener whose security protocol is mapped accordingly in `KAFKA_LISTENER_SECURITY_PROTOCOL_MAP`.
|
||||
- The Docker image `configure` script will log a warning if `KAFKA_OPTS` does not contain the `java.security.auth.login.config` property when SASL listeners are detected.
|
||||
- For more details on SASL mechanisms and configuration, refer to the [Kafka SASL authentication documentation](https://kafka.apache.org/documentation/#security_sasl).
|
||||
|
||||
### Running in SSL mode
|
||||
|
||||
- The recommended way to run in SSL mode is by mounting secrets on `/etc/kafka/secrets` in the Docker container and providing the following configs through environment variables: `KAFKA_SSL_KEYSTORE_FILENAME`, `KAFKA_SSL_KEYSTORE_CREDENTIALS`, `KAFKA_SSL_KEY_CREDENTIALS`, `KAFKA_SSL_TRUSTSTORE_FILENAME` and `KAFKA_SSL_TRUSTSTORE_CREDENTIALS`. This lets the Docker image scripts extract passwords and populate the correct paths in `server.properties`.
|
||||
- Please ensure appropriate `KAFKA_ADVERTISED_LISTENERS` are provided through environment variables to enable SSL mode in the Kafka server, i.e. it should contain an `SSL` listener.
|
||||
- Alternatively, property file input can be used to provide SSL properties.
|
||||
- Make sure you set the location of the truststore and keystore correctly when using file input. See the [file input example](https://github.com/apache/kafka/tree/trunk/docker/examples/docker-compose-files/single-node/file-input) for reference.
|
||||
- Note that the `advertised.listeners` property needs to be provided along with the SSL properties in file input and cannot be provided through an environment variable separately.
|
||||
- In conclusion, SSL properties with `advertised.listeners` should be treated as a group and provided in file input or environment variables in their entirety.
|
||||
- In case SSL properties are provided both through file input and environment variables, environment variable properties will override the file input properties, as mentioned in [Using environment variables](#using-environment-variables).
|
||||
|
||||
### Examples
|
||||
|
||||
Docker Compose examples for single-node and multi-node (combined and isolated mode) clusters, covering plaintext, SSL, SASL and file input setups for both the JVM and the native image, are available in the [Kafka repository](https://github.com/apache/kafka/blob/trunk/docker/examples/README.md).
|
||||
Reference in new issue
Block a user