Rewrite TransactionsBounceTest from Scala to Java and move it to
clients-integration-tests. The transaction status header helpers are
moved to ClientsTestUtils and shared with ProducerIdExpirationTest, and
the Scala helpers in TestUtils that were only used by this test are
removed.
Reviewers: Murali Basani <muralidhar.basani@aiven.io>, Chia-Ping Tsai
<chia7712@gmail.com>
`SubscriptionState#unsubscribe()` resets the listener context to
`NULL_LISTENER`, so calling `unsubscribe()` silently drops the user's
rebalance listener.
KIP-1306 only states that passing null removes the current listener. The
previous implementation also did not reset the listener on
unsubscribe().
Reviewers: ChickenchickenLove <ojt90902@naver.com>, Chia-Ping Tsai
<chia7712@gmail.com>
Since the 4.0 baseline `MetadataVersion` is 3.0, all brokers support
truncation on fetch (`Fetch` v12+ with `LastFetchedEpoch` and
`DivergingEpoch`). Fetcher threads can therefore skip the separate
`OffsetsForLeaderEpochRequest` and detect divergence directly from fetch
responses.
Reviewers: Jun Rao <junrao@gmail.com>, Chia-Ping Tsai
<chia7712@gmail.com>
Fix a bug in `ClientMetricsManager.processPushTelemetryRequest()` where
the `terminating` flag is unconditionally set in the `finally` block,
even when request validation fails.
When a client sends a `PushTelemetry` request with `terminating=true`
but the request fails validation (e.g., stale `subscriptionId` →
`UNKNOWN_SUBSCRIPTION_ID`), the client instance is permanently marked
as terminating. All subsequent requests from that client are rejected
with `INVALID_REQUEST`, making recovery impossible without creating a
new client instance.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Apoorv Mittal <apoorvmittal10@gmail.com>
Move duplicate tests from `ShareHeartbeatRequestManagerTest` and
`ConsumerHeartbeatRequestManagerTest` to
`AbstractHeartbeatRequestManagerTest`.
Reviewers: Lianet Magrans <lmagrans@confluent.io>
Follow-up to (#23227).
Before bootstrap DNS resolution completes,
`coordinatorRequestManager.coordinator()` is empty, causing the
coordinator-unknown path in `maximumTimeToWait()` to return
`heartbeatRequestState.heartbeatIntervalMs()`. However, the heartbeat
interval is initialized to 0 and is not updated until the first
heartbeat response is received. As a result, a member in the JOINING
state reports `maximumTimeToWait() == 0`.
Since the application thread's poll timeout is bounded by this value,
both the application and network threads can busy-spin until the
coordinator is discovered.
Returning `retryBackoffMs` aligns with the behavior discussed in #23227.
Until the coordinator is discovered, no heartbeat can be sent, making
this equivalent to the "nothing to send right now, wait for a backoff"
case for which `FetchRequestManager#maximumTimeToWait()` already returns
the retry backoff. Coordinator discovery itself is also paced by the
coordinator manager's backoff.
Reviewers: Lianet Magrans <lmagrans@confluent.io>
`CommitRequestManager#maximumTimeToWait()` returned the raw remaining
time of the auto-commit timer, which becomes `0` once the interval
elapses. However, `poll()` cannot actually send an auto-commit until
the coordinator is known; it returns `EMPTY` while
`coordinatorRequestManager.coordinator()` is empty. When bootstrap DNS
resolution is still in progress (`bootstrap.resolve.timeout.ms > 0`) and
`auto.commit.interval.ms` is shorter than the resolution window, the
auto-commit timer can expire before the coordinator is discovered.
`remainingMs()` then continues to return `0` indefinitely, since the
timer is not reset until the application thread successfully sends an
auto-commit.
This `0` is propagated to
`ConsumerNetworkThread#cachedMaximumTimeToWait`, which bounds the
`pollTimeout` in `AsyncKafkaConsumer#pollForFetches()` to `0`, causing
the application and network threads to busy-loop for as long as the
coordinator remains unknown.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Andrew Schofield
<aschofield@confluent.io>, TengYao Chi <frankvicky@apache.org>, Lianet
Magrans <lmagrans@confluent.io>
1. Add an evolving tag to BootstrapResolutionException.
2. Add docs stating that setting a positive value for
bootstrap.resolve.timeout.ms is evolving.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, TengYao Chi
<frankvicky@apache.org>, Andrew Schofield <aschofield@confluent.io>
`FetchRequestManager` currently wakes up the `FetchBuffer`
unconditionally whenever `prepareFetchRequests()` returns no requests
to send. This behavior was intended for one specific case: all
currently fetchable partitions already have buffered data waiting to be
consumed. However, the wakeup also happens for any other case that
produces an empty result, including states where no progress can be
made until some external event occurs:
- No partitions are fetchable at all (no assignment yet, invalid
positions, paused, pending revocation/callback). - The
partition's leader is unknown (`metadata.requestUpdate()` was just
triggered). - The target node is in its reconnect backoff window.
- The target node already has an in-flight fetch request outstanding.
- The target node hosts other partitions that already have buffered
data.
Because this wakeup bypasses the caller's normal `retryBackoffMs` delay,
the application thread and the background `ConsumerNetworkThread` can
enter a tight loop: `poll()` issues a fetch-request event, the
background thread finds nothing to send, wakes the buffer, `poll()`
returns immediately, and the cycle repeats without any meaningful
delay.
This is a pre-existing issue introduced with the non-blocking
`AsyncPollEvent` redesign in KAFKA-18376. However, KIP-909's async
bootstrap DNS resolution extends the "no node available" window from
effectively instantaneous to up to `bootstrapResolveTimeoutMs`, making
the busy loop much more visible.
Reviewers: Andrew Schofield <aschofield@confluent.io>, Lianet Magrans
<lmagrans@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>
Enhanced the auto.offset.reset documentation in ConsumerConfig.java to
clearly explain the message-loss risk when using latest after increasing
the partition count, and recommended by_duration:<duration> as a safer
alternative.
Updated `basic-kafka-operations.md` with a recommendation section and a
comparison table for suitable `<duration>` values under both the
consumer (KIP-848) and classic group protocols.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Currently, kafka-producer-perf-test.sh only produces records with null keys, which makes it impossible to benchmark real-world keyed workloads (e.g., compacted topics, semantic partitioning, and stream joins). This PR implements KIP-1299 to introduce key generation support for the producer performance tool.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Followup from:
https://github.com/apache/kafka/pull/22521#discussion_r3491032876
Migrate the streams group command integration tests from
EmbeddedKafkaCluster to the ClusterInstance test framework:
- ResetIntegrationTest.java
- AbstractResetIntegrationTest.java
- ResetIntegrationWithSslTest.java
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Cleanup unused dependencies in `build.gradle`
1. `libs.spotbugs`: he SpotBugs Gradle plugin performs static analysis
using its own bundled jar, it does not require this compile-time
dependency.
2. `libs.jacksonJDK8Datatypes`: Jackson 2.15+ baked Optional support
directly into jackson-databind core.
3. `libs.jmhCoreBenchmarks`: Benchmarks only use jmh-core (annotations,
Blackhole, Runner) and jmh-generator-annprocess. No source file imports
anything from the org.openjdk.jmh.samples or benchmarks package.
4. `libs.bndlib`: Safely removed as no modules currently depend on it.
5. `libs.jacksonDataformatYaml` & `libs.junitPlatformLanucher`: Removed
from individual submodules because they are already globally included in
the runtimeTestLibs array.
6. `libs.caffeine` & `libs.hdrHistogram`: group-coordinator does not use
them
7. `libs.jacksonDataformatCsv`, `libs.jacksonJakartarsJsonProvider`, and
`libs.junitPlatformSuiteEngine`: Removed from specific modules as they
are either no longer directly imported or handled implicitly
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Move `RequestChannelTest.scala` (core) to `RequestTest.java` (server
module), rewriting it in Java as part of the ongoing Scala-to-Java test
migration.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Since `testUnsubscribeDoesNotCommitOffsetsEvenWithAutoCommitEnabled`
enables auto-commit, closing the consumer in the `@AfterEach`
teardown triggers `autoCommitOnClose()`, which calls
`commitSyncAllConsumed()` and blocks waiting on the resulting
`SyncCommitEvent` future:
```java
private void autoCommitOnClose(final Timer timer) {
if (groupMetadata.get().isEmpty() || applicationEventHandler ==
null)
return;
if (autoCommitEnabled) commitSyncAllConsumed(timer); //
sends a blocking SyncCommitEvent
applicationEventHandler.add(new CommitOnCloseEvent());
}
```
Because the test mocks applicationEventHandler and never completes
this event,
close() blocks until the timeout.
This patch completes the commit event during close by calling
completeCommitSyncApplicationEventSuccessfully() at the end of the test
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Follow-up to #20707, which unified bootstrap metadata into the KRaft
zero checkpoint. As a side effect, a directory produced by a newer
`kafka-storage` tool can no longer be started by an older Kafka broker —
even if `--release-version` was set to an older version at format time.
The flag only chooses the initial feature levels; the on-disk checkpoint
layout itself is dictated by the tool's own code version.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Gradle test fixtures to the clients module and removes all dependencies
on sourceSets.test.output. We should revisit other usages in this JIRA.
This will simplify the dependency graph and unblock the upgrade to
Gradle 9.5+
This patch is working on `server` and `storage`
Reviewers: Murali Basani <muralidhar.basani@aiven.io>, Chia-Ping Tsai
<chia7712@gmail.com>
Gradle test fixtures to the clients module and removes all dependencies
on sourceSets.test.output. We should revisit other usages in this JIRA.
This will simplify the dependency graph and unblock the upgrade to
Gradle 9.5+
This patch is working on `group-coordinator` and `share-coordinator`
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Gradle test fixtures to the clients module and removes all dependencies
on sourceSets.test.output. We should revisit other usages in this JIRA.
This will simplify the dependency graph and unblock the upgrade to
Gradle 9.5+
This patch is working on server-common module
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
The new test framework does not yet support the SASL_SSL security
protocol; we should add support for it.
Reviewers: Igor Soarez <i@soarez.me>, Chia-Ping Tsai
<chia7712@gmail.com>
Some methods are only used in a single class; they should be moved into
that class. Java tests should not use TestUtils.scala
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
We have exposed several classes in org.apache.kafka.common.utils as
public APIs, which can be misleading since not all classes in this
package are intended for external use.
So move following class into internal module
- OperatingSystem
- Java
- Exit
- Shell
- LoggingSignalHandler
Reviewers: cychiu <cychiu8@gmail.com>, Chia-Ping Tsai
<chia7712@gmail.com>
We have exposed several classes in org.apache.kafka.common.utils as
public APIs, which can be misleading since not all classes in this
package are intended for external use.
So move ByteBufferUnmapper, BufferSupplier, ChunkedBytesStream into
internal module
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
We have exposed several classes in org.apache.kafka.common.utils as
public APIs, which can be misleading since not all classes in this
package are intended for external use.
To clarify API boundaries and prevent unintended usage, we propose
moving the following classes to an internal package:
- AbstractIterator
- CircularIterator
- CloseableIterator
- CopyOnWriteMap
- FixedOrderMap
- PrimitiveRef
Reviewers: Christo <lolovc@amazon.com>, PoAn Yang <payang@apache.org>
We add a new map to record which topic partitions have experienced
overflow. When an overflow occurs, the next time the group is
processed, we reduce the segment size by a factor of 0.9 to prevent the
overflow from happening again. If the partition still overflows, we
continue to multiply the ratio by 0.9 on subsequent attempts until the
partition is successfully cleaned.
Reviewers: Jun Rao <junrao@gmail.com>, Chia-Ping Tsai
<chia7712@gmail.com>
The PR fixed an NPE when connectorStatusMetrics unregistering task. The
issue will happen if a connector has more than 1 task on one worker and
these tasks fail to start. For example, if the DNS cant resolve producer
URL, tasks will fail to build. If these tasks failing to start,
ConnectorStatusMetricsGroup will unregister them. After first task
unregistered , connectorStatusMetrics removed connector. When second
task calls `recordTaskRemoved()`, connectorStatusMetrics is empty, so
`connectorStatusMetrics.get(connectorTaskId.connector()).close();` will
throw NPE. Like in method `recordTaskAdded()`, this fix checks
connectorStatusMetrics at the beginning of method `recordTaskRemoved()`.
Co-authored-by: Fan Yang <fan365@hotmail.com>
Reviewers: Ding <isDing_L@163.com>, Chia-Ping Tsai <chia7712@gmail.com>
From `testReadWithMinMessage` to
`testFetchLatestTieredTimestampWithRemoteStorage`
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Chia-Ping Tsai
<chia7712@gmail.com>
The following configs are weird because those values are definitely not
READONLY
- leader.replication.throttled.rate
- follower.replication.throttled.rate
- replica.alter.log.dirs.io.max.bytes.per.second
Add them into the `DynamicBrokerConfig` to ensure when we query it's not
READONLY
Reviewers: Lan Ding <isDing_L@163.com>, Chia-Ping Tsai
<chia7712@gmail.com>
Since Kafka 4.2.0 has not been released yet, using str(LATEST_4_2)
causes the e2e tests to fail, so we should remove it.
Reviewers: TengYao Chi <frankvicky@apache.org>, Chia-Ping Tsai
<chia7712@gmail.com>
In Transaction Version 2, strict epoch validation (`markerEpoch >
currentEpoch`) causes hanging transactions in two scenarios:
1. **Coordinator recovery**: When reloading PREPARE_COMMIT/ABORT from
transaction log, retried markers are rejected with
`InvalidProducerEpochException` because they use the same epoch
2. **Network retry**: When marker write succeeds but response is lost,
coordinator retries are rejected for the same reason
Both cases leave transactions permanently hanging in PREPARE state,
causing clients to fail with `CONCURRENT_TRANSACTIONS`.
Detect idempotent marker retries in
`ProducerStateManager.checkProducerEpoch()` by checking three
conditions:
1. Transaction Version ≥ 2
2. markerEpoch == currentEpoch (same epoch)
3. currentTxnFirstOffset is empty (transaction already completed)
When all conditions are met, treat the marker as a successful idempotent
retry instead of throwing an error.
Reviewers: Justine Olshan <jolshan@confluent.io>, TaiJuWu
<tjwu1217@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
FYI: https://github.com/apache/kafka/pull/20175/files#r2572693459
This PR optimizes the logging mechanism in PersisterStateManager to
reduce CPU overhead and fix issues preventing dynamic log level
configuration for internal handlers.
Replaced the usage of getCanonicalName() with the standard class-based
logger initialization (e.g., LoggerFactory.getLogger(...)).
Reviewers: Sushant Mahajan <smahajan@confluent.io>, Chia-Ping Tsai
<chia7712@gmail.com>
While running Kafka e2e tests, various tests were failing with
`TimeoutError('Kafka node failed to stop in 60 seconds')`. In Kafka e2e
tests, we check the PID to ensure the Kafka server has shut down. After
investigating this issue, I found that the Kafka process was a zombie
process in the container:
```bash
ducker@ducker05:/$ jcmd
285 kafka.Kafka /mnt/kafka/kafka.properties
18207 jdk.jcmd/sun.tools.jcmd.JCmd
ducker@ducker05:/$ cat /proc/285/status | grep -i state
State: Z (zombie)
```
This issue is related to [this
change](https://github.com/apache/kafka/pull/17554/files#r1845737954).
When using `CMD ["sudo", "service", "ssh", "start", "-D"]`, PID 1
becomes the SSH service, which does not handle `SIGCHLD` signals and
therefore won't reap zombie processes:
```bash
ducker@ducker05:/$ cat /proc/1/cmdline | tr '\0' ' '
sudo service ssh start -D
```
However, with the old syntax `CMD sudo service ssh start && tail -f
/dev/null`, PID 1 is `/bin/sh`, which is a shell that properly reaps
zombie processes:
```bash
ducker@ducker05:/$ cat /proc/1/cmdline | tr '\0' ' '
/bin/sh -c sudo service ssh start && tail -f /dev/null
```
Use `tini` as PID 1 to properly manage processes and avoid zombie
processes from remaining in the system.
Reviewers: PoAn Yang <payang@apache.org>, TaiJuWu <tjwu1217@gmail.com>,
Chia-Ping Tsai <chia7712@gmail.com>
The openjdk image has been officially deprecated (ref:
https://hub.docker.com/_/openjdk ), so we’ll need to switch to another
base image for our E2E test environment.
I tested this on my local machine using the following command:
```
jdk_version="sapmachine:17-jdk-ubuntu-jammy" \
image_name="ducker-ak:$(git rev-parse --short HEAD)" \
TC_PATHS="tests/kafkatest/tests/client/pluggable_test.py" \
/bin/bash tests/docker/run_tests.sh
```
result:
```
[INFO:2025-11-14 14:40:35,072]: RunnerClient:
kafkatest.tests.client.pluggable_test.PluggableConsumerTest.test_start_stop.metadata_quorum=ISOLATED_KRAFT:
PASS
[WARNING - 2025-11-14 14:40:35,072 - runner_client - log - lineno:459]:
RunnerClient:
kafkatest.tests.client.pluggable_test.PluggableConsumerTest.test_start_stop.metadata_quorum=ISOLATED_KRAFT:
Test requested 4 nodes, used only 3
[WARNING:2025-11-14 14:40:35,072]: RunnerClient:
kafkatest.tests.client.pluggable_test.PluggableConsumerTest.test_start_stop.metadata_quorum=ISOLATED_KRAFT:
Test requested 4 nodes, used only 3
[INFO:2025-11-14 14:40:35,073]: RunnerClient:
kafkatest.tests.client.pluggable_test.PluggableConsumerTest.test_start_stop.metadata_quorum=ISOLATED_KRAFT:
Data: None
================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.12.0
session_id: 2025-11-14--004
run time: 13.568 seconds
tests run: 1
passed: 1
flaky: 0
failed: 0
ignored: 0
================================================================================
test_id:
kafkatest.tests.client.pluggable_test.PluggableConsumerTest.test_start_stop.metadata_quorum=ISOLATED_KRAFT
status: PASS
run time: 13.460 seconds
--------------------------------------------------------------------------------
```
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Ming-Yen Chung
<mingyen066@gmail.com>, PoAn Yang <payang@apache.org>
In Kafka Streams, configuration classes typically follow a fluent API
pattern to ensure a consistent and intuitive developer experience.
However, the current implementation of
`org.apache.kafka.streams.KafkaStreams$CloseOptions` deviates from this
convention by exposing a public constructor, breaking the uniformity
expected across the API.
To address this inconsistency, we propose introducing a new
`CloseOptions` class that adheres to the fluent API style, replacing the
existing implementation. The new class will retain the existing
`timeout(Duration)` and `leaveGroup(boolean)` methods but will enforce
fluent instantiation and configuration. Given the design shift, we will
not maintain backward compatibility with the current class.
This change aligns with the goal of standardizing configuration objects
across Kafka Streams, offering developers a more cohesive and
predictable API.
Reviewers: Bill Bejeck<bbejeck@apache.org>
We can rewrite this class from scala to java and move to server-common
module. To maintain backward compatibility, we should keep the logger
name `state.change.logger`.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Rename org.apache.kafka.server:type=AssignmentsManager and
org.apache.kafka.storage.internals.log.RemoteStorageThreadPool metrics
for the consist, these metrics should be
- `kafka.log.remote:type=...`
- `kafka.server:type=...`
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Jira: https://issues.apache.org/jira/browse/KAFKA-17554
In the previous workflow, the test passes under two conditions:
1. The `t1` thread is waiting for the main thread's `client.wakeup()`.
If successful, `t1` will wake up `t2`, allowing `t2` to complete the
future.
2. If `t1` fails to receive the `client.wakeup()` from the main thread,
`t2` will be woken up by the main thread.
In the previous implementation, we used a `CountDownLatch` to control
the execution of three threads, but it often led to race conditions.
Currently, we have modified it to use two threads to test this scenario.
I run `I=0; while ./gradlew :clients:test --tests
ConsumerNetworkClientTest.testFutureCompletionOutsidePoll --rerun
--fail-fast; do (( I=$I+1 )); echo "Completed run: $I"; sleep 1; done`
and pass 3000+ times.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
All Kafka component register AppInfo metrics to track the application
start time or commit-id etc. These metrics are useful for monitoring and
debugging. However, the AppInfo doesn't provide client-id, which is an
important information for custom metrics reporter.
The AppInfoParser class registers a JMX MBean with the provided
client-id, but when it adds metrics to the Metrics registry, the
client-id is not included. This KIP aims to add the client-id as a tag.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
The current `metadata-shell` find command throws an exception due to
child node `zkMigrationState`.
This interrupts the output and makes the CLI less usable.
Additionally, `zkMigrationState` is no longer used in Kafka 4.x, but it
is still defined under image/features, which causes misleading error
messages.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
We add the three main changes in this PR
- Disallowing null values for most LIST-type configurations makes sense,
since users cannot explicitly set a configuration to null in a
properties file. Therefore, only configurations with a default value of
null should be allowed to accept null.
- Disallowing duplicate values is reasonable, as there are currently no
known configurations in Kafka that require specifying the same value
multiple times. Allowing duplicates is both rare in practice and
potentially confusing to users.
- Disallowing empty list, even though many configurations currently
accept them. In practice, setting an empty list for several of these
configurations can lead to server startup failures or unexpected
behavior. Therefore, enforcing non-empty lists helps prevent
misconfiguration and improves system robustness.
These changes may introduce some backward incompatibility, but this
trade-off is justified by the significant improvements in safety,
consistency, and overall user experience.
Additionally, we introduce two minor adjustments:
- Reclassify some STRING-type configurations as LIST-type, particularly
those using comma-separated values to represent multiple entries. This
change reflects the actual semantics used in Kafka.
- Update the default values for some configurations to better align with
other configs.
These changes will not introduce any compatibility issues.
Reviewers: Jun Rao <junrao@gmail.com>, Chia-Ping Tsai
<chia7712@gmail.com>
This PR removes associated logging within NetworkClient to reduce noise
and streamline the client code.
Reviewers: Ismael Juma <ismael@juma.me.uk>, David Arthur
<mumrah@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
The ReassignPartitionsCommand shows the topic replicas on each broker.
When using the --generate command, it returns the current partition
replica assignment. However, the log directory for each current replica
is always shown as any. This makes it impossible for users to determine
which specific log directory is being used by each replica. Therefore,
we should fix this behavior.
```
Current partition replica assignment
{
"version": 1,
"partitions": [
{
"topic": "test1",
"partition": 0,
"replicas": [
4,
2
],
"log_dirs": [
"any",
"any"
]
}
]
}
```
This PR
```
Current partition replica assignment
{
"version": 1,
"partitions": [
{
"topic": "test1",
"partition": 0,
"replicas": [
4,
2
],
"log_dirs": [
"/tmp/kraft-broker-logs234",
"/tmp/kraft-broker-logs"
]
}
]
}
```
Reviewers: PoAn Yang <payang@apache.org>, Jhen-Yung Hsu
<jhenyunghsu@gmail.com>, TaiJuWu <tjwu1217@gmail.com>, Chia-Ping Tsai
<chia7712@gmail.com>
When remote storage is offline, then the segmentLag and bytesLag metrics
are not recorded. These metrics are useful to know the pending data to
upload when remote storage is down.
Reviewers: TaiJuWu <tjwu1217@gmail.com>, Kamal Chandraprakash
<kamal.chandraprakash@gmail.com>
When writing HTML, it's recommended to use the <code> element instead of
backticks for inline code formatting.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, TengYao Chi
<frankvicky@apache.org>
To maintain code consistency, `MetadataVersion#fromVersionString` uses
`latestTesting()` as the latest version. Therefore, in the tools, we
also need to maintain consistency by updating the outer logging to use
`latestTesting()`.
See the discussion:
https://github.com/apache/kafka/pull/18845#discussion_r1950706791
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
The old approach `OFFSETS_TOPIC_REPLICATION_FACTOR_CONFIG` will override
`OFFSETS_TOPIC_PARTITIONS_CONFIG` config, this behaviour is not
expected, we should fix it.
Reviewers: TengYao Chi <frankvicky@apache.org>
The protocol type; for Connect assignors this is "eager", "compatible",
or "sessioned"
Since `ConnectAssignor` is an interface and the protocol parameter is
restricted to "eager", "compatible", or "sessioned", it aligns with the
existing ConnectProtocolCompatibility enum. Therefore, we can update the
code to use `ConnectProtocolCompatibility` directly.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Use Java to rewrite ProducerSendWhileDeletionTest by new test infra and
move it to client-integration-tests module.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
update the following test to avoid using `becomeLeaderOrFollower`
- testClearPurgatoryOnBecomingFollower
- testDelayedFetchIncludesAbortedTransactions
- testDisabledTransactionVerification
- testFailedBuildRemoteLogAuxStateMetrics
Reviewers: TengYao Chi <kitingiao@gmail.com>, Chia-Ping Tsai
<chia7712@gmail.com>
Use Java to rewrite PlaintextConsumerSubscriptionTest by new test infra
and move it to client-integration-tests module.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Use Java to rewrite BaseConsumerTest, SaslPlainPlaintextConsumerTest by
new test infra and move it to client-integration-tests module, the
BaseConsumerTest test is still used, thus we should not remove now.
Reviewers: PoAn Yang <payang@apache.org>, TengYao Chi
<frankvicky@apache.org>, Chia-Ping Tsai <chia7712@gmail.com>
The main issue was that we forgot to set
`TopicConfig.SEGMENT_BYTES_CONFIG` to at least `1024 * 1024`, which
caused problems in tests with small segment sizes.
To address this, we introduced a new internal config:
`LogConfig.INTERNAL_SEGMENT_BYTES_CONFIG`, allowing us to set smaller
segment bytes specifically for testing purposes.
We also updated the logic so that if a user configures the topic-level
segment bytes without explicitly setting the internal config, the
internal value will no longer be returned to the user.
In addition, we removed
`MetadataLogConfig#METADATA_LOG_SEGMENT_MIN_BYTES_CONFIG` and added
three new internal configurations:
- `INTERNAL_MAX_BATCH_SIZE_IN_BYTES_CONFIG`
- `INTERNAL_MAX_FETCH_SIZE_IN_BYTES_CONFIG`
- `INTERNAL_DELETE_DELAY_MILLIS_CONFIG`
Reviewers: Jun Rao <junrao@gmail.com>, Chia-Ping Tsai
<chia7712@gmail.com>
Use Java to rewrite PlaintextConsumerPollTest by new test infra and move
it to client-integration-tests module.
Reviewers: PoAn Yang <payang@apache.org>, TengYao Chi
<kitingiao@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
Use Java to rewrite `PlaintextConsumerCommitTest` by new test infra and
move it to client-integration-tests module.
Reviewers: PoAn Yang <payang@apache.org>, Chia-Ping Tsai
<chia7712@gmail.com>
Ken Huang