[Kafka] 11. Kafka Client Configurations

 

 

카프카 클라이언트는 애플리케이션이 카프카 브로커와 상호작용할 수 있도록 해주는 라이브러리입니다. 

클라이언트는 주제로 메시지를 생산하고, 주제로부터 메시지를 소비하며, 관리 작업을 수행하는 등의 다양한 작업을 가능하게 합니다. 

카프카 클라이언트는 애플리케이션과 카프카를 연결하여 카프카의 스트리밍 기능을 다른 소프트웨어 시스템에 통합할 수 있도록 합니다.

 

 

1. Best practices in Kafka client configurations

카프카 클라이언트와 작업할 때 모범 사례를 구현하면 애플리케이션 내에서 효율적이고 신뢰할 수 있으며 안전한 메시지 처리를 보장할 수 있습니다. 

고려해야 할 몇 가지 사항은 다음과 같습니다.

 

1.1. Throughput optimization

지연 시간과 처리량 간의 균형을 맞추기 위해 설정을 조정해야 합니다. 

이는 다음과 같은 구성에 영향을 받습니다.

 

Configurations	Best practice	Reason
batch.size, linger.ms	Adjust to balance latency and throughput	Larger batch sizes enhance throughput but increase latency as the producer waits to fill a batch.
fetch.min.bytes, fetch.max.wait.ms	Control data fetched per request	Tuning these reduces the number of fetch requests, improving consumer throughput.
max.poll.records	Control the number of records per poll	Manages the processing load effectively
compression.type	Use to reduce data size	Improves throughput but increases CPU usage
num.stream.threads, cache.max.bytes.buffering	Adjust parallelism and buffer size	Controls execution efficiency and data handling capacity
tasks.max, connector-specific batch size	Control parallelism and batch size	Optimizes the handling and processing of data streams

 

 

1.2. Robust error handling

다음 설정은 카프카 시스템에서 오류 및 다운타임을 효율적으로 처리하는 데 도움을 줍니다.

Configurations	Best practice	Reason
retries, retry.backoff.ms	Implement retry mechanisms with exponential backoff	Ensures message delivery and avoids overloading the server during peak errors
enable.idempotence	Enable to prevent duplicate records	Essential for scenarios requiring exactly-once processing semantics
Transaction APIs	Use for exactly-once semantics	Wraps production and consumption of messages in a transaction to prevent data loss or duplication
Consumer rebalance handling	Design consumers to handle rebalances and commit offsets properly	Ensures continuous processing and data integrity during network errors or broker failures
enable.auto.commit	Set to false so you can manually manage offsets	Allows explicit control over when to commit offsets, reducing data loss
state.dir	Use fast, reliable storage	Minimizes risk of state store corruption and data loss
processing.guarantee	Set to "exactly_once" so records are processed exactly once	Prevents data from being lost or processed multiple times
errors.tolerance, errors.deadletterqueue.topic.name, errors.retry.timeout	Set appropriate error handling policies and configure retry policies	Manages faulty records and reduces data loss during transient failures