Visão Geral

Curso Kafka and OpenSearch Performance Tuning on Kubernetes. Este curso avançado ensina como otimizar performance, latência, throughput, confiabilidade e eficiência de recursos para clusters Apache Kafka e OpenSearch executando em ambientes Kubernetes.
O treinamento é focado em ajustes finos (tuning), estratégias de escalabilidade, configurações avançadas, análise de gargalos, tuning de storage, otimização de ingestão, particionamento, indexação e troubleshooting de performance.

Ao final, o aluno terá domínio sobre como ajustar Kafka e OpenSearch para ambientes de alta demanda, garantindo estabilidade, alto throughput e baixa latência, mesmo em pipelines complexos e distribuídos.

Objetivo

Após realizar este curso Kafka and OpenSearch Performance Tuning on Kubernetes, você será capaz de:

• Otimizar performance do Kafka e OpenSearch em ambientes Kubernetes
• Ajustar configurações de storage, rede, CPU e memória para maximizar throughput
• Identificar e resolver gargalos críticos (bottlenecks)
• Ajustar partições, replicas, índices e mappings
• Aumentar resiliência e estabilidade dos clusters
• Garantir baixa latência e alto volume de ingestão
• Implementar estratégias de scaling horizontal e vertical
• Configurar tuning avançado dos brokers Kafka
• Ajustar caches, thread pools e buffers
• Aplicar boas práticas de tuning para produção

Publico Alvo

• Engenheiros DevOps avançados
• Engenheiros de dados
• Administradores de Kubernetes
• Arquitetos de software
• SREs (Site Reliability Engineers)
• Profissionais que operam Kafka e OpenSearch em produção
• Especialistas em pipelines de dados e event streaming

Pre-Requisitos

• Conhecimentos sólidos de Kubernetes
• Experiência prévia com Kafka e OpenSearch
• Conhecimentos de storage, rede e infraestrutura
• Experiência com Helm, YAML e CI/CD
• Noções avançadas de observabilidade (métricas, logs)

Materiais

Conteúdo Programatico

Module 1 — Foundations of Performance Tuning

1. Understanding throughput vs latency
2. CPU, memory, I/O, and storage fundamentals
3. Kubernetes resource management: limits, requests, QoS
4. Identifying common bottlenecks in Kafka and OpenSearch

Module 2 — Kubernetes Tuning for Data Workloads

1. Optimizing nodes for stateful workloads
2. Storage classes and performance comparison
3. Tuning PersistentVolumes (local-path, SSD, NVMe, network storage)
4. Configuring affinity, anti-affinity, and pod distribution
5. Network tuning and bandwidth considerations

Module 3 — Kafka Performance Tuning

1. Broker-level tuning
2. Log segment sizes, retention, cleanup policies
3. Page cache optimization
4. Tuning producer throughput (batch.size, linger.ms, compression)
5. Tuning consumer performance (fetch sizes, max records)
6. Partition strategy for performance
7. Replication tuning (replica fetchers, ISR management)
8. Kafka Controller tuning
9. Impact of message size and record format

Module 4 — Kafka on Kubernetes Optimization

1. Kafka Operator vs Helm tuning strategies
2. Storage optimization for Kafka logs
3. Broker Pod resource optimization
4. Scaling brokers horizontally
5. Reducing cross-node traffic
6. Avoiding partition hotspots
7. NodeSelector, Taints, and Tolerations for Kafka

Module 5 — OpenSearch Performance Tuning

1. Indexing vs searching performance trade-offs
2. Tuning cluster roles and node types
3. JVM tuning for OpenSearch (heap sizing, GC optimization)
4. Thread pools and queue tuning
5. Shard and replica strategy
6. Refresh interval and index buffer settings
7. Force merge, rollover, and lifecycle policies
8. Mapping optimization for high-velocity ingestion

Module 6 — OpenSearch on Kubernetes Optimization

1. Disk throughput considerations
2. Dedicated nodes for ingestion, search, coordination
3. Scaling horizontally and vertically
4. Handling shard imbalances
5. Reducing cluster state bottlenecks
6. Optimizing OpenSearch Dashboards performance

Module 7 — Performance Testing and Benchmarking

1. Benchmark tools: k6, JMeter, kafka-producer-perf-test, opensearch-benchmark
2. Testing ingestion throughput
3. Measuring search latency
4. Stress tests and chaos scenarios
5. Simulating high-throughput pipelines

Module 8 — Observability for Performance Tuning

1. Key Kafka performance metrics to monitor
2. Key OpenSearch performance metrics to monitor
3. Grafana dashboards for performance visibility
4. Detecting anomalies in ingestion rates
5. Lag, partition imbalance, GC spikes, and shard hotspots
6. Using Kafdrop for real-time performance debugging

Module 9 — Troubleshooting Performance Issues

1. Slow producer/consumer scenarios
2. Troubleshooting low throughput in OpenSearch
3. Fixing slow searches and high latency
4. Diagnosing garbage collection issues
5. Network delays, DNS resolution problems, and cross-zone issues
6. Storage saturation and I/O contention
7. Identifying misconfigurations using metrics and logs

Module 10 — Hands-On Labs

1. Lab 1: Benchmarking Kafka on Kubernetes
2. Lab 2: Benchmarking OpenSearch indexing and search
3. Lab 3: Broker and producer tuning workshop
4. Lab 4: OpenSearch JVM and GC tuning
5. Lab 5: Shard allocation tuning and balancing
6. Lab 6: Tuning network and node resources on Kubernetes
7. Lab 7: Full pipeline performance optimization (Kafka → OpenSearch)
8. Lab 8: Diagnosing and fixing real-world bottlenecks