Visão Geral

Este curso ensina na prática como implantar, configurar e operar OpenSearch e Apache Kafka em clusters Kubernetes, utilizando ferramentas modernas como Helm, Operators e YAML nativo.
Além da implantação do Kafka, o curso inclui a instalação e configuração do Kafdrop, uma interface web para visualizar e monitorar tópicos, mensagens, offsets, consumer groups e brokers dentro do cluster Kafka executando no Kubernetes.

Os participantes aprenderão a integrar OpenSearch e Kafka em pipelines de dados dentro do K8s, operando soluções completas de busca, análise e streaming de eventos.

Objetivo

Após realizar este curso Deploying OpenSearch and Kafka (Kafdrop) on Kubernetes, você será capaz de:

• Implantar e configurar clusters OpenSearch no Kubernetes
• Implantar Kafka no Kubernetes usando Helm Charts e Operators
• Configurar Kafdrop para monitorar Kafka no K8s
• Integrar Kafka e OpenSearch para pipelines de análise e ingestão
• Diagnosticar problemas de conectividade, performance e configuração
• Operar ambientes distribuídos com alta disponibilidade
• Criar arquiteturas cloud-native para stream + indexação
• Utilizar boas práticas de segurança, recursos e observabilidade
• Entender os fundamentos de implantação stateful no Kubernetes

Publico Alvo

• Engenheiros de DevOps
• Engenheiros de dados
• Administradores de Kubernetes
• Desenvolvedores backend e arquitetos de software
• Profissionais que utilizam pipelines de dados em ambientes cloud-native
• Pessoas que desejam dominar implantações distribuídas no K8s

Pre-Requisitos

• Conhecimentos básicos de Kubernetes
• Noções de Docker
• Noções de YAML e Helm
• Conceitos básicos de Apache Kafka
• Noções de arquitetura de logs, observabilidade ou análise de dados

Materiais

Conteúdo Programatico

Module 1 — Introduction to Kubernetes for Data Platforms

1. Kubernetes architecture refresher
2. StatefulSets, PersistentVolumes, StorageClasses
3. ConfigMaps, Secrets, Services, Ingress
4. Challenges of running distributed systems on Kubernetes

Module 2 — Deploying OpenSearch on Kubernetes

1. Overview of OpenSearch architecture
2. Installing OpenSearch using Helm
3. Cluster topology: nodes, roles, replicas
4. Persistence, scaling, and shard allocation
5. OpenSearch Dashboards deployment
6. Validating the cluster health (yellow/green states)

Module 3 — Deploying Apache Kafka on Kubernetes

1. Kafka architecture on Kubernetes
2. Installing Kafka via Helm charts (Bitnami/Strimzi Operator overview)
3. Zookeeper vs KRaft mode
4. Storage and scaling design for Kafka clusters
5. Creating topics using Kubernetes resources
6. Monitoring brokers and partition distribution

Module 4 — Deploying Kafdrop on Kubernetes

1. What is Kafdrop?
2. Installing Kafdrop in Kubernetes
3. Service and Ingress exposure
4. Connecting Kafdrop to Kafka inside the cluster
5. Inspecting topics, partitions, messages, and consumer groups
6. Troubleshooting UI-level connectivity issues

Module 5 — Integrating Kafka and OpenSearch

1. Event streaming workflow
2. Creating producers and consumers for OpenSearch ingestion
3. Common data ingestion patterns
4. Validating message flow using Kafdrop
5. Testing indexing in OpenSearch
6. Handling message transformation and mapping issues

Module 6 — Observability and Monitoring

1. Logs, metrics, and events
2. Using Prometheus & Grafana (overview)
3. Monitoring Kafka cluster health
4. Monitoring OpenSearch performance
5. Identifying bottlenecks in pipelines

Module 7 — Security Essentials

1. TLS for Kafka in Kubernetes
2. Securing OpenSearch with passwords and dashboards
3. Restricting access to Kafdrop
4. Networking policies (NetworkPolicies)
5. Best practices for production security

Module 8 — Troubleshooting and Disaster Recovery

1. Debugging Kafka Pods, brokers, and Zookeeper/KRaft issues
2. Fixing OpenSearch shard issues
3. Dealing with PVC failures
4. Recovering Kafka topics and partitions
5. Cluster scaling and rolling upgrades
6. Common misconfigurations and how to fix them

Module 9 — Hands-On Labs

1. Lab 1: Deploying OpenSearch + Dashboards
2. Lab 2: Deploying Kafka (Helm) on K8s
3. Lab 3: Installing and configuring Kafdrop
4. Lab 4: Creating and inspecting Kafka topics
5. Lab 5: Streaming data into OpenSearch
6. Lab 6: Reading and validating data with Kafdrop
7. Lab 7: Troubleshooting real-world scenarios
8. Lab 8: Scaling and securing the solution