Visão Geral

Este curso ensina como executar, gerenciar e escalar Kafka Connect utilizando Docker e Kubernetes, criando pipelines de dados modernos e altamente resilientes. Você aprenderá desde a containerização e orquestração até boas práticas de produção, incluindo observabilidade, segurança e automação.

Objetivo

Após realizar este curso, você será capaz de:

• Executar Kafka Connect em containers Docker
• Orquestrar clusters de Kafka Connect em Kubernetes
• Configurar pipelines de dados escaláveis e resilientes
• Criar e gerenciar conectores em ambientes distribuídos
• Aplicar práticas de observabilidade, segurança e performance
• Automatizar deploys e atualizações de conectores

Publico Alvo

• Engenheiros de Dados
• Desenvolvedores Backend
• SREs e DevOps
• Arquitetos de Dados
• Profissionais que utilizam Kafka em ambientes distribuídos

Pre-Requisitos

• Conhecimentos básicos de Apache Kafka
• Conhecimentos intermediários em Docker
• Noções de Kubernetes (pods, deployments, configmaps)
• Familiaridade com APIs REST e JSON

Conteúdo Programatico

Module 1 — Introduction to Kafka Connect in Containerized Environments

1. Why run Kafka Connect in containers
2. Overview of Docker + Kubernetes for data engineering
3. Kafka Connect architecture refresher

Module 2 — Running Kafka Connect with Docker

1. Dockerizing Kafka Connect
2. Creating Docker Compose environments
3. Managing plugins in Docker images
4. Deploying connectors through REST API

Module 3 — Advanced Docker Configurations

1. Custom Dockerfiles for Connect workers
2. Persistent volumes for offsets, configs & status
3. Networking considerations
4. Managing connector plugins at scale

Module 4 — Introduction to Kubernetes for Kafka Connect

1. Core K8s concepts for data engineers
2. Deployments, StatefulSets, Services
3. ConfigMaps & Secrets
4. Horizontal Pod Autoscaler basics

Module 5 — Deploying Kafka Connect on Kubernetes

1. Creating Kafka Connect Deployment YAML
2. Plugin management in Kubernetes
3. Configuring distributed mode
4. REST API access inside the cluster

Module 6 — Scaling Kafka Connect in Kubernetes

1. Horizontal scaling with multiple workers
2. Task assignment and rebalancing
3. Failure handling & self-healing
4. Best practices for resource requests/limits

Module 7 — Observability and Monitoring

1. Logging strategies
2. Metrics with Prometheus
3. Dashboards with Grafana
4. Alerting for connector failures

Module 8 — Security for Containerized Kafka Connect

1. Securing API endpoints
2. SSL/TLS configuration
3. Secret management with Kubernetes Secrets
4. Role-based access control (RBAC)

Module 9 — CI/CD and Automation

1. Automating connector deployment with GitOps
2. Using ArgoCD or Flux
3. Automated image builds for Connect workers
4. Rolling updates and versioned connectors

Module 10 — Integrating with Cloud Providers

1. Running Kafka Connect on EKS, GKE, AKS
2. Using cloud-managed Kafka services
3. Best practices for hybrid cloud configurations

Module 11 — Real-World Streaming Pipelines

1. Database → Kafka → Data Warehouse pipeline
2. Managing schema evolution in Kubernetes
3. Error handling, DLQs, and retry strategies

Module 12 — Capstone Project

1. Build a full Kafka Connect cluster on Kubernetes
2. Deploy a source and sink connector
3. Apply monitoring, scaling, and security
4. Final documentation & presentation