Location: Bangalore
Job Type: Full-time
Company: WizzyBox Private Ltd

About Us:
WizzyBox is an innovative company dedicated to delivering scalable, high-performance software solutions for our clients. As we expand, we are looking for a talented Kafka Cluster Engineer to join our team. You will play a crucial role in managing and optimizing our Kafka clusters, ensuring that our data pipelines are reliable, secure, and efficient. If you have experience with Kubernetes, Kafka brokers, Zookeeper, Kafka Connect, and Kafka Streams, we invite you to bring your expertise to WizzyBox!

Roles & Responsibilities:

As a Kafka Cluster Engineer, you will be responsible for managing, maintaining, and optimizing Kafka clusters and related technologies. Your focus will be to ensure high availability, fault tolerance, and performance while handling large-scale data streaming.

1. Kafka Cluster Management:
   • Set up, configure, and maintain Kafka brokers for handling high-volume real-time data streams.
   • Ensure high availability, fault tolerance, and data durability of Kafka clusters through effective architecture and configuration.
   • Monitor and optimize Kafka cluster performance, troubleshooting issues with brokers, producers, consumers, and partitions.
   • Oversee cluster scalability, managing Kafka partitioning and replication strategies to meet changing data demands.
   • Ensure secure communication between Kafka brokers and clients using encryption and authentication mechanisms like SSL and SASL.
2. Kubernetes & Containerization:
   • Manage and orchestrate Kafka clusters using Kubernetes, ensuring seamless deployment, scaling, and operations.
   • Deploy Kafka and Zookeeper on Kubernetes clusters, leveraging containerization tools like Docker.
   • Configure Helm charts for Kafka and Zookeeper deployment, making sure that configurations are consistent and scalable across environments.
   • Implement robust monitoring and logging for Kafka services running on Kubernetes, ensuring minimal downtime and rapid issue resolution.
3. Kafka Connect:
   • Design and manage Kafka Connect for seamless integration of Kafka with external systems, including databases, cloud services, and other messaging systems.
   • Configure and monitor Kafka Connect connectors to ensure reliable data movement between systems.
   • Troubleshoot and optimize connector performance, resolving integration challenges as they arise.
   • Implement and maintain custom Kafka Connectors as required.
4. Kafka Streams & Real-Time Data Processing:
   • Implement Kafka Streams applications for real-time data processing, including data transformation, aggregation, and enrichment.
   • Design and build real-time data pipelines that support high-throughput, low-latency data processing.
   • Ensure data consistency and stream processing performance by monitoring and tuning Kafka Streams applications.
5. Zookeeper Management:
   • Manage Zookeeper clusters for distributed coordination and leader election, ensuring that Kafka brokers are properly synchronized.
   • Troubleshoot and resolve issues related to Zookeeper-based coordination in the Kafka ecosystem.
   • Ensure high availability and consistency of Zookeeper nodes, which are crucial for the Kafka cluster’s stability.
6. Performance Tuning and Optimization:
   • Perform regular health checks and performance tuning of Kafka clusters, brokers, consumers, and producers.
   • Implement and configure retention policies, log compaction, and other Kafka configurations to optimize performance.
   • Monitor Kafka cluster metrics and logs using tools such as Prometheus and Grafana to identify bottlenecks, optimize resource usage, and ensure system reliability.
7. Security & Data Integrity:
   • Ensure data privacy and integrity by implementing robust security measures, including encryption, authentication, and authorization for Kafka clusters.
   • Set up access control using ACLs to manage permissions for Kafka consumers and producers.
   • Conduct regular security audits and vulnerability assessments to protect the Kafka environment.
8. Incident Response & Troubleshooting:
   • Respond promptly to production issues, performing root cause analysis and taking corrective actions to resolve Kafka-related incidents.
   • Implement proactive monitoring and alerting for Kafka clusters, ensuring any performance issues or failures are identified early.
   • Troubleshoot complex distributed systems issues related to Kafka, Zookeeper, and Kubernetes.
9. Documentation & Knowledge Sharing:
   • Document Kafka cluster architecture, configuration, and operational procedures for internal reference and future improvements.
   • Share knowledge of Kafka and related technologies across teams to ensure smooth collaboration and foster a culture of continuous improvement.

Key Skills and Qualifications:

• Proven experience in managing and optimizing Kafka clusters in a production environment.
• Strong understanding of Kafka brokers, Zookeeper, Kafka Connect, and Kafka Streams.
• Hands-on experience deploying and managing Kafka and Zookeeper on Kubernetes using Docker.
• Experience with Helm for managing Kubernetes-based deployments.
• Knowledge of real-time data processing using Kafka Streams and building robust data pipelines.
• Proficiency in monitoring tools like Prometheus, Grafana, or similar for Kafka clusters and Kubernetes environments.
• Strong understanding of distributed systems, including partitioning, replication, and fault tolerance in Kafka.
• Expertise in securing Kafka environments, including SSL/TLS, SASL, and ACL-based access control.
• Experience with CI/CD practices for deploying Kafka clusters and related applications.
• Strong troubleshooting skills in a complex distributed environment.
• Bachelor’s degree in Computer Science, Engineering, or a related field (or equivalent experience).

Preferred Qualifications:

• Experience with cloud platforms (AWS, Azure, GCP) and deploying Kafka in cloud-native environments.
• Familiarity with Kafka security features, including Kerberos, SSL encryption, and authorization models.
• Experience building custom Kafka Connectors for integration with various systems.
• Knowledge of event-driven architectures and how Kafka fits into such systems.
• Familiarity with Kafka Tiered Storage and data retention policies.

Why WizzyBox?

• Join a growing, innovative company that offers challenging projects and opportunities to work with cutting-edge technologies.
• Work in a collaborative and dynamic environment where your contributions will directly impact the product and its growth.
• Competitive salary and benefits package, including flexible work options.
• Opportunities for career growth, learning, and professional development in an emerging field.

If you are passionate about Kafka, distributed systems, and building scalable data architectures, apply now to become a part of the WizzyBox team and make a significant impact on our real-time data platform!

Apply for this job