What We Are Looking For

3-6 month contract

Kafka Master Engineer  (architect) — someone who has at least 7 years of hands-on experience implementing, operating, and supporting Kafka in high-load, mission-critical environments.

This role is not about someone who just knows the theory of Kafka or how to get it running. We’re looking for a real-life practitioner who has already been through multiple large-scale production deployments, understands the pitfalls, and knows 100 ways not to do it — backed by hard-earned lessons, not just documentation.

Current Context

• We currently run on RabbitMQ, and the goal is a full migration to Kafka.
• Kafka will be at the core of our architecture, supporting:
• Trade order processing
• Price feeds
• Payments
• BI and real-time data streams
• Tech stack around it includes C++, Java, Python, PHP, and potentially Go, but the focus of this engagement is Kafka itself.

Performance & Reliability Targets

• High load & Low latency environment;
• Resilience & consistency:
• Zero data loss.
• Zero downtime.
• Strong focus on data ordering/consistency guarantees.

Expertise Required

• Deep knowledge of Kafka internals and architecture.
• Proven experience in designing clusters for very high throughput and low-latency use cases.
• Understanding of hardware, storage, and network configurations and their impact on performance, stability, and failure scenarios.
• Practical experience with replication, partitioning, broker tuning, ISR management, and retention policies at scale.
• Familiarity with side effects and gotchas (e.g., GC pauses, Zookeeper/KRaft issues, producer/consumer tuning, cross-DC replication challenges).
• Strong background in monitoring, alerting, and troubleshooting Kafka under extreme load.

Ideal Profile

• Someone who can challenge assumptions with “this won’t work in production because…” and back it up with evidence.
• A battle-hardened Kafka engineer who has built financial-grade, real-time streaming systems that cannot afford mistakes.