Member Of Technical Staff 3 - Dev

Hungry, Humble, Honest, with Heart.

The Opportunity

This is an opportunity to work on the core distributed metadata layer that powers enterprise-scale infrastructure. As part of the metadata backend team, you will operate in the critical path of cluster lifecycle management and metadata consistency.

In this role, you will tackle fundamental distributed systems challenges — ensuring strong consistency, fault tolerance, high availability, and seamless scalability in large-scale distributed databases. Your contributions will directly influence cluster reliability, performance under failure scenarios, and the ability to scale dynamically without compromising correctness.

This is a systems-heavy role where correctness, performance, and resilience are equally important.

About The Team

CDP Team is responsible for building and maintaining a strictly consistent distributed metadata system derived from Apache Cassandra, along with orchestration mechanisms that manage cluster topology changes.

What We Focus On

• Transforming eventually consistent systems into strongly consistent (CP) architectures using consensus protocols such as Paxos
• Designing mechanisms for safe membership changes in distributed rings
• Handling replication, quorum management, and partition tolerance
• Ensuring data integrity during node additions, removals, failures, and network partitions
• Optimizing performance across storage engines and metadata paths
  
  

Our Engineering Culture

• Deep ownership of correctness and reliability
• Strong emphasis on debugging real-world distributed failures
• Frequent discussions on consensus algorithms, failure modes, and system trade-offs
• Pragmatic decision-making grounded in distributed systems theory
  
  

Your Role

As an MTS3, you will operate as a strong individual contributor with growing technical leadership responsibilities.

Core Responsibilities

• Design & Development
• Architect and implement enhancements to distributed database components that improve scalability, resilience, and operational safety.
• Design safe workflows for cluster membership changes while preserving consistency guarantees.
• Consistency & Consensus
• Work on consensus-driven systems ensuring linearizability and strict correctness.
• Improve mechanisms around quorum coordination, replication flows, and failure recovery.
• Fault Tolerance & Recovery
• Strengthen system behavior under hardware failures, network partitions, disk issues, and partial outages.
• Improve safeguards against metadata corruption and split-brain scenarios.
• Performance & Optimization
• Identify and eliminate bottlenecks in multi-threaded and asynchronous code paths.
• Optimize latency and throughput in metadata-heavy workloads.
• Debugging & Root Cause Analysis
• Investigate complex distributed failures in production environments.
• Analyze logs, metrics, and traces to isolate race conditions, state machine inconsistencies, and performance regressions.
• Technical Leadership
• Lead medium-sized features end-to-end.
• Mentor junior engineers and raise the bar in code quality and design rigor.
• Participate actively in design reviews and architectural discussions.
  

Experience

What You Will Bring

• 4–8+ years of experience in backend or systems engineering.
• Strong exposure to distributed systems, storage engines, or database internals.
  
  

Technical Skills

• Strong programming skills in C++. Familiarity with Java is a plus.
• Deep understanding of:
• Consensus protocols (Paxos/Raft)
• CAP theorem trade-offs
• Replication strategies and quorum-based systems
• Sharding and partitioning
• Failure detection and recovery mechanisms
• Multi-threaded and asynchronous system design
  

Domain Knowledge

• Experience working with distributed databases or key-value stores.
• Understanding of LSM-tree-based storage engines and compaction strategies is desirable.
  
  

Problem-Solving Ability

• Ability to debug large, complex codebases.
• Comfort with production troubleshooting under ambiguous failure conditions.
• Strong analytical mindset when reasoning about state machines and distributed correctness.
  
  

Education

• Bachelor’s or Master’s degree in Computer Science or a related field.
  
  

Work Arrangement

Hybrid: This role operates in a hybrid capacity, blending the benefits of remote work with the advantages of in-person collaboration. In locations where our workplace policy applies (i.e. San Jose, Durham, Mexico City, Bangalore, Pune, Hoofddorp, Belgrade, Barcelona, Singapore, Sydney and Tokyo), employees are expected to work onsite a minimum of 3 days per week to foster collaboration, team alignment, and access to in-office resources. Workplace type may vary based on location and team requirements. Please speak with your recruiter for details. Additional team-specific guidance and norms will be provided by your manager.

--

---