Principal Software Quality Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• Take charge of the comprehensive quality strategy for the distributed database platform, encompassing test architecture, coverage models, release gates, and risk-based prioritization.

• Promote quality across server components, client SDKs, operator and backup tools, and management interfaces.

• Develop testing strategies for replication, rebalancing, partitioning, sharding, consistency, transactions, indexing, query planning, compaction, backup and restore, snapshots, and cross-data center replication.

• Ensure cluster correctness during failures, network partitions, clock skew, disk-full situations, slow disks, rolling upgrades, mixed-version deployments, and prolonged soak tests.

• Create and implement fault-injection scenarios and analyze complex system defects.

• Reproduce defects at the source code level, providing minimal reproducible test cases and root-cause analysis.

• Construct and sustain scalable Python, pytest, API, and end-to-end testing frameworks.

• Guarantee that test suites operate reliably across shared infrastructure and CI/CD environments such as GitHub Actions and Jenkins.

• Minimize flaky tests and enhance regression coverage.

• Lead the integration of AI agents into testing frameworks for test generation, failure triage, and agent reliability assessment.

• Design workflows for agents that analyze logs, traces, metrics, and database states, generating defect reports.

• Engage in feature reviews, requirements discussions, design reviews, and client specifications to establish testability requirements.

• Develop test plans and work alongside development teams to confirm technical designs.

• Transform incidents into regression tests, monitoring checks, and preventive actions.

• Facilitate debugging and defect resolution across Quality Engineering, client, and server engineering teams.

• Mentor senior Quality Engineers and Software Development Engineers in Test, establishing quality standards without direct management responsibilities.

⛳️ Requirements

• 10–18 years of experience in Software Quality Engineering, SDET, or Systems Engineering, showcasing demonstrated Staff or Principal-level technical influence.

• Extensive hands-on experience testing distributed databases or storage systems, ideally NoSQL platforms.

• Strong background in testing replication, partitioning, sharding, consistency, transactions, secondary indexes, backup and restore, and cross-datacenter replication.

• Knowledge of distributed systems fundamentals, including replication protocols, consensus, quorum reads and writes, consistency models, split-brain scenarios, clock skew, and failure/recovery semantics.

• Proficient in reading and debugging server-side code in C, C++, Go, Rust, or Java.

• Familiarity with wire formats, MessagePack, Protocol Buffers, Thrift, and compression codecs.

• Strong automation skills in Python and proficiency in at least one systems programming language.

• Practical experience with fault injection, GDB, chaos testing, process termination, network partitions, latency and packet-loss injection, disk failures, corrupted inputs, version skew, and boundary-value testing.

• Experience across unit, integration, API, contract, end-to-end, performance, scalability, upgrade/downgrade, and soak testing.

• Ability to lead debugging and root-cause investigations across Quality Engineering, client, and server teams.

• Strong CI/CD experience with GitHub Actions, Jenkins, or equivalent tools, and shared test infrastructure.

• Practical experience utilizing AI agents for testing, including agent workflows, tool integration, prompt and evaluation design, and real-world testing frameworks.

• Excellent written and verbal communication skills.

• Preferred: experience in client SDK testing across Python, Java, Go, C, Node.js, and .NET.

• Preferred: knowledge of compatibility matrices, backup and restore, query-engine and secondary-index testing, Kubernetes/database operator testing, cloud infrastructure, and security-focused testing.

🏝️ Benefits

• Comprehensive health, dental, and vision insurance.

• Generous paid time off and flexible work arrangements.

• Opportunities for professional development and continuous learning.

• A vibrant and inclusive company culture.