Principal Software Quality Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• Establish and implement the comprehensive quality strategy for a distributed database platform.

• Take ownership of test architecture, coverage models, release gates, and risk-based prioritization.

• Foster quality across server components, client SDKs, operator and backup tools, as well as management interfaces.

• Develop testing strategies for replication, rebalancing, partitioning, sharding, consistency, transactions, indexes, query planning, compaction, backup and restore, snapshots, and cross-datacenter replication.

• Assess cluster correctness in scenarios involving node failures, network partitions, clock skew, disk-full conditions, slow disks, rolling upgrades, mixed-version deployments, and prolonged soak tests.

• Design and implement fault-injection scenarios while investigating complex system defects.

• Reproduce defects at the source-code level, providing minimal reproducible test cases and conducting root-cause analysis.

• Build and sustain scalable Python/pytest test frameworks, custom harnesses, API testing, and end-to-end automation.

• Ensure the reliability of test suites in shared infrastructure and CI/CD environments, including GitHub Actions and Jenkins.

• Minimize flaky tests, identify noisy failures, and enhance regression coverage.

• Lead the integration of AI agents into testing frameworks for test generation, failure triage, and agent reliability evaluation.

• Design workflows for agents to execute tests, review logs, traces, metrics, and database states, and generate defect reports.

• Create evaluation frameworks utilizing golden datasets, graders, and regression suites.

• Engage in feature reviews, requirements discussions, design reviews, and client specifications to establish testability requirements.

• Develop test plans, work collaboratively with development teams, and transform incidents into regression tests and monitoring checks.

• Facilitate defect resolution among QE, client, and server engineering teams.

• Mentor senior QEs and SDETs and set quality standards without direct people management responsibilities.

⛳️ Requirements

• 10–18 years of experience in Software Quality Engineering, SDET, or Systems Engineering, with a strong emphasis on quality systems and proven Staff or Principal-level technical influence.

• Extensive hands-on experience in testing distributed databases or storage systems, ideally NoSQL platforms such as Cassandra, ScyllaDB, MongoDB, Couchbase, Redis, DynamoDB, or HBase; experience with distributed relational databases is also relevant.

• Strong background in testing replication, partitioning, sharding, consistency, transactions, secondary indexes, backup and restore, and cross-datacenter replication.

• Thorough understanding of distributed systems fundamentals, including replication protocols, consensus, quorum reads and writes, strong and eventual consistency, split-brain scenarios, clock skew, and failure/recovery semantics.

• Capability to read and debug server-side code written in C, C++, Go, Rust, or Java and trace defects back to specific functions and root causes.

• Familiarity with protocol and serialization edge cases, including wire formats, MessagePack, Protocol Buffers, Thrift, and compression codecs.

• Strong software engineering and automation expertise in Python, with proficiency in at least one systems programming language.

• Practical experience in fault injection, GDB, chaos-style testing, process termination, network partitions, latency and packet-loss injection, disk faults, corrupted inputs, version skew, and boundary-value testing.

• Experience across unit, integration, API, contract, end-to-end, performance, scalability, upgrade/downgrade, and soak testing.

• Proven track record of leading debugging and root-cause investigations across QE, client, and server teams.

• Strong CI/CD experience with GitHub Actions, Jenkins, or equivalent tools, and shared test infrastructure.

• Practical experience utilizing AI agents for testing, including agent workflows, tool integration, prompt and evaluation design, and real-world testing framework integration.

• Excellent written and verbal communication skills.

• Preferred: client SDK testing in Python, Java, Go, C, Node.js, and .NET.

• Preferred: compatibility matrices, upgrade/downgrade scenarios, and behavioral consistency across client SDKs.

• Preferred: experience with backup and restore, checkpointing, snapshotting, and WAL/journal correctness at terabyte scale.

• Preferred: query-engine and secondary-index testing.

• Preferred: Kubernetes and database operator testing using KUTTL, Chainsaw, or envtest.

• Preferred: experience with AWS, GCP, or Azure infrastructure, TLS/mTLS, IAM, secrets management, and air-gapped installations.

• Preferred: security testing, fuzzing, CVE reproduction, and memory-safety issue identification.

🏝️ Benefits

• Competitive salary and performance-based incentives.

• Comprehensive health benefits including medical, dental, and vision.

• Generous paid time off and flexible working arrangements.

• Opportunities for professional development and career advancement.

• Collaborative and inclusive company culture.