Senior System Test Engineer

This is a fully remote position, open to applicants in United States, +1 more country.

📋 Description

• Define and lead system-level testing focusing on correctness, reliability, scalability, and performance.

• Prioritize the platform's highest-risk behaviors.

• Develop automated testing frameworks and realistic workloads for data ingestion, concurrent reads and writes, indexing, querying, and maintaining datasets.

• Ensure data integrity and consistency guarantees during concurrency, retries, partial failures, and recovery processes.

• Create fault-injection tests and resilience assessments for process crashes, network issues, storage errors, resource exhaustion, and interrupted operations.

• Design load, stress, and long-duration soak tests.

• Detect regressions in latency, throughput, resource usage, and search quality.

• Test upgrade paths, compatibility, and recovery scenarios.

• Analyze failures through logs, metrics, traces, and source code examination.

• Simplify complex issues to minimal reproductions and drive necessary fixes and regression coverage.

• Develop and enhance reproducible testing environments.

• Integrate reliable and actionable checks into CI and release workflows.

• Utilize AI coding assistants and other AI tools to create test harnesses, explore failure scenarios, analyze outcomes, and expedite debugging.

• Independently validate generated code and conclusions.

• Collaborate closely with engineering teams to convert architectural risks and customer workloads into automated, reproducible tests.

⛳️ Requirements

• Over 10 years of experience in software engineering, system test engineering, or test infrastructure.

• Extensive experience in building or validating distributed databases or large-scale data systems.

• Profound understanding of distributed databases, including consistency models, replication, partitioning, concurrency control, durability, and failure recovery.

• Proven experience in designing tests that verify system invariants and identify data corruption, race conditions, and failures under load or during partial outages.

• Strong programming proficiency in Python and at least one systems language such as Rust, C++, Java, or Go.

• Capability to build maintainable test infrastructure and debug product code effectively.

• Practical experience with fault injection, performance testing, workload generation, and automated testing in distributed settings.

• Demonstrated skill in utilizing AI tools effectively in daily engineering tasks, including coding assistants and agent-based workflows.

• Good judgment in validating AI-generated outputs.

• Experience with Linux, cloud infrastructure, containers, CI/CD, and observability tools.

• Ability to manage ambiguous projects, prioritize risks, communicate findings clearly, and collaborate closely with engineering teams.

• Experience in testing database storage engines, query engines, vector search, or cloud object storage integrations is desirable.

• Familiarity with LanceDB, Lance, Apache Arrow, DataFusion, Parquet, or similar data infrastructure is a plus.

• Experience in property-based testing, fuzzing, differential testing, or distributed consistency testing is a plus.

• Experience with Kubernetes, infrastructure as code, and large-scale test environment orchestration is a plus.

• Contributions to open-source databases, distributed systems, or testing frameworks are advantageous.

• Legally authorized to work in the United States for any employer.

• Residing in the United States or Canada.

🏝️ Benefits

• Equity grants.

• Medical, dental, vision, and life insurance.

• 401(k) retirement plan.

• Flexible Spending Accounts (FSA) and Health Savings Accounts (HSA).

• Commuter benefits.

• Generous paid time off.

• Paid parental leave program for birthing, non-birthing, adoptive, and foster parents.