Automation Test Lead – Data & Platform Engineering

This is a fully remote position, open to applicants in United States.

📋 Description

• Take ownership of a scalable, modular, metadata-driven test automation framework that encompasses batch and streaming data pipelines, APIs/backend services, and comprehensive data product validation.

• Facilitate plug-and-play components, parallel execution, environment isolation, and predictable test executions.

• Develop SQL-based assertions and reconciliations, schema validations, data contracts, lineage/freshness validations, and YAML/JSON configuration-driven test definitions.

• Test for schema drift, backward incompatibility, late-arriving data, partial failures, duplicate/missing/out-of-order events, stress conditions, concurrency, retries, DLQ management, and backpressure scenarios.

• Validate ETL/streaming at scale through row, aggregate, and hash-based reconciliation; perform incremental/backfill validations; assess delivery semantics; and ensure window/time-based correctness.

• Integrate or enhance Great Expectations or Soda, and create custom data-quality validations and dashboards.

• Uphold CI/CD and DataOps practices via pre-merge gates, release blockers, selective/parallel execution, and integration with GitHub Actions/Jenkins.

• Oversee test data management through synthetic generation, masking/anonymization, deterministic datasets, and edge-case simulations.

• Execute pipeline/query benchmarks, concurrency/stress tests, data-skew analysis, and cost/time optimization efforts.

• Conduct PII/PHI exposure evaluations, encryption/access-control validations, retention and audit checks, while supporting GxP/SOX/ISO frameworks.

• Collaborate with data engineers, platform teams, and architects while providing mentorship to engineers.

• Carry out root-cause analyses of production data issues and strive to minimize flaky tests.

⛳️ Requirements

• Proficient in Python (test frameworks, libraries, CLI tools).

• Advanced SQL skills.

• Practical experience with DBT, Airflow, Snowflake, or similar technologies.

• Background in ETL/ELT and data modeling.

• Familiarity with Great Expectations/Soda.

• Experience with lineage and catalog systems.

• Testing experience with Kafka/Kinesis.

• Understanding of delivery semantics.

• Knowledge of Git workflows.

• Experience with CI/CD using Jenkins or GitHub Actions.

• Familiarity with AWS.

• Experience with Infrastructure as Code using Terraform or CloudFormation.

• Over 9 years in automation/QE.

• More than 5 years of experience building frameworks for data platforms at scale.

• Proven track record of enhancing reliability and reducing incidents.

• API contract testing (PACT) experience preferred.

• Basic knowledge of UI automation is a plus.

• Exposure to data mesh/data products is desirable.

• Experience with Prometheus/Grafana is a plus.

• Background in regulated domains such as healthcare, life sciences, or finance is preferred.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Generous paid time off and flexible working hours.

• Opportunities for professional development and training.

• Collaborative and inclusive work culture.

• Access to cutting-edge technology and tools.