Automation Test Lead – Data & Platform Engineering

This is a fully remote position, open to applicants in United States.

📋 Description

• Test Automation Architecture & Strategy: Take ownership of a scalable, modular, and metadata-driven framework that encompasses data pipelines (both batch and streaming), APIs/backend services, and comprehensive data product validation; facilitate plug-and-play components, parallel execution, environment isolation, and deterministic runs.

• Data Testing Framework Engineering: Implement SQL-based assertions/reconciliation, schema validation, data contracts, lineage/freshness validation, and configuration-driven test definitions (YAML/JSON).

• Destructive Testing: Address schema drift, backward incompatibility, late-arriving data, partial failures, duplicate/missing/out-of-order events, as well as stress, concurrency, retries, DLQ handling, and backpressure.

• ETL/Streaming Validation at Scale: Conduct row/aggregate/hash-based reconciliation, incremental/backfill validation, delivery semantics validation, and ensure window/time-based correctness.

• Data Quality & Observability: Integrate or extend Great Expectations or Soda; create custom validations for accuracy, completeness, uniqueness, and timeliness; develop quality dashboards.

• CI/CD & DataOps Enforcement: Implement pre-merge gates, release blockers, selective/parallel test execution, and GitHub Actions/Jenkins integration.

• Test Data Management: Handle synthetic data generation, masking/anonymization, deterministic datasets, and edge case simulation.

• Performance & Reliability Testing: Execute pipeline/query benchmarks, concurrency/stress testing, data skew analysis, and cost/time optimization.

• Security & Compliance: Conduct PII/PHI exposure checks, manage encryption/access control, retention, and audit requirements; support GxP/SOX/ISO frameworks.

• Cross-Functional Quality Leadership: Collaborate with data engineers, platform teams, and architects; mentor fellow engineers.

• Incident Analysis & Prevention: Perform root-cause analysis of production data issues and work to reduce flaky tests.

⛳️ Requirements

• Proficient in Python (test frameworks, libraries, CLI tools) and advanced SQL skills.

• Practical experience with DBT, Airflow, Snowflake, or similar technologies; knowledge of ETL/ELT processes and data modeling.

• Familiarity with Great Expectations/Soda; understanding of lineage and catalog systems.

• Experience with Kafka/Kinesis testing and delivery semantics.

• Knowledge of Git workflows and CI/CD tools (Jenkins, GitHub Actions).

• Proficient in AWS and Infrastructure as Code (IaC) using Terraform/CloudFormation.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Flexible work hours and remote work options.

• Opportunities for professional development and continuous learning.

• Collaborative and inclusive work environment.