Senior Data QA - Onshore

This is a fully remote position, open to applicants in United States.

📋 Description

• Architect, develop, and scale automated testing frameworks from the ground up within Databricks utilizing PySpark, Python, and SQL.

• Create automated assertions for Delta Lake tables, encompassing data drift, schema evolution, and historical data validation through time-travel functions.

• Validate extensive batch and real-time streaming data pipelines using Structured Streaming, ensuring the integrity from source to target.

• Programmatically ensure data lineage, audit logs, and access controls established via Databricks Unity Catalog.

• Lead the integration of automated data quality tests into enterprise CI/CD pipelines employing Azure DevOps, GitHub Actions, Databricks Workflows, APIs, or Airflow.

• Serve as the subject matter expert in data quality, mentoring junior team members, establishing QA standards, and promoting data quality principles.

• Design and perform automated performance and scalability tests on Spark jobs, large clusters, and complex query optimizations.

⛳️ Requirements

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related quantitative field.

• Over 8 years of experience in data engineering, data QA, or software development engineering in test (SDET).

• A minimum of 2 years of specialized experience in architecting test automation within Databricks.

• Proficient in Python and PySpark (DataFrames and SQL APIs) for processing and profiling large datasets.

• Extensive expertise in crafting advanced SQL queries, optimization techniques, and comprehending Spark query execution plans.

• Hands-on expertise with big-data validation libraries, including Great Expectations, pytest, and Delta Live Tables expectations.

• Strong operational knowledge of deploying Databricks on AWS, Azure, or GCP.

• Preferred: Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional.

• Preferred: Experience in validating real-time event-streaming architectures such as Kafka, Event Hubs, or Kinesis.

• Preferred: Solid understanding of DataOps culture, testing infrastructure as code, and data observability principles.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Flexible work hours and remote working options.

• Opportunities for professional development and continuous learning.

• A vibrant company culture that values diversity and inclusion.