Manager, Test Data Engineering Team

This is a fully remote position, open to applicants in Florida, +3 more states.

📋 Description

• Design and construct reusable data quality frameworks that are integrated into Databricks pipelines.

• Create standardized validation libraries including schema checks, freshness, volume, and anomaly detection.

• Define and implement data quality standards and the “Definition of Done” for all pipelines.

• Develop scalable solutions for synthetic data generation and data masking.

• Create replay frameworks to validate pipeline changes against production scenarios.

• Establish automated regression testing harnesses for both batch and streaming pipelines.

• Set up monitoring, alerting, and SLA/SLO frameworks for data pipelines.

• Develop tools to measure and monitor data quality, pipeline health, and reliability metrics.

• Facilitate the proactive identification of data issues within production environments.

• Lead and mentor a team of data quality engineers and test data engineers.

• Collaborate with Data Engineering, Product, and Analytics teams to integrate quality practices.

• Promote the adoption of frameworks and standards across global engineering pods.

• Develop and implement a long-term roadmap for enterprise data quality.

• Prioritize initiatives among multiple stakeholders and global teams.

• Automate validation processes to eliminate manual efforts through engineering best practices.

⛳️ Requirements

• Over 5 years of experience in Data Engineering, Test Data Engineering, or related disciplines.

• At least 2 years in a technical leadership or management position.

• Strong proficiency with Databricks, Spark, distributed data systems, Python, and SQL.

• Experience in building automated validation or data quality frameworks.

• Familiarity with data observability or validation tools (e.g., Great Expectations, Deequ, Soda).

• Understanding of streaming architectures and real-time validation.

• AI orchestration experience is a significant advantage; ability to automate test data practices such as synthetic data generation, data profiling, and pipeline regression testing.

• Solid grasp of anomaly detection and large-scale data validation patterns (e.g., hash/Merkle-based validation), along with knowledge of major TDM tools for data profiling and sampling.

• Ability to manage multiple concurrent projects and establish priorities effectively.

• Experience with JIRA and writing agile user stories and acceptance criteria.

• Willingness to collaborate with cross-functional teams across various time zones and odd hours if required.

• Strong analytical skills are essential; familiarity with user behavior data is a plus.

• Proven track record of successfully launching test data solutions or BI validation applications.

• A versatile, hands-on approach with a strong focus on execution.

• A passion for live music and events is an added bonus.

🏝️ Benefits

• Comprehensive medical, vision, dental, and mental health benefits for you and your family.

• Complimentary concert tickets, generous paid time off including holidays, sick leave, and personal days.

• 401(k) program with company matching and stock reimbursement options.

• New parent programs that include caregiver leave and baby bonuses, as well as support for fertility, adoption, foster care, or surrogacy.

• Opportunities for career and skill development through the School of Live, tuition reimbursement, and student loan repayment assistance.

• Volunteer time off and matching for crowdfunding initiatives.