Manager, Test Data Engineering Team

This is a fully remote position, open to applicants in Florida, +4 more states.

📋 Description

• Lead and develop a team predominantly composed of QA-background engineers into a high-performing, engineering-focused unit.

• Design and create reusable data quality frameworks that are integrated into Databricks pipelines.

• Create standardized validation libraries for checks on schema, freshness, volume, and anomaly detection.

• Define and uphold data quality standards and a “Definition of Done” across various pipelines.

• Develop scalable solutions for generating and masking synthetic data.

• Create replay frameworks to validate pipeline changes against production scenarios.

• Establish automated regression testing harnesses for both batch and streaming pipelines.

• Set up monitoring, alerting, and SLA/SLO frameworks for data pipelines.

• Develop tools for measuring and tracking data quality, pipeline health, and reliability metrics.

• Facilitate proactive identification of data issues in production environments.

• Lead and expand a team of data quality engineers and test data engineers.

• Collaborate with Data Engineering, Product, and Analytics teams to integrate quality practices.

• Promote the adoption of frameworks and standards across global engineering pods.

• Define and implement a long-term roadmap for enterprise data quality.

• Prioritize initiatives among various stakeholders and global teams.

• Streamline manual validation processes through automation and engineering best practices.

• Work with internal stakeholders to establish data validation requirements and create reusable testing frameworks across the organization.

• Manage multiple roadmaps and shared-service backlogs serving diverse engineering teams.

• Collaborate with teams across North America, Europe, and Asia in a globally distributed environment.

⛳️ Requirements

• Over 5 years of experience in Data Engineering, Test Data Engineering, or related disciplines.

• At least 2 years in a technical leadership or management position.

• Strong expertise in Databricks, Spark, distributed data systems, Python, and SQL.

• Experience in building automated validation or data quality frameworks.

• Practical experience with data observability or validation tools such as Great Expectations, Deequ, or Soda.

• Familiarity with streaming architectures and real-time validation.

• AI orchestration experience is a significant advantage.

• Capability to automate synthetic data generation, data profiling, and pipeline regression testing.

• Understanding of anomaly detection and large-scale data validation patterns, including hash/Merkle-based validation.

• Experience with major TDM tools for data profiling and sampling.

• Proficient in JIRA and writing agile user stories and acceptance criteria.

• Willingness to collaborate with cross-functional teams across various time zones and odd hours when required.

• Strong analytical skills.

• Experience with user behavior data is a plus.

• Proven track record of successfully launching test data solutions or BI validation applications.

• A versatile, hands-on approach with a strong focus on execution.

• A passion for live music and events is a plus.

🏝️ Benefits

• Medical, vision, dental, and mental health benefits for you and your family.

• Health care concierge services.

• Flexible Spending Accounts (FSA) or Health Savings Accounts (HSA).

• Complimentary concert tickets.

• Generous paid time off, including holidays, sick leave, and personal days.

• 401(k) program with company match.

• Stock reimbursement program.

• New parent programs that include caregiver leave and baby bonuses.

• Support for fertility, adoption, foster, or surrogacy needs.

• Career and skill development programs through the School of Live.

• Tuition reimbursement options.

• Assistance with student loan repayment.

• Volunteer time off.

• Crowdfunding matching opportunities.

• Reasonable accommodations for qualified individuals with disabilities.

• Reasonable religious accommodations.