Director of Data Engineering

This is a fully remote position, open to applicants in North Carolina.

📋 Description

• Lead a team of data engineers and engineering managers in the development of large-scale, real-time data pipelines using Spark.

• Serve as the primary technical liaison to business leadership, effectively conveying trade-offs between technology and business priorities in both directions.

• Establish the multi-year technical strategy for Data Engineering and communicate this vision to both technical teams and business stakeholders.

• Collaborate with data scientists, analysts, and business stakeholders to transform business requirements into actionable data products for the team.

• Represent Data Engineering in cross-functional and executive planning sessions, fostering alignment among engineering, product, and business leaders.

• Define the architectural direction and standards for data movement across platforms, overseeing the entire process from streaming ingestion through ETL to aggregation and analytics, while ensuring accountability among your managers and teams, and promoting engineering and agile practices that enhance delivery speed and quality.

• Remain engaged with the technical details to inform architectural design decisions, performance optimization, and tradeoff evaluations, providing your team with the context necessary to operate swiftly.

• Collaborate with your teams and the RV Data Platform team to create scalable solutions across distributed systems utilizing AWS, Databricks, and contemporary big data technologies.

• Advocate for a governance-first approach within your team, ensuring that our methods for collecting, transferring, and utilizing data safeguard the privacy of individuals.

• Manage and cultivate the data engineering managers on your team, setting expectations for their coaching, hiring, and team management practices.

• Oversee career development within your organization, establishing growth pathways for both individual contributors and managers, and providing guidance through regular, constructive feedback.

• Hold the team responsible for delivering high-quality, scalable solutions throughout the entire data lifecycle.

⛳️ Requirements

• 8-10 years of experience in data or software engineering, including over 5 years in a management role and at least 1 year managing other managers.

• Proven ability to clearly communicate technical strategies and trade-offs to non-technical executives, quickly establishing trust with cross-functional peers.

• A strong curiosity about the systems and processes that generate our data, with sufficient understanding of the sources to assist your team in enhancing and troubleshooting pipelines rather than viewing data as a black box.

• A history of leading engineering teams within an agile environment, coaching both engineers and managers, and providing teams with enough context to work independently.

• Extensive hands-on experience in developing large-scale Spark applications for both batch and streaming workloads.

• Familiarity with modern lakehouse platforms (such as Databricks) alongside traditional Spark-based pipelines.

• Proficient in the Spark RDD and DataFrame/Dataset APIs, with a focus on DataFrames for querying and data manipulation.

• Experience with Spark Structured Streaming and SparkSQL, including the implementation of broadcast joins.

• Strong programming skills in Python and/or Scala, along with fluency in SQL for handling large-scale datasets.

• Practical experience with AWS data services (such as Kinesis, S3, Lambda, DynamoDB).

• Knowledge of ANSI SQL relational databases (including Oracle, SQL Server, Postgres, MySQL).

• Solid understanding of distributed systems and frameworks for distributed data processing.

🏝️ Benefits

• Health Insurance Coverage (including medical, dental, and vision).

• Life Insurance.

• Short and Long-Term Disability Insurance.

• Flexible Spending Accounts.

• Holiday Pay.

• 401(k) plan with matching contributions.

• Employee Assistance Program.

• Paid Parental Bonding Benefit Program.

• Flexible Paid Time Off (PTO): We recognize the importance of taking time to rest and recharge. Therefore, we offer a generous and flexible PTO policy. Full-time employees earn 20 days of PTO for a full calendar year, increasing to 25 days after five years of service.