Lead Data Platform Engineer

This is a fully remote position, open to applicants in New Jersey.

📋 Description

• Take ownership of the complete technical architecture of the data platform, encompassing ingestion, storage, transformation, orchestration, serving, and observability layers.

• Create and update the architectural vision document for the platform; conduct quarterly architecture reviews to evaluate alignment with organizational objectives and technological trends.

• Define and refine the target-state architecture for the platform, establishing a multi-year technology roadmap in collaboration with engineering leadership.

• Assess emerging technologies and frameworks, providing evidence-based recommendations for adoption.

• Act as the final technical escalation point for intricate design inquiries, inter-team conflicts, and build-vs-buy decisions related to the platform.

• Oversee the planning, execution, and delivery of multi-quarter platform initiatives that involve multiple engineers, cross-functional dependencies, and significant organizational impact.

• Decompose large, ambiguous programs into defined workstreams; assign technical leads for each workstream, establish milestones, and manage inter-team dependencies and risks.

• Initiate projects with a clear framing of the problem, success criteria, architectural constraints, and delivery phases — from exploration to production hardening.

• Maintain stakeholder alignment throughout the delivery process: proactively update on status, highlight trade-offs, and escalate blockers to leadership before they escalate into risks.

• Lead post-mortems and retrospectives for significant initiatives; document and share insights to elevate the organization’s standard for delivery excellence.

• Provide functional oversight within the EDAP team: review technical designs, set quality standards, approve architectural decisions, and ensure consistency in implementation patterns.

• Define, document, and promote platform engineering standards, including coding conventions, testing requirements, CI/CD practices, schema design guidelines, and SLA frameworks.

• Establish and manage the platform's technical review process (design review): triage incoming projects, lead review sessions, and ensure decisions are well-documented and traceable.

• Identify and address technical debt, redundancy, and architectural drift that hinder platform reliability, developer productivity, or scalability.

• Collaborate with security, compliance, and infrastructure teams to ensure platform systems are designed to meet governance, data privacy, and regulatory standards.

• Act as a primary technical mentor for both junior and senior data platform engineers; provide structured guidance on system design, technical communication, and engineering judgment.

• Facilitate design reviews, architecture critiques, and technical deep-dives that enhance the overall skill set of the engineering team.

• Define the standards of technical excellence for each level of engineers and engage in calibration discussions.

• Represent the data platform in engineering-wide forums, all-hands meetings, and external events (conferences, open-source communities, recruiting activities).

• Foster a culture of documentation, reliability, and product-oriented thinking across all data platform engineering functions.

• Engage in goal planning cycles as a technical representative; set engineering-led goals that enhance platform reliability, developer experience, and data quality.

⛳️ Requirements

• 8–14 years of professional experience in software or data engineering, with a minimum of 3 years in a technical lead or principal role on a data platform or distributed systems team.

• Proven track record of leading large-scale, cross-functional data infrastructure projects from inception to production with measurable business outcomes.

• Extensive expertise in distributed data systems, including data warehouse/lakehouse architecture, streaming platforms, large-scale batch processing, and cloud-native data infrastructure.

• Expert-level proficiency in Python and SQL; solid working knowledge of at least one JVM language (Scala or Java) for Spark or Flink development.

• In-depth understanding of modern data warehouse and lakehouse platforms (Snowflake, BigQuery, Redshift, Databricks, or similar), including storage optimization, compute management, and cost governance.

• Strong grasp of open table formats (Apache Iceberg, Delta Lake, or Apache Hudi) and their trade-offs in production lakehouse environments.

• Experience in architecting and operating large-scale streaming pipelines using Apache Kafka, Kinesis, or Pub/Sub, including schema management, consumer group design, and ensuring exactly-once semantics.

• Exceptional communication skills: capable of writing clearly, presenting confidently, and adjusting technical depth for diverse audiences, from individual contributors to senior leaders.

• Experience in establishing engineering standards, review processes, and quality gates across multi-team engineering organizations.

🏝️ Benefits

• Medical, Dental, Vision, and Prescription Coverage (22.5 hours per week or more for full-time and part-time team members).

• Life & AD&D Insurance.

• Short-Term and Long-Term Disability (with options to supplement).

• 403(b) Retirement Plan: Employer match and additional non-elective contributions.

• Paid Time Off (PTO) & Paid Sick Leave.

• Tuition Assistance, Advancement, & Academic Advising.

• Parental, Adoption, and Surrogacy Leave.

• Backup and On-Site Childcare.

• Well-Being Rewards.

• Employee Assistance Program (EAP).

• Fertility Benefits and Healthy Pregnancy Program.

• Flexible Spending & Commuter Accounts.

• Insurance for Pets, Home & Auto, Identity Theft, and Legal matters.