Senior Lead Data Engineer, Content Engineering

This is a fully remote position, open to applicants in New York.

📋 Description

• Construct and Manage Large-Scale Feature Pipelines: Create and oversee batch/streaming pipelines (Spark, Flink, Databricks, Airflow) that generate ML features for ranking models.

• Guarantee Point-in-Time Accuracy: Create feature sets that support unbiased offline training and reliable online inference.

• Build Embedding & Content Pipelines: Develop scalable workflows for metadata, imagery, and multimodal representations; collaborate with Science teams to implement new models.

• Design Data Foundations: Create Delta/Parquet data models and medallion layers, optimizing storage layout and partitioning for latency and cost efficiency.

• Real-Time Engineering: Develop Kafka-based systems for real-time features and user-activity aggregations, ensuring effective handling of out-of-order events and exactly-once semantics.

• Governance & Leadership: Establish data quality standards and schema evolution processes while working across ML pods to convert model requirements into infrastructure.

⛳️ Requirements

• Over 7 years of experience in large-scale data or software engineering.

• Extensive expertise in Spark (PySpark/Scala), Databricks, Airflow, and Kafka.

• Strong proficiency in feature pipelines, temporal joins, and addressing training-serving discrepancies.

• Familiarity with AWS/Azure/GCP and high-performance engines such as Snowflake or Redshift.

• Skilled in programming with Python and SQL emphasizing performance optimization.

• Experience in personalization fields (search, ranking, or recommendation systems).

• Background in supporting petabyte-scale data lakehouses or feature stores.

• Understanding of GenAI/RAG systems, multimodal content, or Delta Live Tables.

• Knowledge in Causal Inference, experimentation signals, or ML evaluation workflows.

• Proficient in using Terraform for managed, repeatable deployments.

🏝️ Benefits

• Medical

• Dental

• Vision

• 401(k) plan

• Life insurance coverage

• Disability benefits

• Tuition assistance program

• PTO