Staff Data Platform Engineer

This is a fully remote position, open to applicants in New York.

📋 Description

• Take charge of the complete data platform for AI research, ensuring continuous ingestion, preprocessing, governance, and export of high-volume image, video, audio, and text data.

• Finalize and enhance the Databricks lakehouse and distributed processing framework.

• Make informed architecture and build-vs-buy choices as data volume increases.

• Develop reusable infrastructure for internal synthetic data generation across various modalities.

• Implement dataset versioning, lineage tracking, metadata management, quality assurance, and stable training/evaluation splits.

• Ensure the reliability, observability, and processing costs of datasets in the pipeline.

• Collaborate with ML researchers to develop self-service tools that expedite model iteration.

• Mentor engineering staff and enhance code review processes, testing, documentation, and runbooks.

• Collaborate with Security and compliance teams on data management, licensing, and access control measures.

• Influence the long-term roadmap of the data platform alongside engineering and research leadership.

• Report directly to the Director of Data.

• Within a year, establish a mature self-service platform, scale synthetic data generation, provide reproducible evaluation datasets, and spearhead high-impact roadmap initiatives.

⛳️ Requirements

• Over 8 years of experience in building data platforms or data infrastructure, including staff or lead-level oversight of production pipelines that generate, ingest, or deliver high-volume data.

• Extensive hands-on experience with distributed processing (Spark, Ray, or similar) and lakehouse platforms, preferably Databricks, utilizing cloud object storage (S3, GCS, or Azure Blob).

• Production experience with Databricks, Unity Catalog, Delta Lake, and Spark on AWS, including performance optimization, access management, and operational troubleshooting.

• Experience with large unstructured media types such as video, audio, or images, including handling of large files, decoding, transcoding, and managing metadata at scale.

• Proven ability to create reproducible ML datasets through versioning, lineage tracking, data quality validation, and orchestration tools (Airflow, Dagster, Databricks Workflows, or similar).

• Proficient in Python and SQL.

• Aptitude to manage ambiguous platform challenges from start to finish on a lean team.

• Experience with infrastructure and deployment automation using Terraform and CI/CD, encompassing AWS storage, IAM, compute, Databricks resources, and distributed processing environments.

• Experience in constructing synthetic data generation pipelines or operating generative models at scale (a plus).

• Background in supporting ML research teams with training and evaluation data, particularly for multimodal models (a plus).

• Experience in security, fraud, trust & safety, or another adversarial domain, or in a Series A or B startup (a plus).

• Must have authorization to work in the U.S. as requested in the application.

• Willingness to relocate if currently located outside of or not near NYC, depending on work arrangement.

🏝️ Benefits

• Comprehensive healthcare plans with 100% premium coverage for employees and partial coverage available for dependents.

• Dental and Vision plans with 100% premium coverage for employees and their dependents.

• Short and long-term disability and life insurance plans with 100% premium coverage for employees.

• FSA/HSA and 401k programs.

• Equity compensation.

• 20 days of paid time off (PTO) annually.

• 12 weeks of Parental Leave.

• Budget for Learning and Development.

• Monthly wellness benefits.

• Annual company-sponsored offsite event.

• Daily in-office lunch via UberEats for employees at the NYC HQ.

• Commuter benefits for NYC HQ employees.

• Remote Fridays for NYC HQ employees.

• Happy Hours and other local events for NYC HQ employees.

• Eligibility for an annual performance bonus.