Senior ML Engineer – VLM

This is a fully remote position, open to applicants in Michigan.

📋 Description

• Take ownership of the offline dataset pipeline — design, implement, test, and deploy cloud-based pipelines that transform logged multi-sensor data into VLM/VLA training datasets, encompassing geometric labels (3D/2D detection, tracking, segmentation, depth) along with semantic, scenario-level, and action/trajectory-grounded annotations.

• Develop VLM-assisted auto-labeling — create open-vocabulary detection, dense captioning, semantic enrichment, and scene/scenario description generation that extend beyond closed-set bounding boxes, leveraging foundation models to enhance annotation scalability and reduce manual labeling costs.

• Produce reasoning-grounded labels — generate language-grounded reasoning and annotations that follow a chain-of-causation style, temporally aligned with ego-motion and trajectories, to facilitate VLA training and elucidate driving behavior.

• Explore and curate the long tail — identify rare, challenging, and high-uncertainty scenarios, and construct curated datasets that significantly enhance downstream VLM/VLA model metrics rather than merely increasing volume.

• Complete the data flywheel — establish dataset schemas, quality metrics, and validation processes; monitor auto-labeling quality against model requirements; and reroute model failures back into re-labeling and retraining cycles.

• Collaborate with the end-to-end model team — co-define dataset specifications with VLM/VLA model developers, maintain the quality standards and delivery schedule, and implement a continuous dataset delivery loop into their training pipelines.

• Scale on cloud infrastructure — create distributed, reproducible pipelines utilizing columnar data formats and distributed computing, adhering to disciplined software practices, version control, and thorough documentation.

• Lead and mentor — act as project lead, support less-experienced engineers, conduct design reviews, establish coding and annotation standards, and ensure alignment across team interfaces with the broader organization.

• Stay updated — monitor the latest advancements in multimodal models, auto-labeling, and end-to-end autonomous driving, and convert relevant research into operational data systems.

⛳️ Requirements

• Highly skilled and proficient in the field; performs complex and significant work with minimal supervision and considerable autonomy for independent judgment.

• Scope of Influence: Anticipated to drive alignment across team interfaces with the rest of the organization. Designs, maintains, and owns team technical solutions while fostering consensus. Provides mentorship and guidance to engineers in the group.

• Bachelor’s Degree in Computer Science, Robotics, Electrical Engineering, or a related technical field, along with competencies typically developed through 6+ years of experience; OR a Master’s Degree in a related technical field with competencies typically acquired through 3+ years of experience.

• Expertise in Computer Vision & Deep Learning — experience in model training and at least two of the following: 2D/3D Object Detection, Tracking, Sensor Fusion, Semantic Segmentation, BEV, Depth Estimation.

• Experience with Multimodal / VLM — hands-on involvement with vision-language models, open-vocabulary or zero-shot recognition, dense captioning, or semantic embeddings/search applied to perception data.

• Proficiency in Model Data Curation — building focused datasets that significantly enhance downstream model performance; large-scale Parquet data processing (Databricks, Daft, Pandas, etc.).

• Familiarity with Distributed ML & data frameworks — experience with PyTorch, Lightning, Ray, Spark, or similar for training and large-scale data processing.

• Experience with Scaled MLOps & Tooling — knowledge of experiment tracking, model registry, MLflow / Weights & Biases, and ML metrics, evaluation, and quality.

• Proficient in Development Tools & Eco-System (at scale) — strong Python software development skills, experience with VDI and cloud-based development environments, CI systems (GitHub Actions), and Docker.

🏝️ Benefits

• A competitive compensation package that includes a bonus component and stock options.

• 100% paid medical, dental, and vision premiums for full-time employees.

• 401K plan with a 6% employer match.

• Flexibility in scheduling and generous paid vacation available immediately after start date.

• Company-wide holiday office closures.

• AD+D and Life Insurance.