Manager, Data Science

This is a fully remote position, open to applicants in California.

📋 Description

• Direct the design and implementation of training and fine-tuning projects for large language models (LLMs), encompassing model selection, training strategies, experimentation, and evaluation.

• Supervise the creation of training datasets, which includes data collection, cleaning, deduplication, annotation, and ensuring quality validation.

• Develop and enhance workflows for supervised fine-tuning and parameter-efficient fine-tuning.

• Utilize preference optimization techniques where relevant.

• Create evaluation frameworks to assess factual accuracy, adherence to instructions, domain relevance, safety, and business-specific performance metrics.

• Identify training-related issues and enhance model quality, training stability, GPU usage, and computational efficiency.

• Lead and mentor data scientists, review their technical outputs, and promote reproducible development practices.

• Collaborate with product, engineering, and domain specialists to define requirements and assist in model deployment and monitoring.

• Oversee project priorities, timelines, and computational resources.

• Present results and trade-offs to stakeholders.

⛳️ Requirements

• In-depth understanding of transformer architectures, attention mechanisms, tokenization, language modeling objectives, as well as pretraining, continued pretraining, and fine-tuning processes.

• Proficient in Python and PyTorch.

• Hands-on experience with Hugging Face Transformers, Datasets, or similar tools.

• Proven ability to implement supervised fine-tuning (SFT), configure training objectives and loss masking, adjust hyperparameters, and select appropriate model checkpoints.

• Practical experience with parameter-efficient fine-tuning (PEFT), including LoRA or QLoRA.

• Capability to construct instruction-response datasets, utilize chat templates, manage sequence lengths and packing, and avoid data leakage and evaluation contamination.

• Experience in training models across multiple GPUs using PyTorch FSDP or DeepSpeed.

• Familiarity with mixed precision, gradient accumulation, and gradient checkpointing techniques.

• Ability to create reliable benchmarks and human evaluations, analyze model errors, and troubleshoot issues related to unstable loss, overfitting, and GPU memory constraints.

• Experience with experiment tracking, dataset and model versioning, checkpoint management, and well-documented training pipelines.

🏝️ Benefits

• Annual incentive bonus.

• Flexible working hours.

• Wellbeing initiatives.

• Shared parental leave.

• Study assistance.

• Sabbaticals.

• Country-specific benefits.

• Support for work/life balance.

• Accommodation or adjustment support for applicants with disabilities.