Manager, Data Science

This is a fully remote position, open to applicants in California.

📋 Description

• Lead the design and implementation of LLM training and fine-tuning initiatives, which encompass model selection, training strategies, experimentation, and evaluation.

• Supervise the preparation of training datasets, including data collection, cleansing, deduplication, annotation, and quality assurance.

• Develop and refine workflows for supervised fine-tuning and parameter-efficient fine-tuning.

• Implement preference optimization techniques where applicable.

• Create evaluation frameworks to assess factual accuracy, adherence to instructions, domain relevance, safety, and performance specific to the business.

• Identify training challenges and enhance model quality, training stability, GPU utilization, and computational efficiency.

• Mentor and manage data scientists, review technical deliverables, and establish reproducible development practices.

• Collaborate with product, engineering, and subject matter experts on requirements, deployment, and monitoring.

• Oversee project priorities, timelines, and computational resources.

• Relay results and trade-offs to stakeholders.

⛳️ Requirements

• Profound understanding of transformer architectures, attention mechanisms, tokenization, language modeling objectives, as well as pretraining, continued pretraining, and fine-tuning.

• Strong proficiency in Python and PyTorch.

• Practical experience with Hugging Face Transformers, Datasets, or similar tools.

• Proven ability to implement supervised fine-tuning (SFT), configure training objectives and loss masking, adjust hyperparameters, and select model checkpoints.

• Practical experience in parameter-efficient fine-tuning (PEFT), including LoRA or QLoRA.

• Capability to construct instruction-response datasets, utilize chat templates, manage sequence lengths and packing, and prevent data leakage and evaluation contamination.

• Experience in training models across multiple GPUs utilizing PyTorch FSDP or DeepSpeed.

• Familiarity with mixed precision, gradient accumulation, and gradient checkpointing.

• Ability to design reliable benchmarks and human evaluations, analyze model errors, and troubleshoot issues related to unstable loss, overfitting, and GPU memory.

• Experience with experiment tracking, dataset and model versioning, checkpoint management, and well-documented training pipelines.

🏝️ Benefits

• Annual incentive bonus.

• Flexible working hours.

• Wellbeing initiatives.

• Shared parental leave.

• Study assistance.

• Sabbaticals.

• Country-specific benefits.

• Disability accommodations during the hiring process.