Senior Data Scientist

This is a fully remote position, open to applicants in California.

📋 Description

• Lead the design and implementation of LLM training and fine-tuning initiatives, encompassing model selection, training strategies, experimentation, and evaluation.

• Supervise the preparation of training datasets through processes such as collection, cleaning, deduplication, annotation, and quality assurance.

• Create and enhance workflows for supervised fine-tuning and parameter-efficient fine-tuning.

• Utilize preference optimization techniques when applicable.

• Develop evaluation frameworks focused on factual accuracy, adherence to instructions, domain relevance, safety, and performance specific to business needs.

• Identify training challenges and enhance model quality, training stability, GPU usage, and computational efficiency.

• Oversee and mentor data scientists.

• Evaluate technical work and promote reproducible development practices.

• Collaborate with product, engineering, and domain specialists to establish requirements and facilitate model deployment and monitoring.

• Prioritize project tasks, manage timelines, and allocate compute resources effectively.

• Convey results and trade-offs to stakeholders.

⛳️ Requirements

• In-depth understanding of transformer architectures, attention mechanisms, tokenization, language modeling objectives, and the distinctions between pretraining, continued pretraining, and fine-tuning.

• Proficient in Python and PyTorch.

• Hands-on experience with Hugging Face Transformers, Datasets, or equivalent tools.

• Proven track record in implementing supervised fine-tuning (SFT), configuring training objectives and loss masking, tuning hyperparameters, and selecting model checkpoints.

• Practical experience with parameter-efficient fine-tuning (PEFT), including techniques such as LoRA or QLoRA.

• Capability to construct instruction-response datasets, utilize chat templates, manage sequence lengths and packing, and prevent data leakage and evaluation contamination.

• Experience in training models across multiple GPUs utilizing PyTorch FSDP or DeepSpeed, with knowledge of mixed precision, gradient accumulation, and gradient checkpointing.

• Ability to design reliable benchmarks and human assessments, analyze model errors, and resolve issues related to unstable loss, overfitting, and GPU memory limitations.

• Familiarity with experiment tracking, dataset and model version control, checkpoint management, and well-documented training pipelines.

• Experience in managing and mentoring data scientists.

• Experience collaborating with product, engineering, and domain experts.

• Ability to effectively manage project priorities, timelines, and compute resources.

• No specific educational qualifications or minimum years of experience required.

🏝️ Benefits

• Annual incentive bonus.

• Initiatives promoting a healthy work/life balance.

• Wellbeing programs.

• Shared parental leave.

• Study assistance.

• Opportunities for sabbaticals.

• Flexible working hours.

• Country-specific benefits.

• Disability accommodations available during the hiring process.