Senior Data Scientist – LLM Evaluation, Business Intelligence

This is a fully remote position, open to applicants in New York, +1 more state.

📋 Description

• Design and conduct evaluations of production LLM pipelines, creating accuracy and quality metrics that assess the performance of our systems on actual clinical tasks.

• Assist in the evaluation of our Patient Trial Evaluation (PTE) pipeline by analyzing prompts and trial-matching logic for accurate reporting, and identifying areas and reasons for failures.

• Evaluate and enhance the performance of our retrieval-augmented generation (RAG) systems, focusing on retrieval quality through to the final output.

• Utilize accuracy metrics to guide cost/quality tradeoffs across trials, prompts, and models, providing the team with a solid foundation for deciding which strategies to implement.

• Create and sustain a library of "best prompts" for recurring clinical concepts, supported by evidence rather than mere intuition.

• Collaborate with AI Engineering to connect evaluation results with production enhancements.

• Develop and manage production-grade data models using SQL and dbt on Databricks, ensuring the reliability, maintainability, and production readiness of analytics logic.

• Contribute to the establishment and growth of a semantic layer that allows for consistent and trustworthy reporting throughout the organization.

• Enhance data governance practices across dbt, Databricks, and Hex, focusing on documentation, testing, version control, and clear ownership of metrics.

• Work alongside non-technical internal stakeholders to create reporting logic, address data inquiries, and convert business requirements into well-defined data products.

• Support both internal and external reporting demands with accurate, well-validated datasets.

• For senior candidates: mentor and assist junior data scientists by providing guidance on evaluation and analytics methodologies, technical best practices, and professional development.

⛳️ Requirements

• A Bachelor's degree or higher in data science, statistics, biostatistics, computer science, mathematics, epidemiology, or a related discipline.

• Over 4 years of experience in a data scientist, analytics engineer, ML/data professional, or similar highly analytical role within life sciences, biotech, healthcare, or a comparable industry.

• Strong comprehension of how production LLM pipelines function, along with practical experience in evaluating LLM or NLP systems—accuracy metrics, error analysis, RAG/retrieval quality, prompt evaluation, or equivalent.

• High proficiency in SQL, including experience in constructing production data models using dbt (or a similar transformation framework).

• Proficiency in at least one programming language (Python preferred; PySpark is a plus).

• Familiarity with modern data platforms like Databricks, and BI/analytics tools such as Hex, or similar technologies.

• Knowledge of data science best practices, including reusability, version control, documentation, testing, code review, and reproducibility.

• Exceptional written and verbal communication skills, with the ability to convert technical concepts into actionable insights for non-technical stakeholders and collaboratively develop reporting logic with them.

• Comfort with AI tools and agentic workflows in everyday tasks.

• Ability to navigate ambiguity and adapt to a mission-driven, fast-paced startup environment.

🏝️ Benefits

• Offers Equity