Senior Data Scientist, Applied AI

This is a fully remote position, open to applicants in Massachusetts.

📋 Description

• Independently develop, test, and enhance workflows for LLM, RAG, and tool-using agents for a designated use case, utilizing maintainable and tested Python code.

• Choose and implement evaluation criteria, baselines, and test datasets in accordance with the team's established evaluation methodology.

• Conduct comparisons of models, embeddings, and retrieval techniques based on quality, reliability, latency, and cost; document recommendations and trade-offs.

• Analyze retrieval misses, incorrect responses, and workflow failures; identify root causes and execute and verify solutions.

• Create and update test cases, benchmark datasets, and regression checks.

• Monitor and analyze performance variations for assigned workflows and suggest and implement enhancements.

• Convey findings and technical trade-offs to engineering, product, and domain collaborators.

⛳️ Requirements

• A bachelor's degree is required; a master's degree in a quantitative or technical discipline is preferred.

• Over 5 years of relevant experience in data science, machine learning, applied AI, software engineering, or a related domain.

• Practical experience in building and assessing LLM applications.

• Proficient in Python programming, software engineering, testing, and analytical problem-solving skills.

• In-depth knowledge of LLMs, embeddings, semantic search, retrieval systems, and RAG architectures.

• Significant hands-on experience with Databricks, including Delta Lake, Unity Catalog, Lakebase, Lakeflow Jobs, and MLflow, for developing and deploying scalable, production-quality data and AI solutions.

• Capability to design representative evaluations for an assigned use case, assess errors, and utilize results to enhance a system.

• Ability to evaluate quality, reliability, latency, and cost when recommending an approach for their projects.

• Effective communication and collaboration skills with both technical and non-technical partners.

• Familiarity with agent frameworks and AI evaluation platforms.

• Experience with healthcare data is preferred but not mandatory.

• This is an individual contributor position with no people-management responsibilities.

• Ability to work independently on an assigned project within the team's established evaluation framework and practices.

• Capacity to escalate decisions regarding changes to the team's evaluation methodology, significant business risks, sensitive data, substantial cost commitments, or production release approvals.

🏝️ Benefits

• Eligibility for company bonus or commission plan, depending on the position.

• Medical insurance coverage.

• Dental insurance coverage.

• Vision insurance coverage.

• Unlimited paid time off.

• 401(k) plan with employer contribution.

• Access to industry-leading products.

• A flexible and dynamic work culture.

• Participation in the Definitive Cares community service group.

• A competitive benefits package.

• 401(k) matching.