Senior AI Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design and implement production AI systems from start to finish (orchestration, retrieval, inference, evaluation, and monitoring) aimed at clinical decision support, document comprehension, and agentic medical companions.

• Create agentic workflows with tailored skills and harnesses, utilizing cutting-edge LLMs and self-hosted models when suitable, while maintaining a preference for function calls over agent invocation where applicable.

• Oversee the development of structured information extraction pipelines from clinical documents, incorporating automated verification systems and human-in-the-loop checkpoints as necessary.

• Establish a robust evaluation framework (golden datasets, LLM-as-judge, regression tests, online A/B evaluations) that scrutinizes every model and prompt modification. Key metrics include accuracy, safety, cost, and latency.

• Focus on latency and cost engineering: implementing streaming, caching, prompt compression, model routing, and inference-cost management.

• Take responsibility for production health: ensuring trace- and span-level observability, prompt versioning, drift detection, cost monitoring, debugging through replay-from-production, PHI-safe logging, and proactively identifying issues before they impact clinicians.

• Operationalize models developed by the AI Scientist team: managing serving, production evaluation, rollback strategies, and creating feedback loops that transform real-world usage into training and evaluation data, rather than unproductive discussions.

• Collaborate with clinicians, the data engineering team, and research partners to convert clinical needs into specifications and deployable systems.

• Train and guide PMs and engineers across Atria on applied AI best practices, from prompt and evaluation design to selecting the most suitable model for the task (which is often not the largest one).

• Define the technical direction for AI engineering at Atria: making architecture choices, deciding between building or buying solutions, and establishing evaluation and observability standards for the team.

• Enhance engineering practices, code reviews, observability, and incident response, consistently and gracefully.

• Conduct technical due diligence on AI/ML vendors: assessing BAA scope, PHI management, sub-processor responsibilities, and intellectual property terms.

• Mentor fellow AI engineers through code reviews, design documents, and architecture decisions, with a steadfast belief that effective systems are developed twice: first poorly, then correctly.

⛳️ Requirements

• A proactive attitude towards shipping. You prefer to have a rough product in production today that you can measure and enhance over a polished product outlined in a design document for next quarter.

• A resourceful mindset. You can quickly learn an unfamiliar tool, framework, or clinical concept and have it operational within a short timeframe.

• A strong commitment to continuous improvement. You engage with others' code, research papers, and post-mortems, viewing mistakes as valuable learning opportunities.

• Practical experience in building and operating LLM-powered or production ML systems: a minimum of 4 years.

• Proficient in Python, with extensive knowledge of at least one production backend framework (e.g., FastAPI, Flask).

• Hands-on experience with LLM orchestration, function/tool invocation, and designing retrieval systems: including chunking, hybrid search, reranking, and the essential components that ensure RAG functions effectively.

• Familiarity with cloud data warehousing solutions (preferably Snowflake) and modern data tools such as dbt, Dagster, or Airflow.

• Proven track record in designing evaluation systems for LLM applications — including those that identified regressions before users noticed them, and those that did not.

• Awareness of safety considerations for LLM applications: guardrails, adversarial testing, and common failure scenarios that may arise unexpectedly.

• Practical knowledge of HIPAA, BAAs, PHI management, and the day-to-day challenges of operating in a regulated environment without compromising morale.

• Excellent written communication skills: crafting design documents, technical RFCs, and accessible documentation that others can follow after several months.

🏝️ Benefits

• Comprehensive health and wellness benefits, fully funded by Atria, effective from the date of hire.

• OneMedical membership available for employees and their dependents, providing access to 24/7 virtual healthcare.

• Support for fertility and family planning.

• Company-sponsored preventive health screenings through partner hospitals (e.g., calcium score).

• Fitness perks, including Wellhub + offerings.

• 401k contributions with a 4% match starting after 6 months of employment.

• Flexible Time Off policy.

• Continuing medical education (CME) and CEU support for professional licensure.