AI Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Continuous LLM Evaluation: Develop and implement a systematic, ongoing process to assess new and emerging LLMs based on accuracy, relevance, speed, and cost — consistently benchmarking them against the specific tasks in our orchestration pipeline to proactively enhance outcomes.

• Eval Framework Development: Create and uphold stringent evaluation frameworks (Evals) to gauge LLM output accuracy, relevance, faithfulness, and speed, with a particular emphasis on minimizing hallucinations in medical record summarization and legal document analysis.

• Proactive Model Transition Planning: Keep track of the LLM landscape across various providers to recognize deprecation timelines and identify appropriate replacement models — and take full ownership of executing those transitions, including the integration of new models into the production pipeline and necessary adjustments to accommodate model behavior.

• AI Pipeline Optimization: Actively implement enhancements to LLM-based orchestration pipelines for document understanding, medical record summarization, case chronology generation, and drafting assistance — overseeing code modifications, deployments, and production validations from inception to completion.

• Cross-Functional Collaboration: Collaborate with product and GTM stakeholders to share model evaluation insights — then personally lead the technical implementation instead of delegating execution to a distinct engineering team.

• End-to-End Implementation Ownership: Assume full responsibility for deploying model changes into production — including writing integration code, managing deployments, executing validation tests, and ensuring a smooth rollout.

• Operational Monitoring: Establish monitoring and observability for model performance in production, benchmarking outputs and costs, detecting drift with ongoing and continuous reporting to management.

• Documentation: Ensure comprehensive documentation of evaluation methodologies, model comparison results, transition decisions, and runbooks for the systems under your purview.

⛳️ Requirements

• Over 5 years of AI/ML engineering experience assessing, fine-tuning, and deploying large language models in production settings — including building and deploying models to cloud (AWS or GCP) infrastructure at scale.

• Practical development and implementation experience with multiple RAG solutions.

• Hands-on expertise in utilizing embedding models and vector databases.

• Practical experience in constructing agentic workflows.

• Extensive familiarity with the LLM ecosystem and the capability to critically evaluate model strengths, weaknesses, and suitability for specific tasks, including cost, quality, speed, and capability tradeoffs.

• Demonstrated experience in designing and executing evaluation frameworks to assess LLM output quality, including accuracy, relevance, and hallucination detection in high-stakes domains (legal, medical, or similar).

• Strong software engineering background with proven experience in developing production-deployed solutions, including LLM orchestration frameworks and multi-model pipelines.

• Comfort in a fast-paced, high-ambiguity environment with strong ownership, tight feedback loops, and a preference for systematic process-building over one-off solutions.

• Outstanding communication skills; ability to distill complex model evaluation results into clear recommendations for engineering, product, and non-technical stakeholders.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Opportunities for professional development and continuous learning.

• Flexible working hours and remote work options.

• Comprehensive health and wellness benefits.

• Collaborative and inclusive work environment.