Applied AI Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop and manage production-level LLM-based agents and assistants.

• Create agent runtime and orchestration, encompassing tool calls, context management, handoffs, failure responses, retries, timeouts, and sandboxing.

• Design evaluation sets, testing frameworks, regression tests, and AI-graded assessments.

• Collaborate with various model providers, oversee version upgrades, and strategize provider fallbacks.

• Monitor agent behavior, track production performance, and allocate AI expenditures to agents and use cases.

• Integrate agents with company data utilizing retrieval (RAG), context design, and prompt engineering.

• Partner with operations teams to assess which workflow tasks agents can reliably execute and when to escalate issues to human personnel.

• Safeguard patient data (PHI), manage agent access, and protect against prompt injection attacks.

• Develop on GCP and collaborate with Data Platform teams.

• Participate in code and design reviews, and mentor engineers who are new to AI.

• Articulate AI trade-offs to product, operations, and clinical stakeholders.

• Perform additional responsibilities as required.

⛳️ Requirements

• Bachelor's degree in Computer Science, Engineering, or a related technical discipline.

• 5 to 15+ years of experience in software engineering.

• Practical experience in building and operating LLM-based systems in production that real users depend on.

• Senior Staff candidates typically possess 2+ years of experience in production LLM.

• In-depth hands-on experience in at least one core area: agent runtime, evaluation, retrieval, or observability and cost.

• Staff and Senior Staff candidates should have expertise in agent runtime or evaluation.

• Senior Staff candidates are expected to have proficiency in a second deep area along with a working knowledge of the remaining areas.

• Experience in assessing AI system performance using evaluation data.

• Strong production engineering skills across testing, debugging, services, APIs, deployment, and on-call responsibilities.

• Experience managing sensitive data with a focus on security.

• Comfort in working within a greenfield environment.

• Ability to communicate effectively with both technical and non-technical partners.

• Preferred experience in healthcare, finance, or insurance sectors.

• Preferred experience with GCP production services.

• Preferred experience in migrating systems between model versions or providers.

• Preferred experience in machine learning beyond LLMs.

• Preferred experience in developing internal platforms utilized by other engineering teams.

• For Senior Staff, preferred experience in guiding a small team's technical direction and scaling a single pod into multiple teams.

🏝️ Benefits

• Medical, Dental, and Vision plans.

• Flexible Spending/Health Savings Accounts.

• Flexible PTO.

• 401(k) + Company Match.

• Life Insurance.

• Pet insurance.