
AI Engineer I
Posted 1 day ago

Posted 1 day ago
This is a fully remote position, open to applicants in California.
• Develop, test, and sustain the Python capabilities and AI agents operational on the platform.
• Maintain an organized skills catalog through version control, documentation, ownership, and the decommissioning of unused skills.
• Create evaluation and regression checks, including test sets, validators, and output format rules to ensure consistent agent quality.
• Oversee agent usage, expenses, and failures, addressing root causes effectively.
• Manage the platform's agent runtime, built on the TypeScript-based Mastra framework, including workflows, tool definitions, memory, and model configurations.
• Maintain MCP servers and connectors that integrate agents with the warehouse, semantic layer, BI tools, and other organizational systems.
• Ensure API services, scheduled jobs, and data pipelines function optimally; manage framework, SDK, and model version upgrades.
• Diagnose and resolve failed jobs, broken connectors, data quality alerts, and issues reported by users.
• Enhance CI/CD processes, logging, and monitoring systems.
• Create runbooks to enable team members to operate the platform efficiently.
• Comprehend how the gold data layer and semantic models contribute to the platform, making minor, reviewed changes as necessary.
• Collaborate with the metric tree to integrate new metrics and relationships into the platform.
• Trace platform metrics back to source data when discrepancies occur.
• Bachelor’s degree in a relevant quantitative, technical, engineering, or business-related discipline, or an equivalent combination of education and experience.
• Generally requires 2 years of relevant experience in AI-native analytics.
• Experience with native AI implementations in the insurance industry is preferred.
• Proficient in Python: able to write, test, and debug modular code independently, ensuring deterministic output.
• Familiar with TypeScript/JavaScript: capable of reading and modifying an existing backend codebase; proficiency in TypeScript is required as Mastra is TS-based.
• Skilled in SQL: able to perform joins, aggregations, and window functions against a warehouse; capable of sanity-checking a metric's output.
• Has developed at least one fully functional LLM application end-to-end, such as a tool-calling agent, a RAG app, or an MCP server; a completed side project is acceptable.
• Understands the distinctions between prompts, tools, retrieval, context limits, and the reasons behind the nondeterministic nature of agent output.
• Possesses an instinct for evaluation: tests agent output against expected responses rather than relying on visual inspection.
• Regularly uses Git and GitHub: experienced in branching, PRs, and code review processes.
• Works with REST or JSON APIs and can interpret API documentation with minimal supervision.
• Understands the concepts of semantic layers, dimensions, and metrics, and how they differ from raw tables.
• Capable of reading a dbt model and YAML configuration, following data lineage effectively.
• Competitive compensation package.
• Flexibility to work remotely from anywhere in the United States for most roles.
• Paid time off including vacation days, sick leave, 9 paid company holidays, and volunteer hours.
• Incentive bonus programs, which may include holiday bonuses, referral bonuses, and performance-based bonuses.
• Comprehensive medical, dental, vision, life, and pet insurance options.
• 401(k) retirement savings plan with company matching contributions.
• A dynamic and engaging work environment.
• Opportunities for career advancement.
• Assistance for educational pursuits.
• Opportunities for both professional and personal development.
• Company recognition programs to acknowledge employee contributions.
• Health and wellbeing resources, including complimentary mental wellbeing therapy/coaching sessions, childcare and eldercare resources, and more.
• An in-person interview may be required during the hiring process.
WBS
WBS Training
NIKI Digital Engineering
CVS Health
Get handpicked remote jobs straight to your inbox weekly.