
Head of AI
Posted Sep 29

Posted Sep 29
This is a fully remote position, open to applicants in Ukraine.
• Design and develop KMA’s internal AI operating system (AI OS)
• Create production LLM architecture incorporating model gateway/router, fallbacks, caching, cost caps, agent orchestration, and skill/agent registry
• Enhance the existing MCP Gateway and resolve its current deficiencies
• Establish both offline and online evaluation and monitoring systems for AI solutions
• Implement LLM-as-judge systems with calibration, pairwise comparison, judge-bias checks, failure taxonomies, drift detection, and degradation alerts
• Configure LLM call tracing for tokens, latency, cache hits, and routing decisions, correlating them with hallucination rates, groundedness, and task success
• Develop AI security measures, including guardrails, tiered permissions, blocking of dangerous access combinations, PII management, secrets protection, and human-in-the-loop controls
• Automate Operations, HR, and Finance processes with AI in collaboration with the VP of Engineering
• Expand the range of tasks that AI can perform without the need for developer involvement
• Make decisions regarding model-routing and build-vs-buy-vs-fine-tune based on cost and quality considerations
• Develop a methodology for assessing ROMI and LTV through holdout design, impact ledger, and baselines
• Build and nurture the AI Integration team
• Engage in the hiring process through recruiter, technical, CTO, and final interviews
• Proven experience in building and launching AI/LLM systems in production, from problem identification to launch and impact assessment
• Familiarity with production LLM architecture: gateway/router, fallbacks, caching, rate limiting, observability, RAG, agents, and deployment methodologies
• Experience working with various LLM providers and understanding the trade-offs between API, managed inference, and self-hosted solutions
• Experience in creating offline/online evaluations, LLM-as-judge, error analysis, failure taxonomies, and drift detection
• Knowledge of data pipelines, PII, data quality, and data lineage
• Understanding of AI security issues: prompt injection, jailbreaks, data exfiltration, permissions, secrets management, and human-in-the-loop processes
• Familiarity with AI governance, the EU AI Act, and GDPR compliance
• A strong grasp of unit economics: token costs, cost efficiency, and the impact of automation
• Knowledge of build-vs-buy-vs-fine-tune considerations and model routing based on cost and quality
• Experience in defining business metrics, establishing baselines, and measuring impact/ROMI
• Hands-on approach with the ability to perform technical tasks independently until they can be effectively delegated
• Capability to influence without direct authority and manage stakeholders beyond direct reporting lines
• Excellent executive communication skills, sound business judgment, and a data-driven mindset
• Experience in building and leading a small technical team
• 2–3+ years of direct AI/LLM production experience
• Minimum of 1 year in a technical team management role
• Fully remote work from anywhere in the free world, with optional access to offices in Kyiv and Warsaw
• Equipment provided regardless of work format and location
• Opportunities for career advancement
• Online yoga sessions held twice a week
• English speaking clubs with a native speaker
• Opportunities to present at internal and external Kiss My Apps events
• Annual budget covering up to 50% of professional conferences, educational events, and training
• Fixed annual benefits budget allowing for 50% coverage of medical insurance, psychotherapy, sports, or dental care
• 40 days of paid vacation and sick leave with no date restrictions
• Support for uninterrupted work during times of war
• Ongoing support for the Defense Forces of Ukraine and involvement in initiatives aimed at Ukraine's recovery
• Option for military reservation post-probation, contingent upon the completion of military registration documents and company capabilities in accordance with applicable laws
CVS Health Careers
Sourcegraph
MCA Connect
Lightly
Get handpicked remote jobs straight to your inbox weekly.