Head of AI

This is a fully remote position, open to applicants in Ukraine.

📋 Description

• Design and develop KMA’s internal AI operating system (AI OS)

• Create production LLM architecture incorporating model gateway/router, fallbacks, caching, cost caps, agent orchestration, and skill/agent registry

• Enhance the existing MCP Gateway and resolve its current deficiencies

• Establish both offline and online evaluation and monitoring systems for AI solutions

• Implement LLM-as-judge systems with calibration, pairwise comparison, judge-bias checks, failure taxonomies, drift detection, and degradation alerts

• Configure LLM call tracing for tokens, latency, cache hits, and routing decisions, correlating them with hallucination rates, groundedness, and task success

• Develop AI security measures, including guardrails, tiered permissions, blocking of dangerous access combinations, PII management, secrets protection, and human-in-the-loop controls

• Automate Operations, HR, and Finance processes with AI in collaboration with the VP of Engineering

• Expand the range of tasks that AI can perform without the need for developer involvement

• Make decisions regarding model-routing and build-vs-buy-vs-fine-tune based on cost and quality considerations

• Develop a methodology for assessing ROMI and LTV through holdout design, impact ledger, and baselines

• Build and nurture the AI Integration team

• Engage in the hiring process through recruiter, technical, CTO, and final interviews

⛳️ Requirements

• Proven experience in building and launching AI/LLM systems in production, from problem identification to launch and impact assessment

• Familiarity with production LLM architecture: gateway/router, fallbacks, caching, rate limiting, observability, RAG, agents, and deployment methodologies

• Experience working with various LLM providers and understanding the trade-offs between API, managed inference, and self-hosted solutions

• Experience in creating offline/online evaluations, LLM-as-judge, error analysis, failure taxonomies, and drift detection

• Knowledge of data pipelines, PII, data quality, and data lineage

• Understanding of AI security issues: prompt injection, jailbreaks, data exfiltration, permissions, secrets management, and human-in-the-loop processes

• Familiarity with AI governance, the EU AI Act, and GDPR compliance

• A strong grasp of unit economics: token costs, cost efficiency, and the impact of automation

• Knowledge of build-vs-buy-vs-fine-tune considerations and model routing based on cost and quality

• Experience in defining business metrics, establishing baselines, and measuring impact/ROMI

• Hands-on approach with the ability to perform technical tasks independently until they can be effectively delegated

• Capability to influence without direct authority and manage stakeholders beyond direct reporting lines

• Excellent executive communication skills, sound business judgment, and a data-driven mindset

• Experience in building and leading a small technical team

• 2–3+ years of direct AI/LLM production experience

• Minimum of 1 year in a technical team management role

🏝️ Benefits

• Fully remote work from anywhere in the free world, with optional access to offices in Kyiv and Warsaw

• Equipment provided regardless of work format and location

• Opportunities for career advancement

• Online yoga sessions held twice a week

• English speaking clubs with a native speaker

• Opportunities to present at internal and external Kiss My Apps events

• Annual budget covering up to 50% of professional conferences, educational events, and training

• Fixed annual benefits budget allowing for 50% coverage of medical insurance, psychotherapy, sports, or dental care

• 40 days of paid vacation and sick leave with no date restrictions

• Support for uninterrupted work during times of war

• Ongoing support for the Defense Forces of Ukraine and involvement in initiatives aimed at Ukraine's recovery

• Option for military reservation post-probation, contingent upon the completion of military registration documents and company capabilities in accordance with applicable laws