Applied AI Engineer – Systems & Reliability

This is a fully remote position, open to applicants anywhere in the world.

📋 Description

• Take charge of evaluation systems and uphold quality standards.

• Develop and sustain evaluation pipelines for essential AI processes including screening, interviews, assessments, and references.

• Establish metrics, benchmarks, and acceptance criteria for AI outputs.

• Monitor performance over time (quality trends, drift, regressions) and ensure visibility of results across the team.

• Foster continuous enhancement of AI performance.

• Identify problems within prompts, workflows, and data pipelines through quantitative analysis and in-depth case studies.

• Design and execute enhancements across:

• prompting strategies.

• model selection, configuration, and fine-tuning.

• input data quality and preprocessing.

• orchestration and workflow design.

• Advance new systems from “working” (80%) to reliable and high-quality (95%+).

• Ensure system reliability, monitoring, and stability.

• Develop and refine monitoring systems for AI (e.g., dashboards, alerts, tracing).

• Identify and mitigate failure modes, breakdown risks, and performance decline.

• Oversee usage, rate limits, and capacity to guarantee stable operation at scale.

• Promote testing, CI, and safe shipping methodologies.

• Integrate AI and prompt testing into CI (e.g., regression tests, golden datasets, staging environments).

• Establish standards and tools so that product and engineering teams can safely deploy without causing regressions.

• Serve as a quality gate for AI-related modifications.

• Manage AI system audits and compliance assistance.

• Prepare for and support internal and external audits (e.g., SOC 2 and beyond).

• Provide evidence, documentation, and artifacts regarding AI system behavior and controls.

• Convert audit findings into tangible improvements in systems and processes.

• Productionize AI workflows (beyond just prototyping them).

• Develop and productionize AI workflows that adhere to established quality and reliability standards.

• Assist product and engineering teams in seamlessly integrating AI into product logic and user experience.

• Ensure new AI functionalities are robust, measurable, and maintainable prior to release.

⛳️ Requirements

• Complete alignment with our Ops Principles (please do not apply if this does not resonate with you).

• Enthusiasm for development in Go.

• Experience with AI/ML systems, LLMs, or data-heavy applications.

• Strong sense of ownership and meticulous attention to detail.

• Deep interest in quality, reliability, and system performance, rather than just feature development.

• Capability to debug intricate systems across prompts, models, and data pipelines.

• Excellent communication and documentation abilities.

• Comfort with enhancing systems and processes, not merely utilizing them.

• Experience with evaluation techniques, metrics, or experimentation is a significant advantage.

• Familiarity with monitoring, CI/CD, and production systems is a plus.

🏝️ Benefits

• Direct responsibility for one of the most vital aspects of the company: AI quality and reliability.

• Collaborate closely with founders on key product and technical decisions.

• Competitive salary along with substantial stock options.

• Educational stipend to facilitate continuous learning and development.

• The most exceptional team to collaborate with (true story!).