LLMOps Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop and manage evaluation suites tailored to specific products based on failure modes, ensuring they are executed on time with every release.

• Ensure cost-effective evaluations as the product line expands.

• Take ownership of the reporting process by sharing evaluation results and platform data with the team and stakeholders.

• Identify unexpected or problematic model behaviors prior to learners encountering them.

• Oversee essential datasets, including classified customer support information.

• Collaborate with the AI Product Manager on product instrumentation and create evaluations using collected data.

• Work alongside Anthropic Engineers, AI engineers, product managers, and data scientists to design effective learning experiences.

• Progress from assisting the technical lead’s designs to taking full responsibility for the analyzer function.

⛳️ Requirements

• Proficient in Python and comfortable with shaping and analyzing untidy LLM-generated data.

• Strong inclination towards data.

• Capability to break down AI products into metrics for success and failure.

• Familiarity with evaluation tools such as Langfuse, RAGAS, DSPy, or similar, with depth in one and awareness of others.

• Proven experience in measuring an AI system objectively in a project.

• Eagerness to take ownership of a function rather than merely executing tasks.

• Excellent communication and collaboration skills.

• Self-motivated to maintain constant communication through web collaboration and video conferencing.

• Preferred time zone: GMT+2.

• Desirable but not mandatory: experience in maintaining cost-effective evaluations at scale, dashboarding and reporting, and classical statistics.

🏝️ Benefits

• Encouragement for continuous capability development and attainment of certifications to reflect personal growth.

• Opportunities for virtual work and collaboration across international locations.