Senior AI Trust Engineer

This is a fully remote position, open to applicants in Spain.

📋 Description

• Design and operate evaluation, observability, and governance mechanisms for AI systems.

• Provide objective evidence of compliance with functional, technical, and business requirements.

• Work throughout the entire lifecycle of solutions, from design to production.

• Collaborate with Software Builders, Agentic Platform Engineers, and AI Architects.

• Implement AI applications using Langfuse or other equivalent tools.

• Capture and analyze complete traces of prompts, responses, tool calls, intermediate steps, latency, tokens, and costs.

• Investigate behaviors and detect degradations or anomalies in production.

• Build dashboards and alerts using Grafana.

• Work with standards such as OpenTelemetry applied to AI systems.

• Design and execute evaluations using Ragas and DeepEval.

• Assess faithfulness, relevance, accuracy, context completeness, hallucinations, and appropriate use of tools.

• Design benchmarks and reference datasets for reproducible version comparisons.

• Apply Human-in-the-Loop and LLM-as-a-Judge methodologies.

• Evaluate robustness and security against biases, prompt injection, and jailbreaks.

• Define objective metrics and acceptance criteria for each use case.

• Integrate AI evaluations into CI/CD pipelines.

• Build quality gates to block changes that do not meet quality thresholds.

• Automate the evaluation of changes in models, prompts, configurations, and agents.

• Develop tools and evaluations in Python.

• Build and maintain an evaluation engine based on versioned datasets.

• Detect regressions and reproducibly compare behavior between versions.

• Ensure traceability of responses, including context, model, prompt version, and intermediate steps.

• Define and analyze quality KPIs and identify improvements.

• Communicate the quality status to both technical and business profiles.

⛳️ Requirements

• Over 4 years of professional experience in AI Quality Engineering, Software Engineering, MLOps, observability, or data/AI engineering.

• Practical experience with generative AI in production.

• Real-world experience with Ragas and/or DeepEval.

• Experience with Langfuse or other observability platforms for LLMs/agents.

• Experience integrating automated evaluations as quality gates in CI/CD.

• Strong knowledge of Python.

• Experience working with LLMs, RAG, and agentic architectures.

• Knowledge of metrics and evaluation methodologies for LLMs.

• Experience with Grafana.

• Knowledge of OpenTelemetry is a plus.

• Understanding of AI system security and robustness.

• Ability to interpret results and translate them into improvement actions.

• Not seeking a traditional QA profile focused solely on functional testing.

🏝️ Benefits

• Opportunity to work in a remote or hybrid work model.

• Flexibility to organize work and promote a balance between personal and professional life.

• Buddy support from day one.

• Annual training plan.

• Official certifications.

• Learning and professional development spaces.

• 100% subsidized health insurance.

• Flexible compensation through Edenred.

• 25 days of annual vacation.

• Additional benefits focused on well-being.

• Close and collaborative environment.

• Commitment to equal pay and opportunities.