AI Software Development Engineer in Test – SDET

This is a fully remote position, open to applicants in Argentina, +1 more country.

📋 Description

• Design, validate, and enhance testing strategies for applications powered by AI, including copilots, chatbots, RAG pipelines, and autonomous AI agents.

• Create AI evaluation frameworks focusing on hallucination detection, groundedness, toxicity and bias, response quality, context relevance, prompt regression, determinism, consistency, safety, and guardrails.

• Construct automated validation pipelines for LLM outputs utilizing structured assertions, semantic scoring, embeddings, and evaluation metrics.

• Generate synthetic and benchmark datasets for AI testing scenarios.

• Assess multi-agent orchestration workflows and the retention of agent memory/context.

• Validate MCP integrations and ensure tool interoperability.

• Engage in Context Engineering, Prompt Engineering, and strategies for Prompt Optimisation.

• Analyze latency, token usage, model cost optimization, throughput, and scalability.

• Develop and sustain scalable UI/API automation frameworks using Python and various testing tools.

• Automate testing for REST APIs, GraphQL APIs, event-driven systems, streaming systems, webhooks, microservices, and MFE architectures.

• Create reusable automation libraries, utilities, and AI-assisted testing workflows.

• Integrate automation within CI/CD pipelines and quality gates.

• Validate RAG architectures, vector databases, embeddings, retrieval accuracy, re-ranking, and vector search relevance.

• Construct or validate AI agents and workflows that involve tool calling, function calling, agent memory, planning/execution, multimodal systems, and structured outputs.

• Maintain quality workflows integrated with CI/CD platforms.

• Operate within Docker and Kubernetes environments.

• Conduct cloud-based testing and deployments across AWS, Azure, OCI, and GCP.

• Validate the reliability and resiliency of distributed systems.

• Monitor AI system behavior using observability, logging, and tracing tools.

• Contribute to shift-left and quality engineering best practices.

• Collaborate with Engineering, AI/ML, Platform, Product, and DevOps teams.

⛳️ Requirements

• Proficient programming skills in Python.

• Solid understanding of Software Testing and Quality Engineering principles.

• Practical experience with AI/LLM testing, Prompt Engineering, Context Engineering, AI evaluations, RAG systems, and AI Agents.

• Familiarity with CI/CD pipelines, Docker & Kubernetes, cloud platforms (AWS/Azure/OCI/GCP), API testing & automation, microservices testing, and UI automation.

• Experience with SQL and data validation.

• Strong debugging, analytical, and troubleshooting abilities.

• Background working in Agile/Scrum environments.

• Preferred experience with Robot Framework.

• Preferred experience with Playwright.

• Knowledge of LangChain, LangGraph, LlamaIndex, CrewAI, Semantic Kernel, and MCP is preferred.

• Hands-on experience with Cursor, GitHub Copilot, Claude, Gemini, and OpenAI APIs is preferred.

• Experience with vector databases and embeddings is preferred.

• Exposure to MLOps workflows and ML lifecycle testing is preferred.

• Understanding of performance testing and security testing is preferred.

• Experience in validating AI governance/compliance requirements is preferred.

• Familiarity with contract testing, chaos testing, or resilience engineering is preferred.

• Knowledge of accessibility testing and cross-browser/device testing is preferred.

• Exposure to synthetic monitoring and production observability is preferred.

• Bachelor’s or Master’s degree in Computer Science, Engineering, AI/ML, or a related field is preferred.

• Equivalent hands-on experience and demonstrable projects are highly valued.

🏝️ Benefits

• Flexible work-from-home, hybrid, or in-office options.

• OSDE 310 Health Care coverage.

• Regular monthly salary reviews.

• Tuition reimbursement opportunities.

• Internet reimbursement provided.

• In-house English lessons available.

• Access to LinkedIn Learning resources.

• Bonus for the referral program.

• Additional days off through F5.

• Annual bonus eligibility.

• Opportunity to work on global projects with a diverse team.

• Opportunities for learning and professional development.

• Access to over 20 affinity groups for networking and connection.

• Mentoring and career growth opportunities.