Software Development Engineer in Test – AI

This is a fully remote position, open to applicants in India.

📋 Description

• Design, validate, and enhance testing strategies for AI-driven applications, including copilots, chatbots, RAG pipelines, and autonomous AI agents.

• Create AI evaluation frameworks aimed at detecting hallucinations, validating groundedness, assessing toxicity and bias, ensuring response quality, context relevance, prompt regression, determinism, consistency, safety, and guardrail validation.

• Develop automated validation pipelines for LLM outputs utilizing structured assertions, semantic scoring, embeddings, and evaluation metrics.

• Generate synthetic and benchmark datasets tailored for AI testing scenarios.

• Examine multi-agent orchestration workflows and the retention of agent memory/context.

• Validate MCP integrations and ensure tool interoperability.

• Engage in Context Engineering, Prompt Engineering, and strategies for Prompt Optimization.

• Assess latency, token usage, model cost optimization, throughput, and scalability.

• Create and manage scalable UI/API automation frameworks using Python alongside tools such as Robot Framework, Playwright, Selenium, and Cypress.

• Automate testing for REST APIs, GraphQL APIs, event-driven systems, streaming systems, webhooks, microservices, and MFE architectures.

• Develop reusable automation libraries, utilities, and AI-assisted testing workflows.

• Integrate automation into CI/CD pipelines and quality gates.

• Validate RAG architectures, focusing on chunking, embedding quality, retrieval accuracy, re-ranking, and relevance in vector search.

• Construct or validate AI agents using LangChain, LangGraph, CrewAI, AutoGen, Semantic Kernel, and OpenAI SDKs.

• Validate tool calling, function calling, agent memory, agent planning/execution, multimodal AI systems, and structured outputs.

• Maintain quality workflows using GitHub Actions, Jenkins, GitLab CI/CD, and Azure DevOps.

• Work within Docker and Kubernetes environments.

• Conduct cloud-based testing and deployments across AWS, Azure, OCI, and GCP.

• Validate the reliability and resiliency of distributed systems.

• Monitor AI system behaviors utilizing observability, logging, and tracing tools.

• Contribute to shift-left methodologies and best practices in quality engineering.

• Collaborate effectively with Engineering, AI/ML, Platform, Product, and DevOps teams.

⛳️ Requirements

• Bachelor's/Master's degree in Computer Science, Engineering, AI/ML, or a related field is preferred.

• Equivalent hands-on experience and demonstrable projects are highly regarded.

• Proficient programming experience in Python is essential.

• Strong foundational knowledge of Software Testing & Quality Engineering principles.

• Practical experience in AI/LLM testing, Prompt Engineering, Context Engineering, AI evaluations, RAG systems, and AI agents.

• Familiarity with CI/CD pipelines, Docker & Kubernetes, cloud platforms (AWS/Azure/OCI/GCP), API testing & automation, microservices testing, and UI automation.

• Experience with SQL and data validation processes.

• Strong debugging, analytical, and troubleshooting capabilities are required.

• Proven experience working in Agile/Scrum environments.

• Experience with Robot Framework is preferred.

• Familiarity with Playwright is an advantage.

• Knowledge of LangChain, LangGraph, LlamaIndex, CrewAI, Semantic Kernel, and MCP is beneficial.

• Hands-on experience with Cursor, GitHub Copilot, Claude, Gemini, and OpenAI APIs.

• Experience with vector databases and embeddings is a plus.

• Exposure to MLOps workflows and testing throughout the ML lifecycle.

• Understanding of performance testing and security testing is desirable.

• Experience in validating AI governance/compliance requirements.

• Familiarity with contract testing, chaos testing, or resilience engineering is advantageous.

• Knowledge of accessibility testing and cross-browser/device testing.

• Exposure to synthetic monitoring and production observability.

• Skills in AI Red Teaming, LLM Security Testing, Agent Reliability & Safety Testing, AI Cost Optimization, Distributed Systems Testing, Test Data Engineering, Feature Flag Testing, AI Observability & Telemetry, Reinforcement Learning feedback workflows, Human-in-the-loop evaluation systems, Knowledge Graph integrations, and Multi-modal AI testing are considered high-impact bonus skills.

• Ability to experiment with AI tools and develop side projects, prototypes, proofs of concept, or contribute to open-source initiatives.

• Capacity to rapidly learn emerging AI frameworks, tools, and architectures.

🏝️ Benefits

• Work-life balance.

• Diversity and inclusion programs.

• Opportunities for learning and development.

• Access to 20+ affinity groups for networking and connection.

• Competitive HMO benefits – 175k MBL with one free dependent after one year of service.

• Punctuality Bonus.

• Generous vacation policy.

• Mentoring and career growth opportunities.