QA Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Titan develops AI software tailored for banks, including specialized small language models, a banking ontology, and trustworthy AI bankers for financial institutions.

• You will be responsible for executing essential tasks: crafting test cases, establishing the evaluation framework, configuring CI/CD gates, and bug triaging in collaboration with the engineering team.

• You will personally design and implement the evaluation framework for LLM and agentic AI outputs across Foundry, Agent Builder, and client-deployed environments.

• You will create assertions, define behavioral contracts, and manage regression baselines for model performance.

• You will develop and uphold the automated test suite, ensuring end-to-end, integration, and regression coverage for backend APIs, document ingestion pipelines, AI inference workflows, and frontend interfaces.

• You will generate test artifacts, audit logs, and process documentation that comply with SOC 2 Type II standards.

⛳️ Requirements

• A minimum of seven years in software QA engineering, including at least two years focused on testing AI or ML systems.

• You have experience writing test cases for LLM outputs, constructing evaluation pipelines from the ground up, and distinguishing between flaky tests and genuinely non-deterministic systems.

• You are proficient in Python and have developed automated test suites using pytest, Playwright, or Selenium.

• Hands-on experience with RAGAS, DeepEval, LangSmith, or similar evaluation tools is required, not merely familiarity with the terms.

• You can trace failures from the application layer to infrastructure and possess sufficient knowledge about Azure, asynchronous systems, and REST APIs to perform this without needing an engineer's assistance.

• You have successfully integrated QA gates into CI/CD pipelines and managed the process from start to finish.

• Experience in fintech, banking, or other regulated sectors is a significant advantage.

• Familiarity with document processing pipelines, multi-agent architectures, RAG validation, or observability tools like Arize or Langfuse will give you an edge.

🏝️ Benefits

• Competitive base salary and substantial equity.

• Remote work opportunity (US), with occasional travel to client locations and team offsites.