AI Quality Assurance Analyst

This is a fully remote position, open to applicants in United States.

📋 Description

• Create and implement testing strategies for AI-driven applications, focusing on both output quality and functional performance.

• Assess AI responses for precision, consistency, hallucinations, and handling of edge cases.

• Test workflows, user interface behavior, and integrations across both internal systems and client-facing tools.

• Identify, document, and monitor defects using standard issue tracking systems (e.g., Jira or similar).

• Collaborate with AI Builders and Product Managers to establish acceptance criteria and quality benchmarks for AI functionalities.

• Develop and sustain repeatable test cases and quality assurance documentation as AI applications progress.

• Perform regression testing when applications are modified or new features are introduced.

• Work closely with the Principal Engineer and product team to identify quality risks early in the development cycle.

• Help define the standards for “good” AI output quality at OneDigital, contributing to the establishment of internal QA benchmarks for LLM-based applications.

• Assist in testing across various business sectors, including Employee Benefits, Retirement, Wealth, and P&C.

⛳️ Requirements

• Experience in a QA, testing, or quality assurance role, with a preference for exposure to AI or LLM-based products.

• Strong capability to assess AI-generated outputs for quality, accuracy, and suitability.

• Familiarity with prompt-driven systems and an understanding of potential failures or unexpected results from language models.

• Experience in writing and executing test cases, along with clearly documenting defects.

• Comfort in working within no-code and low-code environments.

• Excellent written communication skills, with the ability to articulate issues clearly and succinctly.

• Curiosity and flexibility in a rapidly evolving technology landscape.

• Ability to work autonomously and manage multiple testing workflows simultaneously.

🏝️ Benefits

• Health, wellness, retirement, and various financial benefits.

• Paid time off.

• Overtime compensation for non-exempt staff.

• Comprehensive learning and development programs.

• Reimbursement of job-related expenses as per company policy.

• Employee perks and discounts.