QA Analyst – LOIS for Word

This is a fully remote position, open to applicants in California.

📋 Description

• Develop and take responsibility for the test strategy, process, and QA playbook related to the product.

• Advance QA practices by collaborating with product and engineering teams early on, ensuring test plans are prepared prior to feature launches.

• Assess AI output quality based on factors such as accuracy, citation reliability, and properly scoped reasoning.

• Create and uphold evaluation sets that account for non-deterministic output and identify regressions over time.

• Evaluate how AI-generated edits are applied to documents, ensuring accuracy and formatting consistency across various document structures.

• Implement automated guardrails integrated into CI/CD to safeguard critical processes.

• Produce structured, reproducible bug reports.

• Collaborate with legal domain specialists to comprehend how various customer types utilize the product and convert that understanding into test coverage.

• Approach QA with a focus on product and design. If something functions but has room for improvement, we want to know.

• Analyze behavior across product surfaces and document any discrepancies.

⛳️ Requirements

• Over 2 years of QA experience, including ownership of the test strategy for a product or significant feature area.

• Proven track record of establishing QA processes and automation from the ground up in a fast-paced environment.

• Ability to thrive in high ambiguity, where features are rapidly deployed, and you must determine testing methods quickly.

• Strong analytical and problem-solving abilities, with a keen focus on both functional accuracy and AI output quality.

• Capability to reason about non-deterministic systems and define "correctness" when there isn't a single right answer.

• Exceptional written communication skills, with the capacity to craft clear, concise, and actionable bug reports supported by evidence.

• Familiarity with bug tracking tools and test management workflows.

🏝️ Benefits

• Medical, Dental, & Vision Insurance (for full-time employees).

• Competitive & Fair Pay.

• Maternity & paternity leave (for full-time employees).

• Short & long-term disability.

• Opportunity to learn from a dedicated leadership team.

• Top-of-the-line company swag.