Qualitative Evaluation Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Assess the performance of generative models across a variety of tasks, prompts, and modalities, identifying failure modes, regressions, and edge cases that negatively impact product quality.

• Develop and uphold qualitative evaluation frameworks that are both scalable and reusable.

• Convert overarching product objectives into specific evaluative criteria.

• Oversee qualitative studies, comparative analyses, and human-in-the-loop evaluations.

• Transform nuanced assessments into clear feedback that guides fine-tuning, dataset curation, and user experience improvements.

• Collaborate closely with technical artists and engineers to ensure evaluations are consistent with model capabilities and actual use cases.

⛳️ Requirements

• A minimum of 5 years of experience in product evaluation, UX research, model testing, or a related field focused on structured qualitative assessment.

• A Master's degree or higher in Cognitive Science, Human-Computer Interaction, Design Research, Psychology, Media Studies, or a related discipline.

• Strong systems thinking skills: the ability to articulate abstract qualities such as believability or scene coherence in clear evaluative terms.

• Comfort in collaborating with engineers, researchers, and creative professionals.

🏝️ Benefits

• Luma's mission is to create unified general intelligence capable of generating, understanding, and functioning within the physical world.

• Exceptional written communication skills and the capability to distill nuanced judgment into actionable insights.

• Extensive experience with creative workflows for generative models, including animation, filmmaking, digital art, and visual effects.