Expert Software Engineer – AI Evaluation

This is a fully remote position, open to applicants in India.

📋 Description

• Create realistic, multi-step coding and technical challenges that assess AI models' reasoning, coding abilities, and adherence to instructions.

• Develop tasks utilizing codebases, technical specifications, architectural documents, and datasets.

• Analyze AI-generated code and technical content for accuracy, efficiency, clarity, and alignment with real-world engineering practices.

• Evaluate model outputs against established rubrics and ground-truth answers, providing structured feedback.

• Enhance AI comprehension of programming, system architecture, data workflows, and engineering best practices.

• Operate independently and asynchronously, adhering to deadlines while collaborating with the Patronus team.

• Test and refine content to mirror real-world complexities.

• Review the work of other contributors for precision, clarity, and practical applicability.

• Promptly incorporate feedback and identify any gaps.

⛳️ Requirements

• A minimum of 2 years of professional experience in software engineering, excluding internships.

• Proficient in system design, data pipelines, and applications based on machine learning or large language models.

• Previous experience in code reviews, technical writing, AI training, or model evaluation.

• Strong programming proficiency in at least one major language, such as Python, Java, C++, JavaScript, TypeScript, or Go.

• Experience in building, reviewing, and debugging production-quality code.

• Ability to adhere to detailed technical guidelines, work autonomously, respond to feedback, and meet deadlines.

• Exceptional written communication skills and meticulous attention to detail.

• A bachelor's degree in computer science, engineering, or a related field, or equivalent experience.

• All tasks, project files, and solutions must be authored by humans; LLM-generated code, content, or text is not allowed.

• Tasks and solutions should be original and not reproduced from existing published or easily searchable sources.

• Must not include code, assets, or project files from current or previous employers, clients, or studios, or anything subject to an NDA.

• Work must involve fictional studios, projects, and scenarios, with all code and content owned by the contributor.

🏝️ Benefits

• Flexible commitment, typically requiring 15–20 hours per week.

• Can be pursued alongside a full-time job.

• Task-based contract with certain tasks compensated at a flat rate.

• Flexible, project/task-oriented work.