AI Benchmark Engineer – Native Language Specialist, French

This is a fully remote position, open to applicants in France.

📋 Description

• Create, develop, and validate Terminal-Bench tasks for multilingual software challenges.

• Assess coding agents.

• Construct realistic task environments utilizing datasets and files in French, ensuring assets are in the target language.

• Identify AI failure points through prompting and translation work in French.

• Assist in establishing robust reference implementations.

• Develop reliable, deterministic verifier scripts.

• Analyze execution logs and adjust task difficulty from Easy to Very Hard.

• Execute standard Terminal-Bench configurations against Haiku, Sonnet, and Opus model tiers.

• Engage in a four-layer human quality control process: creation, human review, calibration review, and audit.

• Collaborate with automated LLM-based checks to ensure fairness, grammatical accuracy, and benchmark integrity.

⛳️ Requirements

• Over 5 years of industry experience in software engineering.

• Demonstrated success at leading technology firms and/or graduation from prestigious engineering universities.

• Native or near-native fluency in French with a comprehensive understanding of grammar, register, and phrasing rules.

• High proficiency in English.

• Strong skills in Python, standard shell scripting, and data processing.

• Extensive experience with Terminal/CLI-based development workflows.

• Working knowledge of coding agents.

• Deep technical insight into multilingual text-processing challenges.

• Experience with encoding/decoding resilience and Unicode normalization.

• Familiarity with locale-dependent conventions, including collation, casing, and non-Gregorian dates.

• Understanding of text I/O, toolchain interoperability, and safe string operations.

• For applicable languages, knowledge of bidirectional/RTL handling, font fallbacks, and rendering/typography.

• Updated CV in English.

• Successful completion of a GenAI assessment.

• Reliable availability and commitment; most tasks require a minimum of 2 hours per day or 10 hours per week.

• Contractors must adhere to relevant geographic restrictions and are accountable for their own tax obligations.

🏝️ Benefits

• Flexible schedule with the option to work as much or as little as preferred.

• Competitive compensation rates.

• Timely payments.

• Access to a variety of innovative projects.

• Opportunities for portfolio and skills development across different industries and domains.

• Global network of linguists, subject matter experts, and language professionals.

• No fixed hours, mandatory check-ins, or micromanagement.