AI Benchmark Engineer, Native Language Specialist – Arabic

This is a fully remote position, open to applicants in Saudi Arabia.

📋 Description

• Design, construct, and validate Terminal-Bench tasks for multilingual software challenges.

• Assess coding agents.

• Create realistic task environments utilizing Arabic datasets and files that remain in the target language.

• Identify AI failure points in Arabic.

• Develop strong reference implementations.

• Write dependable, deterministic verifier scripts.

• Analyze execution logs and adjust task difficulty from Easy to Very Hard.

• Execute standard Terminal-Bench configurations against Haiku, Sonnet, and Opus model tiers.

• Engage in a four-layer human quality control process encompassing creation, human review, calibration review, and audit.

• Collaborate with automated LLM-based checks to ensure fairness, grammatical accuracy, and benchmark integrity.

• Contribute to LILT’s multilingual AI and human-verified services for enterprises, governments, and AI developers.

⛳️ Requirements

• Over 5 years of industry experience in software engineering.

• Demonstrated success at leading technology companies and/or graduation from top-tier engineering universities.

• Native or near-native fluency in Arabic, with a thorough understanding of grammar, register, and phrasing rules.

• High proficiency in English.

• Strong skills in Python, standard shell scripting, and data processing.

• Extensive experience with Terminal/CLI-based development workflows.

• Familiarity with coding agents.

• In-depth technical knowledge of multilingual text processing challenges.

• Experience in encoding/decoding robustness and Unicode normalization.

• Understanding of locale-dependent conventions, including collation, casing, and non-Gregorian dates.

• Knowledge of text I/O, toolchain interoperability, and safe string operations.

• For Arabic, familiarity with bidirectional/RTL handling, font fallbacks, and rendering/typography in UI or artifacts.

• Reliable availability and commitment.

• Most tasks require a minimum of 2 hours per day or 15–20 hours per week.

• As a 1099 contractor, responsible for one's own tax obligations.

• Candidates must submit an updated CV in English.

• Must complete a GenAI assessment.

🏝️ Benefits

• As an independent contractor, you have the flexibility to work when you want, as much or as little as you desire.

• No fixed hours, check-ins, or micromanaging.

• Competitive rates.

• Timely payments.

• Access to a variety of diverse, innovative projects.

• Opportunity for portfolio and skills development across various industries and domains.

• Supplemental income opportunity with flexible scheduling.

• No benefits such as health insurance, paid time off, or retirement contributions are offered.

• Hours are not guaranteed.