AI Benchmark Engineer, Native Language Specialist – German

This is a fully remote position, open to applicants in Austria.

📋 Description

• Create, develop, and assess Terminal-Bench tasks aimed at addressing multilingual software challenges.

• Assess coding agents for performance and effectiveness.

• Construct realistic task environments utilizing datasets and files in the German language.

• Identify AI failure points through prompting and translation in German.

• Build strong reference implementations for reliable testing.

• Develop dependable, deterministic verifier scripts.

• Examine execution logs for insights and improvements.

• Adjust task difficulty levels from Easy to Very Hard using standard Terminal-Bench configurations.

• Execute tasks across Haiku, Sonnet, and Opus model tiers.

• Engage in a four-layer human quality-control process that includes creation, human review, calibration review, and audit.

• Collaborate with automated LLM-based checks to guarantee fairness, grammatical precision, and benchmark integrity.

• Contribute to LILT’s evaluation suite for multilingual AI and human-verified language technology services.

⛳️ Requirements

• Over 5 years of professional experience in software engineering.

• A proven record at leading technology firms and/or graduation from prestigious engineering universities.

• Native or near-native proficiency in German, with a thorough understanding of grammar, register, and phrasing conventions.

• High proficiency in English.

• Strong expertise in Python programming.

• Proficient in standard shell scripting.

• Strong skills in data processing.

• Extensive experience with Terminal/CLI-based development workflows.

• Familiarity with coding agents.

• In-depth technical knowledge of challenges in multilingual text processing.

• Experience with encoding/decoding robustness and Unicode normalization.

• Understanding of locale-dependent conventions, including collation, casing, and non-Gregorian dates.

• Experience with text I/O, toolchain interoperability, and secure string operations.

• Knowledge of bidirectional/RTL handling, font fallbacks, and rendering/typography in UI or artifacts for specific languages.

• Dependable availability and dedication to timely delivery.

• An updated CV in English.

• Successful completion of a GenAI assessment.

• Geographic restrictions may apply; contractors cannot be engaged in regions subject to international embargo or sanctions.

• Contractors are responsible for their own tax obligations.

🏝️ Benefits

• Flexible scheduling with no fixed hours, check-ins, or micromanagement.

• Competitive compensation rates.

• Prompt payment for services rendered.

• Access to a variety of innovative projects.

• Opportunities for professional portfolio enhancement and skills development.

• A global network of linguists, subject matter experts, and language professionals.

• Additional freelance work opportunities available.

• No benefits such as health insurance, paid time off, or retirement contributions are offered.

• Work from any location at any time, subject to geographic restrictions.

• Most tasks require a minimum availability of 2 hours per day or 10 hours per week.