Member of Technical Staff, Coding Research

This is a fully remote position, open to applicants in New York.

📋 Description

• Design and manage evaluation frameworks for sophisticated coding agents.

• Create benchmark specifications, scoring methods, rubrics, and quality standards.

• Implement stringent methods for assessing coding-model performance across various software engineering tasks.

• Establish objective criteria for correctness, reasoning quality, robustness, and task completion.

• Uphold methodological rigor, reproducibility, and consistency throughout evaluation workflows.

• Generate high-quality datasets, exemplary cases, and structured evaluation protocols.

• Craft technical tasks for the reliable evaluation of cutting-edge coding systems.

• Develop data and evaluation workflows that support model development and iterative enhancements.

• Identify gaps in benchmark or dataset coverage and create new evaluation categories.

• Analyze coding-agent behavior to uncover systematic weaknesses, failure modes, and performance constraints.

• Investigate issues such as incorrect reasoning, implementation errors, tool-use failures, and incomplete task execution.

• Convert findings into actionable recommendations for model training and evaluation.

• Design experiments to test hypotheses regarding coding-model capabilities.

• Construct tools and infrastructure for large-scale experimentation, data generation, review processes, and evaluation pipelines.

• Automate technical procedures to enhance evaluation efficiency and research speed.

• Collaborate with researchers, engineers, and applied AI teams.

• Contribute to technical reports, benchmark studies, research documentation, and external-facing research initiatives.

• Effectively communicate complex technical findings to both specialized and general technical audiences.

⛳️ Requirements

• Strong software engineering background with proficiency in Python, C++, or similar programming languages.

• At least 3 years of experience in software engineering, machine learning, AI research, evaluation, or a related technical field.

• Experience in designing, reviewing, or validating technical assessments, benchmarks, coding tasks, or evaluation methodologies.

• Familiarity with large language models, coding agents, reinforcement learning, model evaluation, or related AI systems.

• Demonstrated ability to build tools, automate workflows, and enhance technical processes through systematic experimentation.

• Strong analytical abilities and capacity to investigate complex model behavior and technical failure modes.

• Excellent written and verbal communication skills.

• Capability to function effectively in fast-paced research environments characterized by significant ambiguity and shifting priorities.

• Experience with advanced AI systems, coding agents, or model-evaluation research is a plus.

• Expertise in designing benchmarks or datasets for large-scale machine learning systems is highly valued.

• Knowledge of agentic workflows, tool use, reinforcement learning, or post-training methodologies is beneficial.

• Publications, open-source contributions, or demonstrated technical leadership in AI, machine learning, or software engineering are advantageous.

• Work must be conducted without utilizing confidential or proprietary information from any employer, client, institution, or other third party.

🏝️ Benefits

• Fully remote work.

• Full-time engagement.

• Opportunity to contribute to pioneering AI research and development.

• Collaboration with researchers, engineers, and applied AI teams.

• Chance to contribute to technical reports, benchmark studies, research documentation, and external-facing research initiatives.