Model Behavior Engineer – Reasoning and Decision Making

This is a fully remote position, open to applicants in California.

📋 Description

• Comprehend the decisions that users and domain experts must make in scientific research and development, as well as in commercial strategy.

• Identify objectives, constraints, available evidence, and critical unknown factors.

• Define and assess effective reasoning through example inspection, expert judgment, and quantitative analysis.

• Develop evaluations for poorly defined problems and ensure that scores indicate significant improvements.

• Explore model and agent failures, transforming observations into testable hypotheses.

• Experiment with approaches like problem decomposition, context, tools, information-seeking, and self-checking.

• Design agent environments and representations for problems, evidence, assumptions, and unresolved queries.

• Create reliable software abstractions, data structures, and component boundaries for reasoning workflows and evaluations.

• Achieve a balance among simplicity, flexibility, reliability, latency, and cost.

• Write maintainable code and engage in design and code reviews.

• Test, monitor, diagnose, and integrate systems into the product.

• Collaborate with engineers, product teams, designers, and domain experts.

• Manage problems from investigation and experimental design to implementation and product use.

• Enhance systems built around existing frontier models, utilizing training or fine-tuning where beneficial.

⛳️ Requirements

• Engaged in rigorous thinking regarding ambiguous issues, including identifying hidden assumptions and evaluating conflicting evidence.

• Possess curiosity across various domains and a capacity to learn unfamiliar subjects.

• Communicate clearly with individuals from diverse areas of expertise.

• Capable of devising valuable experiments in the absence of established benchmarks.

• Skilled at inspecting failures in detail and distinguishing evidence from noise.

• Able to independently write and debug code.

• Experience working with data and model APIs.

• Comfortable using Python or a similar programming language.

• Practically familiar with language models, including their capabilities and limitations.

• Demonstrate strong engineering and architectural judgment.

• Experience in building understandable, usable, and extensible software.

• Effectively utilize coding assistants while validating the resulting design and code.

• Provide evidence through substantial projects, thoughtful experiments, and clear technical explanations.

• Willingness to travel for quarterly retreats and in-person coworking events.

• This role is tailored for candidates contributing at the senior L4 level or above.

🏝️ Benefits

• Flexible work environment with options for office or remote work.

• Quarterly team retreats and coworking events.

• Comprehensive health, dental, vision, and life insurance fully covered for the employee.

• Generous family insurance coverage.

• FSA/HSA options available.

• Flexible vacation policy with a minimum recommendation of 20 days per year.

• Company holidays included.

• Monthly wellbeing stipend of $200.

• 401K plan with a 6% employer match.

• Provision of a new Mac and a $1,000 workstation/home office budget in the first year.

• Annual $500 workstation/home office budget thereafter.

• Quarterly budget of $1,000 for AI experimentation and learning.

• Team administrative assistant available for personal and work-related tasks.

• Above-market equity offered with employee-friendly equity terms.