Applied AI Engineer – Developer Experience

This is a fully remote position, open to applicants in United States.

📋 Description

• Design and conduct experiments focused on tier routing, MCP coverage, permission configuration, repository context quality, and budget headroom.

• Take charge of the analytical component of the measurement program, which includes work classification, evaluation design, longitudinal within-unit analysis, and staggered-adoption estimates.

• Develop and validate LLM-as-judge and classification pipelines utilizing sampling strategies, hand-labeled ground truth, precision/recall measurement, and revalidation.

• Enhance AI capabilities throughout test authoring and maintenance, environment and data setup, migration and modernization, code review assistance, security remediation, and certification evidence assembly.

• Collaborate directly with constrained teams to pinpoint delivery bottlenecks and target capabilities effectively.

• Create evaluations for internal AI capabilities, which encompass golden sets, regression suites, groundedness and answer-quality scoring, cost telemetry, and latency telemetry.

• Identify effective practitioner behaviors, document and teach practices, and publish practices instead of rankings.

• Partner with the platform team on Claude Code configuration, MCP servers, gateway telemetry, and the model registry.

• Present findings to engineering leadership and finance, detailing successful strategies, challenges, delivery constraints, and claim limitations.

⛳️ Requirements

• 7+ years of experience in software engineering and quantitative analysis.

• Hands-on experience with LLM applications, including prompting, tool and function calling, context management, and evaluation.

• Expertise in experimental design and causal inference, covering randomized and quasi-experimental designs, difference-in-differences, instrumental variables, and hierarchical models.

• Proficient in Python and SQL, with familiarity in pandas, statsmodels, scikit-learn, or R.

• Experience in instrumenting and extracting data from operational systems and APIs.

• Strong sampling design skills that can withstand scrutiny.

• Capable of identity resolution across systems and joining disparate data sources.

• Proficient in exploratory analysis, distributions, cohort analysis, time-series analysis, and reporting coverage and limitations.

• Familiarity with the software delivery lifecycle, including code review, CI/CD, test strategy, and release/change management.

• Experience with Git and GitLab instrumentation, including merge request and pipeline data models, diffs and SHAs, merge, squash, rebase, cherry-pick, APIs, and hooks.

• Knowledge of Jira and Confluence integration, including REST APIs, changelogs, page version history, GitLab–Jira development panel, fields, and labels.

• Careful handling of personnel-adjacent data and aggregate reporting.

• Strong communication skills to interact with executives and engineers.

• Preferred: Experience with MCP servers and clients or comparable connector frameworks.

• Preferred: Familiarity with LangGraph, LangChain, Bedrock Agents, Strands, or equivalent agent frameworks.

• Preferred: Experience with enterprise coding-assistant deployment and telemetry.

• Preferred: Knowledge of server-side Git hooks, GitLab CI, and system/webhook-driven capture on a self-managed instance.

• Preferred: Understanding of Confluence and Jira as MCP-connected systems, permission propagation, scoped credentials, and audit logging.

• Preferred: Experience with Ragas, DeepEval, Bedrock model evaluation, LangFuse, Arize, or OpenTelemetry-based tracing.

• Preferred: Familiarity with Amazon Bedrock and AWS cost and usage data.

• Preferred: Experience with DORA, DX Core 4, SPACE, program analysis, test generation, developer tooling research, dbt, Airflow, Dagster, warehouse/lakehouse modeling, BI and visualization, queueing and flow analysis.

🏝️ Benefits

• Global & Multicultural – Diverse perspectives and global collaboration.

• Startup Energy – A fast-moving and impact-driven environment.

• Ownership Mindset – Engineers take ownership of their creations.

• Collaborative & Friendly – An open, curious, and supportive culture.