Applied AI Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Take ownership of the quality, safety, and performance of AI-driven consumer features.

• Manage and enhance system prompts, tool descriptions, and evaluation prompts for Personalized Credit Insights and the AI chat assistant.

• Develop and extend the LLM evaluation framework.

• Transform production failures into test cases and execute regression suites prior to prompt or model modifications.

• Communicate evaluation findings to business stakeholders.

• Assess LLM-based evaluator reliability against human-labeled data using metrics such as precision, recall, and agreement.

• Design, adjust, and oversee guardrails, including classifiers, PII redaction, prompt-injection defenses, and compliance rules for financial advice.

• Spearhead model upgrades by analyzing quality, cost, and latency, deploying safely through A/B testing.

• Lower costs and latency via prompt caching, model routing, and context management.

• Create transcript viewers, labeling workflows, dashboards, and additional tools for Conversation Analysts and QA engineers.

• Collaborate with Analytics, Legal, and Compliance on fairness and bias evaluations.

• Utilize AI coding assistants and agentic workflows to expedite delivery while reviewing and enhancing generated outputs.

⛳️ Requirements

• Bachelor's or Master's degree in Computer Science, Engineering, or a related discipline, with proven software engineering experience.

• Proficient in Python programming.

• Ability to read and contribute to Java (Spring Boot) microservices.

• Demonstrated experience in building and operating LLM-powered applications in a production environment.

• Familiarity with prompt design, tool calling, structured outputs, and streaming techniques.

• Experience in constructing evaluations for LLM systems, including test sets, regression suites, LLM-as-judge approaches, and assessing evaluator quality against human labels.

• Working knowledge of Amazon Bedrock or a similar managed LLM platform.

• Familiarity with AWS services such as S3, Lambda, and DynamoDB.

• Proficient in SQL and comfortable analyzing production data within a data warehouse like Amazon Redshift.

• Experience in production ownership encompassing code review, CI/CD, release management, and on-call support.

• Experience working in Agile/Scrum teams.

• Excellent written and verbal communication skills.

• Experience with guardrail tooling, LLM security testing, RAG and retrieval evaluation, fairness and bias evaluation, model migrations, Datadog, Tableau, or experience in regulated domains is a plus.

🏝️ Benefits

• Highly competitive compensation, benefits, and rewards programs.

• Inclusive company culture.

• Opportunities for professional development and valuable learning experiences.

• Commitment to work/life balance.

• Employee resource groups.

• Social events.

• Equal employment and advancement opportunities.