Applied LLM Engineer

This is a fully remote position, open to applicants in Poland.

📋 Description

• Take ownership of YOlanda’s system prompt, encompassing persona, tone, safety protocols, refusals, and handoff procedures.

• Create and update prompts for V3 user actions such as plan modifications, top-ups, balance inquiries, promotional offers, troubleshooting, and account assistance.

• Craft conversations aimed at achieving a product or service within five interactions.

• Establish the limits of YOlanda’s knowledge and facilitate smooth transitions to YOnC.

• Manage contextual information about users, accounts, and plan catalogs; decompose tasks into sequences and tool invocations.

• Develop and sustain bilingual evaluation sets featuring test cases, success criteria, and trackable metrics.

• Assess the YOlanda/YOnC boundary, verify plan and account details, and conduct red-team exercises on top-up and rewards processes.

• Monitor calls and track prompt versions, models, tokens, cache hits, latency, and expenses.

• Define latency and availability Service Level Objectives (SLOs) and establish alert mechanisms.

• Manage gateway timeouts, retries, and fallback strategies for providers/models.

• Design cacheable prompts and implement semantic, retrieval, and tool-result caching with appropriate scoping and invalidation.

• Optimize token budgets, context truncation, output length, and batch APIs.

• Benchmark and route frontier and open models based on accuracy, latency, and cost considerations.

• Collaborate with engineers operating open models on YOnC compute resources.

• Maintain version control for prompts, models, and configurations in git, ensuring they are reviewed, tested, and ready for rollback.

• Create Continuous Integration (CI) release gates, canary deployments, staged rollouts, and fail-safes.

• Audit monetary and account-related activities, ensuring personal data is redacted before logging or sharing with third-party services.

• Implement runtime safeguards including defenses against prompt injection, tool permissions, spending limits, and human approval checkpoints.

• Set budget limits for features and models with alert systems, translating AI/money/operator-account rules into coding controls.

• Submit pull requests for the LangChain and LangGraph alongside the two responsible engineers.

• Engage daily with models and production systems without direct personnel management.

⛳️ Requirements

• At least two years of experience working with large language models in a production environment with actual traffic.

• Proficiency in Python, including scripting, API client development, data manipulation, and testing.

• Familiarity with Git, JSON, regex, and Markdown.

• Experience deploying production systems utilizing LangChain and LangGraph, with the ability to safely read and modify unfamiliar graphs.

• Proven experience in writing and deploying system prompts and constructing evaluation frameworks.

• Capability to change one variable at a time and document the outcomes.

• Production experience involving tracing, SLOs, fallbacks, and properly invalidated caching.

• Demonstrated improvements in cost and latency with measurable results.

• Practical governance experience in version control, release gates, audit logging, and handling personal data.

• Hands-on experience with retrieval methods including chunking, embeddings, reranking, and grounding responses in sources; understanding when retrieval may not be suitable.

• Exceptional command of written English, encompassing syntax, tone, and subtleties.

• A commitment to users, particularly those on prepaid plans who are mindful of every dollar spent.

🏝️ Benefits

• Flexible working hours.

• Competitive salary.

• Genuine opportunities for personal and professional development.