AI Engineer

This is a fully remote position, open to applicants in Ukraine.

📋 Description

• Advanced Prompt Engineering: Creating intricate and adaptable prompt templates that incorporate conditional logic, while effectively reusing information and context within prompts to enhance generation quality and reasoning.

• Structured Outputs & Schemas: Developing various response formats (JSON mode, function calling, Zod/JSON schemas) to guarantee that AI outputs are predictable and ready for seamless integration into application logic.

• Prompt Engineering & Evaluations: Establishing robust evaluation pipelines and utilizing Langfuse to gather feedback and assess the quality of responses in real time.

• Tracing & Debugging: Conducting in-depth debugging of complex LLM chains with Langfuse traces to pinpoint bottlenecks and optimize for cost, latency, and context window usage.

• AI A/B Testing: Executing systematic experiments across various models via OpenRouter (e.g., comparing Claude 3.5 Sonnet vs. GPT-4o) and analyzing outcomes based on quantitative metrics.

• Data-Driven Decisions: Making deployment choices for new prompts or models exclusively based on quantitative benchmarks and trace data, rather than intuition.

• Output Scoring & Analysis: Creating scoring systems to evaluate the “Problem → Solution” chain and identify the root causes of hallucinations or logical errors using Langfuse analytics.

• Model Performance & Fine-Tuning: Regularly reassessing model performance as new architectures surface and executing fine-tuning when necessary to meet specific domain requirements.

⛳️ Requirements

• Node.js & Next.js: Extensive expertise in the stack to develop reliable services and manage complex LLM-generated data.

• Dynamic Prompting Skills: Demonstrated experience in creating prompts where content is highly reliant on input variables and context injection.

• OpenRouter Experience: Familiarity with unified APIs, managing rate limits, and selecting the most cost-effective models for particular tasks.

• Langfuse (or similar): Knowledge of LLM observability principles — setting up tracing, creating test datasets, and integrating scoring systems.

• Evaluation Methodology: Experience with frameworks such as RAGAS or developing custom “LLM-as-a-judge” systems.

• Analytical Mindset: Capability to convert raw generation logs into actionable business metrics and technical insights.

• Iterative Mindset: Commitment to continuous product enhancement through ongoing feedback loops.

🏝️ Benefits

• Remote Work Environment: Enjoy the flexibility to work from any location at any time, fostering a healthy work-life balance.

• Unlimited PTO: Take advantage of unlimited paid time off to recharge and prioritize your well-being, without the hassle of counting days.

• Paid National Holidays: Relax and celebrate on national holidays with paid time off to rejuvenate.

• Company-provided MacBook: Experience enhanced productivity with high-quality Apple MacBooks provided to all employees who require them.

• Flexible Independent Contractor Agreement: Benefit from flexibility, autonomy, and entrepreneurial opportunities, along with tax advantages, networking possibilities, reduced employment obligations, and the freedom to work from anywhere.