AI Engineer

📋 Description

• Advanced Prompt Engineering: Crafting intricate and dynamic prompt templates that incorporate conditional logic, ensuring efficient reuse of information and context to enhance generation quality and reasoning.

• Structured Outputs & Schemas: Developing various response formats (such as JSON mode, function calling, Zod/JSON schemas) to guarantee that AI outputs are consistent and ready for smooth integration into application logic.

• Prompt Engineering & Evaluations: Creating robust evaluation pipelines and utilizing Langfuse to gather feedback and assess the quality of responses in real-time.

• Tracing & Debugging: Conducting in-depth debugging of complex LLM chains with Langfuse traces to pinpoint bottlenecks and optimize for cost, latency, and context window usage.

• AI A/B Testing: Executing systematic experiments across various models via OpenRouter (e.g., comparing Claude 3.5 Sonnet with GPT-4o) and evaluating results based on quantitative metrics.

• Data-Driven Decisions: Making deployment choices for new prompts or models based strictly on quantitative benchmarks and trace data rather than mere intuition.

• Output Scoring & Analysis: Establishing scoring systems to analyze the “Problem → Solution” chain and identify the underlying causes of hallucinations or logical errors using Langfuse analytics.

• Model Performance & Fine-Tuning: Continuously re-evaluating model performance as new architectures emerge and performing fine-tuning when necessary to meet specific domain requirements.

⛳️ Requirements

• Node.js & Next.js: Extensive expertise in the stack to develop reliable services and manage complex LLM-generated data.

• Dynamic Prompting Skills: Demonstrated experience in creating prompts where content is significantly influenced by input variables and context injection.

• OpenRouter Experience: Familiarity with working on unified APIs, managing rate limits, and choosing the most cost-effective models for specific tasks.

• Langfuse (or similar): Knowledge of LLM observability principles, including setting up tracing, generating test datasets, and integrating scoring systems.

• Evaluation Methodology: Experience with frameworks such as RAGAS or developing custom “LLM-as-a-judge” systems.

• Analytical Mindset: Capability to convert raw generation logs into actionable business metrics and technical insights.

• Iterative Mindset: Commitment to continuous product enhancement through ongoing feedback loops.

• Fluency in Russian and/or Ukrainian.

🏝️ Benefits

• Remote Work Environment: Enjoy the flexibility to work from anywhere, at any time, fostering a healthy work-life balance.

• Unlimited PTO: Take advantage of unlimited paid time off to recharge and prioritize your well-being, free from counting days.

• Paid National Holidays: Relax and celebrate national holidays with paid time off to unwind and rejuvenate.

• Company-provided MacBook: Experience enhanced productivity with high-quality Apple MacBooks provided to all employees who require them.

• Flexible Independent Contractor Agreement: Benefit from flexibility, autonomy, and entrepreneurial opportunities, including tax advantages, networking opportunities, reduced employment obligations, and the freedom to work from anywhere.