This is a fully remote position, open to applicants in Brazil, +4 more countries.

📋 Description

• Design, construct, test, and refine prompts and prompt sequences for production LLM workflows utilized in client delivery and internal automation.

• Develop evaluation harnesses, implement guardrails, and create regression tests to ensure the output quality is measurable and remains consistent over time.

• Integrate LLM components with business systems and data, including APIs, retrieval/RAG pipelines, and structured outputs.

• Collaborate with AI Specialists to transform solution designs into production-ready builds, while documenting patterns for future reuse.

• Optimize for accuracy, cost-efficiency, latency, and safety using real evaluation data, ensuring decisions are data-driven rather than based on intuition.

• Expand Caravel's collection of reusable AI accelerators.

⛳️ Requirements

• A minimum of 2 years of experience in building software or AI/LLM components, including practical experience in crafting and refining prompts for actual applications.

• Solid understanding of LLM fundamentals, such as context windows, few-shot prompting, function/tool calling, structured outputs, and basic RAG principles.

• Proficient in Python (or a similar language) and comfortable working with APIs, JSON, and version control systems (Git).

• Possess a test-and-measure mindset, where you assess output quality rather than make subjective judgments.

• Excellent written English skills and strong documentation practices.

• Ability to work collaboratively across time zones with both onshore and nearshore teams.

• Experience with evaluation frameworks, such as promptfoo, Ragas, or custom evaluation harnesses.

• Familiarity with embeddings, vector databases, and retrieval pipelines.

• Exposure to ERP/CRM systems (e.g., NetSuite, Salesforce) or business process automation.

• Background in agentic frameworks and orchestration.