Senior Prompt Engineer

This is a fully remote position, open to applicants in Portugal.

📋 Description

• Oversee the comprehensive technical migration process for transitioning templates to LLM autoraters.

• Employ Automatic Prompt Generation (APG) tools to develop baseline prompts for intricate parent-child template clusters.

• Execute and manage Automated Prompt Optimization (APO) tools, assess outputs, and identify instances when the APO encounters deadlocks or plateaus.

• Manually compose, test, and enhance prompts to navigate complex template frameworks, address anti-patterns, and manage edge cases where tools may be insufficient or malfunctioning.

• Supervise shadowbot runs to ensure adequate discrepancies (between human and LLM ratings) are recorded, produced, and monitored.

• Evaluate prompt versions against established gold data to consistently assess autorater quality against the human crowd baseline, computing accuracy metrics such as F1 score, precision, and recall.

• Prepare technical launch-readiness justifications (Launch Certification Documentation) for the final launch.

⛳️ Requirements

• Native proficiency in Portuguese and fluency in English.

• Must reside in Portugal.

• Bachelor’s, Master’s, or Doctorate degree in Computer Science, Data Science, Computational Linguistics, Human–Computer Interaction (HCI), Cognitive Science, or a related analytical discipline.

• Minimum of 4 years of experience as a Prompt Engineer, demonstrating expertise in fine-tuning Large Language Models (LLMs) for precise, structured outputs and complex classification tasks, with knowledge of chain-of-thought and few-shot learning.

• Strong skills in recognizing error patterns, analyzing model performance, and utilizing SQL or other data analytics tools.

• Capacity to rapidly learn and master proprietary tools with limited supervision.

• Exceptional verbal and written communication abilities.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Flexible working hours and the option for remote work.

• Opportunities for professional development and continuous learning.

• Collaborative and inclusive work environment.