Senior Prompt Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee the complete technical migration process for transitioning templates to LLM autoraters.

• Employ internal tools to apply prompt engineering strategies that enhance model performance.

• Use Automatic Prompt Generation (APG) tools to develop baseline prompts for intricate parent-child template clusters.

• Execute and monitor the Automated Prompt Optimization (APO) tool, assessing outputs and identifying instances when the APO encounters deadlocks or plateaus.

• Manually create, test, and refine prompts to maneuver through complex template architectures, resolve anti-patterns, and address edge cases.

• Supervise shadowbot runs to ensure that adequate disagreements between human and LLM ratings are recorded, generated, and monitored.

• Test prompt versions against established gold data to consistently evaluate autorater quality, calculating accuracy metrics such as F1 scores, precision, and recall.

• Prepare technical launch readiness justifications (Launch Certification Documentation) for the final phase.

⛳️ Requirements

• Native fluency in English.

• Must reside in the United States.

• Bachelor’s, Master’s, or Doctorate degree in Computer Science, Data Science, Computational Linguistics, Human-Computer Interaction (HCI), Cognitive Science, or a related analytical field.

• A minimum of 4 years' experience as a Prompt Engineer, with proven expertise in fine-tuning Large Language Models (LLMs) for precise, structured outputs, complex classification tasks, and familiarity with chain-of-thought and few-shot learning.

• Strong skills in identifying error patterns, analyzing model performance, and utilizing SQL or other data analytics tools.

• Capability to swiftly learn and master proprietary tools with minimal oversight.

• Exceptional verbal and written communication abilities.

🏝️ Benefits

• Competitive salary and performance-based incentives.

• Flexible work hours and remote work opportunities.

• Comprehensive health, dental, and vision insurance.

• Professional development and continuing education support.

• Collaborative and innovative work environment.