Prompt Systems Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Take charge of designing, testing, and continually enhancing the prompts and system instructions that drive Noodle’s AI agents in the areas of enrollment, learning, and student support.

• Convert operational, pedagogical, and business needs into dependable agent behaviors, encompassing personas, tone, scope limitations, fallbacks, and multi-turn conversation logic.

• Develop prompt architectures for multi-step and chained workflows, including agents augmented with program-specific knowledge bases using RAG.

• Work alongside engineers to configure and launch agents via Noodle’s AI orchestration platform.

• Create engaging three-way chat experiences among learners, campus or Noodle support personnel, and AI assistants.

• Establish and uphold evaluation frameworks that assess accuracy, tone, hallucination rates, task completion, and alignment with educational objectives.

• Utilize Langfuse to oversee production prompt performance, detect regressions, and prioritize enhancements.

• Design red-teaming and adversarial testing protocols.

• Implement prompt versioning practices and maintain a repository of tested, reusable prompt components.

• Collaborate with colleagues and university stakeholders to design, develop, and evaluate agents.

• Partner with enrollment and student success teams to refine companion agents tailored for specific programs and university contexts.

• Document prompt engineering guidelines and best practices for internal teams.

• Monitor model capability changes across OpenAI, Anthropic, and other providers, highlighting opportunities and risks.

• Perform additional duties as required.

⛳️ Requirements

• A minimum of 2 years of experience in designing and refining prompts for LLM-powered applications within a production environment.

• Extensive knowledge of few-shot examples, chain-of-thought, system versus user roles, output formatting constraints, tool-use prompting, and RAG integration.

• Excellent written communication skills.

• Experience in constructing and conducting prompt evaluations, including defining metrics, developing test cases, and analyzing results.

• Comfort in discussing context windows with engineers and Bloom’s Taxonomy with faculty members.

• Familiarity with LLM observability tools like Langfuse or Weights & Biases.

• Knowledge of LTI and LMS platforms, such as Moodle, Canvas, or Instructure.

• Experience with voice agents or multimodal prompting.

• Proficiency in reading and writing basic JavaScript or Python.

• Understanding of APIs from OpenAI, Anthropic, or other model providers.

• Ability to excel in uncertain environments and rapidly iterate based on user feedback.

• Capability to work effectively in a dynamic, fast-paced, team-oriented setting.

• Must be legally authorized to live and work in the United States.

• Required to reside in the United States for the entire duration of employment.

🏝️ Benefits

• Work from our stunning NYC office or the convenience of your home office!

• A computer and charger will be provided by us!

• 401(k) plan with company matching, in addition to equity opportunities.

• Comprehensive medical, dental, and vision insurance for you and your family, including mental health benefits such as Headspace and Talkspace.

• Employer-paid life and disability insurance.

• Optional voluntary coverage, which includes supplemental life, accident, critical illness, and hospital insurance.

• Health Savings Account (HSA), Flexible Spending Accounts (FSA), and commuter benefits.

• Pre-tax benefit options that help you save on eligible healthcare, dependent care, and commuting costs.

• Access to an Employee Assistance Program (EAP).

• Paid parental leave.

• Wellness initiatives and resources available.

• Free subscription to the Calm app.

• Employee discount program.

• 4 weeks of paid vacation, plus 10 paid holidays, and paid sick leave.

• Monthly stipend for mobile connectivity.

• Annual education stipend for ongoing learning.

• Opportunities for growth.