Master's Scholar, LLM, Model Evaluation, Python, Prompt Engineering

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design and conduct experiments to identify innovative approaches for assessing Large Language Models (LLMs) and agents, exploring metrics related to quality, factual accuracy, safety, and compliance with specific criteria.

• Create experimental frameworks for LLM-as-a-Judge, which includes prompt engineering, rubric development, and automated evaluation procedures for scalable analysis of AI-generated outputs, integrating principles from Reinforcement Learning and preference optimization.

• Investigate and implement techniques in preference optimization and reinforcement learning to enhance language models continually, evaluating their effects on quality, resilience, and alignment.

• Execute quantitative assessments of experimental outcomes, document insights, suggest methodological advancements, and play a role in establishing best practices for evaluating and optimizing LLMs.

⛳️ Requirements

• Education: Master’s degree.

• Degree fields: Computer Science, Computer Engineering, Information Systems, Data Science, Statistics, Applied Mathematics, or related disciplines in Computing, Artificial Intelligence, and Machine Learning.

• Proficient in Python.

• Understanding of fundamental Machine Learning and Deep Learning concepts.

• Familiarity with Large Language Models (LLMs) and Generative AI.

• Experience with AI model development libraries such as PyTorch and/or Hugging Face Transformers.

• Capability to read and comprehend scientific literature in English.

• Basic knowledge of experimentation, analysis of results, and model evaluation.

• Experience with LLM evaluation / LLM-as-a-Judge is preferred.

• Experience with RLHF, DPO, or preference optimization is a plus.

• Participation in scientific research or publications in AI or Machine Learning is desirable.

🏝️ Benefits

• Stipend: BRL 9,000.00.

• Availability: 40 hours per week; Duration: 12 months.