Doctoral Research Fellow – Agents, Monitoring, AI Reliability, Model Evaluation

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Chart the execution paths of agents while pinpointing critical areas for log collection and monitoring.

• Create Python pipelines to gather, store, and process execution logs efficiently.

• Establish and implement metrics to assess the agents’ performance, quality, and consistency.

• Develop automated evaluation routines (evals) tailored to various scenarios and use cases.

• Generate analyses and reports on agent performance, highlighting failures and areas for enhancement.

• Assist in the development of dashboards or monitoring tools.

• Document the architecture, metrics, and outcomes achieved throughout the project.

⛳️ Requirements

• Doctorate degree (PhD).

• Relevant fields of study include Data Science, Statistics, Computer Engineering, Computer Science, or similar disciplines.

• Proficiency in Python programming.

• Capability to organize and replicate an LLM-based software project beyond basic usage.

• Understanding of multi-agent systems and their orchestration.

• Experience in extracting, validating, and comparing information within documents.

• Ability to define and interpret evaluation metrics.

• Skill in instrumenting the system with logs, traces, and metrics.

• Competence in technically explaining failures.

• Ability to read research articles and papers, translating concepts into experiments, and articulating results clearly.

• Experience in applied research with AI/ML/LLMs is desirable.

• Proficient in constructing and assessing end-to-end observable systems is desirable.

• Capable of translating operational workflows into agent architecture is desirable.

• Strong scientific writing and verbal communication skills are desirable.

• Exhibits autonomy and experimental maturity is desirable.

🏝️ Benefits

• Stipend: BRL 11,000.00.

• Availability: 40 hours per week.

• Duration: 12 months.