AI Engineer – Governance Intelligence, Third Party

This is a fully remote position, open to applicants in India.

📋 Description

• Develop automated vendor intelligence pipelines that consistently gather and analyze AI system cards, model benchmarks (MMLU, HumanEval, etc.), security disclosures, and publicly available vendor documentation.

• Create synthesis systems that align diverse vendor information with our risk taxonomy—converting technical capabilities into governance-relevant risk indicators.

• Execute quality assessments for generated risk profiles, ensuring precision and identifying when vendor information is inconsistent or outdated.

• Establish adaptive interpretation systems that modify risk evaluations based on organizational context and risk tolerance.

• Investigate and prototype innovative methods for synthesizing information from multiple sources and comparing across vendors.

⛳️ Requirements

• Proven experience in developing LLM-powered applications in production, transitioning from prototype to full-scale implementation.

• In-depth knowledge of document understanding, information extraction, and structured content creation.

• Experience in building data pipelines that manage diverse and evolving source formats—comfortable working with complex real-world data.

• Skilled with contemporary AI development frameworks (LangChain, LlamaIndex) and RAG architectures.

• Strong grasp of prompt engineering, fine-tuning, and the appropriate application of each technique.

• Background in knowledge representation—having created systems that model intricate domains.

• Demonstrated capacity to deliver quickly - able to construct an MVP in days and refine it based on user input.

• Proficient in AI-assisted development tools and methodologies - adept at leveraging AI for enhanced productivity.

• Extensive familiarity with the AI evaluation landscape: model benchmarks (SuperGLUE, BIG-Bench, HELM), safety evaluations (METR, Apollo), and AI system card formats.

• Experience in vendor/third-party risk assessment across various fields (InfoSec, compliance, procurement).

• Developed multi-source aggregation systems that resolve conflicting information and synthesize insights across documents.

• Created systems capable of handling document understanding at scale (10K+ documents).

• Knowledgeable in ontologies, knowledge graphs, or semantic modeling.

• Proven history of automating intricate manual processes through AI.

🏝️ Benefits

• Competitive Salary and Equity

• Health: We provide health, dental, and vision insurance. Additionally, we offer an ergonomic benefit to cover the costs of equipment that helps staff maintain their health while working, both in the office and remotely.

• Coworking: We will reimburse the costs of co-working spaces such as WeWork and in-person meetups.

• Unlimited PTO: Credo AI offers unlimited paid time off to support our employees.

• Generous Parental Leave: We provide up to 12 weeks of paid parental leave.

• 401(k) plan available for employees (US only).