Principal Engineer – Data Retrieval

This is a fully remote position, open to applicants in Colorado, +3 more states.

📋 Description

• Design and expand data acquisition infrastructure that supports NVIDIA’s foundational models.

• Create systems that identify valuable sources, crawl the global web extensively, and provide dependable, high-quality data for training pipelines.

• Oversee the architecture of a cohesive platform that encompasses source discovery, URL indexing, crawl boundaries, predictive ranking, scheduling, distributed fetching, and data ingestion.

• Develop scalable storage and queue systems for frontiers with high-throughput writing, efficient data retrieval, deduplication, and scheduling based on quality, diversity, and freshness.

• Establish a continuous discovery mechanism that extracts links and inputs new candidates into the crawl frontier.

• Create a shared crawl execution layer that reliably and courteously fetches data at scale, including from dynamic websites.

• Develop flexible pipelines for open-web, licensed, and partner content, encompassing text, PDFs, code repositories, and multimodal assets.

• Define consistent interfaces for source connectors.

• Collaborate with parsing and curation teams to maintain valuable code, mathematical content, and structured information, enhancing accepted data yields.

• Enforce provenance and policy compliance in collaboration with legal, privacy, security, and data governance partners.

• Translate publisher preferences, opt-outs, licensing terms, and usage restrictions into auditable downstream controls.

• Ensure production readiness, including capacity planning, monitoring, fault tolerance, recovery, and cost efficiency.

• Instrument the pipeline comprehensively and establish service objectives for throughput, freshness, reliability, and training cutoffs.

• Assess internal crawls and external datasets through comparable pilot programs with research and product teams.

• Deliver reproducible data releases linked to source coverage and processing quality within controlled training experiments.

• Coordinate dependencies across teams while balancing immediate data delivery with long-term platform development.

• Lead architecture reviews and critical implementation efforts.

• Mentor engineers and contribute to team development.

⛳️ Requirements

• Bachelor's, Master's, or PhD in Computer Science, Computer Engineering, or a related technical discipline, or equivalent practical experience.

• Over 15 years of relevant experience in software engineering.

• Proven experience in designing and managing large-scale distributed systems.

• Experience in setting technical direction and delivering complex infrastructure across multiple teams.

• Extensive knowledge of web crawling or closely related discovery, indexing, or ingestion infrastructure.

• Familiarity with distributed scheduling, stateful services, storage solutions, and high-throughput data processing.

• Strong hands-on engineering and system design capabilities.

• Ability to transition between architecture, implementation, performance analysis, and production debugging.

• Experience in constructing batch or streaming pipelines.

• Understanding of consistency, retries, duplicate processing, backpressure, and recovery mechanisms.

• Proven track record of enhancing throughput, latency, resource efficiency, or operational reliability.

• Experience in delivering data platforms tailored for research, machine learning, search, or other demanding technical users.

• Capacity to translate evolving requirements into clear interfaces and measurable engineering outcomes.

• Strong technical communication and leadership skills through influence across engineering, product, research, and governance teams.

• Direct experience in architecting a web-wide crawl platform, scaling crawl frontier storage, or integrating multiple acquisition systems into shared infrastructure.

• Hands-on experience supplying foundation-model pretraining data, including targeted discovery, quality-aware crawl ranking, and acquisition of code or technical content.

• Experience implementing publisher consent, training-use opt-outs, dataset licensing controls, or provenance infrastructure at scale.

• Background in browser-based or agent-based data collection, programmable extraction, and sandboxed environments for tool-use data or post-training evaluation.

• Experience in onboarding licensed or multimodal datasets, building reusable partner integrations, or delivering low-latency streaming feeds for retrieval and grounding.

🏝️ Benefits

• Equity

• Comprehensive benefits package