Principal Engineer – Data Retrieval

This is a fully remote position, open to applicants in Switzerland.

📋 Description

• Design and scale the infrastructure for data acquisition that powers NVIDIA's foundational models, starting with code and STEM data for Nemotron.

• Create and manage a cohesive platform for source discovery, URL indexing, crawl frontiers, predictive ranking, scheduling, distributed fetching, and data ingestion.

• Develop scalable frontier storage and queuing systems that support high-throughput writes, efficient lookups, deduplication, and scheduling based on quality, diversity, and freshness.

• Establish a continuous discovery loop that identifies links and introduces new candidates into the crawl frontier.

• Build a shared crawl execution layer capable of reliable and respectful large-scale fetching, including from dynamic websites.

• Create extensible pipelines for content from the open web, licensed sources, and partnerships, covering text, PDFs, code repositories, and multimodal assets.

• Define stable interfaces for source connectors.

• Collaborate with parsing and curation teams to maintain code, mathematical content, and structured data while enhancing the yield of accepted data.

• Implement provenance tracking and policy enforcement in conjunction with legal, privacy, security, and data governance teams.

• Convert publisher preferences, opt-outs, licensing terms, and usage restrictions into auditable controls downstream.

• Take ownership of production readiness, including capacity planning, observability, fault tolerance, recovery, and cost efficiency.

• Instrument the entire pipeline and establish service objectives for throughput, freshness, reliability, and training cutoffs.

• Collaborate with research and product teams to assess internal crawls and external datasets via comparable pilot projects.

• Provide reproducible data releases that link source coverage and processing quality to controlled training experiments.

• Manage cross-team dependencies and trade-offs between immediate data delivery and long-term platform development.

• Lead architecture reviews, critical implementation tasks, engineer mentoring, and team development.

⛳️ Requirements

• A BS, MS, or PhD in Computer Science, Computer Engineering, or a related technical field, or equivalent experience.

• Over 15 years of relevant experience in software engineering, including the design and operation of large-scale distributed systems, setting technical direction, and delivering complex infrastructure across multiple teams.

• Extensive experience with web crawling or closely related fields such as discovery, indexing, or ingestion infrastructure, involving distributed scheduling, stateful services, storage, and high-throughput data processing.

• Strong practical engineering and system design capabilities.

• Ability to transition seamlessly between architecture, implementation, performance analysis, and production debugging.

• Experience in building batch or streaming pipelines and understanding the dynamics of consistency, retries, duplicate processing, backpressure, and recovery.

• Proven track record of enhancing throughput, latency, resource efficiency, or operational reliability.

• Experience in delivering data platforms for research, machine learning, search, or other demanding technical users.

• Capability to translate evolving requirements into clear interfaces and measurable engineering outcomes.

• Strong technical communication and leadership skills, influencing across engineering, product, research, and governance teams.

• Experience in architecting a web-wide crawl platform, scaling crawl frontier storage, or consolidating multiple acquisition systems into a unified infrastructure.

• Direct experience in providing foundation model pretraining data, including targeted discovery, quality-aware crawl ranking, and acquisition of code or technical content.

• Experience in implementing publisher consent, training-use opt-outs, dataset licensing controls, or provenance infrastructure at scale.

• Background in browser-based or agentic data collection, programmable extraction, and sandboxed environments for tool-use data or post-training evaluation.

• Experience in onboarding licensed or multimodal datasets, building reusable partner integrations, or delivering streaming feeds with low latency for retrieval and grounding.

🏝️ Benefits

• Comprehensive health, dental, and vision insurance.

• Competitive salary and performance-based bonuses.

• Opportunities for professional development and continuous learning.

• Flexible work hours and remote work options.

• A collaborative and innovative work environment.