Principal Engineer, Data Retrieval

About the role

The team builds and runs a single platform for source discovery, crawl scheduling, fetching and ingestion, and needs it pushed to pretraining scale and extended to new model families and data types.

At a glance

What you will do

• Design frontier storage and queueing with high-throughput writes, deduplication and quality, diversity and freshness scheduling
• Build a discovery loop that pulls links out of pages and puts new candidates back into the frontier
• Create a shared crawl execution layer that fetches politely at scale and copes with dynamic sites
• Build pipelines for open web, licensed and partner content: text, PDFs, code repos and multimodal assets
• Define stable connector interfaces so other teams can add sources without touching the platform

What they look for

• 15 or more years in software engineering, with large distributed systems designed and run in production
• Deep background in web crawling or nearby discovery, indexing or ingestion work, including distributed scheduling and stateful services
• Comfort moving between architecture, implementation, performance analysis and production debugging
• Experience with batch or streaming pipelines and the consistency, retry, backpressure and recovery questions they raise

Nice to have

• Track record of improving data yield or processing quality at scale

A typical week

• Architecture reviews and critical implementation work alongside storage and scaling teams
• Partnering with parsing and curation teams on preserving code, math and structured content
• Working with legal, privacy, security and governance partners to turn publisher terms into auditable controls
• Instrumenting the pipeline and setting service objectives for throughput, freshness and reliability
• Coordinating dependencies across teams and weighing fast data delivery against platform work