Research Engineer - Web Crawlers
This role involves building and operating large-scale, distributed web crawlers to source high-quality data for training frontier AI models. The engineer will solve challenges in content extraction, deduplication, and recrawl strategies, while designing pipelines for multilingual and multimedia content. They will also develop tooling for researchers to access and monitor crawled data efficiently.