Web Crawl Engineer – Member of Technical Staff

About the role

As a member of the Data Team at Reflection, you will design, build, and operate large‑scale web crawling systems that continuously collect and process content from billions of URLs. Your work directly supports frontier AI research by ensuring high‑quality, fresh, and diverse web data for model training.

Key responsibilities

• Build and maintain web‑scale crawling infrastructure, including URL discovery, scheduling, and distributed crawl orchestration.
• Develop distributed crawlers that respect site constraints while efficiently acquiring content.
• Implement content extraction, rendering, parsing, and normalization pipelines for diverse web formats.
• Improve crawl coverage, freshness, and quality through measurement, experimentation, and incremental updates.
• Provide observability, monitoring, and reliability tooling for large‑scale crawl operations.

Required profile

• Passionate about web‑scale systems and the challenges of internet data collection.
• Curious about how web data influences AI model performance and willing to iterate based on downstream results.
• Enjoy working at the intersection of distributed systems, data infrastructure, and AI.

Required skills

• Experience building large‑scale web crawling or internet‑scale data collection systems.
• Strong understanding of crawling architectures, URL frontier management, and distributed coordination.
• Proficiency with distributed processing frameworks such as Ray, Spark, Beam, or Flink.
• Familiarity with content extraction, HTML parsing, browser automation, and rendering technologies.
• Expertise in systems engineering: reliability, observability, performance optimization, and debugging.

What we offer

• Top‑tier compensation with salary and equity.
• Stock options for all contributors.
• Comprehensive health, dental, vision, and life coverage plus wellness allowance.
• Daily lunch and dinner provided on‑site.
• Unlimited PTO in the U.S. and generous parental leave.
• Visa sponsorship and support for long‑term immigration pathways.