Software Engineer, Web Crawling
ExaSingapore4 days ago
Full-timeEngineering, Information Technology
Track This Job
Add this job to your tracking list to:
- Monitor application status and updates
- Change status (Applied, Interview, Offer, etc.)
- Add personal notes and comments
- Set reminders for follow-ups
- Track your entire application journey
Save This Job
Add this job to your saved collection to:
- Access easily from your saved jobs dashboard
- Review job details later without searching again
- Compare with other saved opportunities
- Keep a collection of interesting positions
- Receive notifications about saved jobs before they expire
AI-Powered Job Summary
Get a concise overview of key job requirements, responsibilities, and qualifications in seconds.
Pro Tip: Use this feature to quickly decide if a job matches your skills before reading the full description.
Exa is building a search engine from scratch to serve every AI application. We build massive-scale infrastructure to crawl the web, train state-of-the-art embedding models to index it, and develop super high performant vector databases in Rust to search over it. We also own a $5m H200 GPU cluster and routinely run batch jobs with tens of thousands of machines. This isn't your average startup :)
As a Web Crawler engineer, you'd be responsible for crawling the entire web. Basically build Google-scale crawling!
Desired Experience
Compensation Range: $150K - $300K
As a Web Crawler engineer, you'd be responsible for crawling the entire web. Basically build Google-scale crawling!
Desired Experience
- You have extensive experience building and scaling web crawlers, or would be excited to ramp up very quickly
- You have experience with some high performance language (C++, Rust, etc.)
- You’re comfortable optimizing a system to an exceptional degree
- You care about the problem of finding high quality knowledge and recognize how important this is for the world
- Build a distributed crawler that can handle 100M+ pages per day
- Optimize crawl politeness and rate limiting across thousands of domains
- Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures
- Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency
Compensation Range: $150K - $300K
Key Skills
Ranked by relevanceReady to apply?
Join Exa and take your career to the next level!
Application takes less than 5 minutes