Research Crawling Engineer

True Talent Solutions Partners · posted 2 days ago

Open to candidates worldwide

$160k–$250k · Full-time · Senior

See my match for each job — freeApply on the company site

RemoteHunt reads employers' own pages. Free to browse. How we check

What this job asks for

Extracted by RemoteHunt

Must have

•4+ years building web crawlers or data acquisition systems
•Experience with Go, Rust, Python, Java, or C++
•Experience operating data pipelines at terabyte scale
•Deep understanding of HTTP, networking, and browser behavior
•Experience handling anti-bot systems and rate limits

Nice to have

•Experience with headless browsers
•Experience with proxy systems, IP rotation
•Background in data quality evaluation or NLP pipelines

Tools and technologies

Go, Rust, Python, Java, C++, HTTP, Playwright, Puppeteer

Worth checking before you apply

Visa sponsorship is not available

More remote jobs open to candidates in India, Vietnam, Indonesia, Pakistan, Nigeria, Bangladesh, Malaysia, Philippines, Brazil, Colombia, Kenya, Ghana, Egypt, South Africa, Mexico, Argentina, Turkey, Ukraine, Sri Lanka, Nepal, Italy, Spain, Romania, Serbia, Bulgaria, Greece, United Arab Emirates and Portugal

Read the full posting

About the Role

Join a data and infrastructure team building large-scale web data systems for AI research and model development. You will help deliver reliable, high-quality datasets by operating crawlers across complex and changing web environments.

What You'll Do

Build and maintain large-scale crawlers across diverse domains, including social media, travel, and multilingual websites.
Design high-throughput, fault-tolerant collection systems capable of processing millions to billions of URLs per day.
Adapt crawling strategies to anti-bot measures, rate limits, and dynamic or JavaScript-heavy sites.
Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
Build datasets for research and model training in collaboration with research teams.
Monitor crawl coverage, performance, and data quality, and improve systems as web environments change.
Optimize infrastructure for cost, latency, and reliability across cloud and bare-metal environments.

What We're Looking For

Typically 4 or more years of relevant experience building web crawlers or large-scale data acquisition systems. Candidates with a PhD may qualify with 2 or more years.
Strong experience with Go, Rust, Python, Java, or C++, and distributed systems or parallel processing.
Experience operating data pipelines at terabyte scale and processing millions to billions of URLs per day.
Deep understanding of HTTP, networking, and browser behavior, including experience handling anti-bot systems, rate limits, and JavaScript-heavy websites.
Experience with headless browsers such as Playwright or Puppeteer, proxy systems, IP rotation, and request orchestration is valuable.
Background in data quality evaluation, web data processing, NLP pipelines, or machine-learning dataset curation is a plus.
Experience running data workloads on cloud or bare-metal infrastructure; a relevant technical degree is beneficial.