Remote, Córdoba, Argentina · Today · New
Research Crawling Engineer
True Talent Solutions Partners · posted todayFresh: apply early
Open to candidates worldwide
$160k–$250k · Full-time · Senior
Sign up and we score this job against your resume. You never pay to apply.
RemoteHunt reads employers' own pages. Free to browse. How we check
What this job asks for
Extracted by RemoteHuntMust have
- •4+ years building web crawlers or data acquisition systems
- •Experience with distributed data pipelines at terabyte scale
- •Proficiency in Go, Rust, Python, Java, or C++
- •Deep knowledge of HTTP, networking, and browser behavior
- •Experience handling anti-bot systems and dynamic sites
- •Experience with headless browsers (Playwright/
Puppeteer)
Nice to have
- •NLP pipelines or ML dataset curation
- •Experience with cloud platforms or bare-metal infrastructure
- •Degree in computer science or engineering
Tools and technologies
Go, Rust, Python, Java, C++, HTTP, Playwright, Puppeteer, Chrome DevTools Protocol
Worth checking before you apply
- Visa sponsorship is not available
More remote jobs open to candidates in India, Vietnam, Indonesia, Pakistan, Nigeria, Bangladesh, Malaysia, Philippines, Brazil, Colombia, Kenya, Ghana, Egypt, South Africa, Mexico, Argentina, Turkey, Ukraine, Sri Lanka, Nepal, Italy, Spain, Romania, Serbia, Bulgaria, Greece, United Arab Emirates and Portugal
Read the full posting
About the Role
Build large-scale web crawling and data acquisition systems that support AI research and model development. You will work closely with research teams to deliver high-quality web datasets for training and inference.
What You'll Do
-
Build and operate crawlers across diverse domains, including social media, travel, and multilingual websites.
-
Design high-throughput, fault-tolerant collection systems that process millions to billions of URLs per day.
-
Adapt crawling approaches to anti-bot systems, rate limits, and dynamic or JavaScript-heavy websites.
-
Develop large-scale pipelines for cleaning, deduplication, filtering, and normalization of web data.
-
Collaborate with research teams to shape datasets for research and model training.
-
Monitor crawl performance, coverage, and data quality, and respond to changes in web environments.
-
Optimize infrastructure for cost, latency, and reliability across cloud and bare-metal environments.
What We're Looking For
-
Typically 4 or more years of relevant experience building web crawlers or large-scale data acquisition systems. Candidates with a PhD may qualify with 2 or more years; hands-on crawler and pipeline experience is essential.
-
Experience operating distributed data pipelines at terabyte or larger scale, ideally processing millions to billions of URLs per day.
-
Strong proficiency in Go, Rust, Python, Java, or C++ and a solid understanding of distributed systems and parallel processing.
-
Deep knowledge of HTTP, networking, and browser behavior, including experience handling anti-bot systems, rate limits, and dynamic sites.
-
Experience with headless browsers such as Playwright or Puppeteer, Chrome DevTools Protocol, proxy systems, or IP rotation.
-
Experience with web data quality, benchmarking, cleaning, deduplication, and normalization; NLP pipelines or ML dataset curation are a plus.
-
Experience running crawling or data workloads on cloud platforms or bare-metal infrastructure. A degree in computer science, engineering, or a related field is a plus.
Compensation & Benefits
Base salary is $160,000 to $250,000 annually. Visa sponsorship is not available.
Location
Remote.
More remote Data Engineer jobs
See allAnalytics Engineer
Paystone
Ho Chi Minh City, Vietnam (Remote) · 7d ago
Remote, United States · 9d ago
Senior Analytics Engineer
Empower Pharmacy
US Remote · 12d ago
Remote · 15d ago
[C3J] Azure Data Engineer
Software Mind
Warsaw, Masovian Voivodeship, Poland · 21d ago
Get new jobs like this by email
Once a week. Remote data engineer jobs that can hire you in your country.
See how well this job fits your resume
Free, no card.
See my match for each job — free