Data Engineer Web Scraping Crawling

Job Title: Web Crawling Engineer

Experience:3-5 Years

Location: Hyderabad

We are looking for a skilled Web Crawling Engineer with strong expertise in Python and hands-on experience building scalable crawlers and scrapers. The ideal candidate should have deep knowledge of crawling frameworks, anti-blocking techniques, and proxy management, along with experience solving challenges like captchas and rate-limiting.

Key Responsibilities

• Design, develop, and maintain scalable and efficient web crawlers for data extraction.
• Work with Scrapy, Requests, and other Python libraries to implement crawling solutions.
• Handle anti-scraping measures, including IP blocks, rate limits, and captchas.
• Implement and manage proxy rotation and session management strategies.
• Ensure high-quality and structured data extraction, cleaning, and storage.
• Monitor crawler performance, troubleshoot issues, and optimize for reliability and efficiency.
• Collaborate with the data engineering team to integrate crawled data into pipelines.
• Stay updated with the latest tools, techniques, and best practices in web crawling.
  
  

• Strong programming skills in Python.
• Proven experience with Scrapy, Requests, BeautifulSoup, Selenium (if required).
• Hands-on experience solving blocking issues, handling
• Good understanding of proxy usage, rotation, and fingerprinting avoidance techniques.
• Knowledge of HTTP, cookies, headers, sessions, and request/response cycles.
• Experience with databases (SQL/NoSQL) for storing crawled data.
• Strong debugging, problem-solving, and analytical skills.
  
  

• Experience with cloud environments (AWS, GCP, Azure).
• Knowledge of data pipelines and ETL processes.
  
  

• Bachelor’s degree in Computer Science, Information Technology, or equivalent practical experience.
• 3+ years of experience in web crawling or related fields