ABOUT THE EMPLOYER
Our client is a leader in brand protection and channel intelligence, using advanced data collection and machine learning to help businesses automatically detect pricing policy violations, unauthorized sellers, and other online compliance issues. With a rapidly growing client base, our client relies on high-volume, sophisticated web crawlers to power their platform. As these crawlers face increasingly strict anti-bot and anti-scraping countermeasures, the company needs a Security Crawler Engineer to ensure uninterrupted access to valuable online data.
WHAT WILL YOU WORK ON?
As a Security Crawler Engineer, you will be at the forefront of web scraping and security research. Your main goal is to develop and maintain resilient, large-scale crawlers capable of bypassing modern anti-bot systems, CAPTCHAs, and other protective barriers. This role requires a mix of software engineering, reverse engineering, and security expertise to keep pace with constantly evolving website defenses.
- Crawler Development
- Design, build, and optimize web crawlers to scrape high volumes of data from diverse and complex online sources.
- Collaborate with cross-functional teams to ensure crawlers effectively meet business and client needs.
- Research & Bypass Techniques
- Investigate the latest anti-bot and anti-scraping solutions (e.g., Cloudflare, hCaptcha, advanced browser fingerprinting).
- Devise innovative methods to bypass detection algorithms, rate-limiting, CAPTCHAs, and other security measures that hamper data collection.
- Reverse Engineering
- Analyze site structures, security protocols, and detection systems to identify vulnerabilities and develop robust workarounds.
- Deconstruct APIs and obfuscated JavaScript to maintain access to critical data.
- Security & Compliance
- Ensure adherence to legal and ethical standards while performing data collection.
- Maintain secure coding practices and protect sensitive information in the data pipeline.
- Performance & Scalability
- Monitor and optimize crawler performance to handle large-scale operations with minimal downtime.
- Implement strategies to scale and distribute crawling tasks effectively.
- Troubleshooting & Maintenance
- Rapidly diagnose crawler failures caused by site changes or new security measures.
- Maintain clear documentation of processes and provide guidance to junior engineers or support teams.
WHAT SKILLS AND EXPERIENCE ARE WE LOOKING FOR?
- Golang skills: Proficient in Golang. At least 3 years of experience. Familiarity with browser automation tools.
- Security / Reverse Engineering: Demonstrated experience in reverse engineering, packet inspection, and/or bypassing complex anti-bot mechanisms.
- Web Technologies: In-depth knowledge of HTTP/HTTPS, web protocols, and security mechanisms.
- Problem-Solving: Creative approach to circumventing sophisticated security roadblocks.
- Analytical Mindset: Ability to analyze error logs and performance metrics to pinpoint and address crawler issues.
- Experience with Anti-Bot Solutions: Familiarity with Cloudflare, hCaptcha, Distil Networks, or PerimeterX.
- Browser Fingerprinting Techniques: Understanding of client-side detection (canvas fingerprinting, WebGL, etc.).
- Network Protocols & Tools: Proficiency with Wireshark, Fiddler, Burp Suite, or similar traffic analysis utilities.
- Cloud Infrastructure: Hands-on experience with AWS, GCP, or other cloud providers for deploying large-scale crawlers.
- DevOps: Knowledge of CI/CD pipelines, containerization (Docker/Kubernetes), and distributed architectures.
1pm to 10pm (GMT+5)
