Crawlera is a smart downloader designed specifically for web crawling and scraping, removing the headaches of proxy management. It is part of the Scrapinghub platform, the world’s most comprehensive web crawling stack which powers crawls of over 8 billion pages a month. We’re looking for a highly collaborative Technical Lead for Crawlera.

Leading Crawlera’s technical development through the exponential growth that we’re experiencing is just the start. You’ll help set overarching business strategy and goals, identify opportunities, and over time, together with Crawlera’s Product Manager, will have full responsibility for Crawlera as a business.

You should be comfortable not having all the answers but being the person that gets the right people to the table for discussions, and a strong bias towards taking action. Ideally you’ve done this job before, either in an early stage company or a new team/initiative, and have a concrete history of developing world-class team(s).

Job Responsibilities:

Identify and resolve performance and scalability issues with distributed crawling at scale.

Develop, maintain and support a high load distributed system.

Comfortable taking ownership in business critical situations; you enjoy being the “go-to” person.

Lead the team. You’ll have a fair amount of autonomy over this, but it will likely include:

Developing talent and maintaining a strong team culture.

Expanding and maintaining technical and quality standards, in conjunction with broader Engineering expectations.

Translating business intent and requirements into architectural design/decisions to achieve that intent.

Collaborating with other platform teams to provide Crawlera customers the best possible integration to the growing Scrapinghub platform.

Required Skills:

2+ years of production experience with at least two of: Erlang (or other functional programming language), Python, and Go.