Web Scraping Engineer

Intro description:

Legalist is an institutional alternative asset management firm. Founded in 2016 and incubated at Y Combinator, the firm uses data-driven technology to invest in credit assets at scale. We are always looking for talented people to join our team.

Where You Come In:

• Help to design and implement the architecture of a large-scale crawling system

• Design, implement, and maintain various components of our data acquisition infrastructure (building new crawlers, maintain existing crawlers, data cleaners & loaders)

• Work on developing tools to facilitate the scraping at scale, monitor the health of crawlers and ensure data quality of the scraped items. 

• Collaborate with our product and business teams to understand / anticipate requirements to strive for greater functionality and impact in our data gathering systems 

What you’ll be bringing to the team:

• 3+ Years experience with Python for data wrangling and cleaning

• 2+ Years experience with data crawling & scraping at scale (100+ spiders at least)

• Productionized experience with Scrapy is mandatory. Distributed crawling and advanced scrapy experience are a plus.

• Familiarity with scraping libraries and monitoring tools highly recommended (BeautifulSoup, Xpaths, Selenium, Puppeteer, Splash)

• Familiarity with data pipelining to integrate scraped items into existing data pipelines.

• Experience extracting data from multiple disparate sources including HTML, XML, REST, GraphQL, PDF, and spreadsheets.

• Experience running, monitoring and maintaining a large set of broad crawlers (100+ spiders) 

• Sound Knowledge in bypassing Bot Detection Techniques

• Experience using techniques to protect web scrapers against site ban, IP leak, browser crash, CAPTCHA and proxy failure.

• Experience with cloud environments like GCP, AWS, as well as containerization tools like Docker and orchestration such as kubernetes or others.

• Ability to maintain all aspects of a scraping pipeline end to end (building and maintaining spiers, avoiding bot prevention techniques, data cleaning and pipelining, monitoring spider health and performance).

• OOP, SQL and Django ORM basics

Even better if you have, but not necessary:

• Experience with microservices architecture would be a plus.

• Familiarity with message brokers such as Kafka, RabbitMQ, etc

• Experience with DevOps

• Expertise in data warehouse maintenance, specifically with Google BigQuery (ETLs, data sourcing, modeling, cleansing, documentation, and maintenance)

• Familiarity with job scheduling & orchestration frameworks - e.g. Jenkins, Dagster, Prefect