Powered by

Glen De Cauwsemaecker

Lead Crawler Engineer at OTA Insight
Talk title:
How to ensure high quality data while scaling from 100 to 100M requests/day
29th Sep
10:00 am
Who should attend: Data teams involved in day-to-day data exraction
What you will learn: How to stay ahead of the data quality challenges while scaling
The story of OTA Insight is a story of numbers.
Every day OTA gathers data from hundreds of different kinds of data sources, with over 100M requests every day amounting to approximately 9000 GiB of raw data.

This involves the use of datacenter proxies, as well as residential proxies and sometimes no proxies at all or static IPs for integrations where necessary.

In this talk, Glen shares the story of how OTA Insight grew from 100 daily requests to 100,000,000 requests without loss of data quality. 

In this company growth study, Glen outlines:

  • The infrastructural changes required while scaling and how the infrastructure evolved to meet the needs of the company.
  • The technology used by OTA Insight and how this evolved with growing needs
  • A glimpse at the choices made at every step of the growth journey.

This should be an instructive session for anyone involved in scaling web data extraction. The talk will conclude with a reflection on the challenges that Glen foresees for the future. He will give us a peek at how the team at OTA Insight aim to stay on top of the game.

Speaker Bio

Glen De Cauwsemaecker is the Lead Crawler Engineer at OTA Insight, a company of 350 with 67 engineers. At OTA Insight, he has over ten years of experience in driving the technical roadmap and innovation with a team of four other engineers.

Besides researching, developing, and maintaining crawlers, Glen also maintains and creates new projects related to proxy/networking services, backend services, and libraries, automated browser solutions, fingerprinting, tooling, and language agnostic reverse engineering activities.
Glen is also closely involved in the maintenance of almost two hundred different active crawlers, while conducting a lot of R&D for a variety of different purposes.

Glen is a mathematics enthusiast and a conscious father. He is particularly intrigued by networking & security
check linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram