Lead Crawler Engineer at OTA Insight

Presentation would start with a very quick intro on myself and the company (e.g. who am I, what is OTA insight, what kind of data we collect, where we get this data from and how we use it; - If allowed it would also include a tiny plug-in to tell them we’re hiring crawler engineers at junior and senior level, fully remote or on-site (up to the candidates);

The intro would be rather short and go very quickly to a summary slide where we show some numbers (e.g. Every day we gather data from hundreds different kinds of data sources, over 100M requests every day amounting to approximately 9000 GiB (compressed) of raw data.

And this making use of datacenter proxies, residential proxies as well as no proxies or at all or where fitted static IPs for integrations; - It is these stories that set the stage for the story, as this presentation really is a story about how we grew from 100 daily requests to 100,000,000 requests, and that while continuously yielding high quality data, every single day.

The story will fill most of the presentation and takes us to the beginning of the company all the way to today and what we’re working on today to prepare is for tomorrow; - Throughout our journey in this story we’ll go through our infrastructure and how it is involved to help us with this growth. What technology do we use, how did this evolve and what choices did we make at every step. Why did we make decisions and how? - The talk would end with where we see ourselves going as a crawler team, what challenges we foresee for the future and how we aim to stay on top of the game. If the event allows it there can also be an open Q&A.

The presentation would roughly take a maximum of 35 minutes (excluding the opt-in Q&A), but if required we can keep it as short as 15 minutes.

Glen is the Lead Crawler Engineer at OTA Insight (around 350 employees, of which 67 engineers), with over ten years of experience, where he drives the technical roadmap and innovation with a team of four other engineers.

Within this role he not only designs and plans technical endeavours. Besides researching, developing and maintaining crawlers, he also maintains and creates new projects related to proxy/networking services, backend services and libraries, automated browser solutions, fingerprinting, tooling and a trivia of cross-platform and language agnostic reverse engineering activities. As part of all this as well as the maintenance of almost two hundred different active crawlers, there is a lot of R&D involved for a variety of different purposes.

Previous to working at OTA Insight he has worked — From several countries in South America — on operating systems, cloud infrastructure and independent blockchain technology for a decentralised cloud network on the edge.

The first five years of his career he was active in the Game Industry where he worked on Serious VR Games for Training and Medical purposes, Game Engines, Cross platform video games, backend services, tooling, libraries, R&D work and Mobile Games such as Angry Birds Transformers while living in the UK.