Really big data in the real world with Quantcast - A look at display advertising, which illustrates why big data is a step change

As part of my research into Big Data it is sometimes difficult to describe things that convey the scale and capability that is a step change from anything that most of us are used to. So much that goes under the big data banner seems to be just more of what we have done for a long time, with POS data analysis in retail, and CDR analysis in Telco as obvious examples. But every so often you come across something that is really jaw dropping and Quantcast is one such. The scale is staggering and shows a need to operate in real time, so that execution has to be conducted in the time that it takes for you to hit the send button and for the web page to come back with the content you requested, supported by the advertising that Quantcast has identified to the ad server as the most relevant. When it comes to big data, nothing that I have come across so far is as big and demanding as online advertising, and Quantcast are leaders in this field.

There is a seismic shift going on within the online advertising market; not only is it growing in importance at the expense of more traditional channels, but the nature of the advertising displayed is changing rapidly. The advertising world used to be one based upon periodic agreements to buy slots over months – if not quarters – but now that is fast being replaced by real time bidding for each and every slot. Online advertising used to be based on the search request submitted by the consumer, but that is fast being replaced by displaying ads based upon a profile, not of what was requested but by who is doing the requesting. Display advertising is initiated when a consumer submits a request, and has to be concluded quickly enough to allow the correct ads to be brought to their screen with the requested content. That involves making a profile of the user, assessing the options of what to target them with to maximise the chances of them actually buying, and also making a bid for the slot. And to do all of that billions of times a minute and all in the blink of an eye – impressive stuff!

To cope with the requirements of display advertising requires something more than just a bit more of what has gone before: what is required is a revolutionary approach. We are talking about a major step change, it’s mass production on an awesome scale. We are talking about petabytes of data being processed, which is not just about the application of static business rules but is about real time machine learning and optimisation; and all of that has to be brought together as a coherent whole and repeated billions of times a minute 24 by 365.

Quantcast are market leaders in this field based upon business knowledge, technical excellence and an amazing ability to deliver on a vision. Quantcast have built the largest data set of human interests on the planet, so whilst Google and Facebook know a lot about certain aspects of what you do, Quantcast have most of the other data that is of value. What that gives them is the facts on which to build their insight. They can profile a user based upon an understanding of who they are in terms of income, education, lifestyle, family etc. and then use that data to make a judgement on whether or not they are likely to buy.

Online advertising is not Marketing – it is about sales. The advertisers are not interested in brand awareness, they want to see things translated into hard sales, and this is where Quantcast excel. Quantcast have the ability to identify what they term a “Lookalike”. A lookalike is someone who they can say is more like a buyer than a non-purchaser. They do that quickly enough to then allow a bid for the display to be made, which will allow the ads to be requested from the ad server and displayed with the content without any discernable delay to the consumer, and with a very high likelihood that the display will be made to a purchaser.

The scale of all of this is quite staggering. Quantcast are processing over 20+ billion records a day to build their insight. They are handling over 7 billion auction bids a day. This is all continuously growing and undertaken on proprietary software stack assembled from massively redundant arrays of commodity hardware, because at the volumes these guys operate at failure is certain so keep it cheap and use software to ensure you keep running! This I think provides a real insight into what Big data is all about, and shows just what is going on the background when you submit a request on-line. All of this is truly remarkable and Quantcast are deservedly leaders on their chosen field because all aspects of what they do exhibit a fantastic ability to deliver.