Berlin, (Feb 2, 2015) by Holger Pillmann (CTO, LiquidM) – Time is a crucial factor in mobile advertising OpenRTB and many companies are trying to implement new features while staying below the response time limit of 100ms. In this article we describe how LiquidM implemented frequency capping as a distributed micro service using Aerospike and the Go programming language, adding less than 1 ms latency to our system in the process.
OpenRTB systems are bound to the auction time of 100 milliseconds set by the standard, meaning the clocks are constantly ticking in the world of mobile advertising. This limit is end-to-end time, including time on the network. Often the SSP and DSP sit in different datacenters, thus a lot of time has already been used when LiquidM receives a bid request. We must assume that the same time is needed to send our response back, meaning much of the time is already used for communication with our suppliers. Therefore, we put a lot of effort to reach the lowest possible processing times and handle as many requests as possible for our clients.
When we decided to implement a frequency capping solution our plan was clear from the beginning: identify a users device uniquely and count the different ads a user sees. This is straightforward enough, but involves selecting and integrating the perfect database. A database is needed because user data must be persistent and cannot be stored in memory on a single server. The performance requirements for the database is very high – we have to store billions of users and handle over a hundred thousand requests per second. A database also gives us the potential to store additional user related data, which is a huge advantage compared to the usage of matching or counting techniques such as a Bloom filter or HyperLogLog.
Using a database in a low latency trading system, like we run at LiquidM, is not easy. Some suppliers can send you easily 100,000 requests per second (rps) or more, and we are handling millions of users, sites and targeting criteria created by this traffic. At LiquidM, we came to the conclusion that we need a highly scalable, low latency database for this new data driven feature in our OpenRTB stack. We planned the frequency capping solution together with a unique device identification system because it came with three requirements. The first is low read latency due to the ticking auction clock. The second is low write latency due to the possibility that a request with identical device identifiers can arrive in the same instant, still requiring capping to be applied. Finally, we need solutions that scale and keep up with the enormous growth of mobile OpenRTB.
Our investigations brought the Aerospike database to our radar. Our first analysis showed a single machine can easily handle 100,000 rps with a latency below 1ms (including network). The published benchmark showed 1 million rps can be reached on a single server, meaning scaling would obviously not be a big problem. Aerospike is open source and this is always a big plus – LiquidM loves and contributes to open source, bringing low costs and easy debugging during prototyping and development.
When choosing a programming language, we had no limitations as our architecture is based on distributed micro services. We decided to create a new user-based service and utilise user related data to cap ads on frequency and the time between two impressions. We implemented prototypes to send requests to Aerospike in different languages. Go became our choice as it showed the best results in latency and stability. We also saw that our prototype could be refactored into a production system due to the nature of the Go language, which structures programs in modules.
Go is a relatively new open source programming language developed at Google. It is a statically typed compiled language and very different to our previous JRuby-based programs. JRuby was our prefered language due to the ease of expressing and testing complicated logic in Ruby. In the new service, we gained big benefits from the key features of Go, like its strict compiler rules and a native support parallel execution. The Go programming language has a built in concurrency feature called goroutines. These goroutines are lightweight threads, meaning we can execute many parallel asynchronous reads and writes to the Aerospike database. Additionally, Aerospike provide an extremely fast asynchronous Go client for this purpose.
Even though Go was completely new for our team of 3 developers, we implemented the frequency capping solution with great speed. After 3 days, we had finished a working prototype and were completely finished in 3 weeks – including the integration of the new service into our system.
During the beta phase in our production system, we reached 50,000 rps with an average request processing time of 0.3 ms, including calls to Aerospike. The numbers on our production systems showed no increase in latency after the deployment of the frequency capping feature. The system is currently designed to handle 10 times more traffic than in the beta, thus we have been able to increase the amount of RTB traffic we handle for our customers.
We are completely satisfied with the result – Go and Aerospike are the perfect match to power our frequency capping feature, and provide scalability for years to come.
At LiquidM, we are software and technology experts who are passionate about bringing the world’s best products to the rapidly growing mobile advertising industry. We are focused on building the first and only white-labeled Mobile Advertising Management Platform.
Built from the ground up for mobile, LiquidM’s white-labeled Mobile Advertising Management Platform (MAMP) allows mobile media buyers to optimize their campaigns across the full range of premium to performance advertising. LiquidM’s modular, cloud-based SaaS replaces Build-Your-Own (BYO) or inadequate point solutions with a standardized, open platform that is customizable to individual needs.