Sunday, February 22, 2015

Real time analytics example with QBit Microservice Lib.

This is real time analytics example with QBit. This does not mean that there is not machine learning or Hadoop batch jobs (cough, cough, map/reduce) churning data some where. But the churned data is mixed with the data science, pre-chewed caches and counters, and up to the second user activity to make a decision based on data in memory and historic data (blended). The QBit RecommendationService example is the tip of the ice-burg, but it is brute force, exhaustive and fast.

For this example, we are going to imagine a larger app. This app will be a recommendation engine. Think of Cupid.com or DatesRUs.com or iTunes match or NetFlix suggestions or Amazon.com "Customers who bought this also bought these other fine products".

RecommendationService is our CPU intensive service. It will be the recommendation engine. It our hypothetical example RecommendationService is very CPU intensive. It has to iterate through products and user data and pick a match. It is a classic N+1. In our example, we do all of this real time based on the latest user activity, to the last second. What page did that just view? What product did they just bookmark? What product did they buy? What product did their friends buy? What product are people in their same demographic buying now.

UserDataService is our heavy IO service. As much as we would like to, we do know which RecommendationService node a user might land on we can't if we are truly elastic. Are they using an XBox on the west coast, an iPhone in Hawaii, when/how will they hit our recommendation engine who knows. UserDataService manages editing, backup, and syncing user data. UserDataService keeps most users in-memory and also manages replicating and storing user data.

Kafka and Cassandra support, training for AWS EC2 Cassandra 3.0 Training