Big Data is the Answer – What was the Question?
Use cases described in this posting are meant to stimulate ideas of how to apply iterative big data analytics in your own organization and enable your own analytics center of innovation.
First let’s define what makes data Big.
Big Data, Little Data
Unless you have been living in a cave you probably have heard about Big Data. We live in a world of data: transactions, feedback, and realtime interaction with customers, partners, suppliers, and employees.
In addition to brick, click and mobile transactions, the new variable in the mix is Human generated data – explosive growth of blogs/reviews/messages/emails/pictures. The Twitter firehose alone generates 10s of millions of tweets per day and is growing rapidly. Social graphs such as product recommendations based on circle of friends, jobs you may like (linked in), products you have looked at, people who are your contacts etc also create “second order” data that can be mined for sentiment analytics on products or companies or fact discovery.
Another new variable is computer generated data. Computers generate data as byproduct of interacting with people or other with other device. More interactions, more data. This data comes in a variety of formats from semi-structured log files to unstructured binaries. This “exhaust fumes” of data can be extremely valuable. It can be used to understand and track application or service behavior so that we can find patterns, errors or suboptimal user experience. We can mine it for statistical patterns and correlations to generate insights.
However, if you listen to the hype, companies can harness this information learn faster, make better decisions, and stay one step ahead of their competitors. Unfortunately, harnessing big data (and separating the signal-from-noise) is trickier than it looks. It takes a lot of skill and superb understanding of use cases.
Big Data Use Cases
The key to exploiting Big Data Analytics is focusing on a compelling business opportunity as defined by a use case — WHAT (What exactly are we trying to do?)
Use cases are emerging in a variety of industries that illustrate different core competencies around analytics.
Figure below illustrates some Use Cases along two dimensions: data velocity and variety. A Use Case provides a context for a value chain: how to move from “Raw Data -> Aggregated Data -> Intelligence -> Insights -> Decisions -> Operational Impact -> Financial Outcomes -> Value creation.”
Source: SAS and IDC
E-tailing – E-Commerce – Online Retailing Use Cases
- Recommendation engines — increase average order size by recommending complementary products based on predictive analysis for cross-selling.
- Cross-channel analytics — sales attribution, average order value, lifetime value (e.g., how many in-store purchases resulted from a particular recommendation, advertisement or promotion).
- Event analytics — what series of steps (golden path) led to a desired outcome (e.g., purchase, registration).
- Right offer at the right time
Interesting Use Case – Amazon Will Pay Shoppers $5 to Walk Out of Stores Empty-Handed
Interesting use of consumer data entry to power next generation retail price competition…. Amazon is offering consumers up to $5 off on purchases if they compare prices using their mobile phone application in a store. The promotion will serve as a way for Amazon to increase usage of its bar-code-scanning application, while also collecting intelligence on prices in the stores.
Amazon’s Price Check app, which is available for iPhone and Android, allows shoppers to scan a bar code, take a picture of an item or conduct a text search to find the lowest prices. Amazon is also asking consumers to submit the prices of items with the app, so Amazon knows if it is still offering the best prices. A great way to feed data into its learning engine from brick-and-mortar retailers.
This is an interesting trend that should terrify brick-and-mortar retailers. While the real-time “Everyday Low Price” information empowers consumers, it terrifies retailers, who increasingly are feeling like showrooms — shoppers come to to check out the merchandise but ultimately decide to walk out and buy online instead.
Retail/Consumer Use Cases
- Merchandizing and market basket analysis.
- Campaign management and customer loyalty programs.
- Supply-chain management and analytics.
- Event- and behavior-based targeting.
- Market and consumer segmentations.
Predictive analytics is well understood by the retail industry. Retailers want to predict factors that might be important for a buyer to make a purchasing decision before that product ever was stocked on shelves. What if retailers could know exactly what market dynamics were modifying demand curves before they occurred? Imagine the impact on operational efficiencies in terms of inventory cost control, intelligent distribution and routing, and demand projection. The retail Use Cases are quite varied.
Financial Services Use Cases
- Compliance and regulatory reporting
- Risk analysis and management
- Fraud detection and security analytics
- CRM and customer loyalty programs
- Credit risk, scoring and analysis
- High speed Arbitrage trading
- Trade surveillance
- Abnormal trading pattern analysis
Risk Modeling Use Case — A large financial institution took separate data warehouses from multiple departments and combined them into a single global repository in Hadoop for analysis. The bank used the Hadoop cluster to construct a new and more accurate score of the risk in its customer portfolios. The more accurate score allowed the bank to manage its exposure better and to offer each customer better products and advice.
Trade Surveillance Use Case — A large investment bank combines data about the parties that participate in a trade with the complex data that describes relationships among those parties and how they interact with one another. The combination allows the bank to recognize unusual trading activity and to flag it for human review.
Web & Digital Media Services Use Cases
- Large-scale clickstream analytics
- Ad targeting, analysis, forecasting and optimization
- Abuse and click-fraud prevention
- Social graph analysis and profile segmentation
- Campaign management and loyalty programs
Clickstream Use Case – Big Box Retailer need to analyze their clickstream – §3.5 billion records; 71 million unique cookies; 1.7 million targeted ads required per day. The problem: how to improve Return on Ad Spend (ROAS). Also how to speed up the analytics so consumers get more relevant ads quicker, which is especially important during holiday seasons!
Suggestion Use Case. Yelp is growing rapidly and with more than 50 million of monthly visitors and 18 million or reviews the company generates about 400GB of data a day. That data needs to be processed and analyzed. A simple use case is Spelling Suggestions. By looking at millions of misspelled words Yelp uses an algorithm to create suggestions for common misspellings. By looking at typical queries, yelp can list common suggestions for a query even before you finish typing. This is possible because Yelp analyses all the web logs from their websites
Government Use Cases
- Fraud detection
- Threat detection
- Compliance and regulatory analysis
- Energy consumption and carbon footprint management
- Sentiment Analytics
- Mashups – Mobile User Location + Precision Targeting
- Machine-generated data, the exhaust fumes of the Web
Online Dating Use Case: A leading online dating service uses sophisticated analyses to measure the compatibility between individual members, so that it can suggest good matches for a potential relationship.
Big Data Analytics helped customers find romance. The algorithms that power Match.com are not very different than those that are behind LinkedIn.
Social Gaming: Zynga in their S-1 filing claimed that “they process and serve more than a petabyte of content for players every day, a volume of data that is unmatched in the social game industry. We continually analyze game data to optimize our games. We believe that combining data analytics with creative game design enables us to create a superior player experience.”
Health & Life Sciences Use Cases
- Health Insurance fraud detection
- Campaign and sales program optimization
- Brand management
- Patient care quality and program analysis
- Supply-chain management
- Drug discovery and development analysis
Telecommunications Use Cases
- Revenue assurance and price optimization
- Customer churn prevention
- Campaign management and customer loyalty
- Call Detail Record (CDR) analysis
- Network performance and optimization
- Mobile User Location analysis
Telecom Use Case: A large telecommunications provider analyzed call logs and complex data from multiple sources. It used sophisticated predictive models across that data to predict the likelihood that any particular customer would leave. The result: Analytics helped the telecommunications company build more insightful customer relationships and reduce churn.
Utilities Industry Use Cases
Utilities run big, expensive and complicated systems to generate power. Each grid now includes sophisticated sensors that monitor voltage, current, frequency and other important operating characteristics. Efficiency now means paying careful attention to all of the data streaming off of the sensors.
Utilities are now leveraging Hadoop clusters to analyze generation and consumption data via smart meters.
Smart meters – The rollout of smart meters as part of the Smart Grid adoption by utilities everywhere has resulted in a deluge of data flowing at unprecedented levels. Most utilities are ill-prepared to analyze the data once the meters are turned on.
So, What’s the Big Deal?
The big deal is that if analytics is done well as shown in the figure there is room for margin expansion and additional profit.
Big Data is full of valuable, unanswered questions! The challenge is separating the actual predicitive indicators – signal from the noise — in the data.
Companies who compete on analytics and delivering data-driven services tend to iterate quickly on big data. This enables rapid data exploration to identify unknown relationships and trends to create new products and services.
Data overload is going to be a huge challenge for businesses and a headache for decision makers. Public and private sector corporations are going to drown in data — from sales, transactions, pricing, supply chains, discounts, product, customer process, projects, RFID smart tags, tracking of shipments, as well as e-mail, Web traffic and social media. Without a smart use case strategy, a lot of this data will be wasted.
Notes and References
2. A business use case describes what the process does. It is meant to describe in technology-free jargon the business process that is used by its business actors (people or systems external to the process) to achieve their goals . The business use case will describe a process that provides value to the business actor.
- IRS Uses Analytics To Help Collect Delinquent Taxes (informationweek.com)
- Now that data’s going really big, what’s next? (forbes.com)
1) Saama SixthSense for a Saama’s point of view presentation on Big Data and why you need to listen to your Big Data.
2) CIO Analytics: Managing Business Value of IT for a whitepaper on how to measure, Plan and Cost Business Value of IT using a Service Oriented Delivery Model.
3) Big Data is the Answer – What was the Question? for a recorded version of the webinar hosted on February 17, 2012