There comes a point in data analytics where Big Data fails to make the “big” picture, a clear picture.

First off,

Big Data is – large amounts of data. When we dig deep into that data using various programmatic ways, we find find hidden patterns, understand insights and optimize the valuable associations in that data that in normal circumstances are difficult to unearth. E.g. Data about how users interact with their mouse and keyboard strokes on a particular website’s checkout page. This kind of data when analysed for millions of users can unearth certain patterns of user interaction that can help website developers to make changes and convert to more sales.

Small Data is –  less amount of data. It represents specific data attributes which usually need no specific techniques to find any patterns but instead manageable enough to draw conclusions. E.g. Data about your hourly website traffic in a spreadsheet (or e.g. google analytics). This data can easily indicate that which days of the week are usually getting high traffic and which are not so that you can optimize your campaigns on the page.

Problem with Big Data

When Big Data disrupted the business ecosystem with the cutting edge methodology of  predictive study to analyze and track behavioural patterns, businesses began to tilt their attention towards their largely untouched datasets. Big Data is useful to predict customer interests and behaviour by observing collectively and mathematically via algorithmic processing.

Despite the remarkable revolution brought about by Big Data, it’s not a tool that can be entirely relied upon. As the major observations of the mammoth datasets are in fact human behaviour. Apparently human behaviour is a complex abstract where predictions are likely to miss the mark. Since generalizations have quite often (at least historically) been destructive, producing stereotypes and bad judgement call, analysis at the courtesy of Big Data also often produce anomalous results no matter how mathematically correct the applied algorithm claims to be.

What can possibly go wrong?

For instance, Cathy O’Neil recounts of the classic recipe of disaster cooked up by a right-looking “wrong” algorithm in a TED talk from September 2017,

Algorithms are opinions embedded in code. Algorithms are considered to be objective and true and scientific … its a marketing trick!

O’Neil also recounts of how Kiri Soares, a high school principal in Brooklyn, found out that her teachers were being fired when a complex algorithm was employed by her consultants in 2011. The teachers were being scored based upon a “value-added model”. The functioning of this assessment model remained unknown to Soares.  The teachers who were ruled out on the basis of this algorithm were also ridiculed on a public platform for their “bad teaching practices” causing much spur. That’s when someone actually decided to look into the matter and realized that the algorithm had wrongly clustered all the teachers into one set resulting in a random number generator.

Small data to the rescue

Small Data is the real deal in such cases. Why? Because the individual matters. It all revolves around keen observation and applied intuition. While big data deals with overwhelmingly large amounts of data in a stream, Small Data is more oriented towards selecting a small set of meaningful data from a large population and to come up with validations for behaviour. Many companies closely monitor a few selected sets of customers through their cell phone numbers and other social media activities (all with consent of course) to see where does the majority interest lie. They then alter or improve their CRM solutions to meet the need of their customers

Though literally not small, Small Data is comparatively lesser in volume than Big Data. It is easily navigated and can be used even by the large population of the non-data scientists. Small Data is crucial for Enterprise Resource Planning and Customer Relationship Management systems. Especially for the case of startups who are just expanding on a small scale, Small Data being precise and accurate can help them to administer their plans with more realistic insights. Since no hypothesis and guess work is involved, it makes it easier to do deductions.

Martin Lindstrom, branding expert, firmly believes that Small Data is going to be the next revolution in the marketing business. By utilizing small data, businesses can arrive at imperative conclusions without having to navigate through the overwhelming amount of information big data bombards them with on a daily basis.

Small data is the ‘what’ to your problem. If you want to understand ‘why’, then big data is what you seek.

—

You can also be an author on Clarity! Contact us here

Together we spread more Clarity!