Siddharth Sonkar — Published On November 25, 2020 and Last Modified On November 25th, 2020
Beginner Big data Data Engineering


  • Learn what is Big Data and how it is relevant in today’s world
  • Get to know the characteristics of Big Data



The term “Big Data” is a bit of a misnomer since it implies that pre-existing data is somehow small (it isn’t) or that the only challenge is its sheer size (size is one of them, but there are often more).

In short, the term Big Data applies to information that can’t be processed or analyzed using traditional processes or tools.

Know More About Big Data: An Explanation For Everyone - Tech - Mi Community - Xiaomi

Increasingly, organizations today are facing more and more Big Data challenges. They have access to a wealth of information, but they don’t know how to get value out of it because it is sitting in its most raw form or in a semi-structured or unstructured format; and as a result, they don’t even know whether it’s worth keeping (or even able to keep it for that matter).

In this article, we look into the concept of big data and what it is all about.


Table of Contents

  • What is Big Data?
  • Characteristics of Big Data
    • The Volume of Data
    • The Variety of Data
    • The Velocity of Data


What is Big Data

We’re a part of it, every day!

An IBM survey found that over half of the business leaders today realize they don’t have access to the insights they need to do their jobs. Companies are facing these challenges in a climate where they have the ability to store anything and they are generating data like never before in history; combined, this presents a real information challenge.

Big data

It’s a conundrum: today’s business has more access to potential insight than ever before, yet as this potential gold mine of data piles up, the percentage of data the business can process is going down—fast. Quite simply, the Big Data era is in full force today because the world is changing.

Through instrumentation, we’re able to sense more things, and if we can sense it, we tend to try and store it (or at least some of it). Through advances in communications technology, people and things are becoming increasingly interconnected—and not just some of the time, but all of the time. This interconnectivity rate is a runaway train. Generally referred to as machine-to-machine (M2M), interconnectivity is responsible for double-digit year over year (YoY) data growth rates.

Finally, because small integrated circuits are now so inexpensive, we’re able to add intelligence to almost everything. Even something as mundane as a railway car has hundreds of sensors. On a railway car, these sensors track such things as the conditions experienced by the rail car, the state of individual parts, and GPS-based data for shipment tracking and logistics. After train derailments that claimed extensive losses of life, governments introduced regulations that this kind of data be stored and analyzed to prevent future disasters.

Rail cars are also becoming more intelligent: processors have been added to interpret sensor data on parts prone to wear, such as bearings, to identify parts that need repair before they fail and cause further damage—or worse, disaster. But it’s not just the rail cars that are intelligent—the actual rails have sensors every few feet. What’s more, the data storage requirements are for the whole ecosystem: cars, rails, railroad crossing sensors, weather patterns that cause rail movements, and so on.

Now add this to tracking a rail car’s cargo load, arrival and departure times, and you can very quickly see you’ve got a Big Data problem on your hands. Even if every bit of this data was relational (and it’s not), it is all going to be raw and have very different formats, which makes processing it in a traditional relational system impractical or impossible. Rail cars are just one example, but everywhere we look, we see domains with velocity, volume, and variety combining to create the Big Data problem.


What are the Characteristics of Big Data?

Three characteristics define Big Data: volume, variety, and velocity.

Together, these characteristics define “Big Data”. They have created the need for a new class of capabilities to augment the way things are done today to provide a better line of sight and control over our existing knowledge domains and the ability to act on them.

 Big Data Characteristics


1. The Volume of Data

The sheer volume of data being stored today is exploding. In the year 2000, 800,000 petabytes (PB) of data were stored in the world. Of course, a lot of the data that’s being created today isn’t analyzed at all and that’s another problem that needs to be considered. This number is expected to reach 35 zettabytes (ZB) by 2020. Twitter alone generates more than 7 terabytes (TB) of data every day, Facebook 10 TB, and some enterprises generate terabytes of data every hour of every day of the year. It’s no longer unheard of for individual enterprises to have storage clusters holding petabytes of data.

When you stop and think about it, it’s a little wonder we’re drowning in data. We store everything: environmental data, financial data, medical data, surveillance data, and the list goes on and on. For example, taking your smartphone out of your holster generates an event; when your commuter train’s door opens for boarding, that’s an event; check-in for a plane, badge into work, buy a song on iTunes, change the TV channel, take an electronic toll route—every one of these actions generates data.

Okay, you get the point: There’s more data than ever before and all you have to do is look at the terabyte penetration rate for personal home computers as the telltale sign. We used to keep a list of all the data warehouses we knew that surpassed a terabyte almost a decade ago—suffice to say, things have changed when it comes to volume.

As implied by the term “Big Data,” organizations are facing massive volumes of data. Organizations that don’t know how to manage this data are overwhelmed by it. But the opportunity exists, with the right technology platform, to analyze almost all of the data (or at least more of it by identifying the data that’s useful to you) to gain a better understanding of your business, your customers, and the marketplace. And this leads to the current conundrum facing today’s businesses across all industries.

As the amount of data available to the enterprise is on the rise, the percent of data it can process, understand, and analyze is on the decline, thereby creating the blind zone.

What’s in that blind zone?

You don’t know: it might be something great or maybe nothing at all, but the “don’t know” is the problem (or the opportunity, depending on how you look at it). The conversation about data volumes has changed from terabytes to petabytes with an inevitable shift to zettabytes, and all this data can’t be stored in your traditional systems.


2. The Variety of Data

The volume associated with the Big Data phenomena brings along new challenges for data centers trying to deal with it: its variety.

With the explosion of sensors, and smart devices, as well as social collaboration technologies, data in an enterprise has become complex, because it includes not only traditional relational data, but also raw, semi-structured, and unstructured data from web pages, weblog files (including click-stream data), search indexes, social media forums, e-mail, documents, sensor data from active and passive systems, and so on.

What’s more, traditional systems can struggle to store and perform the required analytics to gain understanding from the contents of these logs because much of the information being generated doesn’t lend itself to traditional database technologies. In my experience, although some companies are moving down the path, by and large, most are just beginning to understand the opportunities of Big Data.

Quite simply, variety represents all types of data—a fundamental shift in analysis requirements from traditional structured data to include raw, semi-structured, and unstructured data as part of the decision-making and insight process. Traditional analytic platforms can’t handle variety. However, an organization’s success will rely on its ability to draw insights from the various kinds of data available to it, which includes both traditional and non-traditional.

When we look back at our database careers, sometimes it’s humbling to see that we spent more of our time on just 20 percent of the data: the relational kind that’s neatly formatted and fits ever so nicely into our strict schemas. But the truth of the matter is that 80 percent of the world’s data (and more and more of this data is responsible for setting new velocity and volume records) is unstructured, or semi-structured at best. If you look at a Twitter feed, you’ll see structure in its JSON format—but the actual text is not structured, and understanding that can be rewarding.

Video and picture images aren’t easily or efficiently stored in a relational database, certain event information can dynamically change (such as weather patterns), which isn’t well suited for strict schemas, and more. To capitalize on the Big Data opportunity, enterprises must be able to analyze all types of data, both relational and non-relational: text, sensor data, audio, video, transactional, and more.


3. The Velocity of Data

Just as the sheer volume and variety of data we collect and the store has changed, so, too, has the velocity at which it is generated and needs to be handled. A conventional understanding of velocity typically considers how quickly the data is arriving and stored, and its associated rates of retrieval. While managing all of that quickly is good—and the volumes of data that we are looking at are a consequence of how quickly the data arrives.
To accommodate velocity, a new way of thinking about a problem must start at the inception point of the data. Rather than confining the idea of velocity to the growth rates associated with your data repositories, we suggest you apply this definition to data in motion: The speed at which the data is flowing.
velocity of data

After all, we’re in agreement that today’s enterprises are dealing with petabytes of data instead of terabytes, and the increase in RFID sensors and other information streams has led to a constant flow of data at a pace that has made it impossible for traditional systems to handle. Sometimes, getting an edge over your competition can mean identifying a trend, problem, or opportunity only seconds, or even microseconds, before someone else.

In addition, more and more of the data being produced today has a very short shelf-life, so organizations must be able to analyze this data in near real-time if they hope to find insights in this data. In traditional processing, you can think of running queries against relatively static data: for example, the query “Show me all people living in the ABC flood zone” would result in a single result set to be used as a warning list of an incoming weather pattern. With streams computing, you can execute a process similar to a continuous query that identifies people who are currently “in the ABC flood zones,” but you get continuously updated results because location information from GPS data is refreshed in real-time.

Dealing effectively with Big Data requires that you perform analytics against the volume and variety of data while it is still in motion, not just after it is at rest. Consider examples from tracking neonatal health to financial markets; in every case, they require handling the volume and variety of data in new ways.


End Notes

You can’t afford to sift through all the data that’s available to you in your traditional processes; it’s just too much data with too little known value and too much of a gambled cost. Big Data platforms give you a way to economically store and process all that data and find out what’s valuable and worth exploiting. What’s more, since we talk about analytics for data at rest and data in motion, the actual data from which you can find value is not only broader, but you’re able to use and analyze it more quickly in real-time.

I recommend you go through these articles to get acquainted with tools for big data-

Let us know your thoughts in the comments below.



Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data.

About the Author

Siddharth Sonkar

Our Top Authors

Download Analytics Vidhya App for the Latest blog/Article

One thought on "What is Big Data? A Quick Introduction for Analytics and Data Engineering Beginners"

Utthunga Technologies
Utthunga Technologies says: November 26, 2020 at 1:04 pm
Very Good Information blog Keep Sharing like this Thank You Reply

Leave a Reply Your email address will not be published. Required fields are marked *