The State of Big Data in 2018

In 2018 big data is everywhere: in your pocket, in your wallet, in your car, at home, at work, at the gym—the list goes on ad infinitum.

No one knows the exact size of the world’s pool of data but it is predicted to be in the hundreds of zettabytes (trillion gigabytes). And data is growing exponentially: Cisco estimates that by the end of 2019, the internet of things (IoT) alone will generate more than 500 zettabytes of data per year.

So there are mind-boggling amounts of data out there and this pool is growing exponentially into even more mind-boggling sums. So what?

Well, for one thing, the value of the data extracted from this pool is also growing rapidly. Worldwide revenues for big data and business analytics are set to grow at an average of 12% a year to over $200 billion in 2020, according to IDC.

To give a more tangible example, consider the popular new startup MoviePass. How is it possible for MoviePass to make money allowing customers to see three movies a month for $10—less than the price of one movie ticket?

In a word, data. MoviePass’ parent company, Helios & Matheson, is betting that they can sell data on what movies consumers see at what times to studios and film distributors.

Meanwhile, uses for big data are proliferating. Healthcare, physical sciences, education and retail are starting to dive into the pool of big data. To give one example, data scientists and doctors are developing algorithms with the ability to diagnose diseases.

Worldwide revenues for big data and business analytics are set to grow at an average of 12% a year to over $200 billion in 2020, according to IDC. Tweet This

 
franki-chamaki-682112-unsplash.jpg
 

Democratization of big data

In the past, big data was the sole domain of data scientists because collecting and crunching big data sets required coding skills. Now, however, thanks to the growth in APIs and software-as-a-service (SaaS) data analytics tools, anyone can extract insights from big data.

As revenues from data rise, the costs involved with storing and crunching data continue to fall. Cloud storage, for example, seems to follow what’s known as “Bezos’s Law”: that storage costs fall by 50% every three years.

Meanwhile the costs of building your own data center are lower than ever before. Here at Evo, we found it much cheaper to host our clients’ data by building our own data repository.

Overall, these lower costs and increased services have allowed more small businesses and startups to take advantage of big data.

AI and machine learning

We often hear about AI used for understanding natural language and allowing machines to recognize the world around them. But AI does much more.

Artificial intelligence (AI) and machine learning continue to improve the collection and interpretation of big data. AI can now help with data cleanup and organization, an extremely time-consuming task for data scientists. It can also automatically discover patterns in data, showing data scientists correlations they hadn’t previously considered.

Here at Evo, we employ AI and machine learning for dynamic pricing—automated price suggestions to optimize profits or revenue. We also use it in our inventory allocation engine that optimizes the distribution and redistribution of inventory among stores.

 
mathew-schwartz-792855-unsplash.jpg
 

Big data security

Stakes are high for keeping data safe. According to a Deloitte survey, 73% of consumers would reconsider using a company if it failed to keep their data safe. As a result, the data security market is expected to grow from USD 2.5 billion in 2016 to USD 7 billion by 2022.

With all the growth and improvements in big data comes increasing security risks. As big data use democratizes and more non-experts get direct access to data, privacy and security breaches become more likely.

The proliferation of IoT devices is also a significant security concern because it allows for many more points of attack. Moreover, being new, IoT’s security infrastructure is not yet well developed.

A growing proportion of data security is now handled by AI, which has a number of advantages over human security teams. AI defense systems can quickly and cheaply process years of attack history, learn various defense strategies and search for anomalies in user behavior.

AI is also significantly less expensive than a team of security professionals.

It’s important, however, to remember that AI is only as intelligent as the data it has to learn from. With 90% of the total data pool produced in just the last two years, it’s clear that data is exploding. Expectations for the future benefits of big data and AI should therefore be high.  


About the author

gavekal site pic.jpg

Will Freeman is a content expert at Evo.

He is a former economic journalist and part-time entrepreneur. 

His interests include economic development, China, India, cryptocurrency and blockchain, and financial technology in general.