[Answer] Has “Big Data” Already Become Obsolete?

[Answer] Has “Big Data” Already Become Obsolete?

Rather than saying that “big data” has become obsolete, it may be more accurate to say that it has not truly begun yet. As long as Moore’s Law continues to hold—if electronic technology keeps doubling roughly every 18 months—then the era of big data is still only on its way. The reason is that, as computing technology continues to improve and storage costs continue to fall, people gain more and more capacity to collect larger amounts of data and conduct more fine-grained analysis. However, in traditional data analysis, once the amount of data reaches a certain scale, the results no longer improve further.

Take the simplest example of linear classification. Imagine two types of balls scattered across a plane—red balls and blue balls. We try to separate the two groups as well as possible by drawing a straight line, and then use that line to judge the color of a newly added ball (which may even be wrapped up so its color cannot be seen directly). It is easy to see that because the classification model is extremely simple—just a single straight line—massive amounts of data may not do much to improve the model’s accuracy. This is also the problem traditional data science has run into. The main bottleneck in machine learning—the primary analytical tool of data science—lies here as well: in such cases, more data does not have much additional value.

Deep learning has broken through this bottleneck. Put simply, this method analyzes data through multiple layers and many computational operators, making it possible to build sufficiently complex models and thereby improve analytical power. This approach is also called a neural network, because each operator is tiny and interconnected like a neuron. Of course, the field itself does not really carry a biomimetic meaning; it merely looks somewhat similar to neural structures. Under this learning paradigm, larger datasets can usually lead to higher accuracy, and there is even the possibility that improvements in quantity may produce a qualitative leap in performance. As a result, data scientists’ demand for data has risen sharply, and big-data science has emerged accordingly.

One criticism of deep learning is that, as models become more complex, people can no longer understand the standards by which machines classify things as easily as they can understand a straight line. When there is a black box of understanding, machine learning begins to look like witchcraft in the eyes of some people. For example, if we provide a model with a set of strong essays and weaker essays, after learning from them the machine may be able to score new essays. But those scores are based only on patterns learned from the earlier samples, and the machine cannot provide a detailed explanation for why it gave a particular score. That greatly reduces trust in the result. Recently, however, there has been progress in explaining the principles behind deep learning algorithms, and this may be the first step toward turning deep learning from “witchcraft” into a science with theoretical support.

In any case, with the rapid development of deep learning, big data has likely only just lifted a corner of the curtain; it is still far from having fully arrived. And as deep learning and artificial intelligence—the latter often built on the former—continue to advance rapidly, the scale of data demand will only keep growing. Perhaps only then will the true “era of big data” finally begin.