Big data poses great challenges and opportunities for databases

April 20, 2014

Advances in the technology frontier have resulted in major disruptions and transformations in the massive data processing infrastructures. For the past three decades, classical database management systems, data warehousing and data analysis technologies have been well recognized as effective tools for data management and analysis. More recently, data from different sources and in different format are being collected at unprecedented scale. This gives rise to the so-called 3V characteristics of the big data: volume, velocity and variety. Classical approaches of data warehousing and data analysis are no longer viable to deal with both the scale of data and the sophisticated analysis. This challenge has been labeled as the 'big data' problem. In principle, while earlier DBMSs focused on modeling operational characteristics of enterprises, big data systems are now expected to model vast amounts of heterogeneous and complex data.

Although the massive data pose many challenges and invalidate earlier designs, they provide many great opportunities, and most of all, instead of making decisions based on small sets of data or calibration, decisions can now be made based on the data itself. Various big data applications have emerged, such as Social networking, Enterprise data management, Scientific applications, Mobile computing, Scalable and elastic data management, Scalable data analytics, etc . Meanwhile, many distributed data processing frameworks/systems have been proposed to deal with big data problem. MapReduce is the most successful distributed computing platform whose fundamental idea is to simplify the parallel processing, and has been widely applied. MapReduce systems are good at complex analytics and extract-transform-load tasks at large scale, however it also suffers from its reduced functionality. There also exist many other distributed data processing systems that go beyond the MapReduce framework. These systems have been designed to address various problems not well handled by MapReduce, e.g., Dremel for Interactive analysis, GraphLab for Graph analysis, STORM for stream processing, Spark for memory computing.

The big data presents us the challenges and opportunities in designing new data processing systems for managing and processing the massive data. The potential research topics in this field lie in all phases of data management pipeline that includes data acquisition, data integration, data modeling, query processing, data analysis, etc. Besides, the big data also brings great challenges and opportunities to other computer science disciplines such as system architecture, storage system, system software and software engineering.
See the article:

Big data: the driver for innovation in databases

National Science Review,Volume 1, Issue 1,Pp. 27-30. doi:10.1093/nsr/nwt020

Science China Press

Related Big Data Articles from Brightsurf:

Predicting sports performance with "big data"
Smartphones and wearable devices are not simple accessories for athletes.

Big data could yield big discoveries in archaeology, Brown scholar says
Parker VanValkenburgh, an assistant professor of anthropology, curated a journal issue that explores the opportunities and challenges big data could bring to the field of archaeology.

Army develops big data approach to neuroscience
A big data approach to neuroscience promises to significantly improve our understanding of the relationship between brain activity and performance.

'Big data' for life sciences
Scientists have produced a co-regulation map of the human proteome, which was able to capture relationships between proteins that do not physically interact or co-localize.

Molecular big data, a new weapon for medicine
Being able to visualize the transmission of a virus in real-time during an outbreak, or to better adapt cancer treatment on the basis of the mutations present in a tumor's individual cells are only two examples of what molecular Big Data can bring to medicine and health globally.

Big data says food is too sweet
New research from the Monell Center analyzed nearly 400,000 food reviews posted by Amazon customers to gain real-world insight into the food choices that people make.

Querying big data just got universal
A universal query engine for big data that works across computing platforms could accelerate analytics research.

What 'Big Data' reveals about the diversity of species
'Big data' and large-scale analyses are critical for biodiversity research to find out how animal and plant species are distributed worldwide and how ecosystems function.

Big data takes aim at a big human problem
A James Cook University scientist is part of an international team that's used new 'big data' analysis to achieve a major advance in understanding neurological disorders such as Epilepsy, Alzheimer's and Parkinson's disease.

Small babies, big data
The first week of a newborn's life is a time of rapid biological change as the baby adapts to living outside the womb, suddenly exposed to new bacteria and viruses.

Read More: Big Data News and Big Data Current Events is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to