From molecules to the Milky Way: dealing with the data deluge

November 07, 2007

Most people have a few gigabytes of files on their PC. In the next decade, astronomers expect to be processing 10 million gigabytes of data every hour from the Square Kilometre Array telescope.

And with DNA sequencing getting cheaper, scientists will be data mining possibly hundreds of thousands of personal human genome databases, each of 50 gigabytes.

CSIRO has a new research program aimed at helping science and business cope with masses of data from areas like astronomy, gene sequencing, surveillance, image analysis and climate modelling.

The research program, which began this year, is called 'Terabyte Science' and is named for the data sets that start at terabytes (thousands of gigabytes) in size, which are now commonplace.

"CSIRO recognises that, for its science to be internationally competitive, the organisation needs to be able to analyse large volumes of complex, even intermittently available, data from a broad range of scientific fields," says program leader, Dr John Taylor, from CSIRO Mathematical and Information Sciences.

One aspect of the problem is that methods that work with small data sets don't necessarily work with large ones.

An aim of the program is to develop completely new mathematical approaches and processes for scientists in a range of disciplines to further their research and boost Australia's position as a world science leader.

"Large and complex data is emerging almost everywhere in science and industry and it will hold back Australian research and business if it isn't dealt with in a timely way," Dr Taylor says.

Countries like the US also recognise the challenges, as Dr Taylor has seen first hand in his ten years' working in laboratories there.

"This will need major developments in computer infrastructure and computational tools. It involves IT people, mathematicians and statisticians, image technologists, and other specialists from across CSIRO all working together in a very focussed way," he says.

After a workshop in September, specific research areas have been identified and projects are progressing in advanced manufacturing, high throughput image analysis, modelling ocean biogeochemical cycles, situation analysis and environmental modelling.

CSIRO Australia

Related Astronomy Articles from Brightsurf:

Spitzer space telescope legacy chronicled in Nature Astronomy
A national team of scientists Thursday published in the journal Nature Astronomy two papers that provide an inventory of the major discoveries made possible thanks to Spitzer and offer guidance on where the next generation of explorers should point the James Webb Space Telescope (JWST) when it launches in October 2021.

New technology is a 'science multiplier' for astronomy
A new study has tracked the long-term impact of early seed funding obtained from the National Science Foundation on many key advances in astronomy over the past three decades.

Powerful new AI technique detects and classifies galaxies in astronomy image data
Researchers at UC Santa Cruz have developed a powerful new computer program called Morpheus that can analyze astronomical image data pixel by pixel to identify and classify all of the galaxies and stars in large data sets from astronomy surveys.

Astronomy student discovers 17 new planets, including Earth-sized world
University of British Columbia astronomy student Michelle Kunimoto has discovered 17 new planets, including a potentially habitable, Earth-sized world, by combing through data gathered by NASA's Kepler mission.

Task force recommends changes to increase African-American physics and astronomy students
Due to long-term and systemic issues leading to the consistent exclusion of African-Americans in physics and astronomy, a task force is recommending sweeping changes and calling for awareness into the number and experiences of African-American students studying the fields.

How to observe a 'black hole symphony' using gravitational wave astronomy
New research led by Vanderbilt astrophysicist Karan Jani presents a compelling roadmap for capturing intermediate-mass black hole activity.

Graphene sets the stage for the next generation of THz astronomy detectors
Researchers from Chalmers University of Technology have demonstrated a detector made from graphene that could revolutionize the sensors used in next-generation space telescopes.

3D holograms bringing astronomy to life
Scientists unravelling the mysteries of star cluster formation have taken inspiration from a 19th century magic trick, to help explain their work to the public.

The vibrating universe: Making astronomy accessible to the deaf
Astronomers at the University of California, Riverside, have teamed with teachers at the California School for the Deaf, Riverside, or CSDR, to design an astronomy workshop for students with hearing loss that can be easily used in classrooms, museums, fairs, and other public events.

Prehistoric cave art reveals ancient use of complex astronomy
As far back as 40,000 years ago, humans kept track of time using relatively sophisticated knowledge of the stars

Read More: Astronomy News and Astronomy Current Events is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to