What to do with 15 million gigabytes of data

November 03, 2008

When it is fully up and running, the four massive detectors on the new Large Hadron Collider (LHC) at the CERN particle-physics lab near Geneva are expected to produce up to 15 million gigabytes, aka 15 petabytes, of data every year. Andreas Hirstius, manager of CERN Openlab and the CERN School of Computing, explains in November's Physics World how computer scientists have risen to the challenge of dealing with this unprecedented volume of data.

When CERN staff first considered how they might deal with the large volume of data that the huge collider would produce when its two beams of protons collide, in the mid-1990s, a single gigabyte of disk space still cost a few hundred dollars and CERN's total external connectivity was equivalent to just one of today's broadband connections.

It quickly became clear that computing power at CERN, even taking Moore's Law into account, would be significantly less than that required to analyse LHC data. The solution, it transpired during the 1990s, was to turn to "high-throughput computing" where the focus is not on shifting data as quickly as possible from A to B but rather from shifting as much information as possible between those two points.

High-performance computing is ideal for particle physics because the data produced in the millions of proton-proton collisions are all independent of one another - and can therefore be handled independently. So, rather than using a massive all-in-one mainframe supercomputer to analyse the results, the data can be sent to separate computers, all connected via a network.

From here sprung the LHC Grid. The Grid, which was officially inaugurated last month, is a tiered structure centred on CERN (Tier-0), which is connected by superfast fibre links to 11 Tier-1 centres at places like the Rutherford Appleton Laboratory (RAL) in the UK and Fermilab in the US. More than one CD's worth of data (about 700 MB) can be sent down these fibres to each of the Tier-1 centres every second.

Tier 1 centres then feed down to another 250 regional Tier-2 centres that are in turn accessed by individual researchers through university computer clusters and desktops and laptops (Tier-3).

As Andreas Hirstius writes, "The LHC challenge presented to CERN's computer scientists was as big as the challenges to its engineers and physicists. The computer scientists managed to develop a computing infrastructure that can handle huge amounts of data, thereby fulfilling all of the physicists' requirements and in some cases even going beyond them."

Institute of Physics

---

---

	Present at the Creation: The Story of CERN and the Large Hadron Collider by Amir D. Aczel (Author) The Large Hadron Collider is the biggest, and by far the most powerful, machine ever built. A project of CERN, the European Organization for Nuclear Research, its audacious purpose is to re-create, in a 16.5-mile-long circular tunnel under the French-Swiss countryside, the immensely hot and dense conditions that existed some 13.7 billion years ago within the first trillionth of a second after the fiery birth of our universe. The collider is now crashing protons at record energy levels never created by scientists before, and it will reach even higher levels by 2013. Its superconducting magnets guide two beams of protons in opposite directions around the track. After accelerating the beams to 99.9999991 percent of the speed of light, it collides the protons head-on, annihilating them in a...
	Voyage to the Heart of Matter: The ATLAS Experiment at CERN (Pop-Up Books (Papadakis)) by Emma Sanders (Author), Anton Radevsky (Author) In this unique collaboration between CERN and renowned paper engineer Anton Radevsky, 7000 tonnes of metal, glass, plastic, cables and computer chips leap from the page in miniature pop-up, to tell the story of CERN's quest to understand the birth of the universe. Protons, travelling at nearly the speed of light, collide within the heart of the ATLAS detector, sending out showers of debris to recreate 40 million times a second the conditions that existed millionths of a second after the Big Bang, the event that set our universe in motion. Now all ages can join the ATLAS experiment on this fascinating journey to the beginnings of the universe in this astonishing pop-up book.
	Agent of Cern by Fiction4All hn Smith had just received his Ph.D. in organic chemistry when the Black Druid summoned him to the fantasy world of Trion. There he has to deal with alchemical magic and with swordplay, with chained and barefoot slavegirls (who insist on remaining slavegirls in order to preserve their genius-level intelligence), and - by the way - with a treasonous plot against the King of Cern.
	Collider: The Search for the World's Smallest Particles by Paul Halpern (Author) An accessible look at the hottest topic in physics and the experiments that will transform our understanding of the universeThe biggest news in science today is the Large Hadron Collider, the world's largest and most powerful particle-smasher, and the anticipation of finally discovering the Higgs boson particle. But what is the Higgs boson and why is it often referred to as the God Particle? Why are the Higgs and the LHC so important? Getting a handle on the science behind the LHC can be difficult for anyone without an advanced degree in particle physics, but you don't need to go back to school to learn about it. In Collider, award-winning physicist Paul Halpern provides you with the tools you need to understand what the LHC is and what it hopes to discover.Comprehensive, accessible guide...
	The Quantum Frontier: The Large Hadron Collider by Don Lincoln (Author) The highest-energy particle accelerator ever built, the Large Hadron Collider runs under the border between France and Switzerland. It leapt into action on September 10, 2008, amid unprecedented global press coverage and widespread fears that its energy would create tiny black holes that could destroy the earth. By smashing together particles smaller than atoms, the LHC recreates the conditions hypothesized to have existed just moments after the big bang. Physicists expect it to aid our understanding of how the universe came into being and to show us much about the standard model of particle physics—even possibly proving the existence of the mysterious Higgs boson. In exploring what the collider does and what it might find, Don Lincoln explains what the LHC is likely to teach us about...
	Andri Pol: CERN: European Organization for Nuclear Research (Places of Interest) by Andri Pol (Author) For most people locations that hold a particular importance for the development of our society and for the advancement of science and technology often remain hidden from view. They are separate and protected, such as CERN, the European Organisation for Nuclear Research, close to the city of Geneva. CERN is best known for its giant particle accelerator. Here researchers from around the world take part in a diverse array of fundamental physical research, in the pursuit of knowledge that will perhaps one day revolutionize our understanding of the universe and life on our planet. The Swiss photographer Andri Pol mixed with this multicultural community of researchers and followed their work over an extended period of time. In doing so he created a unique portrait of this fascinating...
	History of CERN, I: Volume I - Launching the European Organization for Nuclear Research by A. Hermann (Author), L. Belloni (Author), U. Mersits (Author), D. Pestre (Author), J. Krige (Author) Describing the history of CERN from its inception in the late 40's up to the mid-60's. The authors have divided these 17-18 years into roughly two successive periods. Volume I deals with the birth and official establishment of the organization and thus covers the years 1949-1954, while Volume II studies the life of the European laboratory during the first twelve years of its existence.
	LEP - The Lord of the Collider Rings at CERN 1980-2000: The Making, Operation and Legacy of the World's Largest Scientific Instrument by Herwig Schopper (Author), Rolf-Dieter Heuer (Foreword) Housed by a 4 m diameter tunnel of 27 km circumference, with huge underground labs and numerous surface facilities, and set up with a precision of 0.1 mm per kilometer, the Large Electron-Positron Collider (LEP) was not only the largest but also one of the most sophisticated scientific research instruments ever created by Man. Located at CERN, near Geneva, LEP was built during the years 1983 - 1989, was operational until 2000, and corroborated the standard model of particle physics through continuous high precision measurements. The Author, director-general of CERN during the crucial period of the construction of LEP, recounts vividly the convoluted decision-making and technical implementation processes - the tunnel alone being a highly challenging geo- and civil engineering project -...
	The Strategic Management of High Technology Contracts: The Case of CERN (Technology, Innovation, Entrepreneurship and Competitive Strategy) by Markus Nordberg (Author), Alain Verbeke (Author) High technology research laboratories are under constant pressure from the governments that support them to generate secondary utilities such as technology transfer and spin-offs. As buyers, such organisations are often used by governments to stimulate innovation by their suppliers, under tight budgetary constraints and within the rigid institutional frameworks applied to public research organisations. This book addresses the design of efficient buyer-supplier contracts within the institutional boundaries faced by the buyer and focuses in particular on vertical buyer-supplier linkages as a source of supplier core competencies in a cost- and technology-driven environment. Based on a study of manufacturing contracts commissioned by the European Laboratory for Particle Physics (CERN), the...
	Frontiers of Particle Beams: Observation, Diagnosis and Correction : Proceedings of a Topical Course Held by the Joint Us-Cern School on Particle Acc (Lecture Notes in Physics) by M. Month (Author), S. Turner (Editor)