Petabytes

Science magazine has a nice article about dark energy by Adrian Cho. But you can't read it unless you subscribe. Except that the nice folks at UC Davis have decided that the article is nice publicity for Tony Tyson and the Large Synoptic Survey Telescope, so they've put the article online for free. See Mark's post for some of the theoretical background. The LSST is an ambitious project -- a proposed giant telescope with a wide-field camera that scans the sky in real time. Every three nights it will complete a survey of the visible sky, providing unprecedented access to astrophysical phenomena in the time domain -- supernovae, asteroids, variable stars, you name it. It will probe dark energy in at least two ways: using Type Ia supernovae as standard candles (which is how the acceleration of the universe was first discovered), and by measuring cosmic structure via weak gravitational lensing. One of the great challenges of the project is the huge amount of data it will produce. We are talking about a petabyte of data per year (pdf) -- about the size of the entire internet archive. To search such a database for some string of characters (say, using "grep") would take several years! It's a tremendous intellectual challenge just to design the ways that such data can be usefully arranged so that we can find what we're looking for. As you might guess, expertise from people like Google is turning out to be very valuable. In fact, the value goes both ways. It turns out that computer companies love to play with astrophysical data, for simple reasons -- it's publicly available, and worth nothing on the open market. We like to think that the data has a loftier kind of worth.