Dr. Luisa Rebull, a Datanaut from the new spring 2018 class, contributed this blog post. She writes from Caltech in sunny California:
NASA archives an awful lot of data in general, of course, but why are there archives of NASA's astronomy data? The main reason is, of course, that there is a lot of blood, sweat, tears, and money that have gone into acquiring those data, and we don't want to lose the data! But today's post is on the wider question of why we havearchives of astronomy data. Lots of the issues I discuss here are also in the February 05, 2017 episode of the Spacepod podcast.
Why are there archives at all? You have some sense of the larger problem if you take a lot of photos. You probably have photos that you took on your cell phone. But what about your partner's or kid's cell phone photos of the same event? And do you have a separate digital camera? Does your partner/kid also have such a camera? Then you might start looking at the files you have amassed on your computer(s). These pictures were taken on this date. Or were they just unloaded on this date? Or was it unloaded to this computer on this date, and that computer on this other date? Wait, whose camera was this? The files have a different numbering system. What if you find an SD disk floating around the bottom of your camera bag or purse? Was that unloaded? Do you have a backup of the photos from the cell phone you just accidentally dropped on the pavement? Maybe your Windows machine just got hijacked by one of those 'ransom' viruses and unless you pay the hackers off, you will lose your last six months of photos. Every time you get a new computer, it takes longer and longer to transfer all your photos.
Even if/when you have sorted all of these kinds of organizational things, you still have problems. Where is the photo you need to find now of this specific event that might have happened in the first six months of 2010? Here in Southern California, the weather is often nice, so remembering that it was a sunny day or that you decided not to take a jacket that day doesn't help that much to weed down the list of possible months in which you should look.
An astronomy archive has to not only keep all the data, and keep it safely, on upgraded operating systems and functioning disks, but also make it easy to search and find what you need quickly. And also, help you when you get stuck.
An astronomy archive has three main purposes, on which I elaborate below:
Archives also need to support both observation planning and mission planning. Particularly in the context of NASA astronomy archives, new mission planning is happening all the time. If you want to plan for future missions, you need to learn from past missions.
There is more science in the archives waiting to be done. We are still learning things from an old NASA mission called IRAS that flew in the 1980s; it was the first all-sky infrared survey. And yet, those data are still being used for new discoveries. New science discoveries are lurking in these archives. You need good tools to make those discoveries. Archives need to continuously develop new tools for working with the data, new ways for filtering the data, new ways to merge data sets (say, across wavelengths or even archives), and (note that this is an important bit) continue to provide user support by experts. It's not just a matter of throwing the data on the web and letting the world have at it. It's having tools to make it easy and lots of documentation that tells you how to use the data, and providing a helpdesk to which you can submit even really detailed questions when you have them.
IRSA is literally running towards petabytes of data. (1000 GB = TB, a terabyte; 1000 TB= PB, a petabyte). We have (invoking my inner Carl Sagan) billions and billions of rows in databases (about 200 billion as I type this).
IRSA has so much data that 10% of all the world-wide refereed journal articles use data that come, at least in part, from data housed at IRSA. That means that probably at least one out of every 10 professional astronomers are working with data right now that originally came from IRSA.
IRSA holds data from, among other missions, the Spitzer Space Telescope. By tracking Spitzer publications, we know that half of the refereed publications from Spitzer come not from the people who originally requested the data, but from people who accessed the data via the archive. Hubble's archive finds a similar result for Hubble. Archives (good archives!) can literally double the scientific productivity of a mission. That's a really powerful statement.
IRSA, as well as other archives, are always developing new tools and new ways of manipulating data. Archives are currently undergoing a transformation from "give me the data and let me take it home to process it" (which is where we've been for decades) to "let me do some processing at the archive and just send me home with part of it." Databases are now getting so large that some part of the analysis needs to be done near the data, e.g., at the archive. These new powerful tools will help everyone (professional astronomers, educators, amateurs) make better use of the archives.
There are other NASA astrophysics archives too -- I'll just list them here for now, and maybe I'll get a chance to expand upon them in future blog posts.
Ronnie has been enthusiastically showcasing NASA data as a member of NASA's Open Data team since 2013. She supports NASA's open source efforts by helping to curate and administrate datasets on NASA's Open Data Portal and Open Source Code Catalog, managing citizen and internal requests for NASA data, contributing to the Space Data Daily Open NASA blog, teaching Datanauts courses, and coordinating logistics and data support for the International Space Apps Challenge hackathons.