IvoaDCP < IVOA

---+ IVOA Data Curation and Preservation

Chair : [[AndreSchaaff][Andre Schaaff]] Vice-Chair : [[TimJenness][Tim Jenness]]

Chairs Emeritus : [[FrancoiseGenova][Francoise Genova]], [[AlbertoAccomazzi][Alberto Accomazzi]], [[BobHanisch][Bob Hanisch]] and [[ReaganMoore][Reagan Moore]]

June 2018: following the nomination of the new chair and vice-chair the updating of this page is on going

---

%TOC%
---

---++ Charter

The Data Curation and Preservation (DCP) Interest Group is established to share best practices and engage IVOA member projects in the long-term curation and preservation of astronomical data. Discussion topics include:

   * Identification of at-risk data and data collections
   * Processes for engaging the community in curation and preservation activities
   * Coordination with strategic partners such as astronomical data centers, research libraries, and scholarly publishers
   * Technology reviews concerning all aspects of curation and preservation (metadata, provenance, data integrity, media migration, replication, disaster recovery, assessment criteria, etc.)
The Interest Group will from time to time produce white papers and/or bring proposed actions to the attention of the IVOA Technical Coordination Group and IVOA Executive.

---++ Mailing-list and membership
   * General working-group discussion [[http://www.ivoa.net/pipermail/datacp/][archive]].
   * Anyone interested in data curation and preservation is welcome to participate.

<!-- People who are currently involved include Bob Hanisch, Francoise Genova, Pepi Fabbiano, Arnold Rots, Wolfgang Voges, Fabio Pasian, and Bob Mann. 

Please add your name to this list! -->

<!--
---++ Background

At the IVOA meeting #11 on 23 June 2004, a request was made for the creation of an Interest Group for Data Curation and Preservation. Reagan Moore (SDSC, USA) and Francoise Genova (CDS, France) were tasked with developing the initial description of the activities for the interest group. The goal of the group is to identify both mechanisms for the long-term preservation of astrophysics collections and sustainability procedures to ensure continued access to astrophysics collections that are at risk.

Mechanisms for long-term preservation address challenges related to:
   * Authenticity, how to guarantee that preservation information (provenance, description, administrative) will remain associated with astrophysics data, while Uniform Content Descriptors evolve and data are moved to new storage systems.
   * Integrity, how to validate that data has not been corrupted, manage the collection name spaces, and manage replicas.
   * Technology evolution, how to guarantee a collection will remain accessible while the underlying storage systems and database technologies evolve.
   * Disaster recovery, how to guarantee that a collection will remain accessible in the event of a natural catastrophe such as an earthquake or hurricane.
Sustainability procedures for continued support address challenges related to:
   * Identification of collections at risk of being lost due to lack of support.
   * Development of procedures for validating whether an at-risk collection should be preserved.
   * Development of procedures for preserving at-risk collections.
   * Development of procedures for identifying communities that will continue to support at-risk collections.
The IVOA Data Curation and Preservation interest group will develop:

(1) A white paper discussing the concepts involved in preservation, and providing a description of existing preservation environments.

(2) A white paper discussing the sustainability procedures.

A combined document will then be circulated to the IVOA as an IVOA Working Draft.

The technologies that are available to build preservation environments come from the grid and digital library communities. Examples include:
   * Storage Resource Broker data grid which provides mechanisms to assert authenticity, validate integrity, manage technology evolution, package data for storage, replicate files, and federate preservation environments.
   * integrated Rule Oriented Data System (iRODS) data grid which automates application of preservation processes. Management policies are expressed as rules that control the execution of preservation processes that are implemented through sets of micro-services. Queries on persistent state information are used to validate assertions about preservation properties such as trustworthiness, authenticity, integrity, and chain of custody.
   * Fedora which provides relationship management that can be used to track UCD evolution
   * DSpace which provide life-cycle management procedures for creating preservation metadata
   * Open Archive Initiative Protocol for Metadata Harvesting for supporting name space registries
   * Metadata Encoding and Transmission Standard for structuring metadata
   * Open Archival Information System for packaging data for preservation (Archival Information Package)
   * Federated data grids which provide deep archives for disaster recovery
   * Dataflow management systems for applying preservation procedures
One expectation is that multiple preservation models may be used, and that interoperability mechanisms will be developed for exchange of data and metadata between preservation systems. This capability will be required if for no other reason than to manage migration between different versions of technology over time.

Interface of the WG with other parts of the IVOA

Preservation environments can be assembled by federating existing collections with deep archives. This means that interactions with existing archives is essential for building a viable system.

Interactions will be needed with the Data Access Layer WG for providing uniform interfaces to the preservation environment, and with the UCD WG to provide relevant discovery information for items in the preservation environment.

Requirements from other WGs

We will seek requirements from other WGs through specification of use cases. A first use case is being defined in collaboration with Interpares (International Research on Permanent Authentic Records in Electronic Systems). The Canadian MOST image collection is being analyzed for preservation requirements. A report will be generated in 1st quarter, 2005. Other examples of interactions with IVOA working groups include integration of preservation environments with processing pipelines, and support for discovery through portals.

Currently the process to provide requirements is by email to the IVOA PE mailing list.

Namespaces

The preservation environment will use at least four name spaces for identifying files:
   * Logical file name. This is a location independent identifier that is unique within a preservation environment.
   * Global Unique Identifier - GUID. This is an identifier that is unique across existing archives.
   * Physical file name. This is the path name on the physical storage system where the data resides.
   * UCDs. Attributes will be associated with each logical file name that can be used to discover relevant data.
The preservation environment provides additional name spaces to describe:
   * Storage resources
   * Users
   * Metadata
By managing each of these name spaces independently from the underlying storage repositories and administrative domains, the preservation environment can control the authenticity and integrity of the collections.

Federation

To ensure preservation against natural disasters, operator error, and malicious users, replicas of the data must be kept with differentiated levels of access. One copy should be accessible by users, a second copy user-accessible copy should be created at a geographically remote site for fault-tolerance, and a third copy should be kept in a deep archive that is not user-accessible. Each of the copies should be managed by an independent metadata catalog. Federated data grids provide this level of support.

The project will track preservation approaches across multiple communities:
   * ERPANET - Electronic Resource Preservation and Access Network, http://www.erpanet.org/
   * DILIGENT - Digital Library Infrastructure on Grid Enabled Technology, http://www.diligentproject.org
   * Global Grid Forum - Preservation Environments Working Group, http://www.gridforum.org
   * Interpares - International Research on Permanent Authentic Records in Electronic Systems
   * Library of Congress National Digital Information Infrastructure and Preservation Program
   * NARA - National Archives and Records Association Transcontinental Persistent Archive Platform
   * UK Digital Curation Center, http://www.dcc.ac.uk/
   * CASPAR - Cultural, Artistic and Scientific knowledge for Preservation, Access and Retrieval, http://www.casparpreserves.eu/
   * PLANETS - Digital Preservation Research and Technology, http://www.planets-project.eu/
   * DPE - Digital Preservation Europe, http://www.digitalpreservationeurope.eu/

-->
---++ Previous meetings

   * May/June 2018, Victoria Interop, InterOpMay2018DCP
   * October 2017, Santiago Interop, InterOpOct2017DCP
   * May 2017, Shanghai, [[InterOpMay2017-DCP][InterOpMay2017DCP]]
   * October 2016, Trieste, InteropOct2016DCP
   * May 2016, Cape Town, [[InterOpMay2016-DCP][InterOpMay2016DCP]]
   * October/November 2015, Sydney, [[InteropOct2015DCP][InterOpOctober2015DCP]]
   * June 2015, Sexten, No session
   * October 2014,Banff, InterOpOct2014SemanticsDCP
   * May 2014, Madrid, 
   * October 2013, Hawaii, InterOpSep2013DCP
   * May 2013, Heidelberg, InterOpMay2013DCP
   * November 2012, Sao Paulo, InterOpOct2012DCP

---++ Links
   * ERPANET - Electronic Resource Preservation and Access Network, http://www.erpanet.org/
   * DILIGENT - Digital Library Infrastructure on Grid Enabled Technology, http://www.diligentproject.org
   * Global Grid Forum - Preservation Environments Working Group, http://www.gridforum.org
   * UK Digital Curation Center, http://www.dcc.ac.uk/

---++ Papers
   * Smith, M., R. Moore, &ldquo;Digital Archive Policies and Trusted Digital Repositories&rdquo;, proceedings of The 2nd International Digital Curation Conference: Digital Data Curation in Practice, November 2006, Glasgow, Scotland.
   * Moore, R., &ldquo;Building Preservation Environments with Data Grid Technology&rdquo;, American Archivist, vol. 69, no. 1, pp. 139-158, July 2006.
   * Rajasekar, A., M. Wan, R. Moore, W. Schroeder, &ldquo;A Prototype Rule-based Distributed Data Management System&rdquo;, HPDC workshop on &ldquo;Next Generation Distributed Data Management&rdquo;, May 2006, Paris, France.
   * Moore, R., J. Jaja, R. Chadduck, "Mitigating Risk of Data Loss in Preservation Environments", NASA / IEEE MSST2005, Thirteenth NASA Goddard / Twenty-second IEEE Conference on Mass Storage Systems and Technologies, April 2005.
   * Moore, R., "Persistent Collections," book chapter in "Databasing the Brain," editors S. H. Koslow and S. Subramaniam, John Wiley & Sons, 2005.
   * Moore, R., R. Marciano, "Prototype Preservation Environments", submitted to Library Trends, Dec. 2004.
   * Moore, R., A. Rajasekar, M. Wan, "Data Grids, Digital Libraries and Persistent Archives: An Integrated Approach to Publishing, Sharing and Archiving Data", submitted to IEEE, Dec. 2004
   * Moore, R., "Building Preservation Environments with Data Grid Technology", submitted to American Archivist, Oct., 2004
   * Moore, R., W. Underwood, "Preservation Environments for Digital Entities," Interpares II report, June 2004.
<!--
      * Set ALLOWTOPICRENAME = IVOA.TWikiAdminGroup
-->
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
ppt	IVOA-Kyoto.ppt	r2 r1	manage	68.5 K	2005-05-19 - 06:04	ReaganMoore	Presentation on Data Curation and Preservation
pdf	Preservation.pdf	r1	manage	46.3 K	2005-05-19 - 07:07	ReaganMoore	Interest Group meeting notes
This topic: IVOA > WebHome > IvoaDCP
Topic revision: r37 - 2018-07-05 - AndreSchaaff