{"id":16111,"date":"2023-06-23T12:00:53","date_gmt":"2023-06-23T10:00:53","guid":{"rendered":"https:\/\/blog.rwth-aachen.de\/itc\/?p=16111"},"modified":"2026-02-10T16:26:19","modified_gmt":"2026-02-10T15:26:19","slug":"archivmigration-projektabschluss","status":"publish","type":"post","link":"https:\/\/blog.rwth-aachen.de\/itc\/en\/2023\/06\/23\/archivmigration-projektabschluss\/","title":{"rendered":"Archive Migration \u2013 Project Completion"},"content":{"rendered":"<div class=\"twoclick_social_bookmarks_post_16111 social_share_privacy clearfix 1.6.4 locale-en_US sprite-en_US\"><\/div><div class=\"twoclick-js\"><script type=\"text\/javascript\">\/* <![CDATA[ *\/\njQuery(document).ready(function($){if($('.twoclick_social_bookmarks_post_16111')){$('.twoclick_social_bookmarks_post_16111').socialSharePrivacy({\"txt_help\":\"Wenn Sie diese Felder durch einen Klick aktivieren, werden Informationen an Facebook, Twitter, Flattr, Xing, t3n, LinkedIn, Pinterest oder Google eventuell ins Ausland \\u00fcbertragen und unter Umst\\u00e4nden auch dort gespeichert. N\\u00e4heres erfahren Sie durch einen Klick auf das <em>i<\\\/em>.\",\"settings_perma\":\"Dauerhaft aktivieren und Daten\\u00fcber-tragung zustimmen:\",\"info_link\":\"http:\\\/\\\/www.heise.de\\\/ct\\\/artikel\\\/2-Klicks-fuer-mehr-Datenschutz-1333879.html\",\"uri\":\"https:\\\/\\\/blog.rwth-aachen.de\\\/itc\\\/en\\\/2023\\\/06\\\/23\\\/archivmigration-projektabschluss\\\/\",\"post_id\":16111,\"post_title_referrer_track\":\"Archive+Migration+%E2%80%93+Project+Completion\",\"display_infobox\":\"on\"});}});\n\/* ]]> *\/<\/script><\/div><p><div id=\"attachment_16126\" style=\"width: 310px\" class=\"wp-caption alignright\"><a href=\"https:\/\/blog.rwth-aachen.de\/itc\/files\/2023\/06\/12085316_20944142-1-scaled.jpg\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-16126\" class=\"size-medium wp-image-16126\" src=\"https:\/\/blog.rwth-aachen.de\/itc\/files\/2023\/06\/12085316_20944142-1-300x200.jpg\" alt=\"Symbol image for project completion archive migration\" width=\"300\" height=\"200\" srcset=\"https:\/\/blog.rwth-aachen.de\/itc\/files\/2023\/06\/12085316_20944142-1-300x200.jpg 300w, https:\/\/blog.rwth-aachen.de\/itc\/files\/2023\/06\/12085316_20944142-1-1024x683.jpg 1024w, https:\/\/blog.rwth-aachen.de\/itc\/files\/2023\/06\/12085316_20944142-1-768x512.jpg 768w, https:\/\/blog.rwth-aachen.de\/itc\/files\/2023\/06\/12085316_20944142-1-1536x1024.jpg 1536w, https:\/\/blog.rwth-aachen.de\/itc\/files\/2023\/06\/12085316_20944142-1-2048x1365.jpg 2048w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><p id=\"caption-attachment-16126\" class=\"wp-caption-text\">Source: <a href=\"https:\/\/de.freepik.com\/vektoren-kostenlos\/organisiertes-archiv-dateien-in-der-datenbank-suchen_12085316.htm#query=forschungsdatentransfer%20illustriert&amp;position=6&amp;from_view=search&amp;track=ais\">Freepik<\/a><\/p><\/div><\/p>\n<p>&#8220;It is done.&#8221; These are the words we can now say about the &#8220;Archive Migration&#8221; project. After more than two and a half years, the last data from the legacy system was migrated to the digital archive or Coscine last week. Now, with the completion of the project, we can review the time and evaluate the progress.<\/p>\n<p><!--more--><\/p>\n<h3><span style=\"color: #00549f;\">The Beginnings<\/span><\/h3>\n<p>The first concept about the upcoming migration was presented in December 2020 and discussed within the IT Center. It quickly became clear that this project would require more than one department to handle the tasks at hand. Based on the original focus on the migration of research data, the project was placed in the &#8220;Research Data Management&#8221; group of the &#8220;IT-PFL&#8221; department (now PDSL). The technical implementation was carried out by the department &#8220;Systems &amp; Operations&#8221;, the public relations work and the communication with the users by the department &#8220;Service &amp; Communication&#8221;. The migration project was presented to various internal and external committees in February 2021, marking the start of the project. Divided into five subprojects, the various work packages could thus begin: Subproject 1 dealt with the concrete, technical migration of the data from the legacy system. Subproject 2 was responsible for the interface to the digital archive (which did not yet exist at this early stage). Subproject 3 created the form that users had to use to classify their archive nodes. Subproject 4 was responsible for the connection to <a href=\"https:\/\/www.coscine.de\/en\/\">Coscine<\/a> and subproject 5 was responsible for the direct and indirect communication with the users.<\/p>\n<h3><span style=\"color: #00549f;\">Milestones and Difficulties Reached Along the Way<\/span><\/h3>\n<p>With the conversion of the TSM archive to read-only access at the beginning of December 2021, an important milestone was reached: From now on, users could classify their archive nodes, which specifically meant that important metadata had to be stored to be able to determine the future storage location. Thus, research data were migrated to Coscine and data from courses and other data were migrated to the digital archive. In parallel, the interfaces to the target systems were tested with the preceding migration of &#8220;simpleArchive&#8221;. Already at this point, it became apparent that the targeted schedule would be difficult to meet.<\/p>\n<p>When we started the actual migration of data in the summer of 2022, we never imagined that we would be in a constant process of developing and adapting the scripts used for the migration to merely transfer data from one system to another until the very end. To communicate the ongoing issues and challenges of the project to the waiting archive node owners, we decided to communicate the circumstances quite openly and transparently in a <a href=\"https:\/\/blog.rwth-aachen.de\/itc\/en\/2022\/09\/21\/projektverzug-archivmigration\/\">blog post<\/a> in September 2022. We also gave users the opportunity to learn directly about the status of the project and their archive node with the publicly accessible reporting page. Internally, we expanded the project group and mobilized various resources at different levels: With more employees, we were able to perform a manual migration of the data for problematic nodes, while at the same time working intensively on the further development of the script for the automated migration. Similarly, the virtual machines were increased to over two dozen to migrate many archive nodes in parallel. This way of working was maintained until the end. Finally, when <a href=\"https:\/\/blog.rwth-aachen.de\/itc\/en\/2023\/01\/20\/alles-hat-ein-ende-auch-das-tsm-backup\/\">TSM backup<\/a> was switched to &#8220;read only&#8221; in January 2023, the limited number of tape drives could be used exclusively for migration, which significantly accelerated progress.<\/p>\n<h3><span style=\"color: #00549f;\">The Groups of People<\/span><\/h3>\n<p>During the project, several groups of people were created and became directly involved in the project. In addition to the actual project team, there was the &#8220;stakeholder group&#8221;, consisting of people from different institutions and with different expertise, who reviewed our processes with their view from the outside and provided important feedback. Throughout the entire duration, the &#8220;project advisory board&#8221; formed an important body, which had the task of informing the project management in regular meetings and participated in decisions on necessary process adjustments.<\/p>\n<p>In addition to the process described, the numbers also reflect the size of this mammoth project: the old archive stored data with a size of over 1.7 PB, distributed over more than 1000 nodes. Of these, half of the nodes were not migrated (about 260 TB) because the users did not want to migrate, or the necessary metadata was not provided. The remaining data is distributed as follows:<\/p>\n<div id=\"attachment_16131\" style=\"width: 310px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/blog.rwth-aachen.de\/itc\/files\/2023\/06\/Zahlen-ArMi_en.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-16131\" class=\"size-medium wp-image-16131\" src=\"https:\/\/blog.rwth-aachen.de\/itc\/files\/2023\/06\/Zahlen-ArMi_en-300x120.png\" alt=\"Numbers of the archive migration project\" width=\"300\" height=\"120\" srcset=\"https:\/\/blog.rwth-aachen.de\/itc\/files\/2023\/06\/Zahlen-ArMi_en-300x120.png 300w, https:\/\/blog.rwth-aachen.de\/itc\/files\/2023\/06\/Zahlen-ArMi_en.png 476w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><p id=\"caption-attachment-16131\" class=\"wp-caption-text\">Source: Own illustration<\/p><\/div>\n<p>In total, more than 600,000,000 objects at almost 1.5 PB were migrated from the TSM archive to Coscine or to the digital archive.<\/p>\n<h3><span style=\"color: #00549f;\">A Heartfelt Thank You<\/span><\/h3>\n<p>This project could not have been accomplished without the active support of the many project participants. Our heartfelt thanks go first and foremost to the members of the project group, who maintained the necessary stamina to migrate even the last bits and bytes into the target systems. Likewise, a big thank you goes to the project advisory board for their very helpful feedback and always constructive discussions. We would especially like to thank the members of the &#8220;Stakeholder Group&#8221;, who gave us the users&#8217; perspective on the project. Finally, we would like to thank the archive users themselves for their patience and understanding that \u00a0as is so often the case not everything always goes smoothly with projects of this size. We have learned a lot for the next migration.<\/p>\n<p>_________________________________<\/p>\n<p>&nbsp;<\/p>\n<p>Responsible for the content of this article are <a href=\"https:\/\/www.itc.rwth-aachen.de\/cms\/it-center\/IT-Center\/Profil\/Team\/~epvp\/Mitarbeiter-CAMPUS-\/?gguid=0x741F3A251551044BB9047AF649DED3B4&amp;allou=1\">Lukas C. Bossert<\/a> and <a href=\"https:\/\/www.itc.rwth-aachen.de\/cms\/it-center\/IT-Center\/Profil\/Team\/~epvp\/Mitarbeiter-CAMPUS-\/?gguid=0xB3655D4C7CF8E5418848BB29AC4CA7E8&amp;allou=1\">Sascha B\u00fccken<\/a><\/p>\n<p>&nbsp;<\/p>","protected":false},"excerpt":{"rendered":"<p>Sorry, this entry is only available in Deutsch.<\/p>\n","protected":false},"author":3675,"featured_media":16113,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"c2c_always_allow_admin_comments":false,"footnotes":""},"categories":[306,1574,316,315],"tags":[209,459,43,46,355,56,714],"class_list":["post-16111","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ankuendigungen","category-fdm","category-projekte-kooperationen","category-services-support","tag-archive-migration","tag-archivknoten","tag-archivmigration","tag-coscine","tag-digitalarchiv","tag-projekt","tag-tsm"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/blog.rwth-aachen.de\/itc\/en\/wp-json\/wp\/v2\/posts\/16111","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.rwth-aachen.de\/itc\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.rwth-aachen.de\/itc\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.rwth-aachen.de\/itc\/en\/wp-json\/wp\/v2\/users\/3675"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.rwth-aachen.de\/itc\/en\/wp-json\/wp\/v2\/comments?post=16111"}],"version-history":[{"count":4,"href":"https:\/\/blog.rwth-aachen.de\/itc\/en\/wp-json\/wp\/v2\/posts\/16111\/revisions"}],"predecessor-version":[{"id":16133,"href":"https:\/\/blog.rwth-aachen.de\/itc\/en\/wp-json\/wp\/v2\/posts\/16111\/revisions\/16133"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.rwth-aachen.de\/itc\/en\/wp-json\/wp\/v2\/media\/16113"}],"wp:attachment":[{"href":"https:\/\/blog.rwth-aachen.de\/itc\/en\/wp-json\/wp\/v2\/media?parent=16111"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.rwth-aachen.de\/itc\/en\/wp-json\/wp\/v2\/categories?post=16111"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.rwth-aachen.de\/itc\/en\/wp-json\/wp\/v2\/tags?post=16111"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}