iRODS

May 28-31, 2024

The 16th Annual iRODS User Group Meeting is your chance to meet iRODS users, Consortium members, and staff to discuss iRODS-enabled applications and discoveries, technologies powered by iRODS, and the future of iRODS and the iRODS Consortium.

We will host 25+ presentations from the user community and the core development team, including use case presentations, live demonstrations, and open discussions about requested iRODS features. We anticipate an audience of 150 participants representing dozens of academic, government, and commercial institutions.

Read through past iRODS User Group Meetings to see representative topics and breadth.

iRODS Training

On Tuesday, May 28, the iRODS Consortium Development Team will provide Beginner and Advanced Training sessions.

Beginner Training

"Getting Started with iRODS" will cover iRODS vocabulary, mental models, capabilities, and basic interactions.

Overview
Four Core Competencies
Planning an iRODS Deployment
Installation
iCommands
Virtualization
Basic Metadata
Basic Rule Engine

Advanced Training

"iRODS Care and Feeding" will cover scoping an iRODS installation, deployment, various integrations, and how they work.

Resource Hierarchies and Composition
Rule Engine Plugins
Data to Compute
Compute to Data
Automated Ingest
Storage Tiering
Indexing
Auditing
Q&A: Troubleshooting, Bottlenecks, & Performance

iRODS Troubleshooting

On Friday, May 31, the iRODS Consortium will host an iRODS Troubleshooting session where participants can get some one-on-one help with an existing or planned iRODS installation or integration.

In-Person Pricing

IN-PERSON TRAINING WORKSHOP + USER GROUP MEETING:

$550
After April 1: $650
Start Date: Tuesday, May 28
End Date: Thursday, May 30

IN-PERSON USER GROUP MEETING ONLY:

$300
After April 1: $350
Start Date: Wednesday, May 29
End Date: Thursday, May 30

OPTIONAL IN-PERSON TROUBLESHOOTING:

$20
Start Date: Friday, May 31
End Date: Friday, May 31

Virtual Pricing

VIRTUAL USER GROUP MEETING ONLY:

$100
Start Date: Wednesday, May 29
End Date: Thursday, May 30

OPTIONAL SWAG:

Key Dates

April 1: Early Bird Pricing Ends
April 1: Deadline for Abstracts for Talks/Papers
May 31: Deadline for Papers

Accepted Talks

iRODS UGM 2024 Keynote
SURF

Safeguard your sensitive data in iRODS using data encryption feature available in GoCommands
Illyoung Choi, Edwin Skidmore, Tony Edgin, and Nirav Merchant
CyVerse / University of Arizona

iRODS has been utilized across various scientific domains for storing and sharing research data. However, certain domains, such as health sciences, require stringent confidentiality measures for utilizing iRODS as their data management solution. Compliance with regulations such as HIPPA (Health Insurance Portability and Accountability Act) necessitates end-to-end encryption, meaning that data must be encrypted for both storage and transmission. While iRODS provides encryption in transit using SSL (Secure Socket Layer), the responsibility for encrypted data storage at rest lies with the iRODS infrastructure provider and presents a significant limitation. Encrypting data before uploading it and decrypting it after download adds additional steps and complexity for manual encryption and decryption process.

To address these multiple challenges and to meet the requirement for encrypted data storage, we have simplified the process of encrypting and decrypting data as built-in capability in GoCommands. GoCommands is a cross platform command-line utility for iRODS, offering data access and management capabilities similar to iCommands. The encryption at rest algorithm employs AES-256-CTR to encrypt both file names and content before uploading, thereby ensuring that data remains encrypted within iRODS. When listing and downloading encrypted files, they are automatically decrypted using the encryption key provided by the user. Since GoCommands and WinSCP share the same encryption algorithm, users who are not familiar with the command-line interface can utilize WinSCP, a GUI-based tool, making it more user-friendly. This end-to-end encryption functionality requires no server-side changes to iRODS, ensuring compatibility with existing iRODS servers.

This new data encryption feature will empower researchers to utilize iRODS for storing and accessing their sensitive data and meeting regulatory compliance, thereby facilitating broader adoption of iRODS across various research domains.

Testing iRODS-based applications: Experiences with Yoda
Claire Saliers, Lazlo Westerhof, Sietse Snel, and Sirjan Kaur
Utrecht University

Yoda is a research data management system based on iRODS that enables researchers to deposit, share, publish and preserve data. It is developed at Utrecht University, and used at multiple universities, as well as other organizations. In this contribution, we share our approach for testing Yoda and planned future work.

Our testing approach primarily revolves around automated scripted testing, encompassing unit tests, integration tests, API tests and UI tests. Additionally, we have explored use of scriptless testing and fuzzing to supplement scripted tests.

Planned future work includes improving test coverage, introducing accessibility tests and exploring ways to improve reliability of UI tests.

Drawing from our experiences, we share recommendations regarding how to improve testability and reliability of iRODS-based applications.

Streamlining iRODS: Kafka-based Data Pipelines
Peter Verraedt and Jo Wijnant
KU Leuven

We sketch how changes in the iRODS catalog can be captured in realtime into Apache Kafka, and how we materialize those master-records as a readonly 'view' in OpenSearch using Apache Flink stream processor. We show that the same tools can be used to continuously monitor project usage within iRODS. As a last application, we sketch how audit logs can be collected and processed, and show future challenges.

iRODS replication management and our future iRODS world at Wellcome Sanger Institute
Sai Poomdla
Wellcome Sanger Institute

At Wellcome Sanger Institute, we manage 20 Petabytes of genomic data using iRODS. We currently use iRODS 4.2.7 in our production (and our tales of upgrading it to a later version) and due to various scenarios, whether it might be in hiccup in the network connection or data object corruption in the transit, we end up with failure in our replica creation process leading to Single, Dirty and Triple replicas. To mitigate these issues, we have developed in-house solutions using python iRODS client. Also, we are evolving into a new iRODS world, where we want to replace our storage servers with single large file-system and allow our users directly to read from this file-system instead of downloading the files into a high-performance file system.

Sharing data in a multi-system multi-role environment centered on iRODS
Claudio Cacciari, Eduard Klapwijk, Simone Mulder, Mark Mulder, Stephan Heunis, Frans van der Zijde, and Wouter Timmer
SURF and Erasmus University Rotterdam

SURF, the cooperative association of Dutch educational and research institutions, offers data infrastructure and services to the research communities. Some of its services are based on iRODS and are often used as building blocks for data platforms. One increasingly common architectural component in those platforms is a web portal where researchers can discover data using project specific queries. Once the data are found, they are made available to the researcher, directly, for example, with a download link or indirectly, triggering a copy to a computing environment where they are analyzed. The implementation of such workflow is time consuming. Its maintenance in the long term is often jeopardized by limited support available within the project and design choices too tailored for that use case makes its adoption by other organizations too difficult. We think that it is possible to model that workflow in a generic way as a re-usable modular component and in a way flexible enough to support even the more stringent requirements associated with sensitive data. The component relies on iRODS and links together multiple web portals and repositories through an API layer based on FastAPI. We present here a proof of concept developed within the GUTS project, in collaboration with the project’s data management team and the research support.

Python package for metadata schemas
Mariana Montes and Ronny Moreas
KU Leuven

In this talk we will present a Python package to validate, write and read structured metadata based on schemas. The schemas are described in a JSON file following the specifications of the metadata schema manager developed in the context of the ManGO portal, but this implementation is independent from the web application. Given a schema, users can provide hierarchical metadata in a Python dictionary, validate its contents and add it to an iRODS data object or collection as AVUs with the appropriate namespacing. The same package allows to read back the AVUs and parse them into a Python dictionary preserving the original hierarchy.

Update: The Intersection Between Policy-Based Data Management and Emerging Health Science Data Standards
Mike Conway
NIH / NIEHS

iRODS is a domain-agnostic platform. The flexibility of iRODS flows from the fact that policy-based data management is at its core. By encoding metadata standards and management policies appropriate to a research domain into an iRODS data grid, an organization can create a powerful tool for data-driven research.

The Health Science domain is increasingly characterized by the co-location of data and compute, often 'bringing the compute to the data' and increasingly in a hybrid environment that spans cloud providers and bridges premises and cloud computing. This overview will look at GA4GH standards that intersect with the policy-based data management capabilities of iRODS for compute-to-data, access control, and federated search.

iRODS-based system turbocharged next-gen sequencing analysis during pandemic and beyond
Robert Verhagen and Erwin van Wieringen
National Institute for Public Health and the Environment (RIVM)

The National Institute for Public Health and the Environment (RIVM) has numerous projects in various scientific domains that generate next generation sequencing data. Bioinformatics plays an important role in analysing and interpreting this sequencing data. To support these analyses we developed a platform that consists of a High Performance Compute (HPC) cluster, a Linux Scientific Workspace for software development and a Data Management System (DMS) based on iRODS. On top of this DMS, we also created a Job Engine: a tightly integrated process automation tool that manages the automated analyses of sequencing data on the HPC.

The development of this robust and scalable platform started two years prior to the COVID-19 pandemic. As the pandemic unfolded, the RIVM found itself tasked with extensive analyses of vast amounts of COVID-19 surveillance data. We were able to quickly and seamlessly scale the system up to meet this increased demand.

For better automation and user experience, we created additional components within and alongside iRODS. The aforementioned Job Engine and a tool to import data from a variety of internal and external sources are used for automation purposes, while an extensive web interface with collection and document viewer functionalities, a data lineage viewer and (meta)data search capabilities were added for easy user interaction. We also created rules to pack and archive iRODS collections to an iRODS consumer at SURF (collaborative organisation for IT in Dutch education and research). With the DMS and these additional components, we have made the data analysis component of many vital processes within the RIVM reproducible.

By leveraging the iRODS software both the efficiency and the quality of data analyses within the RIVM has been improved significantly. The platform has proven to be crucial, notably during the pandemic, for processing otherwise unimaginable amounts of data.

The LEXIS Platform V2 and using iRODS for large scale data management
Martin Golasowski and Mohamad Hayek
IT4Innovations, VSB-TU Ostrava and Leibniz Supercomputing Centre

The LEXIS Platform allows easy access to HPC and Cloud resources with complex workflow orchestration and distributed data management based on iRODS. In our talk we present the improved version of the LEXIS Platform, including centralised metadata index, direct staging from iRODS zones and application container support. Extension of the LEXIS data management features are planned within the Horizon Europe EXA4MIND project, which focuses on handling large amounts of data between various data sources like databases or object storage and HPC infrastructure, including data publication features following FAIR principles. We also describe various use-cases for iRODS in the HPC environment, including using iRODS as data transfer gateway and preservation service based on EUDAT B2SAFE and testing of new iRODS interfaces such as S3 and HTTP API.

Integration of iRODS in a Federated IT Service through HTTP and Python API
Gautier Debaecker
CC-IN2P3

The Federated IT Service (FITS) project, a collaborative endeavor between the IN2P3 (Institut national de physique nucléaire et de physique des particules) computing center and French national HPC Center named IDRIS (Institut du développement et des ressources en informatique scientifique), addresses the challenge of managing the escalating data volumes generated by research infrastructures. The project aims to consolidate computing and storage resources while maintaining control over hosting expenses and minimizing the ecological footprint of digital technologies.

Within the FITS project, iRODS was selected as the storage pooling solution, leveraging its established use within the IN2P3 Computing Centre. This implementation enables project users to seamlessly access their data without being aware of its physical location. The project outlines three primary access interfaces:

1. A 'simple' utilization through icommands for streamlined data sending and processing via the command line interface.

2. A python API-based graphical interface for local data management and processing.

3. A graphical/web interface utilizing HTTP API accessible through a user portal, facilitating user authentication through identity managers such as Indigo IAM or Keycloak. Initial functionalities will include data viewing and collection creation, with plans for expanded features such as group creation, iRODS rule configurations, and more in subsequent phases. While initial constraints of the HTTP API limit direct browser-based data transfers, the system will enable essential data interactions, marking the beginning of an evolving suite of functionalities for enhanced user accessibility and data management within the FITS project framework.

A Rust Library Crate for iRODS
Phillip Davis
Appalachian State University

Rust has a number of features that make it desirable for an iRODS client. For example, its strong type system and the borrow checker provide compile-time guarantees against unexpected runtime errors. I present an iRODS client library written in Rust. Connections are presently coupled with their connection pool, implemented using the deadpool library, but could be used independently with slight additions. However, deadpool allows connections to make limited use of async. Connection pools are generic over iRODS protocol encoding (currently only XML is supported), transport protocol, and authentication strategy. Message serialization and deserialization will only allocate heap memory if users explicitly opt in by using message types which own their data. Finally, TLS/SSL is delegated to operating system implementations using the native-tls library.

Optimizing Data Management for AI: The Synergistic Role of Metadata-Hub and iRODS
David Cerf
GRAU DATA GmbH

Metadata-Hub, when combined with iRODS, becomes a transformative solution for managing escalating volumes of machine-generated and unstructured data. It places embedded metadata at the heart of storage management and data preparation, offering a more efficient, cost-effective, and scalable approach. At the time of data creation, Metadata-Hub extracts embedded metadata, allowing raw data files to be immediately archived to affordable storage solutions. This drastically cuts the need for expensive primary storage. Metadata-Hub becomes the repository for all metadata and makes it readily available. It reduces data preparation time for computational applications, analytics, and artificial intelligence (AI), accelerating the transition from data acquisition to insight derivation. Furthermore, the integration of Metadata-Hub into iRODS provides a robust, contextually-aware framework for automated and intelligent data workflows driven by embedded metadata, including archiving, replication, and distribution.

iRODS Build and Test v9: Automation via GitHub and Kubernetes
Phil Owen, Terrell Russell, Kory Draughn, and Alan King
Renaissance Computing Institute and iRODS Consortium

A reliable and efficient testing environment is vital to foster the quality of iRODS products and enhance developer productivity. It has become a moderately complex and time-consuming undertaking to create, maintain, and execute testing suites across numerous iRODS environment variants in order to comprehensively test iRODS source code.

Typically, unit and topology testing of iRODS components begins with building packages from modified source code. Next, iRODS packages are installed on various target environments that are defined within a matrix of versioned operating systems and DBMS types. Finally, test suites are executed in each environment where log files of all test results are compiled and analyzed.

In this talk we will discuss our newest approach (and future vision) of automating iRODS testing using GitHub and Kubernetes. This discussion will cover some details on how developers will request testing runs, how the build/test process works, and the forensics performed on the test results.

RSpace - iRODS integration: Update and next steps
Rory Macneil and Ander Astudillo
Research Space and SURF

At the 2023 UGM the Stage 1 integration, which ensures that when external files linked to RSpace documents change location, the link integrity is maintained, solving the ‘broken links’ problem and enhancing the reliability of the research record maintained in RSpace, was presented. This year Stage 2 of the integration, which enables export of datasets and associated metadata from RSpace to iRODS, will be presented. This will open the possibility of association of data from RSpace with the potentially vast range of data – e.g. all the data tracked by iRODS and captured in a university’s data storage facilities or all the data in a multi-university or a national project such as a biodiversity initiative. The RSpace data in iRODS will be discoverable via the associated metadata, enabling meaningful association with related data from other research resources, and re-use and repurposing of the data via, e.g., AI engines.

We will also introduce an idea for a Stage 3 integration that will enable RSpace to become a 'research data management front end' for iRODS. The working idea for this concept is as follows: The RSpace Gallery would act as a front end for iRODS to allow upload of data to iRODS (possibly even defaulting to storage in iRODS when files are added to the Gallery). Previewing/inline viewing of certain data types from iRODS should be possible, for example for image files (mirroring the current functionality for files stored in the RSpace Gallery).

Finally, in order to allow retrieval of data from iRODS (outside of RSpace), RSpace should be able to update a specific metadata field in the iRODS catalogue when files are linked to experiment entries in RSpace, so that querying the metadata catalogue for a particular document identifier could retrieve all data associated with that document (similar to seeing the linked documents in RSpace itself).

iRODS Security Challenges Within an Enterprise Environment
Ryan Blome and Simon Cook
Dow

Dow's focus on data security necessitates a tailored approach for our internal users, leading to the development of the Scientific Data Management System (SDMS) Query Tool (SQT) — a user-friendly tool designed to facilitate secure access to specific datasets. The current gap with Metalnx for general users is that there is too much control over modifying data and collections. Additionally, it is difficult to synchronize the iRODS users to our existing Azure Security groups for permission management. This talk outlines the development of a Querying Tool utilizing the iRODS C++ API as a backend to communicate with iRODS. The talk will highlight the need for robust security architecture for Enterprise scale applications and where we are hoping to take the project to in the future (HTTP API, etc.)

ManGO Portal (iRODS PRC based web UI): Feature updates, including extension points and workflows
Paul Borgermans, Mariana Montes, and Ingrid Barcena Roig
KU Leuven

As our Research Data Management user base is growing, so is our in-house software stack including ManGO, the iRODS PRC based web portal. Over the past year, more extension points were added in order to tailor the functions and user experience along the various requirements sets. Also a more generalised base to build workflows / data processing pipelines was added that can be triggered by events inside iRODS as well as external drivers (including ad hoc workflows).

Besides these rather fundamental aspects that make ManGO usuable for many iRODS installations, many smaller features and improvements were added as well.

iRODS HTTP API v0.3.0 with OpenID Connect
Kory Draughn and Martin Flores
iRODS Consortium

The new iRODS HTTP API was introduced last year as an idea to increase developer accessibility to an iRODS namespace, along with early research into how OpenID Connect would fit within the iRODS ecosystem. This talk will share updates through the first three releases, including optimizations and having the iRODS server be an OpenID Connect Protected Resource.

iRODS S3 API v0.2.0 with Multipart
Justin James
iRODS Consortium

The new iRODS S3 API can now present iRODS via the S3 protocol. This talk will share details about the first two releases, the implementation of various endpoints, and the state of Multipart transfers.

iRODS Build and Packaging: 2024 Update
Markus Kitsinger
iRODS Consortium

This talk will provide an update on our journey to 'Normal and Boring' with regard to CMake, libstdc++, and building for various platforms.

DAViDD: Initial data management solution for UNC's READDI AViDD Center
Terrell Russell
iRODS Consortium

The Rapidly Emerging Antiviral Drug Discovery Initiative AViDD Center (READDI-AC) is an NIH-funded public-private partnership focused on developing effective antiviral drugs to combat emerging viruses. The READDI-AC is one of nine Antiviral Drug Discovery (AViDD) Centers funded by the National Institute of Allergy and Infectious Disease (NIAID) at the National Institutes of Health.

The data management preparation, storage, and analysis are handled with DAViDD, a combination of iRODS, the HTTP API, and a custom Angular application. This talk will discuss the development of laboratory and researcher use cases; designing, building, and staging the application; and its early production use.

iRODS Metadata Templates Working Group: Building Blocks and Lessons Learned
Terrell Russell
iRODS Consortium

With starts and stops over more than six years, the working group's efforts have spanned a number of ideas and projects. This talk will cover that history, the group's decision making, and some reusable code.