TRiRODS is a meetup for iRODS users and developers.
TRiRODS works to build a strong sense of community, to offer a space for collaboration and sharing of individual implementations of iRODS, and to further the work on a hardened version of the code.
If you are interested in sharing your work at one of our quarterly meetings, please email info@irods.org.
Wednesday, August 14, 2024
The iRODS Zone Management Tool, written in ReactJS, has been refactored to use the new iRODS HTTP API. Alongside this work, a new Typescript client library has been created to wrap the HTTP API. It comes with testing and packaging.
Wednesday, August 14, 2024
This new Java client library has been designed and developed to wrap the new iRODS HTTP API. It provides java classes to cleanly interact with all of the functionality of the HTTP API and comes with tests and packaging.
Wednesday, August 14, 2024
In this project, information from iRODS was used to populate CKAN and enhance research data management by standardizing metadata and improving data-sharing efficiency, facilitating better compliance with federal requirements and collaboration among research teams.
Wednesday, August 14, 2024
The new iRODS HTTP API provides an easier way for web developers to interact with the iRODS server. This Python wrapper has been designed to expose all of the functionality of the HTTP API in a familiar set of patterns, along with tests and packaging.
Wednesday, February 14, 2024
The iRODS Consortium has been conducting its Authentication Working Group for several years. The product of the working group has been a new client-driven authentication framework and the pam_interactive authentication plugin. This plugin allows iRODS clients to enable complex flows with PAM for authentication with iRODS. This talk will demonstrate an OIDC flow implemented by SURF using the new pam_interactive plugin and give some updates on the state of the project.
Wednesday, December 6, 2023
With the new iRODS S3 API, iRODS can now be presented as S3-compatible storage to existing clients. This talk will quickly cover the requirements, the design, the initial release, and a demo of current functionality.
Wednesday, August 16, 2023
An iRODS Zone can describe its own configuration with a ZoneReport. The schema that defines the ZoneReport is now a bit out of date as the server itself has changed for 4.3.0. Nishant refactored the zone_bundle.json schema and updated the machinery that produces and depends on this format. Issues addressed include the naming of server roles, duplicate plugins, hierarchy information for resources, and self-aware versioning.
Wednesday, August 16, 2023
The iRODS Zone Management Tool has been updated to cover management of iRODS Tickets, the Delay Server, and a number of new health checks.
NFSRODS is a client to iRODS that presents iRODS as the NFSv4.1 protocol. We have found that restarting the server disrupts existing clients because the inodes (file handles) do not remain consistent across restarts. This has now been fixed and provides persistent identifiers.
Wednesday, August 16, 2023
The iRODS Protocol has been pretty stable for nearly 20 years, but not very well documented other than through implementations directly in client library code. This work seeks to provide more approachable and developer-friendly documentation of the iRODS XML Protocol.
Wednesday, August 16, 2023
The Automated Ingest tool is a Python application designed to keep the iRODS catalog up to date with changes in an existing filesystem or S3 bucket. The S3 scanner has implemented the REGISTER_SYNC operation for registering data in-place, but does not yet know how to make a copy of the scanned data into iRODS. This work implements PUT_SYNC to fill out the requested feature.
Wednesday, May 10, 2023
The iRODS Roadmap has seen its first official update since 2017. This talk will cover the recently approved 2023 version and its focus on approachability.
Wednesday, February 15, 2023
The iRODS General Query (GenQuery) interface exists to provide an SQL-like syntax for searching the iRODS namespace. It implements the SELECT and WHERE clauses but does not expose the FROM clause to the user so that the iRODS permission model can be enforced. The joins are determined by the server based on the requested tokens (or columns). This new flex and bison implementation provides a richer SQL-like syntax which allows for more powerful searches to be executed. This talk will include the goals of the updated interface and syntax, its current implementation, and a demonstration.
Wednesday, December 7, 2022
Mentioned at the iRODS User Group Meeting in July 2022, the new rirods R package is nearing its first release. This preview will discuss the design decisions, the implementation, and a demonstration.
Wednesday, August 10, 2022
The iRODS server was originally written in C around 20 years ago. This talk will cover the summer's refactorization of the core server software with modern C++ for increased maintainability, performance, and extensibility.
Wednesday, August 10, 2022
An iRODS server uses a process model which spawns independent agents to service incoming requests. While performant and relatively safe, the agents do not currently interact and will hold stale information when another part of the system changes. This talk covers the work done to automatically reload configuration changes without requiring a server restart.
Wednesday, August 10, 2022
The iRODS Consortium received an issue stating that the AMQP Audit Rule Engine Plugin generated invalid JSON. An investigation revealed that the issue does not originate from the Plugin itself, but rather from incompatibility between two versions of AMQP used in the iRODS demo ELK stack. We discuss the nature of that problem as well as steps forward.
The iRODS REST API did not expose entity rename, delete, repl, or trim functionality, nor did it allow add, remove, or modify of metadata AVUs. I investigated the architecture of the iRODS REST API and created endpoints for these actions.
Wednesday, August 10, 2022
iRODS has been around for a long time. Like other similarly sized projects, iRODS has grown several low-level libraries. This talk will share how a higher level C++ API for iRODS Tickets was created to wrap these existing lower-level calls.
Wednesday, August 10, 2022
This new application is a GUI to an existing Python command line tool. This tool helps us test iRODS to provide confidence to our users around the world. This web application is built by creating a user-friendly front-end and an API that communicates with the existing command line testing environment.
Wednesday, February 16, 2022
This talk introduces a new iRODS backend for MinIO, a scalable, secure, S3-compatible object storage for cloud users. The iRODS backend to MinIO allows users to configure their iRODS login to access their restricted data or anonymous access to public/shared data with an iRODS Ticket. By integrating with MinIO, iRODS users can leverage the user-friendly mode of data access provided by the MinIO user interface and the MinIO object store REST API.
Wednesday, December 1, 2021
The iRODS Testing Environment is a Docker Compose-based environment for running iRODS in a growing variety of configurations. This environment will be used in combination with the iRODS Development Environment in what will become the 8th generation of the iRODS build and test infrastructure.
Wednesday, August 18, 2021
This talk introduces two features that will be included in iRODS 4.3. First, a new facility within the iRODS server to manage background tasks, including (re)starting the Rule Execution (Delay) Server. Second, the Delay Server's new capability to send delayed rules to other servers in the iRODS Zone. These two features provide greater availability and scalability to the iRODS Delay Server mechanism.
Wednesday, August 18, 2021
The iRODS Automated Ingest tool can currently scan in parallel a local filesystem and an S3 bucket for new and updated files. A refactor of the iRODS Automated Ingest tool was completed to better organize the code for easier understanding and adding new capabilities.
Wednesday, May 19, 2021
Supervisory Control and Data Acquisition (SCADA) systems deployed up to now have suffered from well-known and well-publicized design defects in both security and scalability. This work introduces a simplistic SCADA model with InfiniBand for network security and iRODS as a data manipulation subsystem to provide credibility should any intrusions occur.
Mellanox Whitepaper (2012) - Security in Infiniband Fabrics
Wednesday, February 17, 2021
The Python iRODS Client (PRC) is learning some new tricks. First, the iRODS 4.2.8 server provides an API endpoint for atomic updates of multiple metadata operations. Second, an experimental feature in the upcoming iRODS 4.2.9 server provides a facility for parallel file transfer, using multiple connections over port 1247. This presentation will cover both of these new developments as well as how to use them via Python.
Wednesday, December 2, 2020
The iRODS C++ REST API has been discussed for years, but is now ready to show to others. This presentation will explore the different aspects of what is possible with the REST API today and invite discussion about what else it may need.
The iRODS Zone Management Tool (ZMT) is a new client that uses the iRODS C++ REST API. It has a design goal of handling the administrative side of running an iRODS Zone (managing users/groups/resources, etc.).
Wednesday, August 19, 2020
The iRODS Container Storage Interface (CSI) Driver implements the CSI Specification to provide iRODS access to Kubernetes. This allows iRODS to be used as a shared and permanent storage for Kubernetes apps. The iRODS CSI Driver mounts iRODS collections on directory hierarchies in Kubernetes apps by using iRODS clients (iRODS FUSE, NFS client and Davfs2) automatically. Therefore, Kubernetes apps can access data in iRODS without manual data staging or iRODS clients manipulation. This talk covers i) what’s CSI Driver, ii) how the iRODS CSI Driver works, iii) how to use the driver (demo), iv) tips for driver configuration and v) future development plan.
Wednesday, May 20, 2020
The 7th generation of the iRODS test infrastructure uses Jenkins, Docker, and Python to give us confidence in the entire system. Once complete, it will allow Consortium members (and anyone else) to test any changes they would propose to iRODS core and plugins before creating a pull request. With this infrastructure, iRODS is welcoming a more collaborative development process.
Wednesday, February 19, 2020
Historically a single static policy enforcement point, such as acPostProcForPut, was the sole location for all policy implementation. With the addition of a continuation code to the rule engine plugin framework, we may now configure multiple policies to be invoked for any given policy enforcement point. This subsequently allows for a separation of concerns and clean policy implementation. The policy developers now have the ability to separate the "when" (the policy enforcement points) from the "what" (the policy itself). How the policy is then invoked becomes a matter of configuration rather than implementation.
Given this new approach, multiple policies can now be configured together, or composed, without the need to touch the code. For example, the Storage Tiering capability is effectively a collection of several basic policies: Replication, Verification, Retention, and the Violating Object Discovery. All of these policies are configured via metadata annotating root resources, and taken as a whole provide a flexible system for automated data movement.
Wednesday, December 4, 2019
The University of Colorado Boulder Research Computing allocates storage space to end-user projects, and enforces these allocations with size and file limits within the namespace. A recent deployment of StrongBox left us without this capability for our LTFS archive environment; so we have deployed these quotas using the iRODS policy engine.
The new C++ "logical quotas" rule engine plugin was implemented by the iRODS Consortium, in collaboration with CU Boulder, and is driven by metadata stored on iRODS Collections.
Wednesday, August 21, 2019
Slides
Talk on current status of client libraries and interfaces, with a discussion about next steps for Metalnx, REST, and a few other APIs. Also includes conversation about client needs and opportunities, and an update of changes to iRODS APIs and services that could help client development.
Wednesday, May 15, 2019
NFSRODS is an iRODS Client that presents the iRODS virtual file system as NFSv4.1. This allows iRODS to be surfaced into any existing infrastructure with just a mount command, while still preserving the server-side policies enforced by the iRODS API. The demo explains the security model assumptions and shows events from regular file system activity appearing in an iRODS audit dashboard.
Wednesday, February 20, 2019
This is a new effort to provide some familiar abstractions in the form of two libraries, irods::iostreams and irods::filesystem.
Wednesday, February 20, 2019
This talk shows new functionality within the iRODS S3 resource plugin that provides a cacheless channel to an object store.
Wednesday, December 5, 2018
Over the last year, iRODS has released two packaged capabilities that many users are interested in configuring together. This talk will demonstrate using the Automated Ingest Framework and wiring it directly into the Storage Tiering Framework. This approach duplicates most of the functionality of an enterprise hierarchical file system.
Wednesday, August 22, 2018
This talk covers the newly published synchronization policy example from the iRODS Consortium. The demo will use the Python Rule Engine Plugin and the Automated Ingest Framework to monitor a data source for new or changed files and then keeps those changes up-to-date in the iRODS Vault and catalog. With only three configuration variables, this example synchronization policy can easily handle the ingest and sync requirements for most organizations.
Wednesday, February 21, 2018
Jason demonstrated the soon-to-be-released new storage tiering rule engine plugin. This is the first packaged iRODS Capability. Jason also reviewed MungeFS, which is a new user space file system overlay designed to test the storage tiering plugin.
Wednesday, December 6, 2017
DNA sequencing technology has evolved rapidly and resulted in data storage challenges for research organizations. Syngenta’s Next Generation Sequencing platform has been successful at getting data to researchers and analysts. However, there are opportunities to improve the users’ access to the data and the organization’s ability to manage and track data. Towards leveraging these opportunities, Syngenta has implemented a pilot of iRODS that integrates with their current processes and ways of working.
Wednesday, August 23, 2017
Mike spoke about employing iRODS in the mid-tier, lessons learned, and design concepts for the next version of Jargon. In addition, Mike touched on how REST is structured, Discovery Environment, Web interface plans, and how iRODS can be extended.
Wednesday, August 23, 2017
Antoine spoke about what is in version 0.6.0 of the Python iRODS Client (PRC) and what to expect in the next release. We also took a look at an application of PRC to parallelize file transfer into iRODS.
Wednesday, May 17, 2017
Metadata templates represent a significant addition of functionality to iRODS’s handling of metadata. In base iRODS, metadata is stored in the catalog as unadorned attribute-value-unit (AVU) triples of strings. This is flexible for many use cases, but does not afford the grid administrators much control of any stored metadata. By extending iRODS metadata, specifically by associating the AVU triples stored in the catalog with metadata templates which can provide additional information about these metadata elements (or meta-metadata), we can now support a variety of interesting functions.
This talk explains the motivation behind metadata templates, presents a proof of concept implementation of metadata templates, and demonstrates the use of this implementation via their REST API, currently in development.
Wednesday, February 22, 2017
Groups maintaining data repositories at the petabyte-scale are discovering that cataloguing associated metadata is necessary to properly access, recall and analyze data. To be successful they depend on researchers and data curators to design and provide the user defined metadata to be successful. Metalnx was developed to aid researchers with metadata management under iRODS (the Integrated Rule Oriented Data System) and is now an open source project (metalnx.github.io or www.github.com/metalnx)
This talk explains the motivation for this work, demonstrates the operational behavior of Metalnx, provides a preview of our work in data validation, and looks at a model for applying adaptive, hierarchical metadata to research curation.
Wednesday, December 7, 2016
John presented an overview of his library, performance numbers, and a demo of a lightweight GO REST web client built on top of GoRODS. As a surprise, he released version 1.0 near the end of his presentation, pushing live to GitHub.
Wednesday, December 7, 2016
Terrell and Justin showed how the audit plugin works within the iRODS Rule Engine Framework. The AMQP messages generated by every operation in iRODS can be queued, pulled by Logstash into Elasticsearch, and then displayed by Kibana as realtime dashboards of live activity within an iRODS Zone.
If you are interested in sharing your work at one of our quarterly meetings, please email info@irods.org.
Join us at our next meetup, virtually.
Wednesday, Feb 12, 2025
4:00 PM – 5:00 PM ET
Future quarterly meetups will be held in Aug and Dec 2025.
Wednesday, Feb 12, 2025
4:00 PM – 5:00 PM ET
Future quarterly meetups will be held in Aug and Dec 2025.
Previous Presentations
iRODS Zone Management Tool and TypeScript HTTP API Wrapper
By Chris Farber, iRODS ConsortiumWednesday, August 14, 2024
The iRODS Zone Management Tool, written in ReactJS, has been refactored to use the new iRODS HTTP API. Alongside this work, a new Typescript client library has been created to wrap the HTTP API. It comes with testing and packaging.
Chris Farber is an intern with the iRODS Consortium.
Java Client Library for iRODS HTTP API
By Samantha Mau, iRODS ConsortiumWednesday, August 14, 2024
This new Java client library has been designed and developed to wrap the new iRODS HTTP API. It provides java classes to cleanly interact with all of the functionality of the HTTP API and comes with tests and packaging.
Samantha Mau is an intern with the iRODS Consortium.
Populating CKAN Data Commons with iRODS Metadata
By Tanmay Dewangan, University of ArizonaWednesday, August 14, 2024
In this project, information from iRODS was used to populate CKAN and enhance research data management by standardizing metadata and improving data-sharing efficiency, facilitating better compliance with federal requirements and collaboration among research teams.
Tanmay Dewangan is an intern with CyVerse.
Python Wrapper for the iRODS HTTP API
By Christopher Lee, iRODS ConsortiumWednesday, August 14, 2024
The new iRODS HTTP API provides an easier way for web developers to interact with the iRODS server. This Python wrapper has been designed to expose all of the functionality of the HTTP API in a familiar set of patterns, along with tests and packaging.
Christopher Lee is an intern with the iRODS Consortium.
pam_interactive iRODS Authentication Plugin and OIDC
By Alan King, iRODS Consortium and Harry Kodden, SURFWednesday, February 14, 2024
The iRODS Consortium has been conducting its Authentication Working Group for several years. The product of the working group has been a new client-driven authentication framework and the pam_interactive authentication plugin. This plugin allows iRODS clients to enable complex flows with PAM for authentication with iRODS. This talk will demonstrate an OIDC flow implemented by SURF using the new pam_interactive plugin and give some updates on the state of the project.
Alan King is a Senior Software Developer at the iRODS Consortium.
Harry Kodden is a Senior Software Engineer at SURF.
iRODS S3 API: Presenting iRODS as S3
By Terrell Russell, iRODS ConsortiumWednesday, December 6, 2023
With the new iRODS S3 API, iRODS can now be presented as S3-compatible storage to existing clients. This talk will quickly cover the requirements, the design, the initial release, and a demo of current functionality.
Terrell Russell is the Executive Director of the iRODS Consortium.
Updating Zone Report and Other Server Fixes
By Nishant Dash, iRODS ConsortiumWednesday, August 16, 2023
An iRODS Zone can describe its own configuration with a ZoneReport. The schema that defines the ZoneReport is now a bit out of date as the server itself has changed for 4.3.0. Nishant refactored the zone_bundle.json schema and updated the machinery that produces and depends on this format. Issues addressed include the naming of server roles, duplicate plugins, hierarchy information for resources, and self-aware versioning.
Nishant Dash is an intern with the iRODS Consortium.
Zone Management Tool and NFSRODS
By Ganning Xu, iRODS ConsortiumWednesday, August 16, 2023
The iRODS Zone Management Tool has been updated to cover management of iRODS Tickets, the Delay Server, and a number of new health checks.
NFSRODS is a client to iRODS that presents iRODS as the NFSv4.1 protocol. We have found that restarting the server disrupts existing clients because the inodes (file handles) do not remain consistent across restarts. This has now been fixed and provides persistent identifiers.
Ganning Xu is an intern with the iRODS Consortium.
Investigating the iRODS Protocol
By Phillip Davis, iRODS ConsortiumWednesday, August 16, 2023
The iRODS Protocol has been pretty stable for nearly 20 years, but not very well documented other than through implementations directly in client library code. This work seeks to provide more approachable and developer-friendly documentation of the iRODS XML Protocol.
Phillip Davis is an intern with the iRODS Consortium.
Adding S3 transfer capability to iRODS Automated Ingest
By Ajay Krishnamurthy, iRODS ConsortiumWednesday, August 16, 2023
The Automated Ingest tool is a Python application designed to keep the iRODS catalog up to date with changes in an existing filesystem or S3 bucket. The S3 scanner has implemented the REGISTER_SYNC operation for registering data in-place, but does not yet know how to make a copy of the scanned data into iRODS. This work implements PUT_SYNC to fill out the requested feature.
Ajay Krishnamurthy is an intern with the iRODS Consortium.
iRODS 2023 Roadmap
By Kory Draughn, iRODS ConsortiumWednesday, May 10, 2023
The iRODS Roadmap has seen its first official update since 2017. This talk will cover the recently approved 2023 version and its focus on approachability.
Kory Draughn is the Chief Technologist at the iRODS Consortium.
GenQuery2: A richer query interface into the iRODS namespace
By Kory Draughn, iRODS ConsortiumWednesday, February 15, 2023
The iRODS General Query (GenQuery) interface exists to provide an SQL-like syntax for searching the iRODS namespace. It implements the SELECT and WHERE clauses but does not expose the FROM clause to the user so that the iRODS permission model can be enforced. The joins are determined by the server based on the requested tokens (or columns). This new flex and bison implementation provides a richer SQL-like syntax which allows for more powerful searches to be executed. This talk will include the goals of the updated interface and syntax, its current implementation, and a demonstration.
Kory Draughn is the Chief Technologist at the iRODS Consortium.
rirods: A first release of a pure R package for iRODS
By Martin Schobben, earth scientistWednesday, December 7, 2022
Mentioned at the iRODS User Group Meeting in July 2022, the new rirods R package is nearing its first release. This preview will discuss the design decisions, the implementation, and a demonstration.
Martin Schobben is an earth scientist in Vienna, Austria.
Refactor and modernize the main irodsServer
By June Releford, iRODS ConsortiumWednesday, August 10, 2022
The iRODS server was originally written in C around 20 years ago. This talk will cover the summer's refactorization of the core server software with modern C++ for increased maintainability, performance, and extensibility.
June Releford is an intern with the iRODS Consortium.
Add irodsServer configuration live reload
By Violet White, iRODS ConsortiumWednesday, August 10, 2022
An iRODS server uses a process model which spawns independent agents to service incoming requests. While performant and relatively safe, the agents do not currently interact and will hold stale information when another part of the system changes. This talk covers the work done to automatically reload configuration changes without requiring a server restart.
Violet White is an intern with the iRODS Consortium.
AMQP Audit Rule Engine Plugin Investigation and iRODS C++ REST API Additions
By Phillip Davis, iRODS ConsortiumWednesday, August 10, 2022
The iRODS Consortium received an issue stating that the AMQP Audit Rule Engine Plugin generated invalid JSON. An investigation revealed that the issue does not originate from the Plugin itself, but rather from incompatibility between two versions of AMQP used in the iRODS demo ELK stack. We discuss the nature of that problem as well as steps forward.
The iRODS REST API did not expose entity rename, delete, repl, or trim functionality, nor did it allow add, remove, or modify of metadata AVUs. I investigated the architecture of the iRODS REST API and created endpoints for these actions.
Phillip Davis is an intern with the iRODS Consortium.
New Ticket Library
By Awab Masroor, iRODS ConsortiumWednesday, August 10, 2022
iRODS has been around for a long time. Like other similarly sized projects, iRODS has grown several low-level libraries. This talk will share how a higher level C++ API for iRODS Tickets was created to wrap these existing lower-level calls.
Awab Masroor is an intern with the iRODS Consortium.
iRODS Testing Environment Web Application
By Rahqi Sarsour, iRODS ConsortiumWednesday, August 10, 2022
This new application is a GUI to an existing Python command line tool. This tool helps us test iRODS to provide confidence to our users around the world. This web application is built by creating a user-friendly front-end and an API that communicates with the existing command line testing environment.
Rahqi Sarsour is an intern with the iRODS Consortium.
MinIO-Gateway for iRODS
By Illyoung Choi, University of Arizona / CyVerseWednesday, February 16, 2022
This talk introduces a new iRODS backend for MinIO, a scalable, secure, S3-compatible object storage for cloud users. The iRODS backend to MinIO allows users to configure their iRODS login to access their restricted data or anonymous access to public/shared data with an iRODS Ticket. By integrating with MinIO, iRODS users can leverage the user-friendly mode of data access provided by the MinIO user interface and the MinIO object store REST API.
Illyoung Choi is a Research and Development Engineer at CyVerse.org at the University of Arizona. He previously developed the iRODS FUSE client (4.x) and the iRODS Container Storage Interface (CSI) Driver.
iRODS Testing Environment
By Alan King, iRODS ConsortiumWednesday, December 1, 2021
The iRODS Testing Environment is a Docker Compose-based environment for running iRODS in a growing variety of configurations. This environment will be used in combination with the iRODS Development Environment in what will become the 8th generation of the iRODS build and test infrastructure.
Alan King is a Software Developer at the iRODS Consortium.
Delay Server Availability and Scalability (C++)
By Violet White, iRODS ConsortiumWednesday, August 18, 2021
This talk introduces two features that will be included in iRODS 4.3. First, a new facility within the iRODS server to manage background tasks, including (re)starting the Rule Execution (Delay) Server. Second, the Delay Server's new capability to send delayed rules to other servers in the iRODS Zone. These two features provide greater availability and scalability to the iRODS Delay Server mechanism.
Violet White is an intern with the iRODS Consortium.
Ingest Refactor (Python)
By Gabrielle Maxwell, iRODS ConsortiumWednesday, August 18, 2021
The iRODS Automated Ingest tool can currently scan in parallel a local filesystem and an S3 bucket for new and updated files. A refactor of the iRODS Automated Ingest tool was completed to better organize the code for easier understanding and adding new capabilities.
Gabrielle Maxwell is an intern with the iRODS Consortium.
Towards Secure Supervisory Control and Data Acquisition (SCADA) with InfiniBand and iRODS
By David Wade, Integral EngineeringWednesday, May 19, 2021
Supervisory Control and Data Acquisition (SCADA) systems deployed up to now have suffered from well-known and well-publicized design defects in both security and scalability. This work introduces a simplistic SCADA model with InfiniBand for network security and iRODS as a data manipulation subsystem to provide credibility should any intrusions occur.
Mellanox Whitepaper (2012) - Security in Infiniband Fabrics
Dave Wade has led efforts in High Performance Computing for Engineering and Science for over 30
years. Educated at SDSU, Cal Tech, UCLA and UCSD, his instructors included Theodor Geisel, Kip
Thorne, Robert Leighton, Richard Feynman and Jef Raskin. Dave holds a Bachelor of Science degree
in Computer Science from the California State University at Dominguez Hills. He held positions for 22
years with General Atomics in La Jolla, California, involving direction of software and system designs
for HPC and leading and assisting design porting and deployment of parallel algorithms for
computation and data management. After participating with the San Diego Supercomputing Center in
the creation of UNITREE, an Open Source software for Hierarchical Storage Management, Dave
helped to introduce iRODS to a group within GA which spun up to commercialize such for use by the
DOE and DOD. Arguably Dave was the first at the company to build a parallel compute cluster. Dave
has his hands on experience for the components he proposes – in multi core, shared memory,
distributed and parallel file systems and numerical and computational algorithms. In his career at GA
he designed, built, staffed and maintained three HPC lab/centers for the company, led an effort
to define the software layers of a shared memory, parallel coded, embedded system Synthetic Aperture
Radar; and where he retired as a Lead HPC/CFD Engineer using in house and commercial parallel
codes to model flow and forces over the company's Predator line of Unmanned Airborne Vehicles.
Today Dave continues his professional work in HPC as a Director for High Performance Computing at Integral Engineering, a systems and software design engineering firm where he architects systems; leads design, build and deployment efforts of such systems for clients in government, industry and business; all tailored to principles, concepts, components and criteria from High Performance Computing. He speaks and is available to speak at select conferences focusing on HPC each year. In its now 7 year history Integral Engineering has produced designs for high efficiency video codec (HEVC 4 and 16 K) remote image capture, in core analysis and store; detection of fraud in cell phone telecommunications, high throughput data ingestion with data identification targeted to the Internet of Things (IOT), and supervisory control and data acquisition (SCADA).
Today Dave continues his professional work in HPC as a Director for High Performance Computing at Integral Engineering, a systems and software design engineering firm where he architects systems; leads design, build and deployment efforts of such systems for clients in government, industry and business; all tailored to principles, concepts, components and criteria from High Performance Computing. He speaks and is available to speak at select conferences focusing on HPC each year. In its now 7 year history Integral Engineering has produced designs for high efficiency video codec (HEVC 4 and 16 K) remote image capture, in core analysis and store; detection of fraud in cell phone telecommunications, high throughput data ingestion with data identification targeted to the Internet of Things (IOT), and supervisory control and data acquisition (SCADA).
Python iRODS Client: Atomic Metadata and Parallel Transfer
By Daniel Moore, iRODS ConsortiumWednesday, February 17, 2021
The Python iRODS Client (PRC) is learning some new tricks. First, the iRODS 4.2.8 server provides an API endpoint for atomic updates of multiple metadata operations. Second, an experimental feature in the upcoming iRODS 4.2.9 server provides a facility for parallel file transfer, using multiple connections over port 1247. This presentation will cover both of these new developments as well as how to use them via Python.
Daniel Moore is an Applications Engineer at the iRODS Consortium.
iRODS C++ REST API and iRODS Zone Management Tool (ZMT)
By Jason Coposky and Bo Zhou, iRODS ConsortiumWednesday, December 2, 2020
The iRODS C++ REST API has been discussed for years, but is now ready to show to others. This presentation will explore the different aspects of what is possible with the REST API today and invite discussion about what else it may need.
The iRODS Zone Management Tool (ZMT) is a new client that uses the iRODS C++ REST API. It has a design goal of handling the administrative side of running an iRODS Zone (managing users/groups/resources, etc.).
Jason Coposky is the Executive Director of the iRODS Consortium.
Bo Zhou is a web developer at RENCI working with the iRODS Consortium.
iRODS Container Storage Interface (CSI) Driver
By Illyoung Choi, University of Arizona / CyVerseWednesday, August 19, 2020
The iRODS Container Storage Interface (CSI) Driver implements the CSI Specification to provide iRODS access to Kubernetes. This allows iRODS to be used as a shared and permanent storage for Kubernetes apps. The iRODS CSI Driver mounts iRODS collections on directory hierarchies in Kubernetes apps by using iRODS clients (iRODS FUSE, NFS client and Davfs2) automatically. Therefore, Kubernetes apps can access data in iRODS without manual data staging or iRODS clients manipulation. This talk covers i) what’s CSI Driver, ii) how the iRODS CSI Driver works, iii) how to use the driver (demo), iv) tips for driver configuration and v) future development plan.
Illyoung Choi is a Ph.D. candidate in Computer Science at the University of Arizona and works for CyVerse as a research assistant. He previously developed iRODS FUSE client (4.x).
iRODS 7th Generation Build and Test Infrastructure
By Jaspreet Gill, iRODS ConsortiumWednesday, May 20, 2020
The 7th generation of the iRODS test infrastructure uses Jenkins, Docker, and Python to give us confidence in the entire system. Once complete, it will allow Consortium members (and anyone else) to test any changes they would propose to iRODS core and plugins before creating a pull request. With this infrastructure, iRODS is welcoming a more collaborative development process.
Jaspreet Gill is an Automation Engineer at the iRODS Consortium.
iRODS Policy Composition: Principles and Practice
By Jason Coposky, iRODS ConsortiumWednesday, February 19, 2020
Historically a single static policy enforcement point, such as acPostProcForPut, was the sole location for all policy implementation. With the addition of a continuation code to the rule engine plugin framework, we may now configure multiple policies to be invoked for any given policy enforcement point. This subsequently allows for a separation of concerns and clean policy implementation. The policy developers now have the ability to separate the "when" (the policy enforcement points) from the "what" (the policy itself). How the policy is then invoked becomes a matter of configuration rather than implementation.
Given this new approach, multiple policies can now be configured together, or composed, without the need to touch the code. For example, the Storage Tiering capability is effectively a collection of several basic policies: Replication, Verification, Retention, and the Violating Object Discovery. All of these policies are configured via metadata annotating root resources, and taken as a whole provide a flexible system for automated data movement.
Jason Coposky is the Executive Director of the iRODS Consortium.
Deploying Logical Quotas with iRODS at CU Boulder Research Computing
By Jonathon Anderson, CU Boulder Research ComputingWednesday, December 4, 2019
The University of Colorado Boulder Research Computing allocates storage space to end-user projects, and enforces these allocations with size and file limits within the namespace. A recent deployment of StrongBox left us without this capability for our LTFS archive environment; so we have deployed these quotas using the iRODS policy engine.
The new C++ "logical quotas" rule engine plugin was implemented by the iRODS Consortium, in collaboration with CU Boulder, and is driven by metadata stored on iRODS Collections.
Jonathon Anderson is a systems architect and associate director with University of Colorado Boulder Research Computing. His team builds and administers parallel computing and commodity storage services for the research community on campus and with partner institutions in the Rocky Mountain region.
iRODS Client Interfaces Update
By Mike Conway and Deep Patel, National Institute of Environmental Health SciencesWednesday, August 21, 2019
Slides
Talk on current status of client libraries and interfaces, with a discussion about next steps for Metalnx, REST, and a few other APIs. Also includes conversation about client needs and opportunities, and an update of changes to iRODS APIs and services that could help client development.
Mike Conway is a technical architect and developer with the Office of Data Science at NIEHS in RTP. Mike is working to employ FAIR and policy-based data management principles to accelerate data-driven research in environmental health and toxicology through the development of the NIEHS Data Commons.
Deep Patel is a software developer with the Office of Data Science at National Institute of Environmental Health Sciences. He is working to collaborate with bioinformatics to generate and automate next generation sequencing workflow pipelines to achieve FAIR data principles for the NIEHS Data Commons.
NFSRODS (pre-release)
By Terrell Russell, iRODS ConsortiumWednesday, May 15, 2019
NFSRODS is an iRODS Client that presents the iRODS virtual file system as NFSv4.1. This allows iRODS to be surfaced into any existing infrastructure with just a mount command, while still preserving the server-side policies enforced by the iRODS API. The demo explains the security model assumptions and shows events from regular file system activity appearing in an iRODS audit dashboard.
Terrell Russell is the Chief Technologist of the iRODS Consortium.
First Look: irods::iostreams and irods::filesystem Libraries
By Kory Draughn, iRODS ConsortiumWednesday, February 20, 2019
This is a new effort to provide some familiar abstractions in the form of two libraries, irods::iostreams and irods::filesystem.
Kory Draughn is a Software Developer at the iRODS Consortium.
First Look: Cacheless S3 Resource
By Justin James, iRODS ConsortiumWednesday, February 20, 2019
This talk shows new functionality within the iRODS S3 resource plugin that provides a cacheless channel to an object store.
Justin James is an Applications Engineer at the iRODS Consortium.
Together at Last: Automated Ingest to Storage Tiering
By Alan King, iRODS ConsortiumWednesday, December 5, 2018
Over the last year, iRODS has released two packaged capabilities that many users are interested in configuring together. This talk will demonstrate using the Automated Ingest Framework and wiring it directly into the Storage Tiering Framework. This approach duplicates most of the functionality of an enterprise hierarchical file system.
Alan King is a Software Developer at the iRODS Consortium.
iRODS Automated Ingest and Sync Policy Example
By Jason Coposky and Terrell Russell, iRODS ConsortiumWednesday, August 22, 2018
This talk covers the newly published synchronization policy example from the iRODS Consortium. The demo will use the Python Rule Engine Plugin and the Automated Ingest Framework to monitor a data source for new or changed files and then keeps those changes up-to-date in the iRODS Vault and catalog. With only three configuration variables, this example synchronization policy can easily handle the ingest and sync requirements for most organizations.
Jason Coposky is the Executive Director of the iRODS Consortium.
Terrell Russell is the Chief Technologist of the iRODS Consortium.
iRODS Storage Tiering Capability
By Jason Coposky, iRODS ConsortiumWednesday, February 21, 2018
Jason demonstrated the soon-to-be-released new storage tiering rule engine plugin. This is the first packaged iRODS Capability. Jason also reviewed MungeFS, which is a new user space file system overlay designed to test the storage tiering plugin.
Jason Coposky is the Executive Director of the iRODS Consortium.
A Pilot of iRODS for Managing Next Generation Sequencing Data
By Todd Moughamer, SyngentaWednesday, December 6, 2017
DNA sequencing technology has evolved rapidly and resulted in data storage challenges for research organizations. Syngenta’s Next Generation Sequencing platform has been successful at getting data to researchers and analysts. However, there are opportunities to improve the users’ access to the data and the organization’s ability to manage and track data. Towards leveraging these opportunities, Syngenta has implemented a pilot of iRODS that integrates with their current processes and ways of working.
Todd Moughamer is a Senior Computational Biologist and leads the Next Generation Sequence Data Management at Syngenta.
A Developer's Perspective on Extending iRODS to the Mid-Tier
By Mike Conway, RENCIWednesday, August 23, 2017
Mike spoke about employing iRODS in the mid-tier, lessons learned, and design concepts for the next version of Jargon. In addition, Mike touched on how REST is structured, Discovery Environment, Web interface plans, and how iRODS can be extended.
Mike Conway is a Senior Data Science Researcher at RENCI. Conway was part of the Data Intensive Cyber Environments team that invented iRODS and conceptualized policy-based data management, and he is a developer with the iRODS Consortium. He was a lead developer on the NSF DataNet Federation Consortium formed to develop universal solutions to research data management.
Developing a Python Client Library for iRODS
By Antoine de Torcy, iRODS ConsortiumWednesday, August 23, 2017
Antoine spoke about what is in version 0.6.0 of the Python iRODS Client (PRC) and what to expect in the next release. We also took a look at an application of PRC to parallelize file transfer into iRODS.
Antoine is a C++ developer and has been part of the original iRODS development team since 2003. He joined UNC in 2008 and RENCI in 2013. Prior to that he was part of the Data Intensive Computing Environments group (DICE) at the University of California, San Diego. At DICE, Antoine wrote client and server components for iRODS and its precursor the Storage Resource Broker, and he worked on NSF-funded research projects that used iRODS for large scale management of scientific data. Antoine holds an engineering degree in Applied Mathematics and Computer Science from the University of Paris.
Metadata Templates in iRODS
By Rick Skarbez, iRODS ConsortiumWednesday, May 17, 2017
Metadata templates represent a significant addition of functionality to iRODS’s handling of metadata. In base iRODS, metadata is stored in the catalog as unadorned attribute-value-unit (AVU) triples of strings. This is flexible for many use cases, but does not afford the grid administrators much control of any stored metadata. By extending iRODS metadata, specifically by associating the AVU triples stored in the catalog with metadata templates which can provide additional information about these metadata elements (or meta-metadata), we can now support a variety of interesting functions.
This talk explains the motivation behind metadata templates, presents a proof of concept implementation of metadata templates, and demonstrates the use of this implementation via their REST API, currently in development.
Rick Skarbez has a PhD in computer science from the University of North Carolina at Chapel Hill, and is a Systems Programmer for the iRODS Consortium. Rick has been working with iRODS since 2015.
Metalnx, Metadata for the Masses
By Stephen Worth, Dell EMCWednesday, February 22, 2017
Groups maintaining data repositories at the petabyte-scale are discovering that cataloguing associated metadata is necessary to properly access, recall and analyze data. To be successful they depend on researchers and data curators to design and provide the user defined metadata to be successful. Metalnx was developed to aid researchers with metadata management under iRODS (the Integrated Rule Oriented Data System) and is now an open source project (metalnx.github.io or www.github.com/metalnx)
This talk explains the motivation for this work, demonstrates the operational behavior of Metalnx, provides a preview of our work in data validation, and looks at a model for applying adaptive, hierarchical metadata to research curation.
Steve Worth coordinates innovation initiatives across Dell EMC global engineering centers, is a technical liaison between product groups and the global engineering network, and works in big data / digital asset management leveraging user defined metadata. Steve is based out of Dell EMC’s RTP Software Development Center which focuses on data protection, core storage products, and cloud storage virtualization.
GoRODS Demo and 1.0 Release
By John Jacquay, BioTeamWednesday, December 7, 2016
John presented an overview of his library, performance numbers, and a demo of a lightweight GO REST web client built on top of GoRODS. As a surprise, he released version 1.0 near the end of his presentation, pushing live to GitHub.
John Jacquay is a scientific systems engineer at BioTeam.
iRODS Audit (AMQP) Rule Engine Plugin Demo
By Terrell Russell and Justin James, iRODS ConsortiumWednesday, December 7, 2016
Terrell and Justin showed how the audit plugin works within the iRODS Rule Engine Framework. The AMQP messages generated by every operation in iRODS can be queued, pulled by Logstash into Elasticsearch, and then displayed by Kibana as realtime dashboards of live activity within an iRODS Zone.
Terrell is the Chief Technologist of the iRODS Consortium.
Justin is an Applications Engineer at the iRODS Consortium.