Kory Draughn
Chief Technologist
The iRODS Consortium
April 2023

iRODS 2023 Roadmap

With each release of iRODS, the server, libraries, and clients become more stable and featureful. The needs of users and administrators become more complex over time. Users want tools and GUIs that are easy to understand quickly. Administrators want more flexibility and visibility into what the system is doing.

In order to facilitate these needs, the Consortium plans to improve approachability by focusing on the following:

The Server

• Make implementing the iRODS Protocol less complicated by moving to HTTP or designing a new protocol
  • Any move to HTTP will be heavily influenced by the C++ REST API
  • If we choose to design a new protocol, it will be documented and versioned
• Replace the PackStruct serialization library
  • Use an actively maintained and proven serialization library (e.g. Protocol Buffers, Flatbuffers, etc.)
    • Enables a straightforward path to supporting the most popular programming languages
    • API specification is documented as a result of using the library and tools
• Make the server cloud-friendly
  • Improve startup time
  • Improve flexibility around dynamic host names
  • Treat every server as a Catalog Service Provider
  • Centralize configuration
• Improve support for archive files
  • Make archive entries visible in the catalog
    • Design new abstractions, semantics, and APIs
  • Deprecate all functionality stemming from special collections and imcoll

Plugins and Capabilities

• Finish and release the Policy Composition rule engine plugin
• Finish and release the Publishing capability

Libraries

• Add support for the new Authentication Plugin Framework introduced in the iRODS 4.3 series
  • All client-side libraries, regardless of the programming language, must be updated
  • Required for pam_interactive authentication scheme
    • This becomes the default for iCommands

Clients

• Replace iCommands with a new CLI
  • A single binary backed by a plugin architecture allowing its functionality to be extended
  • No memory leaks and proper signal handling
  • Help text and documentation will be consistent in formatting and detail
• Improve Metalnx
  • Add support for the pam_interactive authentication scheme
  • Remove the need for helper database
  • Remove most administrative features
    • Administrative features will be absorbed by the Zone Management Tool
  • Remove unnecessary features
  • Provide sane defaults for all configuration options
• Improve Zone Management Tool
  • Add support for the pam_interactive authentication scheme
  • Allow administrators to manage an entire zone from a single computer
    • Configuration and policy can be modified and applied to one or more servers simultaneously
    • Controls for starting and stopping servers
    • Controls for migrating the Delay server

Terrell Russell
Chief Technologist
The iRODS Consortium
March 2017

iRODS 4.3

The focus for iRODS 4.3 will be on two things: the logging framework, and hardening the existing features and new work introduced in 4.2.

Alongside this work, but not part of the core codebase, the Consortium will be working on supporting features and functionality provided by additional external tools and features delivered as plugins.

Logging

The logging in iRODS needs to be modernized and standardized with regards to best practices around log rotation, timestamps, and human and machine readable formats. We plan to investigate a variety of logging frameworks and select one that delivers flexibility powerful enough to afford easy aggregation of logs from our distributed system.

Hardening

The everpresent backlog of bugfixes and edge cases will continue to be addressed. This release will mark the first of a new tick-tock release pattern, meaning one release will focus on features, and the next will focus more on hardening. 4.2 was the feature release, and 4.3 will be the hardening release.

Multipart Data Objects

As file sizes continue to grow, moving them around monolithically becomes subject to a set of failure modes that are more gracefully handled with the use of smaller files. iRODS will implement multipart data objects whereby large files are broken up into parts before transfer over the network, and then reconstituted on the other end.

Each part of the file will be transferred and tracked as a regular data object within the iRODS protocol. Any individual part that does not transfer correctly can be retransmitted. This will allow iRODS to more gracefully overcome network instability, and affords cleaner restarts of large files. This work will also improve transport speeds by eventually serving as the backbone for multi-target transfers into iRODS and multi-source transfers out of iRODS.

Because each part of a larger file will now be addressable, iRODS will be able to provide POSIX-like semantics for object storage technologies. This means that these object stores can be connected to iRODS in a cache-free configuration.

Filesystem Integration

iRODS strives to serve as the glue that can tie together many existing storage technologies with a unified namespace, discovery mechanism, and policy engine. Usually, when an organization decides they need to incorporate iRODS into their infrastructure, they already have a significant amount of data, usually in disparate physical systems.

iRODS needs a way to keep track of changes occurring on filesystems, where existing scripts, tools, and instruments are making changes outside of iRODS. This can be done for moderate-sized systems with a pattern we have named the "Landing Zone" whereby iRODS watches a particular location in the filesystem and takes action when new files appear. For much larger systems handling a very large number of files concurrently, this approach may become an unreasonable bottleneck.

We are working with Intel's Lustre group to investigate and then implement best practices around keeping the iRODS catalog up-to-date for such high-velocity systems.

Metadata Abstraction

iRODS metadata has traditionally been stored within the relational database known as the iCAT. Permissions on that metadata was originally designed to be readable by everyone because the primary use case was discoverability. As use cases have become more diverse and sophisticated, the metadata is being used to drive policy decisions as well as store additional information that should not be made available to all users of the system.

QueryArrow has been developed to help alleviate this restriction and allow iRODS to store and query metadata in a variety of other database or indexing technologies.

We plan to release the QueryArrow database plugin which will afford this abstraction of metadata.

Jason Coposky
Chief Technologist
The iRODS Consortium
July 2015

iRODS 4.2

For iRODS 4.2, our focus shifted from hardening the core to hardening the software engineering infrastructure and enhancing integration with external services. We chose a standard build system and a compiler across all supported platforms, and we moved the system to a dynamic linking strategy, once and for all allowing for the separation of client and server libraries.

For integration with external services, we worked with existing implementations from our collaborators in our membership and community to capture best practices and standardize message formats and payloads. The resulting Messaging Framework leveraged the new Rule Engine Plugin Framework, which allows for the use of new programming languages as well as special purpose plugins.

Messaging Framework

The messaging framework is a means for iRODS to asynchronously reach out to external services via an AMQP message bus. Several messaging implementations currently exist within the community, namely at CyVerse and DFC. As this technology has become increasingly important, we intend to gather best practices and standardize message formats, serialization technology, topics, and the services expected to be available for those topics.

The message bus is typically reached through iRODS via policy enforcement points (PEPs) in the rule engine. With the addition of the new Rule Engine Plugin Framework, we now have a fine-grained and fast system for message dispatch.

Reference Implementations

The iRODS Consortium provides and supports reference implementations across a variety of use cases and vertical integrations. The first three were auditing, indexing, and metadata storage.

• Auditing System: This reference implementation provides the pieces, as well as the wiring, to handle a full, provably-complete, auditing system of every operation within an iRODS deployment.
• Indexing System: This reference implementation provides an external service to handle partial and fulltext indexing of the data objects and metadata stored in an iRODS deployment.
• Metadata Storage: This reference implementation provides an external service to store large numbers of metadata attributes associated with data stored in an iRODS deployment.

Together, these reference implementations provide a glimpse into the power of iRODS as a cohesive system to track every operation and to provide custom search across arbitrarily detailed amounts of metadata and fulltext.

Rule Engine Plugin Framework

This framework allows the definition of rules in languages other than the Rule Language, such as Python or C++. It also provides the ability to run multiple rule engine plugins and rule bases simultaneously for a given data management operation, which enables the creation of special purpose Rule Engines. Coupled with the use of dynamic policy enforcement points (PEPs), this creates a powerful and versatile system for a variety of applications. In concert with the new Messaging Framework, a special rule engine plugin may publish any and all policy enforcement points and associated data to the AMQP message bus for ingest or action.

Software Engineering Infrastructure

iRODS continues to improve under the hood due to newer software engineering techniques and C++ features being introduced. With the addition of the Rule Engine Plugin framework, new C++ features are now required, and they needed to be included in the infrastructure used to build and deploy iRODS. The new build system has been rewritten using CMake. The build system standardizes on a C++14-capable compiler across all supported platforms. All of the setup, upgrade, and control scripts are standardized on Python. The dynamic libraries for the server and the client have been separated and versioned, and the C API cleanly defined and isolated. This was a major departure from the prior bash and make-based build system but now provides unprecedented flexibility, power, and control as features are added in 4.2+.

Future Direction

The long term vision of the Consortium for iRODS is to create a hardened core broker which invokes services across groups of plugins accessed through a suite of thin and easy-to-use client APIs. iRODS and its source code should focus on data management. A hardened core broker will ultimately allow iRODS to grow in any direction the Membership and Community may envision. To that end, we envision a set of capabilities necessary for iRODS to reach this level of enterprise readiness.

An iRODS Zone should be able to communicate to a client the capabilities of the Zone, and which servers or groups of servers may provide the best service for the given API request and data object or collection in question. A client may request a transformation on a particular data set, a replication of that data to a computation resource or the refresh of an index of the data. The iRODS Zone needs to be able to provide the existence and location of these requested capabilities to the clients or federations in order to best service the request.

iRODS, originally designed around POSIX storage semantics, needs to move to a higher level of storage abstraction. With the emergence of requirements to treat object stores as first class resources, iRODS needs to further encapsulate the underlying storage behavior behind a more abstract storage interface.

Plugin Registry

As iRODS continues to move further toward a full implementation of the plugin model, the plugins themselves will come to depend on features of other plugins. In order to fully realize the ability to ship a set of plugins as a new feature, these plugins need the ability to declare their dependencies to the data grid administrator. Plugins should have the ability to be packaged together as a single unit, and declare to the system both their requirements as well as capabilities.

iRODS plugins, when installed, need a place to record several things: their state, their capabilities, and their dependencies. We envision a registry within the iCAT catalog where plugins may centralize this information. This feature will provide the ability for the Zone to introspect and report its capabilities to the client and other Zones. A plugin may also determine if all dependencies are met and whether it is capable of performing its operations, at which point an administrator may enable or disable the plugin.

Next Generation API

The existing iRODS API relies heavily on the client for various operations, such as recursive collection manipulation. This has caused unnecessary network traffic as well as additional burden on the client API developer, which has historically led to various client APIs in differing states of support. Further, the existing C API is the only client capable of utilizing the binary protocol. In order to bring the support of the various client APIs into lockstep, we intend to standardize on a single serialization and messaging technology which is supported by all target languages. From this point we may begin building a new iRODS API alongside the existing API and bring the support for this new API up to date for all client implementations. We will also be able to reduce the set of exposed storage operations, more in line with object stores, as we push as much code back from the client into the server implementations.

Data Transport Plugins

During the creation of the next generation API, it is our intention to further abstract the transportation of data within iRODS. There is a wealth of existing technologies, such as Aspera and other parallel I/O APIs provided by various storage systems, that can benefit iRODS. One can envision an implementation in which, during a put or get, iRODS simply brokers the operation and selects the best transport mechanism available for the operation. This would permit HPC clusters storing data in iRODS on high performance parallel file systems to take full advantage of the native speed of their storage technologies.

iRODS 4.1

Each release of iRODS begins with a general philosophy which shapes the decisions about the overall goals as well as particular features to include. As of the iRODS Consortium 4.0 release, marking the merge between Enterprise iRODS and the Community Edition, the Consortium has been focused on making this product a stable, reliable, Enterprise Ready data management system. For the 4.1 release this objective limited our scope to activities around hardening the code, fixing bugs, and building Enterprise quality infrastructure around our core product: the iRODS Server.

Hardening

• Resolve static analysis errors
  • over 1200 issues discovered by Coverity
  • continue to keep cppcheck error free
  • continue to keep STACK error free
• Resolve any bugs discovered by our Membership and Community
  • 192 issues in total
• General Improvements - minor architecture updates and clean up of partially implemented features
  • Push the plugin architecture farther into the API
  • Move magic numbers to run-time configuration
  • Remove built-in PostgreSQL DB Vacuum functionality
  • Remove boot user from install script
  • Remove "run_server_as_root" option
  • Remove roles for storageadmin, domainadmin, and rodscurators
  • Remove obfuscation (SIDKey and DBKey)

Enterprise Readiness

• Move configuration to validated JSON - unify configuration file formats into an easy to use, programmatically flexible syntax
  • New schema-driven validated JSON configuration files include
    • server_config.json
    • database_config.json
    • irods_environment.json
    • host_access_control_config.json
    • hosts_config.json
  • Automatically generated on upgrade from 4.0.3
• Zone Introspection - a step toward configuration management for iRODS deployments
  • Generates a validated JSON document describing every server in the Zone
    • Sanitized snapshots of all JSON configuration files
    • Listing of all installed plugins with checksums
    • Listing of all configured resources
    • Contents of rule files in base64 encoding
  • Leveraged by Consortium Continuous Integration for automated deployment and testing of topologies
• Grid-wide control plane - side channel for communication directly with the iRODS Server
  • Per-server or grid-wide operations
    • Status information - validated JSON containing connections, PIDs, server state
    • Gracefully Pause and Resume
    • Gracefully Shutdown
• New Client-Hints API - communicate Server configuration information to the Client
  • Validated JSON document containing
    • StrictACL setting
    • Desired hash-scheme
    • Match hash-scheme policy
    • Listing of installed specific queries
  • Will track iRODS schema versions and grow over time