News / 30 April 2019

iRODS Development Update: April 2019


A pretty full month for development this time - we're in a full sprint until the Netherlands.

It's the last day for submission of UGM2019 abstracts! Register today and submit your talks for Utrecht!

In working towards our 4.3.0 release, we now have parallel building and testing of our many plugins. We've got a prototype of topology testing Dockerized and hope to have Federation completed soon as well. We can start building and testing iRODS in parallel within days. This has been a long time coming.

NFSRODS is all but ready - expect a first release soon. It has already been deployed into an enterprise environment and is being tested for performance and compliance. We simplified the security model to assume identity/user management is already in place. The new model assumes that the user traversing the mount point is already authenticated - and presents that user with a view of their iRODS collections and data objects. It's very clean.

Metalnx 2.0.0 was released a couple weeks ago! Since the merger of Dell and EMC was announced in 2015, this release marks the completion of the contribution of the codebase to the iRODS Consortium. We'll continue to work to make Metalnx even easier to stand up and maintain. It's nice to have a few different iRODS clients production-ready.

We are investigating new transport mechanisms within iRODS. With our new stream operators, we are looking into new approaches that work with UDT, FDT, and RDMA. If you have interest or working knowledge of best practices or design criteria, please get in touch. If there is enough interest, we could schedule a quick conference call (or even spin up a Working Group).

We have isolated and fixed an issue within the irodsReServer that will be included in 4.2.6. Also to be included is some additional work on the new irods::filesystem collection iterator. As we continue to refactor our own code, the new things are being more deeply exercised. Expect this release within the week.

In the run-up to iRODS 4.3.0, we have settled on rsyslog and JSON for the new logging infrastructure. However, the option to run iRODS as a Non-Package Install requires a bit more thinking (rsyslog may not be running or under the control of the user running the iRODS server). We've started a promising conversation around how these options can play well with each other (perhaps a 'foreground' option...). Please share your input in the Google Group.

April Technology Working Group

  • 4.2.6

    • 102 issues open, 61 bugs

  • 4.3.0

    • 112 issues open, 48 bugs
    • clang6, cmake3.11
    • c++17
    • genQuery iterator
    • new logger
    • irods::filesystem
    • irods::iostreams
    • clang format
    • irodsDelayServer
    • Ubuntu18 will be supported
    • Ubuntu14 will not be supported, LTS ends April 2019
    • Considering removal of Non-Package Install optional_arguments
      • due to rise of containers
      • due to rsyslog, logrotate

  • Active Development Work

    • Python iRODS Client (PRC)

    • Cacheless and Detached S3

      • works with current iRODS parallel transfer
      • moved to shared memory for new multi-process transfer
      • slow download speed, working on issue

    • NFSRODS

    • iRODS Capability - Automated Ingest

    • Multipart Transfer, v5 API

    • Lustre Integration

    • Storage Tiering Capability Package

    • Indexing Capability

      • rule engine plugin(s)
      • similar approach as storage tiering
      • initial implementation complete
      • testing beginning
      • first, elasticsearch, both full_text and AVUs

    • Metalnx packaging

    • Cloud Browser

      • multi-dataobject move
      • need to finalize and publish Docker image

    • Continuous Integration (CI)

      • refactoring to use Docker
        • parallelized test runs (down from 8h to 30m)
      • public Jenkins
      • move to pull-request and peer-review
      • adding CockroachDB database plugin
      • adding Ceph RADOS resource plugin
      • adding QueryArrow database plugin
      • adding Nestle R Client Library
      • adding Metalnx
      • adding baton/tears
      • adding automated ingest
      • adding storage tiering
      • adding NFSRODS
      • adding collection mtime rule engine plugin

  • Background Items

    • Samba/CIFS/SMB project to surface iRODS as SMB

    • Python Rule Engine Plugin

      • working with irods/irods master

    • CockroachDB Database Plugin

      • refactored to use UUID
      • testing for concurrent writes
      • testing iRODS schema assumptions
      • not performant enough at this time

    • Swagger REST API

Terrell Russell