News / 30 December 2018

iRODS Development Update: December 2018


Not as much to report this December - everything slows down quite a bit in University land over the holiday break.

Work on the Automated Ingest Framework has proceeded, mostly focused on hardening and testing coverage.

The cacheless and detached storage resource effort is very close to a working implementation for all plugin operations when talking to S3 (through Minio for testing).

And in some experimental work, iRODS can now be installed, configured, and started up while using UUIDs as primary identifiers within the catalog rather than sequences. This is exciting as we continue to target a future consisting of all iRODS servers in the role of catalog provider (via CockroachDB).

2018 has flown by and we are very happy with our progress. With 4-1-stable now EOL, all eyes are focused on 2019, increased adoption, and continued community development of higher-order solutions built on top of iRODS.

Happy New Year everyone!

December Technology Working Group

  • 4.1.12

    • Released November 1, 2018
    • 4-1-stable is now EOL

  • 4.2.5

    • 92 issues, 56 bugs

  • 4.3.0

    • 103 issues open, 47 bugs
    • clang format
    • irodsMonitor
    • genQuery parser

  • New Logger

    • default log levels (currently warn)
      • independently for each of the 10 'types'
        • legacy (rodsLog), server, agent
        • 7 plugins (microservice, resource, api, rule_engine, database, network, auth)
    • syslog default configuration
      • writing to /var/log/irods/irods.log
      • writing to /var/log/irods/irods_delay_queue.log
    • logrotate default configuration
      • weekly, compressed, 26x
    • consider means to point different 'types' to separate log files/locations
    • RFC

  • Cacheless and Detached Storage Resources

    • remove the need for a compound resource and cache management
    • first work on S3
    • plan to do the same with HPSS, unixfilesystem, WOS, UnivMSS
    • HOST_MODE=['archive_attached', 'cacheless_attached', 'cacheless_detached']
    • fetching credentials from somewhere else… TBD
    • can my users BYOB(bucket/identity/credentials)? Only accessible to particular set of users.
    • How would bad authentication to S3 get back to the user?
    • #1849

  • Metadata RFC

    • irods::storage_tiering::
    • filesystem::
    • want to document the usage patterns
    • any other conventions that we should consider?
    • RFC

  • Metadata Templates Working Group

    Will meet on Tuesday, Jan 15 2019, 10am EST

  • Active Development Work

    • Python iRODS Client (PRC)

    • Access Protocols

    • iRODS Capability - Automated Ingest

    • Lustre Integration

    • Python Rule Engine Plugin

      • GenQuery iterator as module, genquery.py
      • now using C++17
      • refactored compilation units (due to boost::python) (reduce rebuild times)
      • working with irods/irods master

    • Storage Tiering Capability Package

    • Metalnx packaging

    • iRODS clients listing

      • adding to website

    • Cacheless and Detached S3

      • whiteboarded
      • working on initial prototype

    • Continuous Integration (CI)

      • refactoring to use Docker
      • public Jenkins
      • move to pull-request and peer-review
      • adding CockroachDB database plugin
      • adding Ceph RADOS resource plugin
      • adding QueryArrow database plugin
      • adding Nestle R Client Library
      • adding Metalnx
      • adding baton/tears
      • adding automated ingest
      • adding storage tiering

  • Ongoing Items

  • Discussion Items

Terrell Russell