News / 21 August 2020

iRODS Development Update: August 2020


Another month from home, another month of development with a great team.

The work on the S3 plugin and connector are complete, as is the work on the parallel transfer engine. The new iRODS Command Line Interface (CLI) is methodically being populated with new subcommands to exercise some of the new parallel endpoints being exposed in the server. We are finalizing the connection between NFSRODS and the Hard Links Rule Engine Plugin to support hard links within the NFSv4.1 mountpoint.

And earlier this week, we hosted the most recent TRiRODS which demonstrated usage of the new iRODS Container Storage Interface (CSI) Driver for Kubernetes.

A final note, we are hiring an additional core C++ software developer(!). If you're reading this and want to work with us, please get in touch.

August Technology Working Group

  • 4.2.9

    • 37 issues, 14 bugs
    • intermediate replicas
    • logical locking

  • 4.2 Backlog

    • 192 issues, 101 bugs

  • 4.3.0

    • 114 issues, 45 bugs
    • clang format
    • new logger (rsyslog or stdout)
    • irodsDelayServer refactoring / with implicit remote()
    • CentOS8 will be supported

  • Metadata Templates Working Group

    • progress towards SwaggerAPI
    • working towards demo of Metalnx-API-iRODS
    • next meeting - August 18, 2020, 10am ET

  • Authentication Working Group

    • building a new API endpoint to provide a flexible authentication mechanism
    • new authentication plugins will drive the flow
    • next meeting - August 25, 2020, 10am ET

  • TRiRODS

    • August 19, 2020, 4pm ET
    • Illyoung Choi, CyVerse
    • iRODS Container Storage Interface (CSI) Driver

  • New Development Work

    • Rule Engine Plugin Framework
      • RULE_ENGINE_SKIP_OPERATION no longer skips post PEPs
        • required to make Hard Links play well with others

  • Active Development Work

    • iRODS Command Line Interface (CLI)

      • single binary
      • eventual replacement for ~50 iCommands
      • ls/put/get/rm/cp

    • Prototype RPC API framework

      • Leverages design from Authentication Working Group
      • Supports synchronous and asynchronous operations
      • Allows for pause, resume, and cancellation of operations
      • Allows for parallel server side behavior

    • Policy Composition

      • releases soon for storage tiering, indexing, publishing

    • NetCDF microservices

      • modernizing packaging for 4.2.x
      • working with CyVerse
      • full HDF5 support could come later

    • Python Rule Engine Plugin

      • fixed map() method of PluginContext, now working for all PEPs
      • worked on memory leaks for multiple/looping microservice calls

    • Metalnx

      • new release soon with search and indexing update
      • want to turn on quotas visibility
      • testbed for metadata templates initial implementation
      • working to eventually remove database dependency
      • awaiting CI

    • Hard Links Rule Engine Plugin

      • patch in progress for 4.2.9.0

    • C++-based REST API

      • working on packaging
      • part of new Data Transfer Nodes pattern

    • C++ S3 API

      • working on packaging
      • part of new Data Transfer Nodes pattern

    • Python iRODS Client (PRC)

      • merged multiple authentication test harness
      • awaiting CI

    • Cacheless S3 Plugin / S3_Connector

      • streaming transport is passing all tests
      • will do some reorganization of the code before release
      • awaiting CI

    • NFSRODS

      • working on correct Hard Links usage
      • awaiting CI

    • Parallel Transfer Engine

      • complete - merged into master

    • Indexing Capability

      • working to index collection metadata
      • working to incorporate geospatial / GeoJSON / bounding boxes
      • second implementation coming for Apache Solr

    • Publishing Capability

      • needs more README
      • awaiting CI

    • Continuous Integration (CI)

      • core
        • ub16 / ub18 / cen7
        • pg / mysql / oracle
        • core / topology / federation / upgrade
      • plugins
        • audit amqp
        • collection mtime rule engine plugin
        • curl
        • hard links
        • indexing
        • kerberos
        • logical quotas
        • metadata guard policy engine
        • s3
        • python rule engine plugin
        • storage tiering
      • plugins (to be added)
        • publishing
      • clients (to be added)
        • baton/tears
        • automated ingest
        • Python iRODS Client (PRC)
        • iRODS Command Line Interface (CLI)
        • Metalnx
        • NFSRODS
      • OS (to be added)
        • CentOS8
        • Ubuntu20
        • SUSE Linux Enterprise Server (SLES)
        • OpenSUSE
      • maybe later
        • CephFS via unixfilesystem
        • gsi
        • Nestle R Client Library
        • Ceph RADOS resource plugin
        • CockroachDB database plugin
        • QueryArrow database plugin

  • Background Items

    • Automated Ingest Capability
      • refactoring to abstract the source filesystem/object
        • considering kafka as source of events (for OMERO via debezium.io)
      • investigating splitting storage backends
        • celery workers -> redis
        • python/iRODS cache -> scylla (C++ distributed keystore)
    • AWS S3 Lambda
      • future release
        • could brute-force confirm checksums to detect renames
        • would persist metadata
    • Unified Storage Tiering Capability
    • Logical Quotas
    • Parallel Filesystem Integration
      • iRODS API plugin
      • have split BeeGFS and Lustre implementations into aggregator and listener
      • testing against known configurations
    • SMBRODS project to surface iRODS as SMB
    • CockroachDB Database Plugin
    • Cloud Browser
    • Member Ticketing System

  • Discussion

Terrell Russell