News / 31 July 2018

iRODS Development Update: July 2018

We're starting to pick up the pace again -- and we've got quite a few updates this month.

Earlier this month we posted An Automated Ingest and Sync Policy Example where we shared a policy set that embodies "nearly every one of our new proofs-of-concept and potential members in the last year." It's quite powerful and requires only a bit of configuration to set up - check it out.

The irods/externals repository has been updated and all the software that iRODS depends on has been bumped to their latest versions for what will become iRODS 4-3-stable. This includes Clang 6.0, CMake 3.11, and Boost 1.67. The repositories at have been populated and if you're building the master branch of iRODS, you can install these new versions alongside your existing irods-externals (the version numbers are included in the package names so they do not collide with one another).

With the landing of 4.3.0-related externals, we are readying for the first run of clang-format and the integration of the work to refactor logging.

One of the limitations of using the Python Rule Engine Plugin has proven to be the lack of an integrated GenQuery iterator (in the iRODS Rule Language, we have LIGQ). We are working on making a module available so GenQuery results can be returned as a Python iterable.

We have begun work on a performance lab to deterministically select the best queueing technology for the automated ingest framework. We have very short jobs (we're usually just issuing a filesystem stat()) and would like to know that any sized filesystem can be scanned in a set amount of time by simply adding more workers. The first release was based on RQ, our current prototype is based on a default Celery implementation, and our other candidates to be tested include RabbitMQ and Bee-Queue.

July Technology Working Group

Terrell Russell