News / 30 September 2018

iRODS Development Update: September 2018

September was a month of polishing and releasing things we've been working on for a while. Testing and production rollouts uncovered a few bugs that were quickly fixed and incorporated. Planning continues for SC18 in Dallas.

iRODS 4.2.4 was released early this month, along with seven plugins for compatibility. It marked iexecmd as deprecated.

Automated Ingest has seen six releases this month, v0.2.0, v0.2.1, v0.2.2, v0.3.0, v0.3.1, and v0.3.2. The v0.2.0 release changed the queuing backend to Celery. The v0.3.0 release added the ability to scan an S3 bucket and provides a significant speedup by chunking the work done in each celery task. Cursory testing suggested we set the default to 50 files_per_task, but as a runtime configuration, we've seen performance improvements up to 300 files_per_task for some server topology and network configurations. This is important because the framework appears to be providing linear scaling (so far). More testing is coming, but it takes time.

The Python iRODS Client v0.8.1 was released. A bug was fixed in filename handling with ampersands and size information is now available for replicas.

The NFSRODS codebase is being readied for its initial release. We are exploring Docker as the primary deployment mechanism.

We are also working to move our CI system from VSphere to Docker. This work would make our Jenkins instance runnable on an individual developer's workstation and reduce our dependency on the limited IP range we currently use.

Work on 4.1.12 is nearly complete. We're so close. It is expected to be the final release in the 4.1.x series. We strongly encourage an upgrade to 4.2+ at this time.

September Technology Working Group

  • 4.2.4

    • released Sept 5, 2018
    • along with seven plugins
  • iRODS Capability - Automated Ingest

    • released v0.2.0, v0.2.1, and v0.2.2

      • based on Celery and Redis
      • handling non-UTF8-encodeable filenames
    • released v0.3.0, v0.3.1, and v0.3.2

      • added S3 scanning
      • added chunking via files_per_task
      • 4M files in 1.5h on GPFS with 2x32 workers
  • 4.1.12

    • 3 issues, 1 bug.
  • 4.2.5

    • 74 issues, 47 bugs.
  • 4.3.0

    • 110 issues open, 48 bugs.
    • moving to new clang/cmake/externals
    • c++17
    • clang format
    • new logging
    • irodsMonitor
    • genQuery parser
  • Metadata Templates Working Group

    Met on September 18 to discuss the storage of schema information in AVUs. The two candidates consist of GUIDs for everything (flexible, but more opaque) vs. a many-to-many mapping from objects to templates.

  • Active Development Work

  • Ongoing Items

Terrell Russell