/ 30 December 2018
iRODS Development Update: December 2018
Not as much to report this December - everything slows down quite a bit in University land over the holiday break.
Work on the Automated Ingest Framework has proceeded, mostly focused on hardening and testing coverage.
The cacheless and detached storage resource effort is very close to a working implementation for all plugin operations when talking to S3 (through Minio for testing).
And in some experimental work, iRODS can now be installed, configured, and started up while using UUIDs as primary identifiers within the catalog rather than sequences. This is exciting as we continue to target a future consisting of all iRODS servers in the role of catalog provider (via CockroachDB).
2018 has flown by and we are very happy with our progress. With 4-1-stable now EOL, all eyes are focused on 2019, increased adoption, and continued community development of higher-order solutions built on top of iRODS.
Happy New Year everyone!
December Technology Working Group
- Released November 1, 2018
- 4-1-stable is now EOL
- 103 issues open, 47 bugs
- clang format
- genQuery parser
- default log levels (currently warn)
- independently for each of the 10 'types'
- legacy (rodsLog), server, agent
- 7 plugins (microservice, resource, api, rule_engine, database, network, auth)
- syslog default configuration
- writing to /var/log/irods/irods.log
- writing to /var/log/irods/irods_delay_queue.log
- logrotate default configuration
- consider means to point different 'types' to separate log files/locations
Cacheless and Detached Storage Resources
- remove the need for a compound resource and cache management
- first work on S3
- plan to do the same with HPSS, unixfilesystem, WOS, UnivMSS
- HOST_MODE=['archive_attached', 'cacheless_attached', 'cacheless_detached']
- fetching credentials from somewhere else… TBD
- can my users BYOB(bucket/identity/credentials)? Only accessible to particular set of users.
- How would bad authentication to S3 get back to the user?
- want to document the usage patterns
- any other conventions that we should consider?
Metadata Templates Working Group
Will meet on Tuesday, Jan 15 2019, 10am EST
Active Development Work
Python iRODS Client (PRC)
iRODS Capability - Automated Ingest
Python Rule Engine Plugin
- GenQuery iterator as module, genquery.py
- now using C++17
- refactored compilation units (due to boost::python) (reduce rebuild times)
- working with irods/irods master
Storage Tiering Capability Package
iRODS clients listing
Cacheless and Detached S3
- working on initial prototype
Continuous Integration (CI)
- refactoring to use Docker
- public Jenkins
- move to pull-request and peer-review
- adding CockroachDB database plugin
- adding Ceph RADOS resource plugin
- adding QueryArrow database plugin
- adding Nestle R Client Library
- adding Metalnx
- adding baton/tears
- adding automated ingest
- adding storage tiering
- Coordinate on release of Jupyter Lab Notebook integration