iRODS Pluggable Rule Engine Demo

I presented at CurateGear 2016 in Chapel Hill, NC last week and demonstrated some of the capabilities of the upcoming iRODS Pluggable Rule Engine. It will be available at part of iRODS 4.2 in a couple months.

The slides are available at

Three Rule Engine Plugins

I demonstrate rules written in the existing iRODS Rule Language, Javascript, and Python.

Additionally, I install an auditing rule engine plugin written in C++ to demonstrate the list of policy enforcement points that are hit during a simple execution of an iRODS Agent (in this case, a basic iput).

Rule Engine Plugin LOC (w/ comments)
iRODS Rule Language 253
Javascript 244
Python 252
Auditing (C++) 157

Each of the rule engine plugins are written in C++ and ~200 lines of code (without comments and whitespace). Given an appropriate rule engine plugin, iRODS rules can be written in any language (both interpreted and compiled).

The rule engine plugin interface to iRODS is defined by four (soon, probably five) operations:

  • start
  • stop
  • rule_exists
  • exec_rule
  • (probably also) prepend_to_rulebase

Each of the three rule engine plugins are running concurrently. Each of the three demonstration rule bases will call rules written in the other two languages.

Policy Enforcement Points

A Policy Enforcement Point (or PEP) is a location in the iRODS Server code where policy can be inserted by an organization. Policy takes the form of a set of rules that are executed by the rule engine.

The following is a sorted list of some of the dynamic PEPs hit by a simple iput by the auditing rule engine plugin:

They represent a pre- and post- hook for every operation in the system. Each is an opportunity to run custom code to enforce an organization’s data management policy.

Consider, for example, a rule to transfer ownership of data objects to the project manager when a user is deleted; the trigger (or PEP) is the deletion of the user.

Rules could be written to extract metadata or pre-process data whenever a file is uploaded to a storage device. Or, upon access to particular data objects, a rule can create a log of the event, send an email notification to the project manager, or perform some other task you need to occur as a result of the data’s access.

I use acPostProcForPut() for this demo.

This is the location in the code that is hit after a file has been uploaded into iRODS. This location in the code has a full picture of the context in which it is executing. It knows the username, the filename, the location on disk, etc.

Three Rule Bases

Each rule engine plugin is demonstrated with an accompanying rule base file written in the associated language. They are ‘turned on’ by listing them in an ordered stanza in /etc/irods/server_config.json:

The following three rule bases are installed into /etc/irods/:

iRODS Rule Language



DEMO – Baseline

To get started and establish a baseline, we run a simple iput without manipulating any of the PEPs in the system.

iput a file into iRODS


An agent was started by the server, the agent received a disconnect from the client, exited, and then the server notes the agent exited.

DEMO 1 – Just Print To rodsLog

First, we overload the base acPostProcForPut() which was a no-op (demonstrated in the baseline). We do this by making sure the name of the function in the javascript rule base is the same (and confirm that the Javascript rule engine plugin is listed above the existing iRODS Rule Language rule engine plugin in server_config.json). By confirming the string is the same, we can see that it is fired when running the same iput puppies.jpg command.

overload the acPostProcForPut() policy enforcement point

remove, then iput the same file into iRODS


Note the first four lines are just the client connecting and then disconnecting during the irm of puppies.jpg.

The next few lines show that the Javascript acPostProcForPut() was triggered. It then called over to the irodsFunc0 function in, followed by a call to pyFunc0 in Then the Javascript function ends, and the agent and server print out their disconnection messages.

This demonstrated calling from language to language with functions with zero parameters. Note that the writeLine function is defined by iRODS out of the box, and did not need to be implemented in any of these additional demo rule bases.

DEMO 2 – Parameters and Error Handling

Next, we overload, again, the acPostProcForPut() PEP by preparing the file. It will take precedence since the Python rule engine plugin is listed before the Javascript rule engine plugin in server_config.json.

overload the acPostProcForPut() policy enforcement point

remove, then iput the same file into iRODS


This result is a bit deeper. The first four lines are the irm again. The next section begins with the Python acPostProcForPut() line, verifying that the Python PEP took precedence.

The Python called irodsFunc1() in which called pyFunc1() back in

Then the Python called jsFunc1() which tried to call doesnotexist() which is not defined in any rule base. The Javascript used a try…catch pattern to handle the error gracefully and return control to the Python rule (note that ‘JAVASCRIPT – END – jsFunc1()’ was never printed).

PLUGIN_ERROR_MISSING_SHARED_OBJECT is emitted due to the iRODS behavior of looking for rules of a certain name, and then falling through to look for microservice plugins of the same name. Since no microservice plugin was found either, the log complains loudly.

After the error handling has returned, the Python acPostProcForPut() reads two session variables from the running context and prints them to the log. You can see that the ‘userNameClient’ is ‘rods’ and that the ‘dataSize’ of puppies.jpg is ‘95891’ bytes. Then the Python exits and the agent and server exit cleanly.

DEMO 3 – EXIF Extraction Demo

This third demo uses the Python Image Library to extract EXIF information from ‘puppies.jpg’ and then save that metadata into the iRODS metadata catalog (iCAT) as AVU (attribute-value-unit) triples. In this case, the Units will be empty.

First, the DEMO 2 acPostProcForPut() name must be changed to something else so that it does not fire. Then, DEMO 3 acPostProcForPut() must be enabled by removing the XXX. Alternately, we could have defined the DEMO 3 PEP before the DEMO 2 PEP. Ultimately, we want to make sure we are running the DEMO 3 PEP when we iput a file.

overload the acPostProcForPut() policy enforcement point

remove, then iput the same file into iRODS


The log here is much less verbose than in DEMO 2. The first four lines represent the irm.

The next section has the physical path and then the logical path of the file in flight (puppies.jpg). Then we see the EXIF complete line. The work in DEMO 3 was done in the database rather than in the rodsLog.

We can see the extracted EXIF information in the database with a separate command.

list the newly extracted and associated metadata

This information be verified by using a third-party tool jhead:

Conclusion and Next Steps

This is an extremely powerful abstraction and will allow iRODS 4.2 to be integrated more tightly into existing infrastructure and workflows. In turn, this means that iRODS will be able to make stronger claims about the data that is under management, rather than calling out to third party code that is harder to verify over time.

The next few candidates for additional rule engine plugins are: Haskell (already under development), Go, PHP, Ruby, and Perl. If you are interested in working on these new rule engine plugins, please get in touch with the iRODS Development Team to coordinate and/or assist.