Add put method for Collections (#219)

* Add put method for Collections

Change defaults for checksum-related API method keywords to False in
order to make behaviour and responsibility for the decision clearer.
API callers are now responsible for setting these flags to True where
they are needed.

Separate checksum retrieval, updates and verification more clearly.

Add a checkum keyword to DataObject::list.

Add checksum size/consistency check for zero-length data objects.

* Add local_checksum parameter to Collection::put

Adding this parameter allows a callable to be supplied which can
obtain per-file checksums during a recursive put operation. e.g. by
reading from an accompanying .md5 file or by deciding, based on file
size, whether to calculate checksums or to insist that an earlier
process supply one pre-calculated.

Improve test coverage of checksum operations.

* Add an existence check for iRODS groups when setting up tests

This check make re-running tests more reliable in cases where the test
teardown has failed to remove test groups from iRODS.

Dockerfile.dev

@@ -17,6 +17,7 @@ RUN apt-get update && \
     git \
     make \
     libbz2-dev \
+    libffi-dev \
     libncurses-dev \
     libreadline-dev \
     libssl-dev \
@@ -43,6 +44,8 @@ COPY ./docker/install_pyenv.sh /app/docker/install_pyenv.sh
 
 RUN /app/docker/install_pyenv.sh
 
+ENV MAKE_OPTS="-j 8"
+
 RUN pyenv install "$PYTHON_VERSION"
 RUN pyenv global "$PYTHON_VERSION"
 

README.md

@@ -265,7 +265,7 @@ distinguish between these circumstances.
 ## Requirements
 
 - The `baton-do` executable from the [baton](https://github.com/wtsi-npg/baton)
-  iRODS client distribution.
+  iRODS client distribution. Version >=4.3.1 is required.
 - The unit tests use the
   [iRODS client icommands](https://github.com/irods/irods_client_icommands)
   clients. These are not required during normal operation.

docker-compose.yml

@@ -3,7 +3,7 @@ services:
   irods-server:
     platform: linux/amd64
     container_name: irods-server
-    image: "ghcr.io/wtsi-npg/ub-16.04-irods-4.2.7:latest"
+    image: "ghcr.io/wtsi-npg/ub-18.04-irods-4.2.11:latest"
     ports:
       - "127.0.0.1:1247:1247"
       - "127.0.0.1:20000-20199:20000-20199"
@@ -26,6 +26,7 @@ services:
     environment:
       IRODS_ENVIRONMENT_FILE: "/home/appuser/.irods/irods_environment.json"
       IRODS_PASSWORD: "irods"
+      IRODS_VERSION: "4.2.11"
     depends_on:
       irods-server:
         condition: service_healthy

src/partisan/icommands.py

@@ -20,6 +20,7 @@
 
 import json
 import os
+import re
 import shlex
 import subprocess
 from io import StringIO
@@ -43,6 +44,18 @@ def rmgroup(name: str):
     _run(cmd)
 
 
+def group_exists(name: str) -> bool:
+    info = iuserinfo(name)
+    for line in info.splitlines():
+        log.debug("Checking line", line=line)
+        if re.match(r"type:\s+rodsgroup", line):
+            log.debug("Group check", exists=True, name=name)
+            return True
+
+    log.debug("Group check", exists=False, name=name)
+    return False
+
+
 def iuserinfo(name: str = None) -> str:
     cmd = ["iuserinfo"]
     if name is not None: