Merge pull request #729 from IBM/develop

chore(release): Bug fixes and enabling caching for import

README.md

@@ -67,6 +67,10 @@ Trestle runs on most all python platforms (e.g. Linux, Mac, Windows) and is avai
 
 Complete documentation, tutorials, and background on compliance can be found [here](https://ibm.github.io/compliance-trestle).
 
+## Demos
+
+A collection of demos utilizing trestle can be found in the related project [compliance-trestle-demos](https://github.com/IBM/compliance-trestle-demos).
+
 ## Development status
 
 Compliance trestle is currently beta. The expectation is that in ongoing work there may be un-announced changes that are breaking within the trestle codebase. With the release of NIST's version 1.0.0 of OSCAL we expect that these changes will be decreasing in size as trestle approaches a 1.0.0 release for itself.

docs/api_reference/trestle.tasks.profile_to_osco.md

@@ -0,0 +1,2 @@
+::: trestle.tasks.profile_to_osco
+handler: python

docs/cli.md

@@ -171,7 +171,7 @@ Passing `-iof` or `--include-optional-fields` will make `trestle create` generat
 
 ## `trestle import`
 
-This command allows users to import existing OSCAL files so that they can be managed using trestle. For example `trestle import -f existing_catalog.json -o my_existing_catalog` will import `existing_catalog.json` into a new folder under `catalogs` as shown below:
+This command allows users to import existing OSCAL files so that they can be managed using trestle. For example `trestle import -f /local_dir/existing_catalog.json -o my_existing_catalog` will import `existing_catalog.json` into a new folder under `catalogs` as shown below:
 
 ```
 .
@@ -184,12 +184,14 @@ This command allows users to import existing OSCAL files so that they can be man
 
 The following options are supported:
 
-- `-f or --file`: specifies the path of an existing OSCAL file.
+- `-f or --file`: specifies the path of an existing OSCAL file or URL to a remote file.
 - `-o or --output`: specifies the name/alias of a model. It is used as the prefix for the output filename under the `dist` directory and for naming the source subdirectories under  `catalogs`, `profiles`, `component-definitions`, `system-security-plans`, `assessment-plans`, `assessment-results` or `plan-of-action-and-milestones`.
 
-The import subcommand can determine the type of the model that is to be imported by the contents of the file.
+The `--file` option may be an absolute or relative path, and it may be a URL.  For details on allowed formats please see the documentation for the `href` command.  The file must be imported from outside the current trestle directory or an error will result.
 
-Finally, during the import process the file must pass the `validate` test described below for the command, `validate`.  If the file does not pass validation a warning will be given describing the nature of the problem and the import will fail.
+The import subcommand can determine the type of the model that is to be imported by the contents of the file.  But the file name must end with an allowed json or yaml extension: `.json, .yaml, .yml`
+
+During the import process the file must pass the `validate` test described below for the command, `validate`.  If the file does not pass validation a warning will be given describing the nature of the problem and the import will fail.
 
 Once a file has been imported it can be split into a rich tree of sub-components as shown at the top of this document.  But the file must be imported first.
 

docs/tutorials/trestle_sample_workflow.md

@@ -14,11 +14,12 @@ The examples shown will work for linux and mac, but Windows will require the fol
 <li>use backslashes `\` for file paths (this is optional in most cases)
 <li>use copy instead of cp (unless you have cp installed)
 <li>use md instead of mkdir (unless you have mkdir installed)
+<li>quotes (') are often not needed unless the text includes spaces, but if quotes are needed they should be double quotes (")
 </ul>
 
 Commands are shown without prompts so they are easy to cut and paste, and responses by trestle are shown with >>> at the start of the line.  In actual usage the >>> would not appear.
 
-Be sure to include the quotes (' ') as shown in the examples, e.g. `merge -e 'catalog.*'`
+Be sure to include the quotes (' ') as shown in the examples, e.g. `merge -e 'catalog.*'` \[On windows you should probably use double quotes (") as needed.\]
 
 In this tutorial you will see sections that contain dropdown that is revealed when you click on them.  Below is an example ("Like this").  Be sure to click on those sections to see their contents - and then close them if you like.
 
@@ -49,22 +50,22 @@ trestle init
 
 ## Step 2: Import a catalog from the trestle sample data directory into your trestle workspace
 
-First download a catalog from the [NIST OSCAL github site](https://github.com/usnistgov/OSCAL).  The specific catalog is [NIST_SP-800-53_rev5_catalog.json](https://raw.githubusercontent.com/usnistgov/oscal-content/master/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json)
+For this tutorial we will use a catalog file from NIST, but we first must `import` it into the trestle workspace.  This can be done either by first downloading the file locally and then importing it, or you can download it directly using its url address.
 
-Save that file on your disk at a location of your choice outside of the trestle directory, and call it `catalog.json`.  The file is approximately 3 MB in size.
+We will import the file directly from the [NIST OSCAL github site](https://github.com/usnistgov/OSCAL).  The specific catalog is [NIST_SP-800-53_rev5_catalog.json](https://raw.githubusercontent.com/usnistgov/oscal-content/master/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json)
 
-Files should be pulled into your trestle workspace using the command `import`, which will check the
-validity of the file including the presence of any duplicate uuid's.  If the file is manually created
-please be sure it conforms with the current OSCAL schema (OSCAL version 1.0.0) and has no defined uuid's that are duplicates.
-If there are any errors the Import will fail and the file must be corrected.
-
-Import the file from the trestle root directory with
+Import the file from the url with the following command:
 
 ```
-trestle import -f MY_DOWNLOAD_DIRECTORY/catalog.json -o mycatalog
+trestle import -f https://raw.githubusercontent.com/usnistgov/oscal-content/master/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json -o mycatalog
 ```
 
-*Here MY_DOWNLOAD_DIRECTORY corresponds to the directory in which you downloaded the catalog.json file.*  On Windows platforms you can use either a forward (/)or backward slash (\\) in the path of the file you are importing.
+As a reminder, you could also have imported the file from a local directory on your file system, or an sftp:// address.  But the file must first be imported to the trestle workspace in order for it to be directly manipulated by trestle as in this tutorial.
+
+The `import` command will also check the
+validity of the file including the presence of any duplicate uuid's.  If the file is manually created
+please be sure it conforms with the current OSCAL schema (OSCAL version 1.0.0) and has no defined uuid's that are duplicates.
+If there are any errors the Import will fail and the file must be corrected.
 
 <br>
 <details>

tests/data/drawio/single_tab_metadata_compressed.drawio

@@ -1,2 +1 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<mxfile host="Electron" modified="2021-05-18T07:20:30.904Z" agent="5.0 (Macintosh; Intel Mac OS X 11_3_1) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/14.5.1 Chrome/89.0.4389.82 Electron/12.0.1 Safari/537.36" etag="O3AE-tferUDKfRkdlYbR" version="14.5.1" type="device"><diagram id="Ul-z1ypJsJkvYuxd73nE" name="Page-1">fZNLb4MwDIB/DcdVEPpYr+362KRVmlptu5rGI6kCQSEddL9+CQkDVrUXiD/biZ9BvMzqjYKCvUqKIiAhrYP4KSAkisKp+VlycWQ+nzmQKk69UQf2/Ac9DD09c4rlwFBLKTQvhvAo8xyPesBAKVkNzb6kGL5aQIpXYH8EcU0/ONXM0Ucy6/gWecral6Pp3GkyaI19JiUDKqseildBvFRSaneSyckmQEIBiSliY+cTxtJf/w3ijA7mBiId3dNhrRXcUt7zbzvThZnVSxSms2snmoMPd6AN+f/iKsx1vwrrGw6ncLdcb/nLfr2YTM676eHt8/3B3+ICa8wOjJfWy37ATpWCysrStpYLHAVkClkRxIs8Ke3PyMIEsEiUOaX29GxLzKC9oeR5anptigzJ6Nr6wLzuzym3b2WogYKGkW+tvrTzouQ5p2izCo17xbjGfQFHq63MhhjGdGa7GzXRNfmh0ljfLFzUa8IGpXlaXYxJt2HOxe8XCb1cddM6HnvGepNKJh6C35D0727S77NvVSt289roelsfr34B</diagram></mxfile>
+<mxfile host="Electron" modified="2021-09-15T00:18:21.745Z" agent="5.0 (Macintosh; Intel Mac OS X 11_5_2) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/14.6.13 Chrome/89.0.4389.128 Electron/12.0.7 Safari/537.36" etag="-7dqLridVgtS7V8w-plG" version="14.6.13" type="device"><diagram id="Ul-z1ypJsJkvYuxd73nE" name="Page-1">fVPbbsIwDP2aPg6VFNh4hXHZpCFNoG2vLvGaoLSp0rCWff0SkqwtCF7a+Pj4bkfJPG9WCkr2JimKiMS0iZLniJDhMJ6Yn0VODpmOYgdkilNPaoEt/0UPBtqRU6x6RC2l0Lzsg3tZFLjXPQyUknWf9i1FP2oJGV4B2z2Ia/STU80c+kQeW3yNPGMh8nAydZocAtlXUjGgsu5AySJK5kpK7V4yPdgCSCwgNU0883zBWHn3PyCO6MDCgEgH93TYaAW3lPfsw2TaNPNmjsJMdulE8/Dp9rQxv2yuwkJ3u7C8YXCIN/Plmr9ul7Px+LiZ7N6/Ph68F5fYmbZjvLJW9gN2qxTUVpZ2tFzgICITyMsomRVpZX9GFiaBWarMK7OvF9tiBsFDxYvMzNo0GdLBNXvHvO7fqLCxctRAQcPAj1afwr4oeSwo2qpiY14zrnFbwt5qa3MhBmM6t9MdnrO77FIoGZXGpgP5rq1QmtDqZCjthTkTf18k9nLdbuso3BzrbCoZexD8hWT/vkl3zn5UQWz39azrXH2y+AM=</diagram></mxfile>