Merge branch 'develop' into 7423-jms-config

doc/release-notes/7263-banner-messages.md

@@ -0,0 +1,7 @@
+## Major Use Cases
+
+- New API for Banners..
+
+## Notes to Admins
+
+The functionality previously provided by the DB settings :StatusMessageHeader and ::StatusMessageText is no longer supported and is now provided through the Manage Banner Messages API. Learn more in the [Native API Guide](https://guides.dataverse.org/en/5.x/api/).

doc/release-notes/7275-aux-files.md

@@ -0,0 +1,3 @@
+## Notes for Tool Developers and Integrators
+
+Experimental endpoints have been added to allow auxiliary files to be added to datafiles. These auxiliary files can be deposited and accessed via API. Later releases will include options for accessing these files through the UI. For more information, see the Auxiliary File Support section of the [Developer Guide](https://guides.dataverse.org/en/5.3/developers/).

doc/release-notes/7418-datasourcedefinition.md

@@ -0,0 +1,88 @@
+## Release Highlights
+
+### Easier Configuration of Database Connections
+
+Dataverse now being able to use up-to-date Java technologies, transforms
+the way how to configure the connection to your PostgreSQL database.
+
+In the past, the configuration of the connection has been quite static
+and not very easy to update. This has been an issue especially for cloud
+and container usage.
+
+Using MicroProfile Config API (#7000, #7418), you can much more easily specify configuration
+details. For an overview of supported options, please see the
+[installation guide](https://guides.dataverse.org/en/5.3/installation/config.html#jvm-options).
+
+Note that some settings have been moved from domain.xml to code such as min and max pool size.
+
+## Notes for Dataverse Installation Administrators
+
+### New JVM Options
+
+- dataverse.db.name
+- dataverse.db.user
+- dataverse.db.password
+- dataverse.db.host
+- dataverse.db.port
+
+<!-- ## Update to Payara Platform 5.2020.6 -->
+<!-- ... -->
+
+<!-- PLACEHOLDER REPLACEMENT TEXT FOR PAYARA UPGRADE NOTE #7417 -->
+🚨 THIS VERSION OF DATAVERSE **REQUIRES** UPGRADING TO PAYARA 5.2020.6. 🚨
+
+<!-- ... -->
+
+## Upgrading from earlier releases
+
+ℹ️ You need to update the Payara Application Server before continuing here. See above.
+
+1. Undeploy the previous version.
+```
+<payara install path>/asadmin list-applications
+<payara install path>/asadmin undeploy dataverse-<version>
+```
+
+(where `<payara install path>` is where Payara 5 is installed, for example: `/usr/local/payara5`)
+
+2. Update your database connection before updating.
+
+Please configure your connection details, replacing all the `${DB_...}`.
+(If you are using a PostgreSQL server on `localhost:5432`, you can omit `dataverse.db.host` and `dataverse.db.port`.)
+
+```
+<payara install path>/asadmin create-system-properties "dataverse.db.user=${DB_USER}"
+<payara install path>/asadmin create-system-properties "dataverse.db.host=${DB_HOST}"
+<payara install path>/asadmin create-system-properties "dataverse.db.port=${DB_PORT}"
+<payara install path>/asadmin create-system-properties "dataverse.db.name=${DB_NAME}"
+echo "AS_ADMIN_ALIASPASSWORD=${DB_PASS}" > /tmp/password.txt
+<payara install path>/asadmin create-password-alias --passwordfile /tmp/password.txt dataverse.db.password
+rm /tmp/password.txt
+```
+
+<!-- PLACE HOLDER FOR EJB TIMER DATABASE RESET NOTE #5345 -->
+
+Now you are safe to delete the old password alias and DB pool:
+```
+<payara install path>/asadmin delete-jdbc-connection-pool --cascade=true dvnDbPool
+<payara install path>/asadmin delete-password-alias db_password_alias
+```
+
+3. Stop payara and remove the generated directory, start.
+```
+service payara stop
+# remove the generated directory:
+rm -rf <payara install path>/payara/domains/domain1/generated
+service payara start
+```
+
+3. Deploy this version.
+```
+<payara install path>/bin/asadmin deploy dataverse-5.3.war
+```
+
+4. Restart Payara
+```
+service payara stop
+service payara start
+```

doc/sphinx-guides/source/api/dataaccess.rst

@@ -40,13 +40,13 @@ A curl example using a DOI (no version):
   export SERVER_URL=https://demo.dataverse.org
   export PERSISTENT_ID=doi:10.70122/FK2/N2XGBJ
 
-  curl -O -J -H "X-Dataverse-key:$API_TOKEN" $SERVER_URL/api/access/dataset/:persistentId/?persistentId=$PERSISTENT_ID
+  curl -L -O -J -H "X-Dataverse-key:$API_TOKEN" $SERVER_URL/api/access/dataset/:persistentId/?persistentId=$PERSISTENT_ID
 
 The fully expanded example above (without environment variables) looks like this:
 
 .. code-block:: bash
 
-  curl -O -J -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx https://demo.dataverse.org/api/access/dataset/:persistentId/?persistentId=doi:10.70122/FK2/N2XGBJ
+  curl -L -O -J -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx https://demo.dataverse.org/api/access/dataset/:persistentId/?persistentId=doi:10.70122/FK2/N2XGBJ
 
 Download By Dataset By Version
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

doc/sphinx-guides/source/api/native-api.rst

@@ -3,7 +3,7 @@ Native API
 
 Dataverse 4 exposes most of its GUI functionality via a REST-based API. This section describes that functionality. Most API endpoints require an API token that can be passed as the ``X-Dataverse-key`` HTTP header or in the URL as the ``key`` query parameter.
 
-.. note:: |CORS| Some API endpoint allow CORS_ (cross-origin resource sharing), which makes them usable from scripts runing in web browsers. These endpoints are marked with a *CORS* badge.
+.. note:: |CORS| Some API endpoint allow CORS_ (cross-origin resource sharing), which makes them usable from scripts running in web browsers. These endpoints are marked with a *CORS* badge.
 
 .. note:: Bash environment variables shown below. The idea is that you can "export" these environment variables before copying and pasting the commands that use them. For example, you can set ``$SERVER_URL`` by running ``export SERVER_URL="https://demo.dataverse.org"`` in your Bash shell. To check if the environment variable was set properly, you can "echo" it (e.g. ``echo $SERVER_URL``). See also :ref:`curl-examples-and-environment-variables`.
 
@@ -2664,6 +2664,48 @@ Delete Database Setting
 Delete the setting under ``name``::
 
   DELETE http://$SERVER/api/admin/settings/$name
+  
+Manage Banner Messages
+~~~~~~~~~~~~~~~~~~~~~~
+
+Communications to users can be handled via banner messages that are displayed at the top of all pages within your Dataverse installation. Two types of banners can be configured:
+
+- A banner message where dismissibleByUser is set to false will be displayed to anyone viewing the application. These messages will be dismissible for a given session but will be displayed in any subsequent session until they are deleted by the Admin. This type of banner message is useful for situations such as upcoming maintenance windows and other downtime.
+- A banner message where dismissibleByUser is set to true is intended to be used in situations where the Admin wants to make sure that all logged in users see a certain notification. These banner messages will only be displayed to users when they are logged in and can be dismissed by the logged in user. Once they have been dismissed by a user, that user will not see the message again. This type of banner message is useful for situations where a message needs to communicated once, such as a minor terms of use update or an update about a new workflow in your Dataverse installation. 
+
+Note that HTML can be included in banner messages.
+
+Add a Banner Message::
+
+  curl -H "Content-type:application/json" -X POST http://$SERVER/api/admin/bannerMessage --upload-file messages.json
+  
+Where ``messages.json`` looks like this::
+
+  {
+    "dismissibleByUser": "true",
+    "messageTexts": [
+    {
+      "lang": "en",
+      "message": "Dismissible Banner Message added via API"
+    },
+    {
+      "lang": "fr",
+      "message": "Message de bannière ajouté via l'API"
+    }
+    ]
+  }
+     
+Get a list of active Banner Messages::
+
+  curl  -X GET http://$SERVER/api/admin/bannerMessage
+  
+Delete a Banner Message by its id::
+
+  curl  -X DELETE http://$SERVER/api/admin/bannerMessage/$id   
+  
+Deactivate a Banner Message by its id (allows you to hide a message while retaining information about which users have dismissed the banner)::
+
+  curl  -X PUT http://$SERVER/api/admin/bannerMessage/$id/deactivate    
 
 List Authentication Provider Factories
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

doc/sphinx-guides/source/developers/aux-file-support.rst

@@ -0,0 +1,36 @@
+Auxiliary File Support
+======================
+
+Auxiliary file support is experimental. Auxiliary files in Dataverse are being added to support depositing and downloading differentially private metadata, as part of the OpenDP project (OpenDP.io). In future versions, this approach may become more broadly used and supported. 
+
+Adding an Auxiliary File to a Datafile
+--------------------------------------
+To add an auxiliary file, specify the primary key of the datafile (FILE_ID), and the formatTag and formatVersion (if applicable) associated with the auxiliary file. There are two form parameters. "Origin" specifies the application/entity that created the auxiliary file, an "isPublic" controls access to downloading the file. If "isPublic" is true, any user can download the file, else, access authorization is based on the access rules as defined for the DataFile itself.
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export FILENAME='auxfile.txt'
+  export FILE_ID='12345'
+  export FORMAT_TAG='dpJson'
+  export FORMAT_VERSION='v1'
+  export SERVER_URL=https://demo.dataverse.org
+ 
+  curl -H X-Dataverse-key:$API_TOKEN -X POST -F "file=@$FILENAME" -F 'origin=myApp' -F 'isPublic=true' "$SERVER_URL/api/access/datafile/$FILE_ID/metadata/$FORMAT_TAG/$FORMAT_VERSION"
+
+You should expect a 200 ("OK") response and JSON with information about your newly uploaded auxiliary file.
+
+Downloading an Auxiliary File that belongs to a Datafile 
+--------------------------------------------------------
+To download an auxiliary file, use the primary key of the datafile, and the
+formatTag and formatVersion (if applicable) associated with the auxiliary file:
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export FILE_ID='12345'
+  export FORMAT_TAG='dpJson'
+  export FORMAT_VERSION='v1'
+
+  curl "$SERVER_URL/api/access/datafile/$FILE_ID/$FORMAT_TAG/$FORMAT_VERSION"

doc/sphinx-guides/source/developers/index.rst

@@ -32,4 +32,5 @@ Developer Guide
    geospatial
    selinux
    big-data-support
+   aux-file-support
    workflows

doc/sphinx-guides/source/installation/config.rst

@@ -1032,6 +1032,57 @@ dataverse.auth.password-reset-timeout-in-minutes
 
 Users have 60 minutes to change their passwords by default. You can adjust this value here.
 
+dataverse.db.name
++++++++++++++++++
+
+The PostgreSQL database name to use for Dataverse.
+
+Defaults to ``dataverse`` (but the installer sets it to ``dvndb``).
+
+Can also be set via *MicroProfile Config API* sources, e.g. the environment variable ``DATAVERSE_DB_NAME``.
+
+dataverse.db.user
++++++++++++++++++
+
+The PostgreSQL user name to connect with.
+
+Defaults to ``dataverse`` (but the installer sets it to ``dvnapp``).
+
+Can also be set via *MicroProfile Config API* sources, e.g. the environment variable ``DATAVERSE_DB_USER``.
+
+dataverse.db.password
++++++++++++++++++++++
+
+The PostgreSQL users password to connect with.
+
+Preferrably use a JVM alias, as passwords in environment variables aren't safe.
+
+.. code-block:: shell
+
+  echo "AS_ADMIN_ALIASPASSWORD=change-me-super-secret" > /tmp/password.txt
+  asadmin create-password-alias --passwordfile /tmp/password.txt dataverse.db.password
+  rm /tmp/password.txt
+
+Can also be set via *MicroProfile Config API* sources, e.g. the environment variable ``DATAVERSE_DB_PASSWORD``.
+
+dataverse.db.host
++++++++++++++++++
+
+The PostgreSQL server to connect to.
+
+Defaults to ``localhost``.
+
+Can also be set via *MicroProfile Config API* sources, e.g. the environment variable ``DATAVERSE_DB_HOST``.
+
+dataverse.db.port
++++++++++++++++++
+
+The PostgreSQL server port to connect to.
+
+Defaults to ``5432``, the default PostgreSQL port.
+
+Can also be set via *MicroProfile Config API* sources, e.g. the environment variable ``DATAVERSE_DB_PORT``.
+
 dataverse.rserve.host
 +++++++++++++++++++++
 
@@ -1168,7 +1219,7 @@ This JVM setting is also part of **handles** configuration. The Handle.Net insta
 .. _dataverse.handlenet.index:
 
 dataverse.handlenet.index
-+++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++
 If you want to use different index than the default 300
 
 .. _dataverse.timerServer:
@@ -1594,22 +1645,6 @@ Make the metrics component on the root dataverse a clickable link to a website w
 
 ``curl -X PUT -d http://metrics.dataverse.example.edu http://localhost:8080/api/admin/settings/:MetricsUrl``
 
-:StatusMessageHeader
-++++++++++++++++++++
-
-For dynamically adding an informational header to the top of every page. StatusMessageText must also be set for a message to show. For example, "For testing only..." at the top of https://demo.dataverse.org is set with this:
-
-``curl -X PUT -d "For testing only..." http://localhost:8080/api/admin/settings/:StatusMessageHeader``
-
-You can make the text clickable and include an additional message in a pop up by setting ``:StatusMessageText``.
-
-:StatusMessageText
-++++++++++++++++++
-
-Alongside the ``:StatusMessageHeader`` you need to add StatusMessageText for the message to show.:
-
-``curl -X PUT -d "This appears in a popup." http://localhost:8080/api/admin/settings/:StatusMessageText``
-
 .. _:MaxFileUploadSizeInBytes:
 
 :MaxFileUploadSizeInBytes
@@ -1666,7 +1701,9 @@ For example, if you want your installation of Dataverse to not attempt to ingest
 :ZipUploadFilesLimit
 ++++++++++++++++++++
 
-Limit the number of files in a zip that Dataverse will accept.
+Limit the number of files in a zip that Dataverse will accept. In the absence of this setting, Dataverse defaults to a limit of 1,000 files per zipfile.
+
+``curl -X PUT -d 2048 http://localhost:8080/api/admin/settings/:ZipUploadFilesLimit``
 
 :SolrHostColonPort
 ++++++++++++++++++

pom.xml

@@ -685,6 +685,7 @@
                     <include>**/*.xml</include>
                     <include>**/firstNames/*.*</include>
                     <include>**/*.xsl</include>
+                    <include>**/*.properties</include>
                 </includes>
             </resource>
         </resources>

scripts/api/data/bannerMessageError.json

@@ -0,0 +1,9 @@
+{
+  "dismissible": "false",
+  "messageTexts": [
+  {
+    "lang": "en",
+    "text": "Invalid json"
+  }
+  ]
+}

scripts/api/data/bannerMessageSample.json

@@ -0,0 +1,13 @@
+{
+  "dismissibleByUser": "false",
+  "messageTexts": [
+  {
+    "lang": "en",
+    "message": "Banner Message added via API"
+  },
+  {
+    "lang": "fr",
+    "message": "Message de bannière ajouté via l'API"
+  }
+  ]
+}

scripts/api/data/bannerMessageTest.json

@@ -0,0 +1,13 @@
+{
+  "dismissibleByUser": "false",
+  "messageTexts": [
+  {
+    "lang": "en",
+    "message": "Banner Message For Deletion"
+  },
+  {
+    "lang": "fr",
+    "message": "Banner Message For Deletion"
+  }
+  ]
+}

scripts/installer/as-setup.sh

@@ -73,7 +73,7 @@ function preliminary_setup()
   ./asadmin $ASADMIN_OPTS create-jvm-options "-XX\:+DisableExplicitGC"
 
   # alias passwords
-  for alias in "rserve_password_alias ${RSERVE_PASS}" "doi_password_alias ${DOI_PASSWORD}" "db_password_alias ${DB_PASS}"
+  for alias in "rserve_password_alias ${RSERVE_PASS}" "doi_password_alias ${DOI_PASSWORD}" "dataverse.db.password ${DB_PASS}"
   do
       set -- $alias
       echo "AS_ADMIN_ALIASPASSWORD=$2" > /tmp/$1.txt
@@ -130,17 +130,11 @@ function final_setup(){
         ./asadmin $ASADMIN_OPTS delete-jvm-options -Xmx512m
         ./asadmin $ASADMIN_OPTS create-jvm-options "-Xmx${MEM_HEAP_SIZE}m"
 
-
-        ./asadmin $ASADMIN_OPTS create-jdbc-connection-pool --restype javax.sql.DataSource \
-                                        --datasourceclassname org.postgresql.ds.PGPoolingDataSource \
-                                        --property create=true:User=$DB_USER:PortNumber=$DB_PORT:databaseName=$DB_NAME:ServerName=$DB_HOST \
-                                        dvnDbPool
-
-       ./asadmin $ASADMIN_OPTS set resources.jdbc-connection-pool.dvnDbPool.property.password='${ALIAS=db_password_alias}'
-
-        ###
-        # Create data sources
-        ./asadmin $ASADMIN_OPTS create-jdbc-resource --connectionpoolid dvnDbPool jdbc/VDCNetDS
+         # Set up the database connection properties
+        ./asadmin $ASADMIN_OPTS create-system-properties "dataverse.db.user=${DB_USER}"
+        ./asadmin $ASADMIN_OPTS create-system-properties "dataverse.db.host=${DB_HOST}"
+        ./asadmin $ASADMIN_OPTS create-system-properties "dataverse.db.port=${DB_PORT}"
+        ./asadmin $ASADMIN_OPTS create-system-properties "dataverse.db.name=${DB_NAME}"
 
         ./asadmin $ASADMIN_OPTS create-jvm-options "\-Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl"