diff --git a/human_experiments/datasette_interface/copy_screenshots_to_server.sh b/human_experiments/datasette_interface/copy_screenshots_to_server.sh new file mode 100755 index 000000000..f8ab82607 --- /dev/null +++ b/human_experiments/datasette_interface/copy_screenshots_to_server.sh @@ -0,0 +1,13 @@ +# Script to copy screenshots to the ivilab webserver. This is meant to be run +# on an IVILab machine on which /data is mounted. + +time rsync \ + -adrPm \ # archive mode, show progress, prune empty directories + --no-inc-recursive \ + --copy-links \ # Copy files and not just symlinks + --info=progress2 \ + --include="*/" \ + --include="*/*/Screenshots/*.png" \ + --include="*/*/screenshots/*.png" \ # Include lowercased variant + --exclude="*" \ + /data/tomcat/raw/LangLab/experiments/study_3_pilot/group ivilab:/var/www/data/tomcat/ diff --git a/human_experiments/datasette_interface/create_datasette_with_plugins_image.sh b/human_experiments/datasette_interface/create_datasette_with_plugins_image.sh index a3ba5f9c7..a5198c78b 100755 --- a/human_experiments/datasette_interface/create_datasette_with_plugins_image.sh +++ b/human_experiments/datasette_interface/create_datasette_with_plugins_image.sh @@ -6,7 +6,9 @@ docker pull datasetteproject/datasette:0.64.3 docker run datasetteproject/datasette \ pip install \ + datasette-render-markdown \ datasette-pretty-json \ - datasette-render-markdown + datasette-copyable \ + datasette-render-image-tags docker commit $(docker ps -lq) datasette-with-plugins diff --git a/human_experiments/datasette_interface/docker-compose.prod.yml b/human_experiments/datasette_interface/docker-compose.prod.yml index 4e0d81f2a..e7a29995b 100644 --- a/human_experiments/datasette_interface/docker-compose.prod.yml +++ b/human_experiments/datasette_interface/docker-compose.prod.yml @@ -8,7 +8,7 @@ services: ports: - "8001:8001" volumes: - - /work/ivilab/public_html/data/tomcat:/mnt + - /tomcat/data:/mnt - .:/repo entrypoint: | diff --git a/human_experiments/datasette_interface/schemacrawler.config.properties b/human_experiments/datasette_interface/schemacrawler.config.properties deleted file mode 100644 index 239cc1f06..000000000 --- a/human_experiments/datasette_interface/schemacrawler.config.properties +++ /dev/null @@ -1,216 +0,0 @@ -# --=----=----=----=----=----=----=----=----=----=----=----=----=----=----=----= -# - SchemaCrawler: Configuration Options -# --=----=----=----=----=----=----=----=----=----=----=----=----=----=----=----= -# -# - Load Options -# ------------------------------------------------------------------------------ -# - Maximum number of system threads to use for loading the database catalog -# - Default: 10 system threads (maximum) -# - NOTE: To force single-threading, override with SC_SINGLE_THREADED=true -# as an environmental variable or Java system property. -#schemacrawler.load.max_threads=10 -# -# - Metadata Retrieval Options -# ------------------------------------------------------------------------------ -# - Override the metadata retrieval strategy -# - This can affect speed, so they are commented out in order to use database -# - specific defaults -# - Default: Hard-coded into each database plugin, otherwise metadata -# - Possible values for each property are none, metadata or data_dictionary_all -# none does not retrieve metadata -# CAUTION: Changing these values could result in an error -#schemacrawler.schema.retrieval.strategy.typeinfo=metadata -#schemacrawler.schema.retrieval.strategy.tables=metadata -#schemacrawler.schema.retrieval.strategy.tableprivileges=metadata -#schemacrawler.schema.retrieval.strategy.tablecolumns=metadata -#schemacrawler.schema.retrieval.strategy.tablecolumnprivileges=metadata -#schemacrawler.schema.retrieval.strategy.primarykeys=metadata -#schemacrawler.schema.retrieval.strategy.indexes=metadata -#schemacrawler.schema.retrieval.strategy.foreignkeys=metadata -#schemacrawler.schema.retrieval.strategy.procedures=metadata -#schemacrawler.schema.retrieval.strategy.procedurecolumns=metadata -#schemacrawler.schema.retrieval.strategy.functions=metadata -#schemacrawler.schema.retrieval.strategy.functioncolumns=metadata -# -# - Limit Options - inclusion rules for database objects -# ------------------------------------------------------------------------------ -# - Regular expression schema pattern to filter -# - schema names -# - Default: .* for include, for exclude -# - IMPORTANT: Please uncomment the follow patterns only for -# - database that support schemas. SQLite for example does -# - not support schemas -#schemacrawler.schema.pattern.include=.* -#schemacrawler.schema.pattern.exclude= -# - Regular expression table and column name pattern to filter table -# - and column names -# - Column regular expression to match fully qualified column names, -# - in the form "CATALOGNAME.SCHEMANAME.TABLENAME.COLUMNNAME" -# - Default: .* for include, for exclude -#schemacrawler.table.pattern.include=.* -#schemacrawler.table.pattern.exclude= -#schemacrawler.column.pattern.include=.* -#schemacrawler.column.pattern.exclude= -# - Regular expression routine and routine parameter name pattern to filter -# - routine and routine parameter names -# - Default: .* for include, for exclude -#schemacrawler.routine.pattern.include= -#schemacrawler.routine.pattern.exclude=.* -#schemacrawler.routine.inout.pattern.include=.* -#schemacrawler.routine.inout.pattern.exclude= -# - Regular expression synonym pattern to filter -# - synonym names -# - Default: for include, .* for exclude -#schemacrawler.synonym.pattern.include= -#schemacrawler.synonym.pattern.exclude=.* -# - Regular expression sequence pattern to filter -# - sequence names -# - Default: for include, .* for exclude -#schemacrawler.sequence.pattern.include= -#schemacrawler.sequence.pattern.exclude=.* -# -# - Grep Options - inclusion rules -# ------------------------------------------------------------------------------ -# - Include patterns for tables -# - Default: .* for include, for exclude -#schemacrawler.grep.table.pattern.include=.* -#schemacrawler.grep.table.pattern.exclude= -# - Include patterns for table columns -# - Default: .* for include, for exclude -#schemacrawler.grep.column.pattern.include=.* -#schemacrawler.grep.column.pattern.exclude= -# - Include patterns for routine parameters -# - Default: .* for include, for exclude -#schemacrawler.grep.routine.inout.pattern.include=.* -#schemacrawler.grep.routine.inout.pattern.exclude= -# - Include patterns for table and routine definitions -# - Default: .* for include, for exclude -#schemacrawler.grep.definition.pattern.include=.* -#schemacrawler.grep.definition.pattern.exclude= -# -# - Sorting Options -# ------------------------------------------------------------------------------ -# - Sort orders for objects -#schemacrawler.format.sort_alphabetically.tables=true -#schemacrawler.format.sort_alphabetically.table_columns=false -#schemacrawler.format.sort_alphabetically.table_foreignkeys=false -#schemacrawler.format.sort_alphabetically.table_indexes=false -#schemacrawler.format.sort_alphabetically.routines=true -#schemacrawler.format.sort_alphabetically.routine_columns=false -# -# - Show Options - text output formatting -# ------------------------------------------------------------------------------ -# - Controls generation of the SchemaCrawler header and footer in output -# - Default: false -#schemacrawler.format.no_header=false -#schemacrawler.format.no_footer=false -#schemacrawler.format.no_schemacrawler_info=false -#schemacrawler.format.show_database_info=false -#schemacrawler.format.show_jdbc_driver_info=false -# - Controls display of remarks for tables and columns in output -# - Default: false -#schemacrawler.format.hide_remarks=false -# - Shows all object names with the catalog and schema names, for easier comparison -# - across different schemas -# - Default: false -#schemacrawler.format.show_unqualified_names=false -# - Shows standard column types instead of database specific column types -# - Default: false -#schemacrawler.format.show_standard_column_type_names=false -# - Shows ordinal numbers for columns -# - Default: false -#schemacrawler.format.show_ordinal_numbers=false -# - Hides table row counts, even if they are loaded -# - Default: false -#schemacrawler.format.hide_table_row_counts=false -# - Hide database objects -#schemacrawler.format.hide_routines=false -#schemacrawler.format.hide_schemas=false -#schemacrawler.format.hide_sequences=false -#schemacrawler.format.hide_synonyms=false -#schemacrawler.format.hide_tables=false -# - Hide specific parts text output for dependant database objects -#schemacrawler.format.hide_alternatekey_names=false -#schemacrawler.format.hide_foreignkey_names=false -#schemacrawler.format.hide_index_names=false -#schemacrawler.format.hide_primarykey_names=false -#schemacrawler.format.hide_routine_specific_names=false -#schemacrawler.format.hide_constraint_names=false -#schemacrawler.format.hide_trigger_names=false -#schemacrawler.format.hide_weakassociation_names=false -# - If foreign key names, constraint names, trigger names, -# - specific names for routines, or index and primary key names -# - are not explicitly provided while creating a schema, most -# - database systems assign default names. These names can show -# - up as spurious diffs in SchemaCrawler output. -# - All of these are hidden with the --portable-names -# - command-line option. For more control, use the following -# - options. -# - Hides foreign key names, constraint names, trigger names, -# - specific names for routines, index and primary key names -# - Default: false -#schemacrawler.format.hide_primarykey_names=false -#schemacrawler.format.hide_foreignkey_names=false -#schemacrawler.format.hide_alternatekeys_names=false -#schemacrawler.format.hide_weakassociation_names=false -#schemacrawler.format.hide_index_names=false -#schemacrawler.format.hide_trigger_names=false -#schemacrawler.format.hide_routine_specific_names=false -#schemacrawler.format.hide_constraint_names=false -# Specifies how to quote (delimit) database object names in text output -# Options are -# - quote_none - Do not quote any database object names -# - quote_all - Always quote database object names -# - quote_if_special_characters - Only quote database object names -# if they contain special characters -# - quote_if_special_characters_and_reserved_words - Quote database object names -# if they contain special characters or SQL 2003 reserved words -# - Default: quote_if_special_characters_and_reserved_words -#schemacrawler.format.identifier_quoting_strategy=quote_if_special_characters_and_reserved_words -# - Does not color-code catalog and schema names. -# - Default: false -#schemacrawler.format.no_schema_colors=false -# - Encoding of input files, such as Apache Velocity templates -# - Default: UTF-8 -#schemacrawler.encoding.input=UTF-8 -# - Encoding of SchemaCrawler output files -# - Default: UTF-8 -#schemacrawler.encoding.output=UTF-8 -# -# - Diagram Options -# - (some diagram options may be controlled by text formatting options) -# ------------------------------------------------------------------------------ -# - Show a crow's foot symbol to indicate cardinality -# - Default: true -#schemacrawler.graph.show.primarykey.cardinality=true -#schemacrawler.graph.show.foreignkey.cardinality=true -#schemacrawler.graph.show.foreignkey.filtered_tables=true -# -# - Graph attributes for Graphviz, supporting graph, node and edge -# - See https://www.graphviz.org/doc/info/attrs.html -schemacrawler.graph.graphviz.graph.rankdir=RL -schemacrawler.graph.graphviz.graph.labeljust=r -schemacrawler.graph.graphviz.graph.fontname=Helvetica -#schemacrawler.graph.graphviz.graph.splines=ortho -schemacrawler.graph.graphviz.graph.ranksep=1.5 -schemacrawler.graph.graphviz.node.fontname=Helvetica -#schemacrawler.graph.graphviz.node.shape=none -schemacrawler.graph.graphviz.edge.fontname=Helvetica -# - Additional options for Graphviz, to control diagram generation -# - See https://www.graphviz.org/doc/info/command.html -schemacrawler.graph.graphviz_opts=-Gdpi=220 -# -# - Data Output Options -# ------------------------------------------------------------------------------ -# - Whether to show data from CLOB and BLOB objects -# - Default: false -#schemacrawler.data.show_lobs=false -# -# -# --=----=----=----=----=----=----=----=----=----=----=----=----=----=----=----= -# Queries -# --=----=----=----=----=----=----=----=----=----=----=----=----=----=----=----= -# Define your own named queries, which then become SchemaCrawler command -hsqldb.tables=SELECT * FROM INFORMATION_SCHEMA.SYSTEM_TABLES -tables.select=SELECT ${columns} FROM ${table} ORDER BY ${columns} -tables.drop=DROP ${tabletype} ${table} diff --git a/human_experiments/datasette_interface/templates/base.html b/human_experiments/datasette_interface/templates/base.html index 715cfa272..ce81536bc 100644 --- a/human_experiments/datasette_interface/templates/base.html +++ b/human_experiments/datasette_interface/templates/base.html @@ -78,7 +78,7 @@ {% endif %} {% endblock %} -
+
diff --git a/human_experiments/datasette_interface/templates/index.html b/human_experiments/datasette_interface/templates/index.html index fcc12a5d4..52db25b37 100644 --- a/human_experiments/datasette_interface/templates/index.html +++ b/human_experiments/datasette_interface/templates/index.html @@ -20,20 +20,18 @@

{{ metadata.title or "Datasette" }}{% if private %} 🔒{% endif %}

The dataset currently is served via a [single SQLite database](/tomcat), but we may add additional ones in the future. -Currently, there are tables for the participant information, group session, -raw fNIRS data, the testbed messages, and the baseline tasks. More -tables (EEG, EKG, GSR, eye tracking, vocalic features, demographic data, -questionnaires) will be added in the near future. - For users who are not familiar with SQL, we will provide some sample SQL queries to demonstrate how to use this database and the associated Datasette interface. For more advanced users, feel free to try different queries, use the programmatic APIs provided by Datasette, or simply [download the whole SQLite -database](https://tomcat.ivilab.org/tomcat.db). +database](https://tomcat.ivilab.org/tomcat.db). Note that queries that take +too long to execute (i.e., > 10 seconds) will be aborted, in order to keep the +site running smoothly. If you would like to perform such complex queries, we +recommend downloading the database locally to perform them. -The database schema diagram is available here: [Database +The entity relationship diagram describing the database schema diagram is available here: [ERD diagram](/assets/db_diagram.png). Sign up for our [mailing list](/mailing-list) to get updates on the dataset! diff --git a/human_experiments/datasette_interface/templates/pages/derived-data-products.html b/human_experiments/datasette_interface/templates/pages/derived-data-products.html index 228662067..b544a0e0e 100644 --- a/human_experiments/datasette_interface/templates/pages/derived-data-products.html +++ b/human_experiments/datasette_interface/templates/pages/derived-data-products.html @@ -11,17 +11,22 @@ Documentation for the derived data products is in sections 4 and 5 of the [data products document](http://ivilab.cs.arizona.edu/data/tomcat/data_products.pdf). - -- [Filtered synchronized data (1.1 -TB)](https://ivilab.cs.arizona.edu/ivilab/data/tomcat/derived/release_2023_07_17_19.tar.gz). -Uploaded 2023-07-19. See [here](/updates/2023-07-19) for details. +- [Filtered synchronized data (generated 2023-08-28) (836 +GB)](https://ivilab.cs.arizona.edu/ivilab/data/tomcat/derived/release_2023_08_28_17.tar.gz) ## Older versions -We always recommend using the latest version of the derived data products. +We always recommend using the latest version of the derived data products, as +older versions may have issues that are fixed in the newest versions, and the +documentation is only accurate for the newest version. + However, we will continue to provide the older versions for as long as we feasibly can. +- [Filtered synchronized data (1.1 +TB)](https://ivilab.cs.arizona.edu/ivilab/data/tomcat/derived/release_2023_07_17_19.tar.gz). +Uploaded 2023-07-19. See [here](/updates/2023-07-19) for details. + - [Filtered synchronized data (168 GB)](https://ivilab.cs.arizona.edu/ivilab/data/tomcat/derived/filtered_synchronized_data.tar.gz) Uploaded 2023-06-14.