Docspell Nightly

Commits

• 0c432e3: Update sbt-scalafix to 0.13.0 (eikek-scala-steward) #2803

Docspell 0.42.0

What’s Changed

🚀 Features

• Nix module: secure config file @ivanbrennan (#2699)
• Add logout-url option to Nix module @VTimofeenko (#2646)
• Add package option to Nix modules @VTimofeenko (#2630)
• Add Japanese Vertical Support Branch for Tesseract and Ocrmypdf OCR @tenpai-git (#2505)
• Extend nix setup, including dev environments @eikek (#2539)
• Extend config for external commands @eikek (#2536)

🐛 Bug Fixes

• Honor default value true for collectOutput @eikek (#2663)
• FIx extracting addons with only a single file @eikek (#2660)
• Move arg-mappings underneath command section @eikek (#2659)
• Change docker base images to 3.19.1 @eikek (#2519)
• fix(webapp): downstream keep-alive events to backend preventing timeout @nekrondev (#2500)
• Fix some Kubernetes Kustomize deployment issues @TheAnachronism (#2501)
• tesseract english data is now a separate package @programmerq (#2479)

💚 Maintenance

• Update flake @eikek (#2736)
• Adding CJK and Custom Mapping Documentation @tenpai-git (#2669)
• docs(development.md): Fix minor typos @ChanceHarrison (#2613)
• Add page to website about contributing to docs @ChanceHarrison (#2612)
• Fix renamed elm package @eikek (#2547)
• Redocly tailwind setup @eikek (#2540)
• Pushing a minor fix to the bash commands. @tenpai-git (#2523)
• Add documentation for backup and restore process for PostgreSQL. @tenpai-git (#2508)

🧱 Dependencies

• Update dependency @fontsource/montserrat to v5.0.19 @renovate (#2734)
• Update dependency tailwindcss to v3.4.10 @renovate (#2733)
• Update dependency cssnano to v7.0.5 @renovate (#2726)
• Update postgres Docker tag to v16.4 @renovate (#2725)
• Update dependency tailwindcss to v3.4.9 @renovate (#2724)
• Update dependency tailwindcss to v3.4.8 @renovate (#2723)
• Update dependency postcss to v8.4.41 @renovate (#2722)
• Update dependency autoprefixer to v10.4.20 @renovate (#2719)
• Update dependency tailwindcss to v3.4.7 @renovate (#2714)
• Update dependency postcss to v8.4.40 @renovate (#2713)
• Update alpine Docker tag to v3.20.2 @renovate (#2712)
• Update mariadb-java-client to 3.4.1 @eikek-scala-steward (#2708)
• Update dependency @fortawesome/fontawesome-free to v6.6.0 @renovate (#2707)
• Update dependency tailwindcss to v3.4.6 @renovate (#2706)
• Update dependency tailwindcss to v3.4.5 @renovate (#2704)
• Update circe-yaml to 0.15.3 @eikek-scala-steward (#2703)
• Update jsoup to 1.18.1 @eikek-scala-steward (#2702)
• Bump braces from 3.0.2 to 3.0.3 in /modules/webapp @dependabot (#2684)
• Update sbt to 1.10.1 @eikek-scala-steward (#2700)
• Update dependency cssnano to v7.0.4 @renovate (#2698)
• Update dependency postcss to v8.4.39 @renovate (#2695)
• Update sbt-sonatype to 3.11.0 @eikek-scala-steward (#2693)
• Update circe-yaml to 0.15.2 @eikek-scala-steward (#2691)
• Update alpine Docker tag to v3.20.1 @renovate (#2689)
• Update dependency cssnano to v7.0.3 @renovate (#2687)
• Update scalafmt-core to 3.8.2 @eikek-scala-steward (#2686)
• Update testcontainers-scala-mariadb, ... to 0.41.4 @eikek-scala-steward (#2685)
• Update actions/checkout action to v4.1.7 @renovate (#2683)
• Update pureconfig, pureconfig-ip4s to 0.17.7 @eikek-scala-steward (#2680)
• Update imageio-jpeg, imageio-tiff to 3.11.0 @eikek-scala-steward (#2679)
• Update scala-java-time to 2.6.0 @eikek-scala-steward (#2677)
• Update dependency tailwindcss to v3.4.4 @renovate (#2676)
• Update dependency cssnano to v7.0.2 @renovate (#2675)
• Update scribe, scribe-slf4j2 to 3.15.0 @eikek-scala-steward (#2674)
• Update scribe, scribe-slf4j2 to 3.14.0 @eikek-scala-steward (#2671)
• Update swagger-ui to 5.17.14 @eikek-scala-steward (#2670)
• Update dependency flag-icons to v7.2.3 @renovate (#2665)
• Update munit to 1.0.0, munit-cats-effect to 2.0.0 @eikek (#2664)
• Update calev-circe, calev-core, calev-fs2 to 0.7.3 @eikek-scala-steward (#2662)
• Update swagger-ui to 5.17.11 @eikek-scala-steward (#2657)
• Update scodec-bits to 1.2.0 @eikek-scala-steward (#2655)
• Update mariadb-java-client to 3.4.0 @eikek-scala-steward (#2653)
• Update sourcecode to 0.4.2 @eikek-scala-steward (#2651)
• Update flyway-core, ... to 10.13.0 @eikek-scala-steward (#2652)
• chore(deps): update alpine docker tag to v3.20.0 @renovate (#2645)
• chore(deps): update dependency flag-icons to v7.2.2 @renovate (#2644)
• chore(deps): update actions/checkout action to v4.1.6 @renovate (#2640)
• chore(deps): update cachix/install-nix-action action to v27 @renovate (#2639)
• Update scribe, scribe-slf4j2 to 3.13.5 @eikek-scala-steward (#2635)
• chore(deps): update postgres docker tag to v16.3 @renovate (#2634)
• chore(deps): update actions/checkout action to v4.1.5 @renovate (#2633)
• Update jwt-circe to 10.0.1 @eikek-scala-steward (#2631)
• Update io, sbt to 1.10.0 @eikek-scala-steward (#2628)
• Update http4s-circe, http4s-dsl, ... to 0.23.27 @eikek-scala-steward (#2626)
• Update scala-library to 2.13.14 @eikek-scala-steward (#2625)
• Update sbt-scalafix to 0.12.1 @eikek-scala-steward (#2624)
• Update flyway-core, ... to 10.12.0 @eikek-scala-steward (#2622)
• chore(deps): update dependency cssnano to v7.0.1 @renovate (#2621)
• Update swagger-ui to 5.17.2 @eikek-scala-steward (#2619)
• chore(deps): update dependency tailwindcss to v3.4.3 @renovate (#2594)
• chore(deps): update actions/checkout action to v4.1.4 @renovate (#2615)
• Update scribe, scribe-slf4j2 to 3.13.4 @eikek-scala-steward (#2611)
• Update dependency cssnano to v7 @renovate (#2610)
• Update swagger-ui to 5.17.0 @eikek-scala-steward (#2606)
• Update actions/checkout action to v4.1.3 @renovate (#2604)
• Update dependency @fontsource/montserrat to v5.0.18 @renovate (#2602)
• Update stanford-corenlp to 4.5.7 @eikek-scala-steward (#2601)
• Update flyway-core, ... to 10.11.1 @eikek-scala-steward (#2599)
• Update scribe, scribe-slf4j2 to 3.13.3 @eikek-scala-steward (#2598)
• Update swagger-ui to 5.15.2-1 @eikek-scala-steward (#2597)
• Update icu4j to 75.1 @eikek-scala-steward (#2595)
• Update swagger-ui to 5.15.2 @eikek-scala-steward (#2596)
• Update sourcecode to 0.4.1 @eikek-scala-steward (#2593)
• Update sourcecode to 0.4.0 @eikek-scala-steward (#2591)
• Update jcl-over-slf4j to 2.0.13 @eikek-scala-steward (#2588)
• Update sbt-native-packager to 1.10.0 @eikek-scala-steward (#2587)
• Update swagger-ui to 5.15.1 @eikek-scala-steward (#2586)
• Update swagger-ui to 5.15.0 @eikek-scala-steward (#2585)
• Update commons-io to 2.16.1 @eikek-scala-steward (#2583)
• Update tika-core to 2.9.2 @eikek-scala-steward (#2579)
• Update dependency @fortawesome/fontawesome-free to v6.5.2 @renovate (#2578)
• Update swagger-ui to 5.13.0 @eikek-scala-steward (#2577)
• Update scalafmt-core to 3.8.1 @eikek-scala-steward (#2576)
• Update commons-io to 2.16.0 @eikek-scala-steward (#2575)
• Update fs2-core, fs2-io to 3.10.2 @eikek-scala-steward (#2574)
• Update dependency tailwindcss to v3.4.3 @renovate (#2571)
• Update dependency tailwindcss to v3.4.2 @renovate (#2570)
• Update swagger-ui to 5.12.2 @eikek-scala-steward (#2569)
• Update dependency flag-icons to v7.2.1 @renovate (#2568)
• Update dependency cssnano to v6.1.2 @renovate (#2567)
• Update fs2-core, fs2-io to 3.10.1 @eikek-scala-steward (#2565)
• Update scribe, scribe-slf4j2 to 3.13.2 @eikek-scala-steward (#2563)
• Update calev-circe, calev-core, calev-fs2 to 0.7.2 @eikek-scala-steward (#2562)
• Update dependency postcss to v8.4.38 @renovate (#2561)
• Update dependency autoprefixer to v10.4.19 @renovate (#2560)
• Update dependency postcss-import to v16.1.0 @renovate (#2559)
• Update dependency cssnano to v6.1.1 @renovate (#2558)
• Update sbt-scalajs, scalajs-compiler, ... to 1.16.0 @eikek-scala-steward (#2556)
• Update scribe, scribe-slf4j2 to 3.13.1 @eikek-scala-steward (#2555)
• Update sbt-buildinfo to 0.12.0 @eikek-scala-steward (#2554)
• Update dependency postcss to v8.4.37 @renovate (#2553)
• Update swagger-ui to 5.12.0 @eikek-scala-steward (#2552)
• Update fs2-core, fs2-io to 3.10.0 @eikek-scala-steward (#2551)
• Update dependency postcss to v8.4.36 @renovate (#2549)
• Update pdfbox to 3.0.2 @eikek-scala-steward (#2544)
• Update swagger-ui to 5.11.10 @eikek-scala-steward (#2541)
• Update postgresql to 42.7.3 @eikek-scala-steward (#2546)
• Update flyway-core, ... to 10.10.0 @eikek-scala-steward (#2545)
• Update sbt-github-pages to 0.14.0 @eikek-scala-steward (#2525)
• Update dependency autoprefixer to v10.4.18 @renovate (#2532)
• Update kittens to 3.3.0 @eikek-scala-steward (#2537)
• Update dependency cssnano to v6.1.0 @renovate (#2511)
• Update cachix/install-nix-action action to v26 @renovate (#2534)
• Update dependency @fontsource/montserrat to v5.0.17 @renovate (#2529)
• Update flyway-core, ... to 10.9.1 @eikek-scala-steward (#2533)
• Update http4s-circe, http4s-dsl, ... to 0.23.26 @eikek-scala-steward (#2527)
• Update cats-effect to 3.5.4 @eikek-scala-steward (#2528)
• Update flyway-core, ... to 10.9.0 @eikek-scala-steward (#2526)
• Update scala-library to 2.13.13 @eikek-scala-steward (#2515)
• Update sbt-scalafix to 0.12.0 @eikek-scala-steward (#2522)
• Update swagger-ui to 5.11.8 @eikek-scala-steward (#2520)
• Update postgres Docker tag to v16.2 @renovate (#2493)
• Update pureconfig, pureconfig-ip4s to 0.17.6 @eikek-scala-steward (#2512)
• Update sbt to 1.9.9 @eikek-scala-steward (#2513)
• Update mariadb-java-client to 3.3.3 @eikek-scala-steward (#2506)
• Update postgresql to 42.7.2 @eikek-scala-steward (#2507)
• Update dependency postcss-import to v16.0.1 @renovate (#2496)
• Update flyway-core, ... to 10.8.1 @eikek-scala-steward (#2499)
• Update emil-common, emil-doobie, ... to 0.17.0 @eikek-scala-steward (#2494)
• Update flyway-core, ... to 10.7.2 @eikek-scala-steward (#2492)
• Update dependency postcss to v8.4.35 @renovate (#2491)
• Update jcl-over-slf4j to 2.0.12 @eikek-scala-steward (...

Docspell 0.41.0

🎉 Changelog • Documentation • Milestone 🎉

• Add khmer language (#2011)
• Replace blaze with ember (http stack) (#2221, #2348)
• Updated several dependencies (#2348, #2354, 2355)
• Fix AddonExecutionResult monoid (#2363)
• Setting item date from addons (#2345)
• Fix sql queries where a space was missing (#2367, #2368)
• Change AWS region for minio file backend (#2386)
• Allow additional metadata to be passed on upload for addons and hooks (#2334)
• Add support for Slovak language
• Fix possibility to configure too large session-valid values (#2309)
• Consider emails when flattenArchives is true (#2063)
• Allow new file upload without hitting reset (#2323)

Solr needs analysis-extra module

With the addition of khmer language, solr needs now to load its analysis-extra module. It is (usually) packaged with the solr distribution, so it only requires to change the startup command. The docker-compose file has been changed accordingly. You might need to check your setup.

H2 Update

The in-process database H2 was updated in this release: from 2.1.214 to 2.2.224. You need to migrate your data to make it work with the new release. Exporting using the old version and importing using the new version should work fine. For details, please refer to the h2 documentation. There is a script tools/h2-util.sh to help with this process. Also this comment might be useful.

Big thanks to:

• @eikek
• @madduck
• @mprasil
• @rehanone
• @v6ak
• @xshadowlegendx

Docspell 0.40.0

🎉 Changelog • Documentation • Milestone 🎉

This release contains some bug fixes and improvements.

• Add Ukrainian language (#1835, @GooRoo)
• webui: normalize , to . for numeric fields (#1975)
• improve error reporting when file backend fails (#1976)
• docker: remove exlpicit zlib install (#1863)
• nix: implement nix flake (#1869, @VTimofeenko)
• nix: replace wkhtmltopdf with weasyprint (#1873, #1895, @VTimofeenko)
• fix listing shares with no name (#1840)
• fix html conversion of text files (#1915)
• fix: remove test-scoped dependencies from artifacts (#1856)
• fix deletin users (#1941)
• fix notification for collective scoped tasks (#1987)

Docspell 0.39.0

🎉 Changelog • Documentation • Milestone 🎉

This release contains some bug fixes, updates docker images and includes some internal change to get a bit forward with #585 (user groups).

• Allow to set extracted content (#1775) You can now fix OCRed text this way.
• Improve handling mixes of OIDC and local accounts (#1827). It is now possible to use the same account locally and via OIDC.
• Add Estonian language (#1646)
• Updated docker images to new alpine and openjdk, fixing #1736 (#1713) by @jberggg and @eikek
  • drops support for arm7 (#1719)
  • introduces weasyprint as an alternative to wkhtmltopdf for converting html files to pdf
• Fix docspell's query to remove attach.count. It has been removed a while ago, but the query language didn't reflect it (#1758)
• Fix search for linked items (#1808)
• Fix item selection after merging (#1809)
• Internal changes to how a collective is referenced as a preparation for #585 (#1686)
• Update H2 to 2.1.x, incompatible to the previous used version (#1690)

Warning

The database structure changed substantially in this release. Please make sure to create a backup of your database before attempting the upgrade!

Docker

There are two major changes for docker users: First ARM7 support has been dropped, because it was too much work to maintain alongside the other architectures. Second the images have been updated to the latest alpine linux, which requires to sacrifice using wkhtmltopdf as a tool to convert HTML to PDF (often used for processing emails).

The joex image doesn't have the wkhtmltopdf binary anymore, because it is not available for alpine linux. Instead weasyprint has been added. Docspell by default still uses wkhtmltopdf, because I found it has better results. But you can now switch to weasyprint and if you use the provided docker images you have to.

There is a new config that you need to set - when using environment variables:

DOCSPELL_JOEX_CONVERT_HTML__CONVERTER=weasyprint

Just add it to the env variables in the docker-compose.yml. If you use a config file, add this to it:

docspell.joex {
  convert.html-converter = "weasyprint"
}

File Backends

The internal change on how collectives are now referenced requires to adopt the files accordingly. If you have your files in the database, all is being migrated automatically on first start.

For other file backends, the files must be migrated manually. The difference is that from now on a collective is referenced by a unique number and not by its name anymore. You can look at the table collective to see which number was assigned to a collective and then

• for a filebackend, simply move the folder with a collective name to its corresponding number
• for s3 backend the same must happen, using some s3 client (maybe this one)

H2

If you use H2 as a database, there are some manual steps required. H2 was bumped from 1.4.x to 2.1.x and the new version cannot read the database files of the old version.

Additionally, one of the changesets for H2 used a now illegal syntax and had to be changed. This will lead to checksum mismatch errors when starting up.

Creating and restoring a dump, the script tools/h2-util.sh can be used. The H2 version can be specified with an environment variable H2_VERSION to easily create a dump in one version and restore in another.

To fix the changeset, you could simply run this sed command on the dump before restoring:

sed -i 's,175554607,-276220379,g' docspell-dump-h2.sql

But this could potentially change not only the checksum, but other things in the dump. It is not very likely, though. A more safe alternative is to use a text editor and find the correct place (in table flyway_schema_history) to change or just set database-schema.repair-schema = true in the config file or use the env variables

DOCSPELL_SERVER_BACKEND_DATABASE__SCHEMA_REPAIR__SCHEMA=true

and startup only the restserver one time to have the checksum fixed. Another safe variant is to run this update statement on your h2 database:

UPDATE flyway_schema_history set checksum = -276220379 WHERE version = '1.9.3';

Docspell 0.38.0

🎉 Changelog • Documentation • Milestone 🎉

This release fixes some issues related to OpenID connect integration and addons.

• Allow to skip login page if a single OIDC provider is configured (#1640)
• Add config to force OCR on a PDF even if text would be available (#1628)
• Improvements for OIDC integration (#1619, #1545)
• Improve unzipping/zipping files (#1603)
• Fix starting a background task to install addon (#1601)
• Add more database indexes to improve search (#1584)
• Add paging to the share view (#1582)
• Remove unused code (#1581)

Docspell 0.37.0

🎉 Changelog • Documentation • Milestone 🎉

This is more a bug fix release, targeting mainly fixes related to addons.

• Fix e-mail address input field (#1565)
• Overhaul of search when fulltext search is applied. Fixes #1363.
• Return when a search exceeds server configured limit for page size (#1358)
• Several fixes related to addons (#1566, #1573)

Docspell 0.36.0

🎉 Changelog • Documentation • Milestone 🎉

This release allows to extend Docspell with custom scripts/programs. This is still an experimental feature, I expect more changes coming. But you are invited to give it a try 😄!

• Extend docspell via addons (experimental feature) (#1550) There are two addons to play with
  • rotate-pdf-addon for rotating pdf files
  • audio-files-addon can provide basic support for audio files
• Adds Lithuanian and Polish to document languages (#1540, #1345)
• Quick guide for adding more languages (#942)
• Make dashboard boxes headlines bold (#1531)
• Improve logging config; allow to specify loggers and their level (#1535)
• Allow for auth tokens to optionally be issued with an validity (#1533, #1534)

---

Note to docker users: The docker image of the previous release contained a "broken" zlib library that affected checksum generation for the db migrations (see #1517). The image of this release has an updated version of this system library. This means that it is quite likely that those who were not affected by this in 0.35.0 (for example, you started with a fresh db) - could now be affected. If you get these errors, try to "repair" the migrations. Apply the following settings in the docker-compose file for one start:

DOCSPELL_SERVER_BACKEND_DATABASE__SCHEMA_REPAIR__SCHEMA=true
DOCSPELL_SERVER_BACKEND_DATABASE__SCHEMA_RUN__FIXUP__MIGRATIONS=true
DOCSPELL_SERVER_BACKEND_DATABASE__SCHEMA_RUN__MAIN__MIGRATIONS=true

Start up one time and stop again. Then remove these settings again.

Docspell 0.35.0

🎉 Changelog • Documentation • Milestone 🎉

• Download multiple documents as zip (#1093). The webui allows to download multiple files as a zip archive. The zip file is created at the server and cached for a while.
• New project ds4e providing utilises to access Docspell from Emacs
• Increase size for password fields in the database (#1508)
• Hide the delete button in new notifciation channel forms (#1506)
• Fix logging (#1468), non-errors were logged as errors.
• Apply the migration fix from last version only from 0.32.0 onwards (#1469)
• Fix typos in UI (#1510, @monnypython)
• Add support for Postgres FTS in nix module (#1512, @LeSuisse)

Docspell 0.34.0

🎉 Changelog • Documentation • Milestone 🎉

• French translation (#1440): The UI is now available in French! Thanks to @jgirardet.
• Link Items (#528): Link items together directly (without using custom fields) and see all related items quickly on the search page.
• Import mails recursively (#1260): The scanmailbox task can now go through folders recursively. Thanks to @Moerfi666 and @seijikun.
• Fulltext search via PostgreSQL (#424): Another backend for full-text search was added that is backed by PostgreSQL's text search feature. You can now choose between SOLR and PostgreSQL for fulltext search.
• More file backends (#1379): Next to storing the files in the database, you can now choose to store them in a S3 compatible storage or use the filesystem directly.
• Flat zip upload (#1366): An option has been added to the upload metadata that will unpack zip files and process each entry separately (instead of treating a zip file as a single item).
• Fix to upload large files (#1339): Uploaded files are not loaded entirely into memory allowing to upload large files.
• Fix previously published db migration (#1454, #1436): Unfortunately, there was an db migration in the last release that caused problems on some installations. A fix for this migration has been added.
• Fix build to run on windows (#1449, #1461): Building docspell on windows has been improved by @arittner.

For some of the new features, you need the new version of dsc.