Bugfix Release

Yet another batch of fixes and improvements for v4: last stop before v4.1!

Cluster & Scalability

• Fixed many clustering side-effects (service start locking, grpc error status management)
• Fixed Mongo-specific issues or missing features (logs filtering, logs cursor, indexer truncate to size, etc)

UX

• Virtualize Image Gallery view to avoid overloading the server
• Abort XHR when leaving console dashboard
• Use websocket to keep users list in sync (under Console > People) - should also avoid conflicts when saving
• Upgrade forms "variant" where missing

Other

• Datasource fix "internal path" on custom Minio server
• Clean "orphan" tasks and taskslogs in scheduler
• buf mod update for protobuf libraries
• Expose "SSL" config in mailer (by opposition to StartTLS).

Change log

You can find a summary of the change log here.

Bugfix Release

Lots of bugfixes and improvements for v4.

Core

• Cluster : use a unique id for metrics, use a shared lock for starting all nodes at once, fixed ETCD lease issue
• Typo in users cache could degrade performances for concurrent requests
• CELLS_WEB_RATE_LIMIT environment applied on data APIs (s3). It was only applied on REST APIs before.
• Anonymous download feature was broken
• Workspace sometimes temporarily vanishing from left menu
• Fix documentation for Sites Configuration + Let's Encrypt
• Fix documentation for SQL migration possible issue
• Better handling of time desynchronization error to avoid client-side loop
• Re-enabled caddy logging : errors only by default, access+errors when Cells is in debug mode
• Rework Metrics, Grafana dashboard and missing services in metrics.

Datasources

• Structured datasources FS modification not detected
• S3 structured datasources regression (re-creation of .pydio instead of using s3 empty objects as introduced in v4
• S3 datasources cannot copy files bigger than 5GB from an other DS as it does not switch to Multipart
• S3 : issue when using an internal path inside the bucket
• Rehash operation shows a progress and can be cancelled for big files, also now triggered automatically after reindexation
• An error was displayed when remove an object service, and restarting a new datasource at the same location could show minio errors

Cells Sync Compat

• Added checks for compatibility between v4 server and Cells Sync latest build, as hashing mechanism (used for quick file comparison) have changed.

Cells Flows & Enterprise

• Auto-open parameters panel if mandatory ones are not set
• Purge Activities default Flow does not have the CompactDB flag
• Big update of the documentation, along with Smart Forms documentation
• Auto-apply search engine content indexation
• Custom CSS not persisting after closing branding tool
• Simplify SmartForm submission with WebHooks
• Full Documentation update

Change log

You can find a summary of the change log here.

Bugfix Release

New batch of improvements for v4 branch

Fixes & Improvements

• Deduplicate objects services if they share a same PeerAddress to avoid v3 => v4 migration issues
• Fix possible concurrent map read/write in configs
• Better retries on sql transaction deadlocks
• Email digest notification: show clickable links for documents
• Scheduler tasks and tasks logs are not properly cleared when using MongoDB
• Publish prometheus target discovery with a custom endpoint (protected with basic auth)
• Rewire bigCache metrics and add new metrics for system info
• [Ent] Fix Glob Path condition in Security Policies

[Ent] Cells Flows

• New debug mode displaying messages flowing directly in the graph
• Ability to wait for parallel actions to be done and merge their output into a new action
• New playground for live-testing Anko script
• New "Hook" mechanism to define custom API endpoints and custom UX buttons to trigger a Flow
• Better handling of job "mandatory" parameters
• Queue selectors in the throttled dispatchers (like actions), to avoid applying tons of selectors parallel

Change log

You can find a summary of the change log here.

Bugfix Release

As a result of active testing and community reports, we continue to correct issues and improve v4.

Important Bugfixes

• New files detection on structured datasource (local FS) was partially broken
• Fixed a possible concurrent map access during massive uploads
• Refix sync task cancellation for long datasource resync
• Appending query params to URL breaks public links

Minor improvements

• Exif rotation correct support for avatar pictures
• Make sure binaries are not indexed by search engine
• Improve share screen tabs display
• Scheduler: much better log display and debugging level
• Fixed Scp Put action (support for templates)

Feature

• Integrate Uppy.io to directly capture webcam/mic/screencast

Change log

You can find a summary of the change log here.

Bugfix Release

This weekly release fixes glitches on v4 branch and introduces a couple of new features.

Bugfixes

• Important issue with non-ascii characters breaking headers
• Issues with structured datasources and events
• Various ETCD glitches when services are stopped or leases are lost (cluster mode)

New

• Interface libraries updates: pdfjs (better support of signed pdf), soundmanager (add flac support)
• New parameter on flat datasources to automatically distribute files inside subfolder(s)
• New filter types for log levels.

Change log

You can find a summary of the change log here.

Bugfix Release

First batch of fixes for v4

Change log

You can find a summary of the change log here.

v4.0.1

Cells v4 is a major leap forward for clustered deployments. It features a brand-new microservices engine for unparalleled performance, and a new dependency management making it much easier to delegate core services to cloud standards.

Pydio Cells 4.0 is fully cloud-ready

To get cloud- and multi-node ready, we’ve retired our original microservices framework, which was complex and limiting, to use our own. Working from a clean slate, we developed a brand new, truly stateless instance of Cells 4.0 that decouples and externalizes storage and service layers to create an image that can easily be distributed and replicated.

Cells 4.0’s new simplified core works in single-node as well as complex multi-node configurations – all the way from Raspberry Pi to Kubernetes clusters.

Codebase Major Changes

The V4 branch was a long-term development project that started with our desire to adopt Go Modules. This redesign led to a big reduction in our dependencies, a huge simplification of the architecture and, as a result, a lot of interesting features!

The code now relies on Go 1.18+ and Go Modules, and the master branch is now main. The main dependencies are updated to more recent versions, including Caddy, Hydra and Minio, etc. We now make a better use of resources within a process by sharing servers (http|gRPC) between services. This greatly reduces the number of network ports used by Cells and it has a huge impact on performances.

Resources

• Upgrade Instructions : Must Read!
• Support: Forum
• Administration Guide : Quick Start
• Developer Docs: API, Knowledge Base
• Show your love: Star on Github, share on Twitter or follow us on LinkedIn!

Contributions

This release comes with fully translated Brazilian version thanks to Claudio Ferreira and Japanese version thanks to Satoshi Yuza.

If you want to help us and participate by adding translation to your language, it is really easy: just navigate to Pydio Cells project in Crowdin, create an account and get started!

Change log

You can find a summary of the change log here.

Cells v4.0.0-rc7

New relase candidate for v4. Genaral Availability is really getting closer 🤞!

• Fixing fork deregister
• Scheduler: RPC error handling
• Fix a few forgotten messages + almost 100% Brazilian thanks to @filhocf
• Build ARM binaries from AMD build agent
• Fix keystore permissions on Windows (upgrade v3 => v4)
• Recover websocket handler errors (closed chan)
• Fix missing uuid for hash copy
• Scheduler: clean tasks/subscriber code, simplify ListJobs dao signature
• Capture context cancelled while listing endpoints in sync, allowing to stop long indexation processes
• Add mutexes on ACL internals
• Monitor zaps slices / remove hard limit on roles listing (cherry pick from v3)
• Avoid any login issue after upgrade by using a new local token name.
• Improve README for docker/compose/ha

Learn more about v4 here

Change log

You can find a summary of the change log here.

Cells v4 release candidate

New Release Candidate for v4.

See previous release candidate RC5 for information about v4 major changes.

Learn more about the new clustering approach in the updated Administration Guide. Currently a work-in-progress.

This sprint was focused on cleaning the new APIs, cluster deployment glitches and fixing various bugs.

• API: handle new path parameters vs. body parameters constraints implied by OpenAPI updates
• Checks and warning on "public" address usage : finally use loopback as default for binding grpc and other internal services
• Fix various users creations glitches (CLI)
• Refactoring: clean use of deprecated ioutil package, refactor sync lib for better context/error handling
• [Cluster] Recheck or implement TLS support on all external dependencies (mysql, etcd, nats, redis, mongo)
• [Cluster] Fix Queue mechanism in Nats broker
• [Cluster] Update docker-compose to use a dedicated minio node for storage

Change log

You can find a summary of the change log here.

Cells v4 release candidate

Release Candidate 5 for Cells v4

Cells v4 is a major leap forward for clustered deployments. It features a brand-new microservices engine for unparalleled performance, and a new dependency management making it much easier to delegate core services (such as configs, registry, caches, etc.) to cloud standards.

RC3, RC4, RC5

We have released many packages since RC2 to validate and fix many glitches. This is really getting closer to stability !

For developers, one important change occurred during the last weeks: we now rely on Go18 (tested on Go19) to build the code. All dependencies versions were bumped for security. This should not impact users or API consumers though.

We also re-organized some CLI commands to make the hierarchy more consistent.

Moving to Go modules at last

The V4 branch was a long-term development project that started with our desire to adopt Go modules. Cells was written at a time when modules were not yet part of the language (we used the vendor folder...). This made the migration to modules complex: Go's auto-migration tool was not usable for us (it simply crashed), and we ended up recreating the code base from scratch using modules, re-adding our libraries and dependencies one by one.

As expected, this redesign led to a big reduction in our dependencies, a huge simplification of the architecture and, as a result, a lot of interesting features!

To make a long story short:

• We got rid of the microservices framework we were using (Micro), regaining control of the gRPC layer.
• We updated the main dependencies to their latest versions, including Caddy, Hydra and Minio.
• We make a better use of resources within a process by sharing "servers" (http|gRPC) between "services". This greatly reduces the number of network ports used by Cells. This has a huge impact on performance, typically for single-node deployments.
• These "servers" and "services" are much better managed and can be more easily started/stopped on different nodes, making cluster deployments easier than ever (see below).

MongoDB as a drop-in replacement for all services based on on-file storages

Historically we've been using BoltDB and BleveSearch and we like it. They are pure GO key/value stores and indexers and it allows Cells to provide full indexing functionality without external dependencies. By default, the search engine, activity stream or logs use these JSON document shops to provide rich, out-of-the-box functionality. But these stores are disk and memory intensive, and while they are suitable for small and medium-sized deployments, they create bottlenecks for large deployments.

We therefore looked at alternatives for implementing new 'drivers' for the data abstraction layer of each of these services, and chose MongoDB as a feature-rich, scalable and indexed JSON document store. All services using BoltDB/Bleve as storage now offer an alternative MongoDB implementation, a migration path from one another, and the ability to scale horizontally drastically.

File-based storage is still a very good option for small/medium instances, avoiding the need to manage another dependency, but the Cells installation steps will now offer to configure a MongoDB connection to avoid the need to migrate data in the long term. Note that Mongo does not replace MySQL storages, that DB is still required for Cells.

Cluster Me Please!

Cells was developed from day one as a set of microservices, but we had to face the fact that deploying Cells in a multi-node, highly available environment was extremely complex and almost nobody could really make it work... The v4 was the perfect time to tackle this problem!

We took a step back, learned our lesson from v1 to v3, and looked closely at cloud-native DevOps best practices (yes K8s, we're looking at you). The main objective was: how to create a fully stateless instance of Cells (image, container, you name it...) that can be easily distributed and replicated.

Similar to the move from BoltDB to Mongo, we implemented DAOs to decouple and externalize many layers, making Cells V4 finally cloud-ready. To achieve that without re-inventing the wheel, Cells V4 stands on the shoulders of giants :

• ETCD for configs and services registry
• NATS for message broadcasting (pub/sub)
• REDIS for shared cache
• MONGO for JSON documents
• HASHICORP VAULT for secrets and certificates management

Again, all these are optional and Cells can still be deployed as a standalone, dependency-free binary on a Rasperry Pi (even the older 32bits versions) !

Migration and testing

Single-node deployments

Upgrade process is standard and should be straight-forward (and we would really love to hear from you on that).
There are a couple of important notes during this upgrade :

• Hydra JWKs will be regenerated in the DB, with effect of invalidating all existing authentication token. You will be logged out after upgrade, and if you are using Personal Access Tokens, you will have to regenerate new ones.
• Cells Sites Bind URL should not use a domain name but should declare a binding PORT, eventually including the Network Interface IP. If you have connection issue after upgrade, make sure to edit sites (cells configure sites) to bind to e.g. 0.0.0.0:8080 instead of domain.name:8080

Migrating Bolt/Bleve storages to Mongo

Migrate from existing on-file storage to MongoDB using the following steps:

• Install MongoDB, currently tested against version 5.0.X, prepare a Mongo database for cells data
• Stop cells, as Bolt/Bleve files must not be opened by the application during the migration process
• Use cells admin config db add command to configure a connection:
  • Setup connection using mongo connection string like mongodb://user:pass@ip:port/dbname
  • Accept prompt for using this connection as default document DSN
  • Accept prompt to perform data migration from existing bolt/bleve files to mongo. This can take some time.
• Now restart Cells. You should see "Successfully pinged and connected to MongoDB" in the logs.
• As search engine data are not migrated, you have to relaunch indexation on the pydio.grpc.search service using cells admin resync --service=pydio.grpc.search --path=/

Now you should be good to go. Try searching for * in Cells search engine, you should have blazing fast results.

Cluster deployments

We will provide dedicated blog posts on this topic very soon. You can already have a look at this sample Docker Compose file that shows the required dependencies and how to specify their endpoints to Cells using environment variables.

Change log

You can find a summary of the change log here.