Attachments in CouchDB 4.x?

[LarsVoith]

What is the plan for attachments in 4.0?

• Will attachments be allowed?

• What will the maximum attachment size be?

• Will this influence the document size?

• Will the API remain the same?

[wohali]

Hi @LarsVoith ,

Will attachments be allowed?

Yes. Discussion was here, from 2019.02: https://lists.apache.org/thread.html/4a7f29d2356349fa684fca199a6d0817d06e741fba483fd83f6ee59b%40%3Cdev.couchdb.apache.org%3E

IBM/Cloudant intend to develop attachments stored separately in an object store "in the cloud", but as you can see from the thread, agree some sort of native attachment functionality is valuable to retain. So, there is agreement.

What will the maximum attachment size be?

Undecided at this time. It will likely be lower than earlier CouchDBs because of technical limitations.

Will this influence the document size?

Unclear, but given how we write documents into FDB, probably no.

Will the API remain the same?

Yes, the intent for 4.0 is no API changes unless absolutely necessary, with any desired API changes likely to occur in the 5.0+ timeframe (if at all). The freeze on the API is to make the transition to 4.0 less painful.

[cluxter]

How about future performance? Upload/download performance of attachments is terrible right now, it's exponentially declining and becomes barely usable above ~1 Gb from what I remember from my benchmarks (like 1 Mb/s on localhost).

As you stated here, "Right now, large attachments (>16MB of attachments per JSON document) aren't a first order design scenario for CouchDB internal storage or so-called "internal replication" between nodes in a cluster. That needs to be resolved before thinking about any sort of "external" replication enhancements that specifically address large files."

I understand that CouchDB, a document oriented database system, is not an object-based storage system (like MinIO is for example, or AWS S3) and that the challenges of these two storage systems are really different. That said, I think it makes sense from a functional perspective to be able to upload/download attachments to/from the same storage interface, which itself is capable to replicate the attachments on all the instances.

Now let me be clear: MinIO and similar systems do object storage very well and it kind of feels like implementing this feature properly in CouchDB would actually be reimplementing MinIO.

So in order to decide wether we should keep or drop the attachment system in CouchDB, we should identify its potential added value. Here is my list:

• only one system to maintain instead of two. This means easier backups and less complexity on Ops;
• only one interface to work with. This means less complexity for developers. CouchDB already has a great HTTP interface and an authentication mechanism which we can leverage for serving files;
• binary file replication follows documents on all the CouchDB instances, which increases reliability. This is what I like the most: I like having the binary file and a document binded to it, both replicated (quite) at the same time.

These are the most obvious advantages but there are probably more.
Now I know this creates a lot of complexities that would need to be addressed: how to replicate files properly without clogging up the whole system for example.

But for now, I think we should focus on the value proposition from a functional point of view, also called use cases. Which is pretty much the opposite of your sentence: "That needs to be resolved before thinking about any sort of "external" replication enhancements that specifically address large files."

Because if there is no real value proposition for the developer/end user, then it becomes pointless to keep it at all. So I would suggest to first focus on is it something that should be kept? If yes, then how could we do it? Not the opposite.

EDIT: it seems that I'm not the only one sharing this need and I deviated from my original question: what about performance?