-
-
Notifications
You must be signed in to change notification settings - Fork 161
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[16.0] [IMP] fs_attachment: store attachments linked to different model/fields to different FS storages #269
[16.0] [IMP] fs_attachment: store attachments linked to different model/fields to different FS storages #269
Commits on Aug 24, 2023
-
Create base_attachment_object_storage to extract common code to store…
… implementations
Configuration menu - View commit details
-
Copy full SHA for 7d186d7 - Browse repository at this point
Copy the full SHA 7d186d7View commit details -
Abstract object storage in attachment_s3
Using the base_attachment_object_storage module, the same way attachment_swift is done. Fixed a few issues along the way in attachment_swift.
Configuration menu - View commit details
-
Copy full SHA for e901f7d - Browse repository at this point
Copy the full SHA e901f7dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 75cac64 - Browse repository at this point
Copy the full SHA 75cac64View commit details -
Configuration menu - View commit details
-
Copy full SHA for 125822d - Browse repository at this point
Copy the full SHA 125822dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 71471a2 - Browse repository at this point
Copy the full SHA 71471a2View commit details -
Ensure that migration of files is commited before deleting files
When moving attachments from the filestore to an object storage, the filesystem files will be deleted only after the commit, so if the transaction is rollbacked, we still have the local files for another try.
Configuration menu - View commit details
-
Copy full SHA for e953901 - Browse repository at this point
Copy the full SHA e953901View commit details -
Fix attachments stored in FS instead of object storage
Assume the following situation: * We have installed addons base, sale and attachment_s3 (hence base_attachment_object_storage as dependency) * All attachments are in S3 already * We run an upgrade of the 'base' addon, 'sale' is upgraded before attachment_s3 in the order of loading. * Sale updates the icon of the Sale menu * As attachment_s3 is not loaded yet, the attachment is created in the filestore Now if we don't persist the filestore or use different servers, we'll lose the images of the menus (or any attachment loaded by the install/upgrade of an addon). The implemented solution is to move the attachments from the filestore to the object storage at the loading of the module. However, this operation can take time and it shouldn't be run by 2 processes at the same time, so we want to detect if the module is loaded during a normal odoo startup or when some addons have been upgraded. There is nothing anymore at this point which allow us to know that modules just have been upgraded except... in the caller frame (load_modules). We have to rely on the inpect module and get the caller frame, which is not recommended, but seems the only way, besides, it's not called often and if _register_hook was called from another place, it would have no effect (unless the other place has a variable 'update_module' too).
Configuration menu - View commit details
-
Copy full SHA for 9973ddc - Browse repository at this point
Copy the full SHA 9973ddcView commit details -
Document a weird domain which is there for a reason
The reason being: https://github.com/odoo/odoo/blob/9032617120138848c63b3cfa5d1913c5e5ad76db/odoo/addons/base/ir/ir_attachment.py#L344-L347 I nearly deleted this domain but it was too weird to be there for no reason. A comment explaining the issue was really missing.
Configuration menu - View commit details
-
Copy full SHA for d0a012b - Browse repository at this point
Copy the full SHA d0a012bView commit details -
Configuration menu - View commit details
-
Copy full SHA for a58053c - Browse repository at this point
Copy the full SHA a58053cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1bb73d0 - Browse repository at this point
Copy the full SHA 1bb73d0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 64f59cb - Browse repository at this point
Copy the full SHA 64f59cbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 19e153e - Browse repository at this point
Copy the full SHA 19e153eView commit details -
Configuration menu - View commit details
-
Copy full SHA for e3a9c1d - Browse repository at this point
Copy the full SHA e3a9c1dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 45e6295 - Browse repository at this point
Copy the full SHA 45e6295View commit details -
Configuration menu - View commit details
-
Copy full SHA for d2ce94c - Browse repository at this point
Copy the full SHA d2ce94cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9b2fea7 - Browse repository at this point
Copy the full SHA 9b2fea7View commit details -
Configuration menu - View commit details
-
Copy full SHA for b7757a2 - Browse repository at this point
Copy the full SHA b7757a2View commit details -
Add method to force storage of special attachments to DB
Some attachments (e.g. image_small, image_medium) are stored in DB instead of the object storage for faster access. In some situations, we may have pushed all these files on the Object Storage (migration from a filesystem to object storage) and want to bring back these attachments from the object storage to the database. This method is not called anywhere but can be called by RPC or scripts.
Configuration menu - View commit details
-
Copy full SHA for 2da4d11 - Browse repository at this point
Copy the full SHA 2da4d11View commit details -
Rework and fix storage forced in database
The initial issue that triggered this rework is that the forced storage in database was working only on writes, and was never applied on attachment creations. This feature is used to store small files that need to be read in a fast way in database rather than in the object storage. Reading a file from the object storage can take 150-200ms, which is fine for downloading a PDF file or a single image, but not if you need 40 thumbnails. Down the path to make a correction, I found that: * the logic to force storage was called in `_inverse_datas`, which is not called during a create * odoo implemented a new method `_get_datas_related_values`, which is a model method that receive only the data and the mimetype, and return the attachment values and write the file to the correct place The `_get_datas_related_values` is where we want to plug this special storage, as it is called for create and write, and already handle the values and conditional write. But using this method, we have less information than before about the attachment, so let's review the different criterias we had before: * res_model: we were using it to always store attachments related to 'ir.ui.view' in db, because assets are related to this model. However, we don't really need to check this: we should store any javascript and css documents in database. * exclude res_model: we could have an exclusion list, to tell that for instance, for mail.message, we should never store any image in db. We don't have this information anymore, but I think it was never used and added "in case of". Because the default configuration is "mail.mail" and "mail.message" and I couldn't find any attachment with such res_model in any of our biggest databases. So this is removed. * mimetype and data (size) are the last criteria and we still have them The new system is only based on mimetype and data size and I think it's actually more versatile. Previously, we could set a global size and include mimetypes, but we couldn't say "I want to store all images below 50KB and all files of type X below 10KB". Now, we have a single system parameter with a dict configuration (`ir_attachment.storage.force.database`) defaulting to: {"image/": 51200, "application/javascript": 0, "text/css": 0} Assets have a limit of zero, which means they will all be stored in the database whatever their size is. Overall, this is a great simplification of the module too, as the method `_get_datas_related_values` integrates it better in the base calls of IrAttachment. Note for upgrade: I doubt we customized the previous system parameters which are now obsolete, but if yes, the configuration may need to be moved to `ir_attachment.storage.force.database`. For the record, the params were: * mimetypes.list.storedb (default: image) * file.maxsize.storedb (default: 51200) * excluded.models.storedb (mail.message,mail.mail), no equivalent now The method IrAttachment.force_storage_to_db_for_special_fields() should be called through a migration script on existing databases to move the attachments back into the database.
Configuration menu - View commit details
-
Copy full SHA for 76fd84a - Browse repository at this point
Copy the full SHA 76fd84aView commit details -
Configuration menu - View commit details
-
Copy full SHA for cdadd97 - Browse repository at this point
Copy the full SHA cdadd97View commit details -
Configuration menu - View commit details
-
Copy full SHA for d37faaa - Browse repository at this point
Copy the full SHA d37faaaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 522f4a8 - Browse repository at this point
Copy the full SHA 522f4a8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8e3b9af - Browse repository at this point
Copy the full SHA 8e3b9afView commit details -
Update manifest files to be consistent inbetween them
The main goal is to be able to easily do grep and sed when we do mass update on them
Configuration menu - View commit details
-
Copy full SHA for ba4e88a - Browse repository at this point
Copy the full SHA ba4e88aView commit details -
Configuration menu - View commit details
-
Copy full SHA for e3e5481 - Browse repository at this point
Copy the full SHA e3e5481View commit details -
Configuration menu - View commit details
-
Copy full SHA for e9bc08e - Browse repository at this point
Copy the full SHA e9bc08eView commit details -
Configuration menu - View commit details
-
Copy full SHA for abebd6a - Browse repository at this point
Copy the full SHA abebd6aView commit details -
Configuration menu - View commit details
-
Copy full SHA for acc4811 - Browse repository at this point
Copy the full SHA acc4811View commit details -
Configuration menu - View commit details
-
Copy full SHA for 33be0cd - Browse repository at this point
Copy the full SHA 33be0cdView commit details -
feat: remove after method (OCA#393)
* fix: azure reading in stream monkey patch documents
Configuration menu - View commit details
-
Copy full SHA for 68e614b - Browse repository at this point
Copy the full SHA 68e614bView commit details -
Configuration menu - View commit details
-
Copy full SHA for ca197b1 - Browse repository at this point
Copy the full SHA ca197b1View commit details -
Configuration menu - View commit details
-
Copy full SHA for fbd71df - Browse repository at this point
Copy the full SHA fbd71dfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 748130e - Browse repository at this point
Copy the full SHA 748130eView commit details -
Configuration menu - View commit details
-
Copy full SHA for cc44584 - Browse repository at this point
Copy the full SHA cc44584View commit details -
Configuration menu - View commit details
-
Copy full SHA for 95e2003 - Browse repository at this point
Copy the full SHA 95e2003View commit details -
Configuration menu - View commit details
-
Copy full SHA for d77de21 - Browse repository at this point
Copy the full SHA d77de21View commit details -
Configuration menu - View commit details
-
Copy full SHA for ccbe944 - Browse repository at this point
Copy the full SHA ccbe944View commit details -
[IMP] fs_attachment: Speedup install
Avoid recompute of new columns when installed on an existing database
Configuration menu - View commit details
-
Copy full SHA for 0617a2f - Browse repository at this point
Copy the full SHA 0617a2fView commit details -
[IMP] fs_attachment: Server Environement support
Allows to provide configuration parameters through server environement files.
Configuration menu - View commit details
-
Copy full SHA for d1ea120 - Browse repository at this point
Copy the full SHA d1ea120View commit details -
[IMP] fs_attachment: Simplify code.
Remove code used to try to read file from the root filesystem and write into the specialized filesystem. This code was used to try to provide a way to manage staging environments by reusing the same filesystem storage but with a different directory_path depending of the environement. A simpler method is to configure use a different filesystem storage by environement. If a production database is restored in pre production env, you can declare a new filesystem storage with a different code to store the attachements by default and configure the filesystem storage from the production with information allowing to read documents stored in it but not to modify or delete existing documents. This make the implementation far more simple.
Configuration menu - View commit details
-
Copy full SHA for a87a27b - Browse repository at this point
Copy the full SHA a87a27bView commit details -
[FIX] fs_attachment: No new registry creation
To create a new cursor, just ask to the current registry.... Loading a registry is very time consuming and could lead to deadlocks...
Configuration menu - View commit details
-
Copy full SHA for f4a64c1 - Browse repository at this point
Copy the full SHA f4a64c1View commit details -
[IMP] fs_attachment: store attachments linked to different model/fiel…
…ds to different FS storages
Configuration menu - View commit details
-
Copy full SHA for e493887 - Browse repository at this point
Copy the full SHA e493887View commit details