Add a field to limit the size of uploading content #1701

git-hyagi · 2024-07-12T19:45:30Z

closes: #532

lubosmj

Was the goal to read blob data in smaller chunks and in case we are blocking the API for too long, we raise an error?

pulp_container/pulp_container/app/registry_api.py

Lines 1353 to 1360 in 25abfd6

    
           while True: 
        
               subchunk = chunk.read(2000000) 
        
               if not subchunk: 
        
                   break 
        
               temp_file.write(subchunk) 
        
               size += len(subchunk) 
        
               for algorithm in Artifact.DIGEST_FIELDS: 
        
                   hashers[algorithm].update(subchunk)

I see a threshold defined for config blobs: https://github.com/containers/image/blob/1dbd8fbbe51653e8a304122804431b07a1060d06/internal/image/oci.go#L63-L83. But, I could not find any thresholds for regular blobs. Can we investigate this?

lubosmj · 2024-07-14T20:27:28Z

pulp_container/app/registry_api.py

@@ -938,6 +942,10 @@ def put(self, request, path, pk=None):
        repository.pending_blobs.add(blob)
        return BlobResponse(blob, path, 201, request)

+    def _verify_payload_size(self, distribution, chunk):
+        if distribution.max_payload_size and chunk.reader.length > distribution.max_payload_size:
+            raise PayloadTooLarge()


Is this exception properly rendered to the container's API format?

ref:

pulp_container/pulp_container/app/registry_api.py

Line 266 in 25abfd6

def handle_exception(self, exc):

Some of the exceptions are properly documented at https://docker-docs.uclv.cu/registry/spec/api/#blob-upload.

Ahn... my bad, I didn't know about the spec for errors.
I also found this doc https://github.com/opencontainers/distribution-spec/blob/v1.0.1/spec.md#error-codes that will be helpful.

lubosmj · 2024-07-14T20:28:19Z

pulp_container/app/models.py

@@ -751,6 +751,8 @@ class ContainerDistribution(Distribution, AutoAddObjPermsMixin):
        null=True,
    )

+    max_payload_size = models.IntegerField(null=True)


I do not think we want to make this parameter configurable. I vote for having a constant for this.

git-hyagi · 2024-07-18T19:40:07Z

Was the goal to read blob data in smaller chunks and in case we are blocking the API for too long, we raise an error?

Hum... when I was writing this PR I was thinking about a feature to allow a maximum size for blob layers and deny it if the limit is exceeded.
Did I misunderstand the goal of the issue?

I see a threshold defined for config blobs: https://github.com/containers/image/blob/1dbd8fbbe51653e8a304122804431b07a1060d06/internal/image/oci.go#L63-L83. But, I could not find any thresholds for regular blobs. Can we investigate this?

I also couldn't find a limit for regular blobs. From what I could understand, they limit only the size of buffered "resources" (mainifests, config blobs, signatures, etc.) to avoid OOM issues:
containers/image@61096ab
"Restrict the sizes of blobs which are copied into memory such as the
manifest, the config, signatures, etc. This will protect consumers of
c/image from rogue or hijacked registries that return too big blobs in
hope to cause an OOM DOS attack."

lubosmj · 2024-07-18T19:51:50Z

Okay, I think we can follow that path. Let's then focus on manifests, config blobs, and signatures exclusively.

git-hyagi · 2024-07-23T13:26:56Z

After an internal discussion, here is the conclusion we got:
Considering that a proper installation of Pulp should have a reverse proxy and if an upload request is too big, the damage (denial of service) on the reverse proxy is already done, we should define the limits on the reverse proxy, not on Pulp.
Here is a draft of a nginx config to limit the manifest size:

location ~* /v2/.*/manifests/.*$ {
	client_max_body_size 1m;
}

ipanova · 2024-08-14T13:42:51Z

This change makes sense to me, however besides manifest directive we need to also cap extensions/v2/(?P<path>.+)/signatures endpoint. This is an alternative way to upload skopeo produced atomic signatures.

Config blobs are left out but i don't see an easy way to set 4mb limit specifically on them given that they are in location ~* /v2/.*/blobs/.*$ I would be ok to proceed only with signatures and manifests.

One thing I do not understand, if the malicious user can create big manifest that would lead to big memory consumption, what prevents him to just create big blob? There is no limit set on the blobs.

ipanova · 2024-08-14T13:50:52Z

Skimming through commit description containers/image@61096ab changes, they are mostly targeted on the client side, so the client during pull operation is not susceptible to DDoS attack.
This reads to me as registry itself does not have limits on upload. If that's the case, than the only place where we need to introduce a safeguard is sync operation. Because here we act as a client so would be good to read only first 4mb of the manifests/config blobs/signatures and then raise an error.

ipanova · 2024-08-14T13:57:32Z

Hm, apparently the changes were applied also on some upload calls like https://github.com/containers/image/blob/61096ab72530cc9216b50d9fc3bee90b53704f68/docker/docker_image_dest.go#L630

ipanova

is this still needed https://github.com/pulp/pulp_container/blob/main/pulp_container/app/registry_api.py#L1454

ipanova · 2024-08-27T09:38:54Z

CHANGES/532.feature

@@ -0,0 +1,2 @@
+Added a limit to the size of manifests and signatures during sync tasks and


can you be specific and mention that this is a 4mb limit?

Can you also explain why this limit was added?

ipanova · 2024-08-27T09:41:48Z

pulp_container/app/tasks/sync_stages.py

@@ -545,6 +549,14 @@ async def create_signatures(self, man_dc, signature_source):
                        "Error: {} {}".format(signature_url, exc.status, exc.message)
                    )

+                if not is_signature_size_valid(signature_download_result.path):
+                    log.info(
+                        "Signature size is not valid, the max allowed size is {}.".format(


how about 'signature body size exceeded maximum allowed size of X'

ipanova · 2024-08-27T09:49:18Z

pulp_container/app/tasks/sync_stages.py

+                            SIGNATURE_PAYLOAD_MAX_SIZE
+                        )
+                    )
+                    raise ManifestSignatureInvalid(digest=man_digest_reformatted)


when used, this exception is raised to the container client

ipanova · 2024-08-27T09:54:48Z

pulp_container/app/tasks/sync_stages.py

@@ -566,7 +578,12 @@ async def create_signatures(self, man_dc, signature_source):
            # signature extensions endpoint does not like any unnecessary headers to be sent
            await signatures_downloader.run(extra_data={"headers": {}})
            with open(signatures_downloader.path) as signatures_fd:


you have the signatures_downloader.path why not calling 'is_valid_size' before open/reading it?

you're missing check for manifests

pulp_container/pulp_container/app/tasks/sync_stages.py

Line 70 in 60a6fbc

with open(response.path, "rb") as content_file:

ipanova · 2024-08-27T10:03:48Z

pulp_container/app/utils.py

+
+def _is_manifest_size_valid(manifests):
+    for manifest in manifests:
+        if manifest.get("size") > MANIFEST_PAYLOAD_MAX_SIZE:


i would not trust what's written on the manifest list json file other then actually checking the manifest file size

ipanova · 2024-08-27T10:06:06Z

pulp_container/app/webserver_snippets/nginx.conf

@@ -38,3 +38,25 @@ location /token/ {
    proxy_redirect off;
    proxy_pass http://pulp-api;
 }
+
+location ~* /v2/.*/manifests/.*$ {


what about apache snippet?

ipanova · 2024-08-27T10:17:56Z

for the sync workflow, sometimes we skip tag or entire manifest list when the signature is missing or one image manifest from the list is unsigned. do we want to also skip those objects that are exceeding the size limit instead of failing whole sync?

ekohl

I'm not familiar with the code base, but @git-hyagi asked about the Apache config. There we couldn't find a config that applied LimitRequestBody to proxied requests.

My suggestion is to enforce the checks in the application itself. Having the proxy (nginx, apache or whatever) can be considered an optimization, but Pulp remains responsible for enforcing it.

I have added some inline observations that I think should make it a bit more efficient, but that is all based on familiarity with Django & django-rest-framework that was mostly built up around 10 years ago. I read some relevant information, but it's possible I missed things.

ekohl · 2024-09-06T11:32:33Z

pulp_container/app/downloaders.py

+
+        total_size = 0
+        buffer = b""
+        async for chunk in response.content.iter_chunked(MEGABYTE):


I'm not familiar with the codebase, but would it make sense to send HTTP 413 Request Entity Too Large before you start reading by inspecting the Content-Length header (which I already saw was done elsewhere).

hum... it seems like a good idea, but my concern is that it is easy to modify the header and bypass the verification, so I'm not sure if we can trust only in the Content-Length value, what do you think?

If you really want to be safe, you can apply both. First check the header isn't bigger than the maximum size and then make you read at most the Content-Length size. Though I think Content-Length is an optional header so if it's not present, you should keep the current code anyway.

I'd recommend reading aiohttp's documentation on what you can (and can't) assume.

ekohl · 2024-09-06T11:49:00Z

pulp_container/app/registry_api.py

@@ -1426,7 +1425,7 @@ def put(self, request, path, pk):
        except models.Manifest.DoesNotExist:
            raise ManifestNotFound(reference=pk)

-        signature_payload = request.META["wsgi.input"].read(SIGNATURE_PAYLOAD_MAX_SIZE)
+        signature_payload = request.META["wsgi.input"].read()


It surprises me that this uses such a low level implementation. If I read the code right, request is a rest_framework.request.Request which has a method request.stream to get it as io.BytesIO. Isn't that a much more portable interface? Then I think you can actually use signature_dict = json.load(request.stream) to save copying the full response into memory an additional time.

And if you want to guard it, you can inspect the length explicitly prior to loading and send HTTP 413. See _load_stream() for how they read the length:
https://github.com/encode/django-rest-framework/blob/f593f5752c45e06147231bbfd74c02384791e074/rest_framework/request.py#L297-L314

My primary reason to avoid wsgi.input (and this isn't the only occurrence) is that it blocks you from moving to ASGI and eventually more performant application servers.

@git-hyagi, can you also address this comment, please? It is worth the effort.

ekohl · 2024-09-11T22:35:47Z

pulp_container/app/downloaders.py

+            total_size += len(chunk)
+            if max_body_size and total_size > max_body_size:
+                await self.finalize()
+                raise InvalidRequest(resource_body_size_exceeded_msg(content_type, max_body_size))


Does this result in a HTTP 413 error?

I wonder about the impact of this change. Does it mean that pulp-to-pulp syncing (or any syncing in general) will not work when someone already uploaded/synced a larger content unit? I am still a bit hesitant to include this change.

Does this result in a HTTP 413 error?

No 😢 it is following the oci distribution spec https://github.com/opencontainers/distribution-spec/blob/v1.0.1/spec.md#error-codes (which does not have a specific error for request entity too large)

I wonder about the impact of this change. Does it mean that pulp-to-pulp syncing (or any syncing in general) will not work when someone already uploaded/synced a larger content unit? I am still a bit hesitant to include this change.

I modified the PR to allow defining these limits in the settings.py (instead of being constants). By default, if these values are not defined, no limit will be enforced, ensuring that the functionality remains the same as before.

You can also see the API doc here: https://distribution.github.io/distribution/spec/api/. How is this InvalidRequest rendered on the client side when using podman?

hum... this is a sample output that I received while trying to push a manifest with an invalid size:

Writing manifest to image destination WARN[0010] Failed, retrying in 1s ... (1/3). Error: writing manifest: uploading manifest latest to pulp-container:5001/test: unknown: Manifest body size exceeded the maximum allowed size of 10 bytes. ... WARN[0014] Failed, retrying in 1s ... (2/3). Error: writing manifest: uploading manifest latest to pulp-container:5001/test: unknown: Manifest body size exceeded the maximum allowed size of 10 bytes. ... WARN[0019] Failed, retrying in 1s ... (3/3). Error: writing manifest: uploading manifest latest to pulp-container:5001/test: unknown: Manifest body size exceeded the maximum allowed size of 10 bytes. ... Error: writing manifest: uploading manifest latest to pulp-container:5001/test: unknown: Manifest body size exceeded the maximum allowed size of 10 bytes.

This is what dockerhub returns in case I am trying to upload a 4MB+ large file.

tr -dc 'a-zA-Z' < /dev/urandom | head -c 5194304 > output.txt

cat output.txt | http --auth-type=jwt --auth=$TOKEN_AUTH PUT https://registry-1.docker.io/v2/lmjachky/pulp/manifests/new

HTTP/1.1 400 Bad Request content-length: 110 content-type: application/json date: Thu, 19 Sep 2024 09:25:15 GMT docker-distribution-api-version: registry/2.0 docker-ratelimit-source: 2a02:8308:b093:8e00:62b:28c5:e440:e0b9 strict-transport-security: max-age=31536000 { "errors": [ { "code": "MANIFEST_INVALID", "detail": "http: request body too large", "message": "manifest invalid" } ] }

We should be returning a formatted error response. Similarly to our existing MANIFEST_INVALID:

pulp_container/pulp_container/app/exceptions.py

Line 107 in bb7e18c

detail={

.

No 😢 it is following the oci distribution spec https://github.com/opencontainers/distribution-spec/blob/v1.0.1/spec.md#error-codes (which does not have a specific error for request entity too large)

It says 4xx code:

A 4XX response code from the registry MAY return a body in any format.

So I don't think it forbids using code 413. Did you intend to link to https://github.com/opencontainers/distribution-spec/blob/v1.0.1/spec.md#determining-support which does list error codes in the Failure column (and supports what you said)?

It says 4xx code:

A 4XX response code from the registry MAY return a body in any format.

So I don't think it forbids using code 413.

hum.... this is a good point.
It seems like this doc is a little bit confusing. I thought that 413 was not included in the spec because the "Error Codes" section states:

The code field MUST be one of the following:

and there isn't an entity too large error in the table.

Perhaps worth raising an issue on the spec to get an opinion?

lubosmj · 2024-09-19T09:39:44Z

Creating a manifest larger than 4MB is quite challenging. Currently, registries have this 4MB limit adopted for uploading. However, it is possible that we may encounter hitting the limit in the future frequently. Given this, should we consider removing the restriction if there is a chance it might be lifted later on? Alternatively, would it be better to maintain the current limit and allow administrators to fiddle with this constraint? Are the administrators ever going to touch the proposed setting? So many questions... I am sorry but I am leaning towards closing the PR and issue completely. Do you mind bringing in more voices?

git-hyagi · 2024-09-19T14:20:59Z

I agree that we still have a lot of questions and, with the OCI spec supporting a wide range of different "artifacts", it is possible that we may encounter some problems to fine tune a good value for these constraints and adding this setting (allowing admins to change the limit value) could only bring more code to be maintained for something that no one ever would need/change.
So I am ok with closing it.

lubosmj · 2024-09-20T13:14:55Z

CHANGES/532.feature

+Added support to the `MANIFEST_PAYLOAD_MAX_SIZE` and `SIGNATURE_PAYLOAD_MAX_SIZE` settings to define
+limits (for the size of Manifests and Signatures) to protect against OOM DoS attacks during synchronization tasks
+and image uploads.
+Additionally, the Nginx snippet has been updated to enforce the limit for these endpoints.
+Modified the internal logic of Blob uploads to read the receiving layers in chunks,
+thereby reducing the memory footprint of the process.


We should refine this changelog to make it shorter, simpler, and more user-friendly. For instance, we can omit details like "Modified the internal logic of Blob uploads to read the receiving layers in chunks, thereby reducing the memory footprint of the process".

lubosmj

Based on the discussion we had, you should now update the code to start returning 413 in case the processed entity is too large (https://github.com/pulp/pulp_container/pull/1701/files#r1766498485).

More observations:

The settings should be defined in the settings.py file. We should not run settings.get("MANIFEST_PAYLOAD_MAX_SIZE", None) in "heavy-duty" situations.
Once again, please, do not forget to use request.stream in other places as well.
It is up to you what you decide to do with web-server snippets. I think we should drop any changes made to the nginx.conf file and keep the apache config untouched.

ipanova · 2024-09-26T10:39:45Z

3. It is up to you what you decide to do with web-server snippets. I think we should drop any changes made to the `nginx.conf` file and keep the apache config untouched.

+1 to revert changes made to the snippets.

lubosmj · 2024-10-04T08:02:10Z

People are storing PNGs inside configs' labels. 🙄

I bet manifests' annotations will be next.

Adds new settings to limit the size of manifests and signatures as a safeguard to avoid DDoS attack during sync and upload operations. Modify the blob upload to read the layers in chunks. closes: pulp#532

lubosmj reviewed Jul 14, 2024

View reviewed changes

git-hyagi marked this pull request as draft July 23, 2024 14:11

git-hyagi force-pushed the limit-payload-size branch 2 times, most recently from 2845307 to 60a6fbc Compare August 26, 2024 15:27

git-hyagi marked this pull request as ready for review August 26, 2024 16:28

ipanova reviewed Aug 27, 2024

View reviewed changes

git-hyagi marked this pull request as draft August 27, 2024 18:00

git-hyagi force-pushed the limit-payload-size branch from 60a6fbc to edbaaf1 Compare September 2, 2024 18:04

ekohl reviewed Sep 6, 2024

View reviewed changes

git-hyagi force-pushed the limit-payload-size branch 2 times, most recently from 904a3a4 to cedb1d1 Compare September 11, 2024 21:32

ekohl reviewed Sep 11, 2024

View reviewed changes

git-hyagi force-pushed the limit-payload-size branch from cedb1d1 to 452a2fa Compare September 12, 2024 21:23

git-hyagi marked this pull request as ready for review September 13, 2024 13:59

git-hyagi force-pushed the limit-payload-size branch 3 times, most recently from 19732ef to 884f7df Compare September 18, 2024 18:30

lubosmj reviewed Sep 20, 2024

View reviewed changes

lubosmj requested changes Sep 26, 2024

View reviewed changes

Limit the size of manifests/signatures sync/upload

a24678e

Adds new settings to limit the size of manifests and signatures as a safeguard to avoid DDoS attack during sync and upload operations. Modify the blob upload to read the layers in chunks. closes: pulp#532

git-hyagi force-pushed the limit-payload-size branch from 884f7df to a24678e Compare October 4, 2024 19:52

github-actions bot added the no-changelog label Oct 4, 2024

	while True:
	subchunk = chunk.read(2000000)
	if not subchunk:
	break
	temp_file.write(subchunk)
	size += len(subchunk)
	for algorithm in Artifact.DIGEST_FIELDS:
	hashers[algorithm].update(subchunk)

		@@ -0,0 +1,2 @@
		Added a limit to the size of manifests and signatures during sync tasks and

Add a field to limit the size of uploading content #1701

Are you sure you want to change the base?

Add a field to limit the size of uploading content #1701

Conversation

git-hyagi commented Jul 12, 2024

lubosmj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

git-hyagi commented Jul 18, 2024

lubosmj commented Jul 18, 2024

git-hyagi commented Jul 23, 2024

ipanova commented Aug 14, 2024

ipanova commented Aug 14, 2024 • edited Loading

ipanova commented Aug 14, 2024

ipanova left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ipanova commented Aug 27, 2024

ekohl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lubosmj commented Sep 19, 2024

git-hyagi commented Sep 19, 2024

Choose a reason for hiding this comment

lubosmj left a comment

Choose a reason for hiding this comment

ipanova commented Sep 26, 2024

lubosmj commented Oct 4, 2024

ipanova commented Aug 14, 2024 •

edited

Loading