Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Requesting unfiltered manifest yields a 504 #3892

Open
achave11-ucsc opened this issue Feb 24, 2022 · 6 comments
Open

Requesting unfiltered manifest yields a 504 #3892

achave11-ucsc opened this issue Feb 24, 2022 · 6 comments
Labels
+ [priority] High bug [type] A defect preventing use of the system as specified manifests [subject] Generation and contents of manifests orange [process] Done by the Azul team spike:5 [process] Spike estimate of five points

Comments

@achave11-ucsc
Copy link
Member

achave11-ucsc commented Feb 24, 2022 •
edited
Loading

When requesting a compact manifest with no filters in prod2 a 504 Gateway Timeout response is returned.

$ http 'https://service.azul2.data.humancellatlas.org/manifest/files?format=compact&catalog=dcp13'
HTTP/1.1 504 Gateway Timeout
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 61
Content-Type: application/json
Date: Thu, 24 Feb 2022 18:13:22 GMT
Via: 1.1 8cbd4665d0e455e41084d4437d5b6422.cloudfront.net (CloudFront)
X-Amz-Cf-Id: dkNuWwf9ML2NVqdoJStCLzt0r7Hlue2_qTRr7_zP3LoxDTzW66rq9g==
X-Amz-Cf-Pop: LAX53-P1
X-Cache: Error from cloudfront
x-amz-apigw-id: ODyZ1HFnIAMFh7g=
x-amzn-ErrorType: InternalServerErrorException
x-amzn-RequestId: eb134ebe-2bbe-421a-af6a-9e7e578de47f

{
    "message": "Endpoint request timed out"
}

Note that the service logs for this request display no malfunction.
CloudWatch Logs Insights
region: us-east-1
log-group-names: /aws/lambda/azul-service-prod2
start-time: -3600s
end-time: 0s
query-string:

fields @timestamp, @message
| filter @requestId like "e772004d-5ec6-4362-a3cc-3f8fab13a2c8"
| sort project_id desc
| limit 2000
@timestamp @message
2022-02-24 18:13:23.187 [INFO] 2022-02-24T18:13:23.186Z e772004d-5ec6-4362-a3cc-3f8fab13a2c8 POST https://search-azul-index-prod2-f7axew6smhdyuga32dcjapmlla.us-east-1.es.amazonaws.com:443/azul_v2_prod2_dcp13_files_aggregate/_search [status:200 request:26.166s]
2022-02-24 18:13:23.187 [INFO] 2022-02-24T18:13:23.187Z e772004d-5ec6-4362-a3cc-3f8fab13a2c8 Manifest content hash 394521012 was computed in 26.167s using filters MutableFilters(explicit={}, source_ids={'da6948ca-ae2d-47ff-ac5a-8e1312dc3337', '7a817f0e-325e-425b-86cf-82c343925942', 'db44f677-294e-49cc-ad76-1a46fb7870ca', '549f58b0-5a7a-426b-a437-f61ad269896f', 'd36473b1-5e33-4b7a-85fc-488ecc964f78', '7f53c348-d385-43a6-8511-741e193f6b42', '03409b70-6310-402a-a264-9147ea43ae9b', '252accc8-1625-4fa9-b1ca-fc75f09b977c', 'c4d27025-74ef-4cb4-b3a7-6c0287c68bd4', 'e829d562-0ba7-45c2-9417-deb9df4a5c7e', '7ea25eae-af9b-43fe-becb-bf1e8cf177e9', 'd8811777-be06-410d-a9bd-71b6db05a8f9', 'b9563933-e3fb-4513-922b-99c15b39124d', '7e0be183-5d66-4ccf-be28-659a71dd16e5', '5da1ec4d-a462-4bf0-9ee1-4b76d8199241', 'd549b819-669e-4870-836d-5a07d8f036c4', 'd0d1c365-b46d-4ec8-b174-6cae5cd147fa', 'dee17f6d-8c5c-4f0f-b692-1277be521c91', 'b1e0587c-4a90-464d-9f36-f47b486a180e', 'fe93e46d-0784-4849-ab22-b0db620085ee', '1d28b559-e71f-49c7-a94d-4caa5d7892c1', 'bc7f344d-ff18-4a10-8f59-518324ad4126', '655b04a5-8bf1-46e8-bc9c-4b123d20989f', '4f22b28d-aef3-4ac3-bd3e-89ac755a2134', '0bd26623-ed1e-4d89-b704-56dfa1528178', 'b96f1a14-3c04-41fd-aa1e-deb143c56776', '330a500e-fdfd-48c7-8707-b2197110c9de', 'e6138837-e42d-47bb-9c45-38c3f72008cf', 'd2864afc-800d-4f59-b3e9-081fdaec2b6a', 'a4c1112d-362a-47d8-af65-e5c02837b7b7', '82336861-0e39-497f-bdbe-03923f5f36ab', '52cd4265-318a-4f20-b9a4-b24643600820', '0d71ab0f-b712-453f-9639-2cc02930f0e5', '482c4d78-348d-4ecf-9c55-b0453cfcda9c', '61dd201a-d4ab-42df-8f06-dd905877ba68', '91720058-04fd-4bea-a1f3-ad42af15b0a4', '2fae997c-1801-4405-9697-386883f713d8', '67dae2c2-b525-45f1-a702-9c2e303bf84f', '26622890-b33f-4c7d-a91e-83e75bda6106', 'd4757bd5-fb24-4569-8349-171998f24681', '8c21d8f2-ea8b-4007-a27f-ad3de33087be', 'd71bb717-1117-4e81-8400-1bcb2b9f30b1', '9e5cb4da-66b5-4345-bd5c-c84320a7e166', '4023b698-22f9-4672-acf2-e83a6e13ccbf', '878898dd-dae2-425d-90c7-4f6ce2857286', '7d7c58c7-5e1b-4a10-b608-82ccefbc5828', '2bb7a279-d3d0-4e1b-b1eb-3cf0ba721b0c', '041b6b2c-3fa8-4512-b70d-ef4bf89f4ab4', 'deece79d-9f4a-4baf-9fd1-d1de0c6bbe86', '09c8d0c7-35c7-45cb-b4f8-1790c7992f3c', '4ae0b895-72ca-4e26-8411-07626323f9df', '53daf091-c545-4eae-8cbe-fae1e6e73c8c', '54e2bac5-e4c3-4d6b-8e32-1cade76f1c18', '1c10a88a-51dc-4df5-a088-c3afa38a246c', '88dd4a69-b01d-4781-a3ca-828ffc51f56f', 'cefed032-b60e-4097-b435-d0b25dd6ab78', '76bf0b9a-0589-4a0b-a738-a3410604013e', 'deb0ddec-b8c0-40e2-a2c2-695b171628e5', 'b84fad3b-3df3-4942-823c-26a41e2d5ae5', 'a395c817-826a-41c5-a692-d52352b7a5a1', 'c70aabfa-8199-4b7b-b222-b2ef8035ab32', '6ccb6287-26b3-4f47-bd64-dcf68dcea27b', 'c9f3337c-53ae-4709-bb3a-723ec418eeda', '2f60a6b8-036b-435a-8959-45c077f21168', 'bd75ce6a-9a03-40cc-a6b3-f424393f35dd', '0481ef14-8c6f-4e16-bc71-211423bc8d90', '480f466a-3559-4f38-b8af-87f2715d81b9', 'fa8ed927-175f-4389-bd4c-bc3941cf997c', 'e2224510-f0bd-46ba-a208-57810244fba6', '66f7f93a-fbbc-411e-aa3b-1fc15b5a6826', 'b4f7d772-1684-440a-83fd-885e0d13ce0d', '84da8ac1-1608-4c8b-a6ae-fa624aa124fb', '0f37ca34-6639-4058-a842-d69bb5ef057b', 'b35caf6f-7196-4d29-89f9-f26576c05ab8', 'd480f97f-510b-4961-8d66-4491cedd9c9d', '9de724a5-5254-45ba-abb5-0494d43af861', '401d7ddf-5192-4cf6-915b-c3c9443e213b', '5afd43d9-bfbf-43d7-b904-3ad94e7e87d9', '1c626617-ad99-4c85-bb60-8c83b550985c', '34511e5e-eb16-4c34-99da-a0404538a009', 'd6b9db1e-7334-4143-a35a-05f82940dbbd', '2c4218ee-884f-4a75-89db-8bbbd3164804', '2f9182e0-a6a8-4834-996e-2f6512c5eccb', 'e05547a9-dc28-48fd-90af-8d0d8659a6ef', 'faf1d08b-b176-4a94-a6eb-869f0af5038b', 'aa2e93a6-e6fe-4aa0-8bae-2a12aa322699', 'e8d3a212-6eef-4b40-af3b-abe9cadb0a07', '820f2b64-23ca-4b6a-9a94-c5b74b9cbbc1', 'e717e0f8-5a6f-45a9-8f82-c04f37191eea', '03e1d8cc-1772-449a-bf21-36d6f887aab6', 'cdbc4cf8-448b-4a02-ba5e-ab6ede56627a', '948679cb-5017-4f7a-bffa-bac32d47c364', '4addf51d-b7f3-43fa-bc27-9649d4a0f9c2', 'aeb2bd6b-e22b-404e-b18a-ea478a6c3419', '21dd063b-ede2-4028-9c68-14c8a9e26ab6', 'bac19b85-e0e7-4f3c-9ca3-d94c335f1f9a', '5318962b-8fda-40fb-9714-4fd27a33081e', '0c04238e-e8f4-4691-80bb-c684797aff83', 'b1aa61c0-f993-4896-8a0f-15e6b309d30b', 'f760ac1c-3986-49f0-8fec-aae4cded23ac', '8c6314aa-ff6a-4198-9f9e-eff7aa029ca9', 'f9493de0-5261-4a39-b5ba-e32a661083b7', '45832780-0746-4979-9adc-418e94f85de8', 'b9aa492e-056b-490d-956a-950d33a4b852', '6ec8f9f9-e838-4267-8604-e3d37ecfb1c5', '6d39533c-1f7f-494e-9cee-40ed2f801e19', 'e7cadc9b-f3ef-49d5-b3e3-f6029b75716a', '5695fb29-fb6a-4ad6-8738-43621fd94588', 'f380e38a-ad98-4c2c-ba4a-db09da82fb6a', '85955f79-3a92-424e-b195-3f53b924b5e2', '2c1b9d96-74b6-42ac-bb2f-c45a3968db09', 'bbc476e1-5f7a-47fd-aaab-1a5bcd461b16', '3d9e97f0-f187-403e-8192-9f6f97ae54f3', '7db34ac4-6074-4cc0-abc0-6235d1be316c', '85cc91ad-a33f-4f21-b7fb-80f800cec8ff', '25d8cc3a-9aed-476c-a069-b23aa1a8f2c5', '48a0f35e-c458-4756-b3bc-e8aeba57e0e6', 'd31df398-b23f-48ea-a2f3-1ec5bc3537f1', '39c9f820-0f16-498a-94a8-bfc8317aa0e5', 'd49eeacd-131c-4f7c-97cb-578b3f9c53fe', '6cae24be-377a-4bbf-afee-8f0fad930ca0', '22c36f8b-3acf-4388-9adc-369b22913548', '86273555-105f-4dde-8af1-bea78350b622', '70998740-fba3-4f41-b50b-7d9447ec2ba0', '4c6a1b7e-d4e4-436f-9678-8789390a94f0', '5e6ea544-f51e-4730-90e8-dd97421c869d', '6bf6e46a-e398-4f5a-9ca4-d29f7b069e83', '1cf7544b-d2d5-41e3-ba0b-e27086878778', '68ecbef7-ca33-4423-baf6-a190c4457148', '6a9b080f-00ec-4673-9e5c-77b251233851', '777386b8-7110-49de-95c1-b29fb86728e8', '1477238e-068d-4b69-bff9-25939ef02e61', 'aa277f3e-2ccb-4929-a37f-1fc3ea9808d9', 'f2c40da8-66a6-4671-9161-4aac96dd3b62', 'b1c13070-aa08-41b1-a680-dc14103c703a', '5a0909be-59ac-4550-ae8c-31d57f363876', 'b1e0c970-1bce-4e57-b707-8cdf09db08c1', 'ff2e5d3c-e8d2-4cea-953f-9a53d244b098', 'e7fc5d85-70a1-4a7c-81cf-952d0b3ba1e8', '748772a3-6e84-4296-b926-5f9071dd3f55', '25712305-8de7-49dc-bb14-de5d8298d8ff', 'cd9a498b-cbf9-4de3-9435-9c61cfc4d160', '79c67fb9-a77b-4922-9d01-92fd93aff87f', '375c5fee-3c45-475f-803d-acd151eacd47', '2ef5da40-9ae1-4fba-b6ae-64701eca9e1b', '556dfecc-11fe-453d-8214-34dc16c05c8e', 'afbe6b24-2cd3-490e-b4f5-2cbbfb96ad0f', '1542673a-aeda-48a4-81c5-77c7a8f763af', '20d43128-6914-4b76-b402-2d0100d1f052', '6bc1de69-d841-49e3-94c5-d25775e41433', '7f7af9e1-259c-4a92-9b3c-e290ef09e352', 'c8f4520b-74b9-44ed-b2e0-140a25aab5ba', '95927c86-9d3b-480f-9b2b-452956cd9e72', 'eef5612d-6b4f-414d-87cd-2dc97229f0cd', '77137192-811a-425b-903b-ef83932fff35', 'e50bc732-c172-4df4-ba6d-415424dd41a8', 'fdc5d07b-a861-4b27-b539-d0b555d6621d', 'b8f00be9-f7b3-463b-8030-13b9275e394e', '532a2687-966b-430c-a194-921428832df7', '8f00c4a9-d4e5-4e33-9110-3879264eccac', 'e01ea88e-00d8-464e-a750-8769ea11d934', 'e04f12c0-29ee-41ef-b8bb-40bbfea3c89e', 'bce60b88-654e-44de-b6cd-9a5d5079833c', '6083072c-4597-42ea-a65d-49d4e6a270a0', 'fa0ea5d3-a529-4731-a0de-c17f33885971', '4d333d73-7728-4c8c-b615-d2ddefa37eb0', '3e8e9a1d-f249-41e7-8a01-ef289ab42241', '564e84c1-bf16-4dad-995b-fbcd6a4bf4d1', '6d331dda-b1fa-4f59-bde5-7304d4e5cdbf', 'e29906d7-7d5f-4383-9e47-f3aa67a5db79', 'c9df902d-7e83-46f4-9529-86e82007e242', '37ca958f-b4fb-4f56-8d28-a6f36492a5f7', 'bbe3aa2e-42b0-44db-83c8-a4b9ec8c31ab', 'ec82832e-27d7-464b-9e88-9d1fe395f396', '8f623603-410c-46e1-82aa-b017b94171e2', 'f0896197-e9ba-41e9-97da-3d4c09f406c6', '4cb28d9f-a289-49e8-b629-3f644f7cbe70', '65e7e128-9582-447e-b72d-bb1c32919ba4', 'f49bbbcf-f081-496f-8a21-51a245bc91e0', '4322719f-556b-4588-a1ce-3cb5682f883d', 'fef7ec23-d73a-4aa2-a64e-2cdbd0b47559', '8ef8b1c8-a6d3-4a64-8af2-f505f4e67274', '64c9299e-3949-4475-ab3e-47449c382ae0', 'a378880a-a24c-4df8-b8c0-dab2775edd9c', '3ccf79bc-89b8-4448-96e9-605e05d0e7b4', 'da25c98b-6b2a-48d9-98a4-2efa6080b158', '2c2a61a6-1b06-4770-b37d-621a68b2bffa', 'd96b2c52-c943-4f17-99ff-049b03bb46ba', '9b60f33e-b9d1-4b66-95e5-b40e42012660', 'e6a81e9a-4e5a-4480-89b7-1b5abe072ee9', '2efd52e7-0096-4a0c-9b0a-18b872eb062c', 'b4a798c1-2d39-49ae-998e-80adbd2de5c4', '9a362ef6-abbb-4094-97fb-36a2526be52b', '1d985148-e272-4fb3-8750-6df5a42d233b', 'b5cd9eed-a005-4584-baef-1f1bde61fcb1', 'c25f206f-c5de-48f2-9dfa-7ab41221d6eb', 'e0258c32-9baf-4e78-8318-d0b73233b744', '88d0385d-63fd-40cf-bcae-5eb530e65fc8', '112b212a-72a8-46fe-b678-78ea12d38869', '80779470-39e2-4e4a-b979-94494dd27902', '43f89fcc-8cd3-40d2-a2a3-aa43b2ce86c5', '809549a8-63e4-44bd-bc8b-2772c7349164', '5d00d70c-7edc-444f-a825-83dbc1b26403', '6be53a51-c58b-46c5-b454-8d9a9f6bb48a', '9670836c-7eb9-4000-b54a-b5be81421fa1', '3047ec0b-695e-4c1f-8c20-bf060b5f72aa', '2c253d58-c23a-417a-98a3-497a78d6c47d', '3f5afa2e-e491-44b7-b829-a5af58bc9fd9'}).
2022-02-24 18:13:23.339 [DEBUG] 2022-02-24T18:13:23.339Z e772004d-5ec6-4362-a3cc-3f8fab13a2c8 Manifest object expires in 539196.660758 seconds, on 2022-03-03 00:00:00+00:00
2022-02-24 18:13:23.358 [DEBUG] 2022-02-24T18:13:23.358Z e772004d-5ec6-4362-a3cc-3f8fab13a2c8 Returning 302 response with headers {"Location": "https://edu-ucsc-gi-azul-dcp2-prod-storage-prod2.s3.amazonaws.com/manifests/aa352adb-2626-503d-96ca-391ee03742ba.tsv?response-content-disposition=attachment%3Bfilename%3D%22hca-manifest-aa352adb-2626-503d-96ca-391ee03742ba.tsv%22&AWSAccessKeyId=&Signature=ZuDuJAapNzDxxlDLZr0luVKJlYs%3D&x-amz-security-token=IQoJb3JpZ2luX2VjEDoaCXVzLWVhc3QtMSJHMEUCIQCITDRO7BtSSCtSHI0jVR8RElQzpGgt%2Fl4H6h9eekh2xgIgPTuwTZloKyP%2BRugCjHzO0t4pVeLemaqPOj9VAoPNQnAqoAIIk%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgw1NDI3NTQ1ODkzMjYiDEVQVrriymGVUos0sCr0AS5Ae1xhjhj%2FpR3IDvoiPTbePWJ76CyDkhXD6Jzo%2BBTjefBEKCQMAf2rH6qfZAbgnnx5o0fSNTPeZ81EbQ%2Be4L%2FcsGRBd4UMGfrV%2BOQyYi5mH0dS0728bgfAIBuIh1jC6EKxd5tq2pSxUde1rVoD6wFgeISDRiUhlJmjFTce2J0fQr%2FevNaOrlcN%2FO4KEBEb7oaOHHi3SgufFJfCTzo26KlBb2NNdWBlAV22GKAXaf4svPS2E9fOZDA%2BEifumdNcGO1m8GCa8FPUIgTBM3bArYHPFYpOaaJ%2BCnKrHvqjzE73v%2B27oXVTEJ6qUmTlIq4%2FPtT5Q3ow0YDfkAY6mgHsslKmaf2h8586U3Zi1tY4ODL8Iqxff3TTJr%2BjCXF%2B1ELKqLX2wyEDBIVUQ74ennoXbK6JhhWqWthf6YIss4SBhofVkywyBNeqyQ1E9FX9YjLKJwSf8BEUo6fVXJvfcETnyt8AIC7acDLTr8S%2BAumP%2B%2B8cyJx0hsqdH87wVKXGXgZZv3CPLALA08sqgo9tszJQZBufICvZtNa3&Expires=1645730003", "Access-Control-Allow-Origin": "*", "Access-Control-Allow-Headers": "Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key"}. See next line for the first 1024 characters of the body.

Download the manifest in cmd.exe with `curl` using:

curl.exe --location --output "hca-manifest-aa352adb-2626-503d-96ca-391ee03742ba.tsv" "https://edu-ucsc-gi-azul-dcp2-prod-storage-prod2.s3.amazonaws.com/manifests/aa352adb-2626-503d-96ca-391ee03742ba.tsv?response-content-disposition=attachment%3Bfilename%3D%22hca-manifest-aa352adb-2626-503d-96ca-391ee03742ba.tsv%22&AWSAccessKeyId=&Signature=ZuDuJAapNzDxxlDLZr0luVKJlYs%3D&x-amz-security-token=IQoJb3JpZ2luX2VjEDoaCXVzLWVhc3QtMSJHMEUCIQCITDRO7BtSSCtSHI0jVR8RElQzpGgt%2Fl4H6h9eekh2xgIgPTuwTZloKyP%2BRugCjHzO0t4pVeLemaqPOj9VAoPNQnAqoAIIk%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgw1NDI3NTQ1ODkzMjYiDEVQVrriymGVUos0sCr0AS5Ae1xhjhj%2FpR3IDvoiPTbePWJ76CyDkhXD6Jzo%2BBTjefBEKCQMAf2rH6qfZAbgnnx5o0fSNTPeZ81EbQ%2Be4L%2FcsGRBd4UMGfrV%2BOQyYi5mH0dS0728bgfAIBuIh1jC6EKxd5tq2pSxUde1rVoD6wFgeISDRiUhlJmjFTce2J0fQr%2FevNaOrlcN%2FO4KEBEb7oaOHHi3SgufFJfCTzo26KlBb2NNdWBlAV22GKAXaf4svPS2E9fOZDA%2BEifumdNcGO1m8GCa8FPUIgTBM3bArYHPFYpOaaJ%2BCnKrHvqjzE73v%2B27oXVTEJ6qUmTlIq
2022-02-24 18:13:23.359 END RequestId: e772004d-5ec6-4362-a3cc-3f8fab13a2c8
2022-02-24 18:13:23.359 REPORT RequestId: e772004d-5ec6-4362-a3cc-3f8fab13a2c8 Duration: 30231.25 ms Billed Duration: 30232 ms Memory Size: 2048 MB Max Memory Used: 121 MB
2022-02-24 18:12:53.123 START RequestId: e772004d-5ec6-4362-a3cc-3f8fab13a2c8 Version: $LATEST
2022-02-24 18:12:53.134 [INFO] 2022-02-24T18:12:53.128Z e772004d-5ec6-4362-a3cc-3f8fab13a2c8 Received GET request for '/manifest/files', with query {"catalog": "dcp13", "format": "compact"} and headers {"accept": "*/*", "accept-encoding": "gzip, deflate", "cloudfront-forwarded-proto": "https", "cloudfront-is-desktop-viewer": "true", "cloudfront-is-mobile-viewer": "false", "cloudfront-is-smarttv-viewer": "false", "cloudfront-is-tablet-viewer": "false", "cloudfront-viewer-country": "US", "host": "service.azul2.data.humancellatlas.org", "user-agent": "HTTPie/2.2.0", "via": "1.1 8cbd4665d0e455e41084d4437d5b6422.cloudfront.net (CloudFront)", "x-amz-cf-id": "dkNuWwf9ML2NVqdoJStCLzt0r7Hlue2_qTRr7_zP3LoxDTzW66rq9g==", "x-amzn-trace-id": "Root=1-6217caa5-01c0154417f401b3367284c6", "x-forwarded-for": "76.89.218.122, 130.176.213.18", "x-forwarded-port": "443", "x-forwarded-proto": "https"}.
2022-02-24 18:12:53.134 [INFO] 2022-02-24T18:12:53.128Z e772004d-5ec6-4362-a3cc-3f8fab13a2c8 Did not authenticate request.
2022-02-24 18:12:53.159 [DEBUG] 2022-02-24T18:12:53.159Z e772004d-5ec6-4362-a3cc-3f8fab13a2c8 _request('GET', 'https://data.terra.bio/api/repository/v1/snapshots', fields={'offset': 0, 'limit': 200}, headers=None, body=None)
2022-02-24 18:12:54.509 [DEBUG] 2022-02-24T18:12:54.509Z e772004d-5ec6-4362-a3cc-3f8fab13a2c8 _request(…) -> b'{"total":259,"filteredTotal":259,"items":[{"id":"3ea18ed4-17fd-4209-b3b9-dfcb3f2f05a9","name":"hca_prod_df88f39f01a84b5b92f43177d6c0f242__20211004_20211004","description":"Create snapshot hca_prod_df88f39f01a84b5b92f43177d6c0f242__20211004_20211004","cr...'
2022-02-24 18:12:54.511 [DEBUG] 2022-02-24T18:12:54.511Z e772004d-5ec6-4362-a3cc-3f8fab13a2c8 _request('GET', 'https://data.terra.bio/api/repository/v1/snapshots', fields={'offset': 200, 'limit': 200}, headers=None, body=None)
2022-02-24 18:12:55.757 [DEBUG] 2022-02-24T18:12:55.757Z e772004d-5ec6-4362-a3cc-3f8fab13a2c8 _request(…) -> b'{"total":259,"filteredTotal":259,"items":[{"id":"6d39533c-1f7f-494e-9cee-40ed2f801e19","name":"hca_prod_d012d4768f8c4ff389d6ebbe22c1b5c1__20220119_dcp2_20220121_dcp12","description":"Create snapshot hca_prod_d012d4768f8c4ff389d6ebbe22c1b5c1__20220119_dc...'
2022-02-24 18:12:55.758 [DEBUG] 2022-02-24T18:12:55.758Z e772004d-5ec6-4362-a3cc-3f8fab13a2c8 _request('GET', 'https://data.terra.bio/api/repository/v1/snapshots', fields={'offset': 259, 'limit': 200}, headers=None, body=None)
2022-02-24 18:12:57.001 [DEBUG] 2022-02-24T18:12:57.001Z e772004d-5ec6-4362-a3cc-3f8fab13a2c8 _request(…) -> b'{"total":259,"filteredTotal":259,"items":[]}'
2022-02-24 18:12:57.019 [DEBUG] 2022-02-24T18:12:57.019Z e772004d-5ec6-4362-a3cc-3f8fab13a2c8 Computing content hash for manifest using filters MutableFilters(explicit={}, source_ids={'da6948ca-ae2d-47ff-ac5a-8e1312dc3337', '7a817f0e-325e-425b-86cf-82c343925942', 'db44f677-294e-49cc-ad76-1a46fb7870ca', '549f58b0-5a7a-426b-a437-f61ad269896f', 'd36473b1-5e33-4b7a-85fc-488ecc964f78', '7f53c348-d385-43a6-8511-741e193f6b42', '03409b70-6310-402a-a264-9147ea43ae9b', '252accc8-1625-4fa9-b1ca-fc75f09b977c', 'c4d27025-74ef-4cb4-b3a7-6c0287c68bd4', 'e829d562-0ba7-45c2-9417-deb9df4a5c7e', '7ea25eae-af9b-43fe-becb-bf1e8cf177e9', 'd8811777-be06-410d-a9bd-71b6db05a8f9', 'b9563933-e3fb-4513-922b-99c15b39124d', '7e0be183-5d66-4ccf-be28-659a71dd16e5', '5da1ec4d-a462-4bf0-9ee1-4b76d8199241', 'd549b819-669e-4870-836d-5a07d8f036c4', 'd0d1c365-b46d-4ec8-b174-6cae5cd147fa', 'dee17f6d-8c5c-4f0f-b692-1277be521c91', 'b1e0587c-4a90-464d-9f36-f47b486a180e', 'fe93e46d-0784-4849-ab22-b0db620085ee', '1d28b559-e71f-49c7-a94d-4caa5d7892c1', 'bc7f344d-ff18-4a10-8f59-518324ad4126', '655b04a5-8bf1-46e8-bc9c-4b123d20989f', '4f22b28d-aef3-4ac3-bd3e-89ac755a2134', '0bd26623-ed1e-4d89-b704-56dfa1528178', 'b96f1a14-3c04-41fd-aa1e-deb143c56776', '330a500e-fdfd-48c7-8707-b2197110c9de', 'e6138837-e42d-47bb-9c45-38c3f72008cf', 'd2864afc-800d-4f59-b3e9-081fdaec2b6a', 'a4c1112d-362a-47d8-af65-e5c02837b7b7', '82336861-0e39-497f-bdbe-03923f5f36ab', '52cd4265-318a-4f20-b9a4-b24643600820', '0d71ab0f-b712-453f-9639-2cc02930f0e5', '482c4d78-348d-4ecf-9c55-b0453cfcda9c', '61dd201a-d4ab-42df-8f06-dd905877ba68', '91720058-04fd-4bea-a1f3-ad42af15b0a4', '2fae997c-1801-4405-9697-386883f713d8', '67dae2c2-b525-45f1-a702-9c2e303bf84f', '26622890-b33f-4c7d-a91e-83e75bda6106', 'd4757bd5-fb24-4569-8349-171998f24681', '8c21d8f2-ea8b-4007-a27f-ad3de33087be', 'd71bb717-1117-4e81-8400-1bcb2b9f30b1', '9e5cb4da-66b5-4345-bd5c-c84320a7e166', '4023b698-22f9-4672-acf2-e83a6e13ccbf', '878898dd-dae2-425d-90c7-4f6ce2857286', '7d7c58c7-5e1b-4a10-b608-82ccefbc5828', '2bb7a279-d3d0-4e1b-b1eb-3cf0ba721b0c', '041b6b2c-3fa8-4512-b70d-ef4bf89f4ab4', 'deece79d-9f4a-4baf-9fd1-d1de0c6bbe86', '09c8d0c7-35c7-45cb-b4f8-1790c7992f3c', '4ae0b895-72ca-4e26-8411-07626323f9df', '53daf091-c545-4eae-8cbe-fae1e6e73c8c', '54e2bac5-e4c3-4d6b-8e32-1cade76f1c18', '1c10a88a-51dc-4df5-a088-c3afa38a246c', '88dd4a69-b01d-4781-a3ca-828ffc51f56f', 'cefed032-b60e-4097-b435-d0b25dd6ab78', '76bf0b9a-0589-4a0b-a738-a3410604013e', 'deb0ddec-b8c0-40e2-a2c2-695b171628e5', 'b84fad3b-3df3-4942-823c-26a41e2d5ae5', 'a395c817-826a-41c5-a692-d52352b7a5a1', 'c70aabfa-8199-4b7b-b222-b2ef8035ab32', '6ccb6287-26b3-4f47-bd64-dcf68dcea27b', 'c9f3337c-53ae-4709-bb3a-723ec418eeda', '2f60a6b8-036b-435a-8959-45c077f21168', 'bd75ce6a-9a03-40cc-a6b3-f424393f35dd', '0481ef14-8c6f-4e16-bc71-211423bc8d90', '480f466a-3559-4f38-b8af-87f2715d81b9', 'fa8ed927-175f-4389-bd4c-bc3941cf997c', 'e2224510-f0bd-46ba-a208-57810244fba6', '66f7f93a-fbbc-411e-aa3b-1fc15b5a6826', 'b4f7d772-1684-440a-83fd-885e0d13ce0d', '84da8ac1-1608-4c8b-a6ae-fa624aa124fb', '0f37ca34-6639-4058-a842-d69bb5ef057b', 'b35caf6f-7196-4d29-89f9-f26576c05ab8', 'd480f97f-510b-4961-8d66-4491cedd9c9d', '9de724a5-5254-45ba-abb5-0494d43af861', '401d7ddf-5192-4cf6-915b-c3c9443e213b', '5afd43d9-bfbf-43d7-b904-3ad94e7e87d9', '1c626617-ad99-4c85-bb60-8c83b550985c', '34511e5e-eb16-4c34-99da-a0404538a009', 'd6b9db1e-7334-4143-a35a-05f82940dbbd', '2c4218ee-884f-4a75-89db-8bbbd3164804', '2f9182e0-a6a8-4834-996e-2f6512c5eccb', 'e05547a9-dc28-48fd-90af-8d0d8659a6ef', 'faf1d08b-b176-4a94-a6eb-869f0af5038b', 'aa2e93a6-e6fe-4aa0-8bae-2a12aa322699', 'e8d3a212-6eef-4b40-af3b-abe9cadb0a07', '820f2b64-23ca-4b6a-9a94-c5b74b9cbbc1', 'e717e0f8-5a6f-45a9-8f82-c04f37191eea', '03e1d8cc-1772-449a-bf21-36d6f887aab6', 'cdbc4cf8-448b-4a02-ba5e-ab6ede56627a', '948679cb-5017-4f7a-bffa-bac32d47c364', '4addf51d-b7f3-43fa-bc27-9649d4a0f9c2', 'aeb2bd6b-e22b-404e-b18a-ea478a6c3419', '21dd063b-ede2-4028-9c68-14c8a9e26ab6', 'bac19b85-e0e7-4f3c-9ca3-d94c335f1f9a', '5318962b-8fda-40fb-9714-4fd27a33081e', '0c04238e-e8f4-4691-80bb-c684797aff83', 'b1aa61c0-f993-4896-8a0f-15e6b309d30b', 'f760ac1c-3986-49f0-8fec-aae4cded23ac', '8c6314aa-ff6a-4198-9f9e-eff7aa029ca9', 'f9493de0-5261-4a39-b5ba-e32a661083b7', '45832780-0746-4979-9adc-418e94f85de8', 'b9aa492e-056b-490d-956a-950d33a4b852', '6ec8f9f9-e838-4267-8604-e3d37ecfb1c5', '6d39533c-1f7f-494e-9cee-40ed2f801e19', 'e7cadc9b-f3ef-49d5-b3e3-f6029b75716a', '5695fb29-fb6a-4ad6-8738-43621fd94588', 'f380e38a-ad98-4c2c-ba4a-db09da82fb6a', '85955f79-3a92-424e-b195-3f53b924b5e2', '2c1b9d96-74b6-42ac-bb2f-c45a3968db09', 'bbc476e1-5f7a-47fd-aaab-1a5bcd461b16', '3d9e97f0-f187-403e-8192-9f6f97ae54f3', '7db34ac4-6074-4cc0-abc0-6235d1be316c', '85cc91ad-a33f-4f21-b7fb-80f800cec8ff', '25d8cc3a-9aed-476c-a069-b23aa1a8f2c5', '48a0f35e-c458-4756-b3bc-e8aeba57e0e6', 'd31df398-b23f-48ea-a2f3-1ec5bc3537f1', '39c9f820-0f16-498a-94a8-bfc8317aa0e5', 'd49eeacd-131c-4f7c-97cb-578b3f9c53fe', '6cae24be-377a-4bbf-afee-8f0fad930ca0', '22c36f8b-3acf-4388-9adc-369b22913548', '86273555-105f-4dde-8af1-bea78350b622', '70998740-fba3-4f41-b50b-7d9447ec2ba0', '4c6a1b7e-d4e4-436f-9678-8789390a94f0', '5e6ea544-f51e-4730-90e8-dd97421c869d', '6bf6e46a-e398-4f5a-9ca4-d29f7b069e83', '1cf7544b-d2d5-41e3-ba0b-e27086878778', '68ecbef7-ca33-4423-baf6-a190c4457148', '6a9b080f-00ec-4673-9e5c-77b251233851', '777386b8-7110-49de-95c1-b29fb86728e8', '1477238e-068d-4b69-bff9-25939ef02e61', 'aa277f3e-2ccb-4929-a37f-1fc3ea9808d9', 'f2c40da8-66a6-4671-9161-4aac96dd3b62', 'b1c13070-aa08-41b1-a680-dc14103c703a', '5a0909be-59ac-4550-ae8c-31d57f363876', 'b1e0c970-1bce-4e57-b707-8cdf09db08c1', 'ff2e5d3c-e8d2-4cea-953f-9a53d244b098', 'e7fc5d85-70a1-4a7c-81cf-952d0b3ba1e8', '748772a3-6e84-4296-b926-5f9071dd3f55', '25712305-8de7-49dc-bb14-de5d8298d8ff', 'cd9a498b-cbf9-4de3-9435-9c61cfc4d160', '79c67fb9-a77b-4922-9d01-92fd93aff87f', '375c5fee-3c45-475f-803d-acd151eacd47', '2ef5da40-9ae1-4fba-b6ae-64701eca9e1b', '556dfecc-11fe-453d-8214-34dc16c05c8e', 'afbe6b24-2cd3-490e-b4f5-2cbbfb96ad0f', '1542673a-aeda-48a4-81c5-77c7a8f763af', '20d43128-6914-4b76-b402-2d0100d1f052', '6bc1de69-d841-49e3-94c5-d25775e41433', '7f7af9e1-259c-4a92-9b3c-e290ef09e352', 'c8f4520b-74b9-44ed-b2e0-140a25aab5ba', '95927c86-9d3b-480f-9b2b-452956cd9e72', 'eef5612d-6b4f-414d-87cd-2dc97229f0cd', '77137192-811a-425b-903b-ef83932fff35', 'e50bc732-c172-4df4-ba6d-415424dd41a8', 'fdc5d07b-a861-4b27-b539-d0b555d6621d', 'b8f00be9-f7b3-463b-8030-13b9275e394e', '532a2687-966b-430c-a194-921428832df7', '8f00c4a9-d4e5-4e33-9110-3879264eccac', 'e01ea88e-00d8-464e-a750-8769ea11d934', 'e04f12c0-29ee-41ef-b8bb-40bbfea3c89e', 'bce60b88-654e-44de-b6cd-9a5d5079833c', '6083072c-4597-42ea-a65d-49d4e6a270a0', 'fa0ea5d3-a529-4731-a0de-c17f33885971', '4d333d73-7728-4c8c-b615-d2ddefa37eb0', '3e8e9a1d-f249-41e7-8a01-ef289ab42241', '564e84c1-bf16-4dad-995b-fbcd6a4bf4d1', '6d331dda-b1fa-4f59-bde5-7304d4e5cdbf', 'e29906d7-7d5f-4383-9e47-f3aa67a5db79', 'c9df902d-7e83-46f4-9529-86e82007e242', '37ca958f-b4fb-4f56-8d28-a6f36492a5f7', 'bbe3aa2e-42b0-44db-83c8-a4b9ec8c31ab', 'ec82832e-27d7-464b-9e88-9d1fe395f396', '8f623603-410c-46e1-82aa-b017b94171e2', 'f0896197-e9ba-41e9-97da-3d4c09f406c6', '4cb28d9f-a289-49e8-b629-3f644f7cbe70', '65e7e128-9582-447e-b72d-bb1c32919ba4', 'f49bbbcf-f081-496f-8a21-51a245bc91e0', '4322719f-556b-4588-a1ce-3cb5682f883d', 'fef7ec23-d73a-4aa2-a64e-2cdbd0b47559', '8ef8b1c8-a6d3-4a64-8af2-f505f4e67274', '64c9299e-3949-4475-ab3e-47449c382ae0', 'a378880a-a24c-4df8-b8c0-dab2775edd9c', '3ccf79bc-89b8-4448-96e9-605e05d0e7b4', 'da25c98b-6b2a-48d9-98a4-2efa6080b158', '2c2a61a6-1b06-4770-b37d-621a68b2bffa', 'd96b2c52-c943-4f17-99ff-049b03bb46ba', '9b60f33e-b9d1-4b66-95e5-b40e42012660', 'e6a81e9a-4e5a-4480-89b7-1b5abe072ee9', '2efd52e7-0096-4a0c-9b0a-18b872eb062c', 'b4a798c1-2d39-49ae-998e-80adbd2de5c4', '9a362ef6-abbb-4094-97fb-36a2526be52b', '1d985148-e272-4fb3-8750-6df5a42d233b', 'b5cd9eed-a005-4584-baef-1f1bde61fcb1', 'c25f206f-c5de-48f2-9dfa-7ab41221d6eb', 'e0258c32-9baf-4e78-8318-d0b73233b744', '88d0385d-63fd-40cf-bcae-5eb530e65fc8', '112b212a-72a8-46fe-b678-78ea12d38869', '80779470-39e2-4e4a-b979-94494dd27902', '43f89fcc-8cd3-40d2-a2a3-aa43b2ce86c5', '809549a8-63e4-44bd-bc8b-2772c7349164', '5d00d70c-7edc-444f-a825-83dbc1b26403', '6be53a51-c58b-46c5-b454-8d9a9f6bb48a', '9670836c-7eb9-4000-b54a-b5be81421fa1', '3047ec0b-695e-4c1f-8c20-bf060b5f72aa', '2c253d58-c23a-417a-98a3-497a78d6c47d', '3f5afa2e-e491-44b7-b829-a5af58bc9fd9'}) ...
@achave11-ucsc achave11-ucsc added the orange [process] Done by the Azul team label Feb 24, 2022
@melainalegaspi melainalegaspi changed the title Requesting compact manifest in prod2 yields a 504 Status Code Requesting compact manifest without filters yields a 504 Feb 24, 2022
@melainalegaspi melainalegaspi added bug [type] A defect preventing use of the system as specified manifests [subject] Generation and contents of manifests labels Feb 24, 2022
@melainalegaspi
Copy link

melainalegaspi commented Feb 24, 2022 •
edited
Loading

@hannes-ucsc to propose browser improvement regarding large manifests taking a long time and to have Azul expose an estimated manifest creation time in the summary response.

Also, @hannes-ucsc to come up with a way to accelerate the content hash check (move some of the check to indexing time).

@melainalegaspi
Copy link

First @achave11 spike to eliminate a TDR slowness (listing sources) as a contributing factor.

@melainalegaspi melainalegaspi added the spike:3 [process] Spike estimate of three points label Feb 24, 2022
@achave11-ucsc
Copy link
Member Author

When looking at the above request and a few others, the following times were observed for the requests that list the TDR sources.

        requestId: e772004d-5ec6-4362-a3cc-3f8fab13a2c8, the first request to list sources started @
        2022-02-24T18:12:53.159Z        and the last response was received @
        2022-02-24T18:12:57.001Z        taking a total of 3.842 secs to complete.
                                        Content hash gen. 26.167s

        requestId=3238ce58-65da-4774-b485-f4a97979d1da, the first request to list sources started @
        2022-02-24T15:02:25.959         and the last response was received @
        2022-02-24T15:02:30.259         taking a total of 4.300 secs to complete.
                                        Content hash gen. 22.397s

        requestId=2ffd1eec-caf6-440b-9d47-645adab0298e, the first request to list sources started @
        2022-02-24T16:01:12.977         and the last response was received @
        2022-02-24T16:01:16.798         taking a total of 3.821
                                        Content hash gen. 0.0  / was already cached.

        # Experimental request made the day this ticket was spiked 03/07/22
        requestId=851638a9-925f-4e3b-8ef6-38a98e924a41, the first request to list sources started @
        2022-03-07T16:02:20.680         and the last response was reveived @
        2022-03-07T16:02:21.543         taking a total of 0.863 secs to complete.
                                        Content hash gen. 26.284s

In the above findings, it is apparent that the content hash generation takes longest in comparison to the listing of TDR sources.

@melainalegaspi
Copy link

@hannes-ucsc : "Great. That confirms that TDR is not the culprit here. Back to me for coming up with a solution to accelerate the hash generation."

@melainalegaspi melainalegaspi added spike:5 [process] Spike estimate of five points and removed spike:3 [process] Spike estimate of three points labels Mar 8, 2022
@hannes-ucsc
Copy link
Member

hannes-ucsc commented Mar 14, 2022 •
edited
Loading

Since we're not using incremental indexing anymore, there is currently is no way the index could change without a change to the configured set of snapshots to be indexed. So for now it is safe to assume that a hash of the set of the snapshot names is a good enough approximation of the content hash. The hash should be insensitive to ordering differences, obviously.

Long term, we should hash the bundle reference from all contributions into the projects index and change ManifestGenerator.manifest_content_hash to look at the projects index instead of the files index, but still applying the same filters. This would have more false positives than the current solution, but be orders of magnitude faster. A false positive would be caused by a change to the metadata associated with a file in a project even though the filters restrict the manifest to exclude that file. The project-level hash would be different. The file-based hash as implemented today would be the same. It remains to be verified that this approach can't have false negatives. A false negative could be caused by set(projects_matching_filter) not being a superset of set(file.project for file in files_matching_filter).

@hannes-ucsc hannes-ucsc removed their assignment Mar 14, 2022
@bvizzier-ucsc bvizzier-ucsc added the + [priority] High label Oct 17, 2024
@dsotirho-ucsc
Copy link
Contributor

A similar situation occurred on 2024-11-13 this time with a filter that specifies all file types and species.

@dsotirho-ucsc dsotirho-ucsc changed the title Requesting compact manifest without filters yields a 504 Requesting manifest without filters yields a 504 Nov 13, 2024
@dsotirho-ucsc dsotirho-ucsc changed the title Requesting manifest without filters yields a 504 Requesting unfiltered manifest yields a 504 Nov 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
+ [priority] High bug [type] A defect preventing use of the system as specified manifests [subject] Generation and contents of manifests orange [process] Done by the Azul team spike:5 [process] Spike estimate of five points
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@hannes-ucsc @bvizzier-ucsc @achave11-ucsc @melainalegaspi @dsotirho-ucsc