{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":332076075,"defaultBranch":"main","name":"enos","ownerLogin":"hashicorp","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2021-01-22T22:23:01.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/761456?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1722545334.0","currentOid":""},"activityList":{"items":[{"before":"2c2c0d0e37d34c4bf0ab35e8decdb6b13f751696","after":"fc4c6d61c53a6b88aa5dc1889706b6decec4cafb","ref":"refs/heads/artifact-manifest/main/utterly-relevant-moccasin","pushedAt":"2024-08-15T20:20:44.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"Update artifact manifest license header\n\nMatch .copywrite.hcl","shortMessageHtmlLink":"Update artifact manifest license header"}},{"before":"7d6c524682dc6e17d3f189913321d362a1a490cc","after":"2c2c0d0e37d34c4bf0ab35e8decdb6b13f751696","ref":"refs/heads/artifact-manifest/main/utterly-relevant-moccasin","pushedAt":"2024-08-02T17:27:43.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"shore","name":"brian shore","path":"/shore","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/801275?s=80&v=4"},"commit":{"message":"Update artifact manifest license header\n\nMatch .copywrite.hcl","shortMessageHtmlLink":"Update artifact manifest license header"}},{"before":"ef6dccf8ee74ba0c611c463c3521982978506cac","after":"7d6c524682dc6e17d3f189913321d362a1a490cc","ref":"refs/heads/artifact-manifest/main/utterly-relevant-moccasin","pushedAt":"2024-08-01T21:53:34.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"jeanneryan","name":"Jeanne Franco","path":"/jeanneryan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/64046531?s=80&v=4"},"commit":{"message":"add copyright header","shortMessageHtmlLink":"add copyright header"}},{"before":"f0c86a86e6c2d003f914c38795dee29e7816ce5d","after":null,"ref":"refs/heads/ryan/fix-terraform-cli-path","pushedAt":"2024-08-01T20:48:54.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"}},{"before":"62f3a69167d60d429b21aa2a55aa17e0286a7960","after":"0243a3e4d6f2a1f868bbe12326a9e38f76274a57","ref":"refs/heads/main","pushedAt":"2024-08-01T20:48:52.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"terraform_cli: support `path` and `env` and in `terraform_cli` blocks (#152)\n\nAt some point during the serial to multiplex rewrite of Enos, we stopped\r\nrespecting the special `path` and `env` attributes in `terraform_cli`\r\nblocks.\r\n\r\nI discovered this when attempting to test an older version of Terraform\r\nwith `enos` by setting the `path` and it was still resolving the latest\r\nversion which it found via the `PATH`.\r\n\r\nThis resolves that by configuring the Terraform executor in the\r\nworkspace with any special `terraform_cli` variables that have been set\r\nin the flight plan.\r\n\r\n* Configure the Terraform executor with the scenario \"terraform_cli\"\r\n attributes if necessary\r\n* Regenerate protobuf with latest generator\r\n* Update Go modules\r\n* Bump version\r\n\r\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"terraform_cli: support path and env and in terraform_cli blocks (…"}},{"before":"62f3a69167d60d429b21aa2a55aa17e0286a7960","after":"ef6dccf8ee74ba0c611c463c3521982978506cac","ref":"refs/heads/artifact-manifest/main/utterly-relevant-moccasin","pushedAt":"2024-08-01T17:47:05.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"jeanneryan","name":"Jeanne Franco","path":"/jeanneryan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/64046531?s=80&v=4"},"commit":{"message":"Add artifacts manifest (automatically generated)","shortMessageHtmlLink":"Add artifacts manifest (automatically generated)"}},{"before":null,"after":"62f3a69167d60d429b21aa2a55aa17e0286a7960","ref":"refs/heads/artifact-manifest/main/utterly-relevant-moccasin","pushedAt":"2024-08-01T17:47:04.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"jeanneryan","name":"Jeanne Franco","path":"/jeanneryan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/64046531?s=80&v=4"},"commit":{"message":"version: bump version to 0.0.32 (#151)\n\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"version: bump version to 0.0.32 (#151)"}},{"before":"8096df954623f41ae718573cbed232750611833d","after":"f0c86a86e6c2d003f914c38795dee29e7816ce5d","ref":"refs/heads/ryan/fix-terraform-cli-path","pushedAt":"2024-07-11T16:57:08.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"terraform_cli: support `path` and `env` and in `terraform_cli` blocks\n\nAt some point during the serial to multiplex rewrite of Enos, we stopped\nrespecting the special `path` and `env` attributes in `terraform_cli`\nblocks.\n\nI discovered this when attempting to test an older version of Terraform\nwith `enos` by setting the `path` and it was still resolving the latest\nversion which it found via the `PATH`.\n\nThis resolves that by configuring the Terraform executor in the\nworkspace with any special `terraform_cli` variables that have been set\nin the flight plan.\n\n* Configure the Terraform executor with the scenario \"terraform_cli\"\n attributes if necessary\n* Regenerate protobuf with latest generator\n* Update Go modules\n* Bump version\n\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"terraform_cli: support path and env and in terraform_cli blocks"}},{"before":null,"after":"9390a2a9f344889975fa8030ee2a0a8d3d68c195","ref":"refs/heads/dependabot/go_modules/google.golang.org/grpc-1.64.1","pushedAt":"2024-07-09T21:47:47.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"},"commit":{"message":"Bump google.golang.org/grpc from 1.64.0 to 1.64.1\n\nBumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.64.0 to 1.64.1.\n- [Release notes](https://github.com/grpc/grpc-go/releases)\n- [Commits](https://github.com/grpc/grpc-go/compare/v1.64.0...v1.64.1)\n\n---\nupdated-dependencies:\n- dependency-name: google.golang.org/grpc\n dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] ","shortMessageHtmlLink":"Bump google.golang.org/grpc from 1.64.0 to 1.64.1"}},{"before":null,"after":"8096df954623f41ae718573cbed232750611833d","ref":"refs/heads/ryan/fix-terraform-cli-path","pushedAt":"2024-07-03T20:27:49.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"terraform_cli: support `path` and `env` and in `terraform_cli` blocks\n\nAt some point during the serial to multiplex rewrite of Enos we stopped\nrespecting the special `path` and `env` attributes in `terraform_cli`\nblocks.\n\nI discovered this when attempting to test an older version of Terraform\nwith enos by setting the `path` and it was still resolving the latest\nversion which it found via the `PATH`.\n\nThis resolves that by configuring the Terraform executor in the\nworkspace with any special `terraform_cli` variables that have been set\nin the flight plan.\n\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"terraform_cli: support path and env and in terraform_cli blocks"}},{"before":"035042488628981e4b6d92eb5bafd818b3f4026d","after":null,"ref":"refs/heads/ryan/bump-version","pushedAt":"2024-06-24T21:18:22.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"}},{"before":"5984ea2aa38419fad370d25eee3ebc69ca750da6","after":"62f3a69167d60d429b21aa2a55aa17e0286a7960","ref":"refs/heads/main","pushedAt":"2024-06-24T21:18:21.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"version: bump version to 0.0.32 (#151)\n\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"version: bump version to 0.0.32 (#151)"}},{"before":null,"after":"035042488628981e4b6d92eb5bafd818b3f4026d","ref":"refs/heads/ryan/bump-version","pushedAt":"2024-06-24T21:10:42.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"version: bump version to 0.0.32\n\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"version: bump version to 0.0.32"}},{"before":"4adbdd0c3fe8b2a5225ac5c132e0dbc79490910e","after":null,"ref":"refs/heads/ryan/update-docker-packaging","pushedAt":"2024-06-24T20:12:42.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"}},{"before":"b609725ecdc1125727ed772bd46c13a5666547c7","after":"5984ea2aa38419fad370d25eee3ebc69ca750da6","ref":"refs/heads/main","pushedAt":"2024-06-24T20:12:41.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"actions: update hashicorp/actions-docker-build to v2 (#150)\n\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"actions: update hashicorp/actions-docker-build to v2 (#150)"}},{"before":null,"after":"4adbdd0c3fe8b2a5225ac5c132e0dbc79490910e","ref":"refs/heads/ryan/update-docker-packaging","pushedAt":"2024-06-24T20:07:11.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"actions: update hashicorp/actions-docker-build to v2\n\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"actions: update hashicorp/actions-docker-build to v2"}},{"before":"27d9bb704f404046ba719fd8651847e728af568b","after":null,"ref":"refs/heads/compliance/add-headers","pushedAt":"2024-06-24T18:25:36.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"}},{"before":"114d3b9eeab3c675958ce17615a4d90227273fda","after":null,"ref":"refs/heads/ryan/scenario-sample-memory","pushedAt":"2024-06-24T18:24:59.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"}},{"before":"a4eff3d02b56f019fc74d02de8bab8f07890d87a","after":"b609725ecdc1125727ed772bd46c13a5666547c7","ref":"refs/heads/main","pushedAt":"2024-06-24T18:24:58.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"VAULT-28313: Consider available memory when validating samples (#149)\n\nFurther improve our sample validation by considering our available\r\nmemory when determining the total number on concurrent workers to make\r\navailable to our sample validator.\r\n\r\nPreviously we'd only consider the number of CPU's which could cause\r\nunreliability on machines with less memory than we'd expect.\r\n\r\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"VAULT-28313: Consider available memory when validating samples (#149)"}},{"before":"fd54f9523745a790d9173e336f6692ba2fe68491","after":"114d3b9eeab3c675958ce17615a4d90227273fda","ref":"refs/heads/ryan/scenario-sample-memory","pushedAt":"2024-06-24T18:20:00.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"VAULT-28313: Consider available memory when validating samples\n\nFurther improve our sample validation by considering our available\nmemory when determining the total number on concurrent workers to make\navailable to our sample validator.\n\nPreviously we'd only consider the number of CPU's, which could cause\nunreliability on machines with less memory than we'd expect.\n\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"VAULT-28313: Consider available memory when validating samples"}},{"before":"c69ef5141d6436f2515281089db64ae5ba1a3aa9","after":"fd54f9523745a790d9173e336f6692ba2fe68491","ref":"refs/heads/ryan/scenario-sample-memory","pushedAt":"2024-06-24T18:16:25.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"VAULT-28313: Consider available memory when validating samples\n\nFurther improve our sample validation by considering our available\nmemory when determining the total number on concurrent workers to make\navailable to our sample validator.\n\nPreviously we'd only consider the number of CPU's, which could cause\nunreliability on machines with less memory than we'd expect.\n\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"VAULT-28313: Consider available memory when validating samples"}},{"before":"d0507833df34b403af3762ce5a1c4fd6daea9f46","after":"c69ef5141d6436f2515281089db64ae5ba1a3aa9","ref":"refs/heads/ryan/scenario-sample-memory","pushedAt":"2024-06-24T17:53:52.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"VAULT-28313: Consider available memory when validating samples\n\nFurther improve our sample validation by considering our available\nmemory when determining the total number on concurrent workers to make\navailable to our sample validator.\n\nPreviously we'd only consider the number of CPU's, which could cause\nunreliability on machines with less memory than we'd expect.\n\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"VAULT-28313: Consider available memory when validating samples"}},{"before":null,"after":"d0507833df34b403af3762ce5a1c4fd6daea9f46","ref":"refs/heads/ryan/scenario-sample-memory","pushedAt":"2024-06-24T17:52:06.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"VAULT-28313: Consider available memory when validating samples\n\nFurther improve our sample validation by considering our available\nmemory when determining the total number on concurrent workers to make\navailable to our sample validator.\n\nPreviously we'd only consider the number of CPU's, which could cause\nunreliability on machines with less memory than we'd expect.\n\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"VAULT-28313: Consider available memory when validating samples"}},{"before":null,"after":"27d9bb704f404046ba719fd8651847e728af568b","ref":"refs/heads/compliance/add-headers","pushedAt":"2024-06-24T16:27:24.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"hashicorp-copywrite[bot]","name":null,"path":"/apps/hashicorp-copywrite","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/761456?s=80&v=4"},"commit":{"message":"[COMPLIANCE] Add Copyright and License Headers","shortMessageHtmlLink":"[COMPLIANCE] Add Copyright and License Headers"}},{"before":"5b8ea24ce069b1aa812341284434276824f9dc6b","after":null,"ref":"refs/heads/vault-28313","pushedAt":"2024-06-24T15:36:13.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"}},{"before":"96c036c93e59ebc09f379767332f001fb3a98f2f","after":"a4eff3d02b56f019fc74d02de8bab8f07890d87a","ref":"refs/heads/main","pushedAt":"2024-06-24T15:36:12.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"[VAULT-28313] enos: decode scenarios iteratively (#147)\n\nSignificantly improve the speeed and reliability of `list` and `validate`\r\nin large repositories. While the prior implementation did list and\r\nvalidate in parallel, it also retained whole sets in memory even if they\r\nwere not being used. Holding onto those references was catastrophic on\r\nmachines with scarce resources. We replace the old implementation with a\r\nscenario decoder that's an iterator which allows the caller to handle the\r\nscenario if necessary or allow it to be garbage collected when we're\r\nthrough with it.\r\n\r\nThis improves our memory usage with `list` and `validate` significantly.\r\nWhen testing `validate` against the Vault `replication` scenario with our\r\nprior implementation, our memory usage grew linerally until reaching\r\n~11GB. Now our memory stays flat in the 300-400mb range.\r\n\r\nOverall it improves listing wall clock speed by 10% and our validate\r\nspeed by 15% when validating a subset of scenarios. On my machine you\r\ncouldn't successfully validate all of them with the prior implementation\r\nand now it works just fine.\r\n\r\nThere's still room for improvement on listing speed. If we cared less\r\nabout format padding we could change the UI to be incremental for the\r\nbasic UI and then render the scenarios on screen while they are being\r\ndecoded. I chose not to do that as it's outside the scope of what we're\r\ntrying to resolve.\r\n\r\nThere's also significant memory usage when creating a sample frame and\r\nvalidating and/or making an observation it. There are likely subsequent\r\nimprovements we could make there but those don't appear to be a blocker\r\nto fixing our validator.\r\n\r\nBackwards incompatible changes:\r\n * We now return an error when attempting to list scenarios in a\r\n directory without any defined scenario blocks.\r\n * We now sort our matrix block by variant and elements and the ordering\r\n can be slightly different than before. I doubt anybody would notice\r\n but it's possible.\r\n * We don't guarantee perfectly sorted output when listing as it is done\r\n in parallel and we aren't caching and sorting.\r\n\r\nChanges:\r\n * Create a scenario decoder that implements an iterator pattern\r\n * Utilize the iterator pattern in various endpoints and tests\r\n * Fix a race in the warnings as errors check acceptance test\r\n * Bump version\r\n * Pin to the latest actions to resolve deprecation warnings\r\n\r\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"[VAULT-28313] enos: decode scenarios iteratively (#147)"}},{"before":"4c58961b9c2d4dec75424f68b476092674f0a19f","after":"5b8ea24ce069b1aa812341284434276824f9dc6b","ref":"refs/heads/vault-28313","pushedAt":"2024-06-20T22:16:30.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"[VAULT-28313] enos: decode scenarios iteratively\n\nSignificantly improve the speeed and reliability of `list` and `validate`\nin large repositories. While the prior implementation did list and\nvalidate in parallel, it also retained whole sets in memory even if they\nwere not being used. Holding onto those references was catastrophic on\nmachines with scarce resources. We replace the old implementation with a\nscenario decoder that's an iterator which allows the caller to handle the\nscenario if necessary or allow it to be garbage collected when we're\nthrough with it.\n\nThis improves our memory usage with `list` and `validate` significantly.\nWhen testing `validate` against the Vault `replication` scenario with our\nprior implementation, our memory usage grew linerally until reaching\n~11GB. Now our memory stays flat in the 300-400mb range.\n\nOverall it improves listing wall clock speed by 10% and our validate\nspeed by 15% when validating a subset of scenarios. On my machine you\ncouldn't successfully validate all of them with the prior implementation\nand now it works just fine.\n\nThere's still room for improvement on listing speed. If we cared less\nabout format padding we could change the UI to be incremental for the\nbasic UI and then render the scenarios on screen while they are being\ndecoded. I chose not to do that as it's outside the scope of what we're\ntrying to resolve.\n\nThere's also significant memory usage when creating a sample frame and\nvalidating and/or making an observation it. There are likely subsequent\nimprovements we could make there but those don't appear to be a blocker\nto fixing our validator.\n\nBackwards incompatible changes:\n * We now return an error when attempting to list scenarios in a\n directory without any defined scenario blocks.\n * We now sort our matrix block by variant and elements and the ordering\n can be slightly different than before. I doubt anybody would notice\n but it's possible.\n * We don't guarantee perfectly sorted output when listing as it is done\n in parallel and we aren't caching and sorting.\n\nChanges:\n * Create a scenario decoder that implements an iterator pattern\n * Utilize the iterator pattern in various endpoints and tests\n * Fix a race in the warnings as errors check acceptance test\n * Bump version\n * Pin to the latest actions to resolve deprecation warnings\n\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"[VAULT-28313] enos: decode scenarios iteratively"}},{"before":"a0d798882b4646c41cf00fa7558516841d9669db","after":"4c58961b9c2d4dec75424f68b476092674f0a19f","ref":"refs/heads/vault-28313","pushedAt":"2024-06-20T21:35:24.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"[VAULT-28313] enos: decode scenarios iteratively\n\nSignificantly improve the speeed and reliability of `list` and `validate`\nin large repositories. While the prior implementation did list and\nvalidate in parallel, it also retained whole sets in memory even if they\nwere not being used. Holding onto those references was catastrophic on\nmachines with scarce resources. We replace the old implementation with a\nscenario decoder that's an iterator which allows the caller to handle the\nscenario if necessary or allow it to be garbage collected when we're\nthrough with it.\n\nThis improves our memory usage with `list` and `validate` significantly.\nWhen testing `validate` against the Vault `replication` scenario with our\nprior implementation, our memory usage grew linerally until reaching\n~11GB. Now our memory stays flat in the 300-400mb range.\n\nOverall it improves listing wall clock speed by 10% and our validate\nspeed by 15% when validating a subset of scenarios. On my machine you\ncouldn't successfully validate all of them with the prior implementation\nand now it works just fine.\n\nThere's still room for improvement on listing speed. If we cared less\nabout format padding we could change the UI to be incremental for the\nbasic UI and then render the scenarios on screen while they are being\ndecoded. I chose not to do that as it's outside the scope of what we're\ntrying to resolve.\n\nThere's also significant memory usage when creating a sample frame and\nvalidating and/or making an observation it. There are likely subsequent\nimprovements we could make there but those don't appear to be a blocker\nto fixing our validator.\n\nBackwards incompatible changes:\n * We now return an error when attempting to list scenarios in a\n directory without any defined scenario blocks.\n * We now sort our matrix block by variant and elements and the ordering\n can be slightly different than before. I doubt anybody would notice\n but it's possible.\n * We don't guarantee perfectly sorted output when listing as it is done\n in parallel and we aren't caching and sorting.\n\nChanges:\n * Create a scenario decoder that implements an iterator pattern\n * Utilize the iterator pattern in various endpoints and tests\n * Fix a race in the warnings as errors check acceptance test\n * Bump version\n * Pin to the latest actions to resolve deprecation warnings\n\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"[VAULT-28313] enos: decode scenarios iteratively"}},{"before":"094a08869befafc2a6860a644acb5126861c5427","after":"a0d798882b4646c41cf00fa7558516841d9669db","ref":"refs/heads/vault-28313","pushedAt":"2024-06-20T21:32:57.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"[VAULT-28313] enos: decode scenarios iteratively\n\nSignificantly improve the speeed and reliability of `list` and `validate`\nin large repositories. While the prior implementation did list and\nvalidate in parallel, it also retained whole sets in memory even if they\nwere not being used. Holding onto those references was catastrophic on\nmachines with scarce resources. We replace the old implementation with a\nscenario decoder that's an iterator which allows the caller to handle the\nscenario if necessary or allow it to be garbage collected when we're\nthrough with it.\n\nThis improves our memory usage with `list` and `validate` significantly.\nWhen testing `validate` against the Vault `replication` scenario with our\nprior implementation, our memory usage grew linerally until reaching\n~11GB. Now our memory stays flat in the 300-400mb range.\n\nOverall it improves listing wall clock speed by 10% and our validate\nspeed by 15% when validating a subset of scenarios. On my machine you\ncouldn't successfully validate all of them with the prior implementation\nand now it works just fine.\n\nThere's still room for improvement on listing speed. If we cared less\nabout format padding we could change the UI to be incremental for the\nbasic UI and then render the scenarios on screen while they are being\ndecoded. I chose not to do that as it's outside the scope of what we're\ntrying to resolve.\n\nThere's also significant memory usage when creating a sample frame and\nvalidating and/or making an observation it. There are likely subsequent\nimprovements we could make there but those don't appear to be a blocker\nto fixing our validator.\n\nBackwards incompatible changes:\n * We now return an error when attempting to list scenarios in a\n directory without any defined scenario blocks.\n * We now sort our matrix block by variant and elements and the ordering\n can be slightly different than before. I doubt anybody would notice\n but it's possible.\n * We don't guarantee perfectly sorted output when listing as it is done\n in parallel and we aren't caching and sorting.\n\nChanges:\n * Create a scenario decoder that implements an iterator pattern\n * Utilize the iterator pattern in various endpoints and tests\n * Fix a race in the warnings as errors check acceptance test\n * Bump version\n * Pin to the latest actions to resolve deprecation warnings\n\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"[VAULT-28313] enos: decode scenarios iteratively"}},{"before":"65b8796861e02835257a2ed46346670f9bd7cc5f","after":"094a08869befafc2a6860a644acb5126861c5427","ref":"refs/heads/vault-28313","pushedAt":"2024-06-20T21:26:04.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ryancragun","name":"Ryan Cragun","path":"/ryancragun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65058?s=80&v=4"},"commit":{"message":"[VAULT-28313] enos: decode scenarios iteratively\n\nSignificantly improve the speeed and reliability of list and validating\nin large repositories. The prior implementation was parallel but also\nreturned whole sets in memory even if they were not being used. Holding\nonto those references was catastrophic on machines with scarce\nresources. We replace the old collection with a scenario decoder that's\nan iterator which allows the caller to handle the scenario if necessary\nor allow it to be garbage collected when we're through with it.\n\nThis improves our memory usage when list and validating significantly.\nWhen test validating the Vault `replication` scenario with our prior\nimplementation, our memory usage would grow linerally while validating\nthe scenarios. Now our memory stays in the 300-400mb range now instead\nof growing upwards of 11GB.\n\nOverall it improves listing wall clock speed by 8-10% and our validation\nspeed by 15% when validating a subset of scenarios. On my machine you\ncouldn't succesfully validate all of them with the prior implementation\nand now it works just fine.\n\nWe also improved our failure diagnostics output as we'll only return the\na single set of decode diagnostics on failure rather than any that might\nget written to channel from parallel workers.\n\nThere's still room for improvement on speed listing if we care less about\nformat padding as we could change the UI to be incremental for the basic\nUI and then render the scenarios on screen while they are being decoded.\nI chose not to do that as it's outside the scope of what we're trying\nto resolve.\n\nThere's also significant memory usage when creating a sample frame and\nvalidating/observing it. There are likely subsequent improvements we\ncould make there but those don't appear to be a blocker to fixing our\nvalidator.\n\nBackwards incompatible changes:\n * We now return an error when attempting to list scenarios in a\n directory without any defined scenario blocks.\n * We now sort our matrix block by variant and elements and the\n ordering can be slightly different than before. I doubt anybody\n would know.\n * We don't guarantee perfectly sorted output when listing as it\n is done in parallel and we aren't caching and sorting.\n\n* Create a scenario decoder that implements an iterator pattern\n* Utilize the iterator pattern in various endpoints and tests\n* Fix a race in the warnings as errors check acceptance test\n* Bump version\n* Pin to the latest actions to resolve deprecation warnings\n\nSigned-off-by: Ryan Cragun ","shortMessageHtmlLink":"[VAULT-28313] enos: decode scenarios iteratively"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEm4vyEgA","startCursor":null,"endCursor":null}},"title":"Activity · hashicorp/enos"}