From f385f46e32bf47bf5c992d19a54c7933d8c55e00 Mon Sep 17 00:00:00 2001 From: joshua-armory Date: Wed, 16 Oct 2024 15:56:03 -0400 Subject: [PATCH 1/3] readded armory kb documents --- ...ssion-errors-when-enabling-new-services.md | 12 + ...und-github-access-and-rate-limit-issues.md | 15 + ...e-without-application-write-permissions.md | 10 + ...nt-endpoints-to-help-in-troubleshooting.md | 22 ++ ...oubleshooting-armory-scale-agent-issues.md | 32 +++ ...-spinnaker-rest-api-using-httpie-and-jq.md | 81 ++++++ ...e-have-incorrect-dynamic-lookup-results.md | 13 + ...members-to-application-times-out-in-gcp.md | 16 ++ ...l-policies-and-access-to-an-eks-cluster.md | 15 + ...adjust-clean-up-timing-for-armory-agent.md | 13 + ...eption--timeout-exceeded-for-operation-.md | 22 ++ .../applying-custom-sizes-to-services.md | 35 +++ ...known-kind-apiservice-in-apiservice-....md | 13 + ...gent-testing,-investigation,-and-checks.md | 83 ++++++ ...-as-user-already-exists-status-code-409.md | 18 ++ ...als-did-not-have-\"xxxxxx-xxxxx'-scope.md" | 33 +++ ...y-and-disaster-recovery-sla-information.md | 30 ++ ...s-deployment-as-a-service-cdaas-support.md | 150 ++++++++++ kb/armory/General/armory-diagnostics.md | 8 + kb/armory/General/armory-halyard-overview.md | 7 + ...iority-zero-p0-case-handling-procedures.md | 2 +- .../General/armory-product-demo-videos.md | 7 + .../General/armory-release-definitions.md | 8 + ...tion-hours-of-operation-slas-procedures.md | 264 ++++++++++++++++++ .../General/authenticate-against-jenkins.md | 30 ++ ...entication-timeout-results-in-401-error.md | 18 ++ ...ration-with-file-based-role-definitions.md | 48 ++++ ...-of-specific-applications-in-spinnaker-.md | 35 +++ ...ed-w--onlyspinnakermanaged-set-to-false.md | 11 + ...billing-budget-alerts-and-cost-explorer.md | 98 +++++++ ...nts,-when-multiple-accounts-are-defined.md | 12 + .../aws-lambda-&-custom-webhook-stages.md | 198 +++++++++++++ ...es-not-populate-the-existing-functions-.md | 10 + .../General/aws-rds-certificate-update.md | 39 +++ ...st-stage-fails-without-an-error-message.md | 20 ++ ...age-timeouts-caused-by-external-factors.md | 89 ++++++ ...itbucket-cloud-vs-bitbucket-on-premises.md | 10 + ...-unknown-bitbucket-event-type-in-dinghy.md | 16 ++ ...-armory-console-be-retricted-using-rbac.md | 17 ++ ...es\342\200\213-example--aws-cloudwatch.md" | 14 + kb/armory/General/cancel-a-stuck-pipeline.md | 36 +++ ...dd-custom-kind-to-delete-manifest-stage.md | 15 + ...ne-ebs-volumes-when-deploying-baked-ami.md | 10 + ...ed-with-github-pull-request-validations.md | 15 + ...-them-or-adding-an-image-with-same-name.md | 13 + ...t-see-pipeline-history-for-changes-made.md | 10 + .../General/capacity-provider-concepts.md | 17 ++ ...ry's-support-bundle-w--automated-script.md | 44 +++ .../General/cases-and-change-requests.md | 16 ++ ...nfigmap-not-reflected-in-new-deployment.md | 33 +++ ...ker-services-using-okhttp-java-services.md | 11 + ...tions-older-than-a-specific-time-period.md | 25 ++ ...-account-disappearing-from-spinnaker-ui.md | 10 + ...edagentscheduler---unable-to-run-agents.md | 13 + ...-up-armory-agent-during-initial-install.md | 14 + ...is-not-caching-subnets-or-server-groups.md | 30 ++ ...clouddriver-logs-show-slow-sql-warnings.md | 25 ++ ...le,-request-timed-out-after-10000ms.\".md" | 15 + ...ivate-github-instance-due-to-ssl-errors.md | 32 +++ ...ng-a-stoppage,-databasechangelock-error.md | 12 + ...too-long-when-deploying-with-app-engine.md | 31 ++ ...ve-reports---armory-continuous-delivery.md | 17 ++ ...-exposures-cve-reports---armory-plugins.md | 16 ++ kb/armory/General/compliance-and-auditing.md | 17 ++ ...ith-bakes-fail-packer-behavior-in-rosco.md | 16 ++ .../configure-elasticache-with-spinnaker.md | 60 ++++ ...apture-heapdumps-for-spinnaker-services.md | 75 +++++ ...se-dynamic-github-authentication-tokens.md | 56 ++++ ...ternal-account-configuration-with-vault.md | 141 ++++++++++ ...-for-pipeline-credentials-not-found*\".md" | 14 + ...send-a-shorter-and-more-compact-message.md | 30 ++ ...-services-to-connect-to-redis-over-mtls.md | 215 ++++++++++++++ ...to-use-internal-certificate-authorities.md | 72 +++++ ...onfiguring-vim-editor-to-work-with-yaml.md | 25 ++ ...onment-debugging-id-with-armory-support.md | 14 + .../container-image-cannot-be-selected.md | 40 +++ ...on$streamexception--received-data-frame.md | 16 ++ ...spinnaker-plugins-and-donations-of-code.md | 31 ++ ...e-aws-iam-roles-from-a-terraform-script.md | 40 +++ ...figmaps-and-secrets-from-artifact-files.md | 82 ++++++ ...creating-a-change-request-in-servicenow.md | 20 ++ ...cases-and-case-management-in-servicenow.md | 37 +++ .../creating-custom-deployment-strategies.md | 18 ++ .../critical-technical-notifications.md | 17 ++ ...cannot-access-properties-of-application.md | 12 + ...\"access-denied-to-application\"-error.md" | 16 ++ ...ot-have-access-to-create-change-request.md | 9 + ...unseting-deprecated-teams-api-endpoints.md | 19 ++ ...it--spinnaker-rce-vulnerability-in-gate.md | 16 ++ ...og4shell---log4j-vulnerability-analysis.md | 16 ++ ...-22965---spring-framework-rce-w--jdk-9+.md | 14 + ...889---apache-commons-text-vulnerability.md | 17 ++ ...e-recommendations-for-running-spinnaker.md | 20 +- ...ging-enabled-into-a-crash-loop-restart-.md | 14 + .../General/debugging-deployment-errors.md | 12 + .../debugging-slow-orca-performance.md | 12 + ...ine-application-attributes-using-dinghy.md | 71 +++++ .../delayed-image-push-pipeline-trigger.md | 10 + .../delete-pipeline-executions-in-redis.md | 31 ++ ...even-if-skip-spel-evaluation-is-checked.md | 12 + ...m2-deployment-resources-from-kubernetes.md | 12 + .../deploy-stage-missing-from-pipeline.md | 10 + ...-spinnaker-and-managing-secrets-in-helm.md | 21 ++ ...n-list-customresourcedefinition-null\".md" | 140 ++++++++++ ...eption--resolve-deploy-source-manifest-.md | 36 +++ ...--does-not-exist\"-error.md" | 18 ++ ...r-service--does-not-exist.md | 14 + ...b-missing-previously-available-options.md" | 14 + ...deployments-exceeding-progress-deadline.md | 10 + ...eck-the-environment's-spinnaker-version.md | 12 + ...t-phase-takes-and-the-service-boot-time.md | 14 + ...ts-and-choosing-the-right-artifact-type.md | 14 + ...assed-to-actual-spinnaker-pipeline-json.md | 12 + ...in-the-same-dinghyfile-using-pipelineid.md | 26 ++ ...n-customers-are-using-a-custom-endpoint.md | 12 + ...ss-of-connectivity-loss-of-relationship.md | 17 ++ ...te-pipelines-in-armory-2.19.8-and-prior.md | 13 + ...-does-not-update-pipelines-in-spinnaker.md | 11 + ...is-always-failed-when-using-yaml-parser.md | 11 + ...tries-to-validate-readme.md-as-a-module.md | 16 ++ ...ghy-multi-repo-strategy-with-repoconfig.md | 54 ++++ ...d-to-application-access-rights-verified.md | 19 ++ ...g-pipeline-when-local-module-is-changed.md | 10 + ....lang.nosuchmethoderror--void-org.jsoup.md | 14 + ...l-response,-rather-than-a-json-response.md | 13 + ...ates-do-not-trigger-slack-notifications.md | 11 + ...create-update-pipelines-forbidden-error.md | 58 ++++ ...apped-environments-and-specify-the-repo.md | 66 +++++ ...a-feature-that-requires-registration\".md" | 29 ++ ...-echo-pod-unable-to-start-armory-2.21.2.md | 31 ++ ...and-specifying-the-protocols-to-be-used.md | 69 +++++ ...atching-repositories-breaking-pipelines.md | 11 + ...ied-service--amazon-s3;-status-code-403.md | 36 +++ ...s-accounts-with-github-cannot-use-token.md | 10 + .../General/echo-sends-duplicate-triggers.md | 11 + ...-supporting-ephemeralstorage-in-fargate.md | 20 ++ ...group-names-for-cluster....are-taken\".md" | 15 + ...ls-when-using-external-launch-type-task.md | 50 ++++ kb/armory/General/ecs-deployment-failures.md | 16 ++ ...t-read-property-'label'-of-undefined\".md" | 23 ++ ...rror-when-migrating-from-redis-to-mysql.md | 14 + ...discovered-by-spinnaker-or-armory-agent.md | 25 ++ .../General/enable-aws-cloudformation.md | 115 ++++++++ ...uthentication-for-spinnaker-via-halyard.md | 21 ++ ...g-mode-in-spinnaker-environment-service.md | 21 ++ ...-pipelines-to-run-as-mutually-exclusive.md | 41 +++ ...river-caching-to-resolve-caching-issues.md | 35 +++ ...bling-the-managed-pipeline-templates-ui.md | 33 +++ .../error---credential-not-found-aws.md | 11 + ...pplication-because-it-has-server-groups.md | 14 + ...-pipeline-module.json-invalid-character.md | 39 +++ ...e-ui--exception-resolve-target-manifest.md | 16 ++ ...nifest-kind-csidriver-with-armory-agent.md | 23 ++ ...s-configured-to-\"ignore-the-failure\".md" | 11 + ...lastic-cache-with-encryption-in-transit.md | 11 + ...uration-hints,-orca-clouddriver-front50.md | 16 ++ ...unt-configuration-to-external-s3-bucket.md | 21 ++ kb/armory/General/escalating-a-case.md | 18 ++ .../exception-starting-plugin-in-orca.md | 14 + ...-run-job-stages-are-running-in-parallel.md | 13 + ...-sent-by-spinnaker-customwebhook-stages.md | 98 +++++++ ...ayed-or-take-a-lot-more-time-to-execute.md | 76 +++++ kb/armory/General/exposing-spinnaker.md | 7 + ...error-while-saving-canary-configuration.md | 13 + ...ng-issues-when-deploying-cloud-services.md | 11 + ...o-stabilize.-unexpected-task-failure\".md" | 22 ++ ...and-check-role-provider-for-permissions.md | 10 + ...-to-start-in-a-cloudfoundry-environment.md | 14 + .../flushall-on-spinnaker's-redis-cache.md | 18 ++ kb/armory/General/flushing-gate-sessions.md | 39 +++ ...spinnaker-version-from-2.20.1-to-2.23.5.md | 82 ++++++ ...-not-available-to-the-monitoring-daemon.md | 12 + ...le-querying-target-pool-instance-health.md | 25 ++ ...idden--http-status-code-403;-transport-.md | 24 ++ ...-ec2-instance-with-pipeline-expressions.md | 12 + ...ase-branch-from-\"master\"-to-\"main\".md" | 12 + ...ferring-to-mixed-case-applications-fail.md | 11 + ...nes,-due-to-plugins-missing-unavailable.md | 21 ++ ...ions-do-not-update-github-commit-status.md | 23 ++ ...multiple-artifacts-on-trigger-.tf-files.md | 35 +++ ...cts-that-have-been-added-and-can't-edit.md | 10 + .../google-cloud-provider---private-gke.md | 13 + ...b-config-artifact-for-google-app-engine.md | 12 + ...form-gpg-rotation-codecov-vulnerability.md | 18 ++ ...ub-emails-after-every-dinghyfile-update.md | 12 + ...n-deploying-a-configmap-secret-too-long.md | 17 ++ .../hitting-igor's-caching-thresholds.md | 16 ++ ...ttp-https-redirects-with-authentication.md | 31 ++ .../General/iam-auth-on-pods-via-irsa.md | 11 +- ...line-triggers-500-internal-server-error.md | 10 + ...cenow-troubleshooting-servicenow-admins.md | 20 ++ .../General/incorrect-oauth2.0-redirects.md | 10 + ...it-timeouts-on-agent-clouddriver-plugin.md | 61 ++++ ...d-agent-to-avoid-network-latency-issues.md | 57 ++++ ...e-without-issue,-but-loops-upon-timeout.md | 11 + ...json-canary-config-in-the-spinnaker-ui.md" | 11 + ...-not-showing-correct-deployment-details.md | 10 + ...-larger-than-51200-bytes-as-an-artifact.md | 11 + ...cs-from-observability-plugin-to-datadog.md | 17 ++ ...iated-login-does-not-work-with-saml-2.0.md | 11 + ...-but-the-deployment-shows-as-succeeded-.md | 23 ++ .../General/instance-registration-teardown.md | 7 +- kb/armory/General/integration-with-nexus.md | 36 +++ ...ent-after-upgrading-the-armory-operator.md | 27 ++ ...d-token-with-named-profiles-assume-role.md | 11 + ...ccess-to-certain-envrionments-with-rbac.md | 14 + ...information-about-setting-up-monitoring.md | 19 ++ ...o-monitor-the-jenkins-job-tomcat-server.md | 13 + kb/armory/General/jenkins-timeout-issues.md | 32 +++ ...ors-index--1-out-of-bounds-for-length-0.md | 26 ++ ...hooks-payload-values-directory-specific.md | 67 +++++ ...on-outages-with-kuberenetes-auth-method.md | 53 ++++ .../kayenta-404s-in-armory-spinnaker.md | 16 ++ ...are-not-visible-in-spinnaker-console-ui.md | 45 +++ ...ppearing-after-deletion-in-spinnaker-ui.md | 10 + ...s-a-new-service-old-service-not-deleted.md | 10 + .../General/kubernetes-v1-provider-removed.md | 18 ++ ...for-aws-exception--no-message-available.md | 19 ++ ...-errors-due-to-timeouts-and-400-errors..md | 11 + ...rvice-svc-spinnaker-demo-does-not-exist.md | 13 + ...min-access-users-when-using-\"run-as\".md" | 14 + .../long-force-cache-refresh-kubernetes.md | 10 + ...rora-for-spinnaker\342\204\242-no-helm.md" | 225 +++++++++++++++ ...plunk-despite-setting-up-hec-connection.md | 10 + ...clouddriver's-redis-to-its-own-instance.md | 10 + ...plugin-from-the-policy-engine-extension.md | 39 +++ ...n-rosco-image-got-exception-create-bake.md | 49 ++++ ...user-field-for-triggers-in-spinnaker-ui.md | 12 + kb/armory/General/monitored-deploy-rollout.md | 15 + ...communication-failure-when-using-tls1.1.md | 23 ++ ...ling-back-spinnaker-undo-renamed-values.md | 18 ++ ...driver-and-agent-causing-timeout-issues.md | 60 ++++ ...er-'<'-code-60--expected-a-valid-value-.md | 18 ++ .../General/no-expected-artifacts-field.md | 10 + ...g-api-version-\"networking.k8s.io-v1\".md" | 29 ++ .../no-plugins-error-on-service-startup.md | 14 + ...not-get-propagated-while-using-operator.md | 13 + ...ception-error-with-armory-cloud-enabled.md | 21 ++ kb/armory/General/oauth-setup-for-github.md | 7 + ...er-tab-and-a-slow--deck-ui-cluster-page.md | 49 ++++ ...lic-articles-and-cases---internal-users.md | 24 ++ ...es-not-working-after-spinnaker-upgrade-.md | 16 ++ ...r--failed-to-shutdown-server-gracefully.md | 17 ++ ...o-supply-the-complete-certificate-chain.md | 11 + ...connect-to-halyard-due-to-a-tcp-timeout.md | 12 + ...er-upgrade-in-an-air-gapped-environment.md | 13 + ...ing-error-message-about-modified-object.md | 11 + .../orca-mishandling-spel-expressions.md | 19 ++ ...rca-stuck-in-crashloopbackoff-db--mysql.md | 23 ++ ...-memory-usage---tables-information-dive.md | 28 ++ ...is-too-large---mysql-max_allowed_packet.md | 11 + .../pagerduty---setting-up-for-new-users.md | 63 +++++ ...otifications-integration-with-spinnaker.md | 64 +++++ ...s-with-kubernetes-manifest-&-using-spel.md | 15 + kb/armory/General/password-resets.md | 9 + ...ght-kubernetes-deployments-in-spinnaker.md | 18 ++ ...-keys-in-common-with-target-workload\".md" | 12 + .../General/pipeline-randomly-cancelled.md | 10 + ..."wait-for-manifest-to-stabilize\"-task.md" | 15 + ...anonymous-without-sufficient-permission.md | 10 + ...ugh-it-has-been-explicitly-set-to-false.md | 20 ++ ...es-being-triggered-with-stale-artifacts.md | 10 + ...rd-fail-with-'not-authorized'-exception.md | 12 + .../pod-details-do-not-show-for-some-pods.md | 14 + ...unhealthy-state-due-to-zombie-processes.md | 10 + ...or-scripts-remain-after-completed-state.md | 11 + ...missing-spinnaker.persistence.pipelines.md | 11 + ...-engine---opa-for-styra-das-deployments.md | 14 + ...policy-engine-vx.x.x'-to-plugins-folder.md | 34 +++ ...t-a-function-when-accessing-project-tab.md | 22 ++ .../policy-engine-regoscript-examples.md | 14 + ...8s-auth-method-and-updating-to-k8s-1.21.md | 52 ++++ ...-customer-service-portal-administrators.md | 34 +++ ...roducing-artifacts-in-the-webhook-stage.md | 28 ++ ...ry-support-for-troubleshooting-purposes.md | 16 ++ ...alidation-error---file-not-found-module.md | 10 + kb/armory/General/quiet-the-echo-service.md | 75 +++++ ...to-sql-migration--can't-see-old-history.md | 12 + kb/armory/General/reduce-kubeconfig-size.md | 10 + ...w-copy-and-does-not-delete-the-original.md | 11 + .../renaming-an-application-in-spinnaker.md | 11 + ...lidate-dinghy-pipelines-with-armory-cli.md | 15 + ...sometimes-will-work,-sometimes-will-not.md | 20 ++ ...uster-wide-dockerhub-rate-limit-warning.md | 35 +++ ...rror--got-\"map\",-expected-\"string\".md" | 22 ++ ...n-the-ui-when-migrating-to-armory-agent.md | 13 + ...gers-many-jenkins-jobs-when-using-redis.md | 12 + .../General/restrict-application-creation.md | 81 ++++++ ...-into-specific-namespace-in-eks-cluster.md | 10 + ...unt\"-metric-unavailable-in-prometheus.md" | 15 + ...h-causing-a-failed-pipeline-erratically.md | 11 + ...n-a-generic-shell-script-with-spinnaker.md | 92 ++++++ .../General/run-job---producing-artifacts.md | 71 +++++ ...-does-not-show-results-of-multi-failure.md | 29 ++ ...pipeline-artifacts-from-previous-stages.md | 14 + ...xecute-due-to-write-permissions-in-fiat.md | 11 + ...s3-buckets-for-front50-revision-history.md | 12 + ...-to-servicenow-service-portal-migration.md | 22 ++ ...ions-folder-with-execution-id-times-out.md | 16 ++ ...dinghy-deployments-occurring-via-github.md | 24 ++ .../serverless-support-with-spinnaker.md | 12 + ...rvicenow---add-new-version-of-spinnaker.md | 23 ++ ...change-requests,-navigation-and-anatomy.md | 11 + ...t-support-cases,-navigation-and-anatomy.md | 26 ++ ...e-tickets-across-organization-divisions.md | 14 + .../servicenow---attaching-kb-articles.md | 13 + ...w---create-edit-knowledge-base-articles.md | 82 ++++++ ...-change-requests-on-behalf-of-customers.md | 30 ++ .../General/servicenow---creating-users.md | 21 ++ ...now---customer-account-company-creation.md | 20 ++ ...calation-response-for-support-engineers.md | 24 ++ ...a-resource-case,-change-request,-etc....md | 13 + ...icenow---navigation-of-the-backend-site.md | 32 +++ ...icenow---notifications-on-case-comments.md | 21 ++ ...rs---opening-and-managing-support-cases.md | 24 ++ .../servicenow---using-lists-and-views.md | 35 +++ ...-service-portal-customer-administrators.md | 19 ++ .../General/set-execution-history-lookback.md | 13 + ...atus-check-on-pr-from-a-pipeline-status.md | 12 + ...ting-an-sql-cleanup-job-for-clouddriver.md | 25 ++ .../setting-up-logging-without-tokens.md | 26 ++ .../shared-configuration-repository.md | 16 ++ ...innaker-ui-failed-to-evaluate-not-found.md | 14 + ...if-a-template-json-is-used-in-aws-bakes.md | 20 ++ .../specify-a-subnet-during-bake-stage.md | 8 + ...interpreting--with-terraform-templates-.md | 21 ++ ...e--must-be-of-type-string--\"integer\".md" | 10 + ...fiat-enabled-exception-monitor-pipeline.md | 18 ++ kb/armory/General/spinnaker-and-helm.md | 7 + .../General/spinnaker-cloud-provider-limit.md | 10 + ...rtifacts-and-resources-when-redeploying.md | 10 + ...pengine-flex-due-to-name-being-too-long.md | 37 +++ ...when-spel-expressions-are-commented-out.md | 11 + ...-when-used-memory->-'maxmemory\"-redis.md" | 11 + .../General/spinnaker-kubernetes-v2-faqs.md | 47 ++++ ...to-scaling-group-after-a-new-deployment.md | 25 ++ ...e-account-names-across-both-aws-and-ecs.md | 28 ++ ...es-not-displaying-in-infrastructure-tab.md | 10 + ...ration,-but-not-v4.x-artifact-not-found.md | 13 + ...ale-agent--increasing-grpc-message-size.md | 38 +++ ...o-spinnaker-2.19.x-from-2.18.x-or-lower.md | 13 + ...lost-splunk-crashed-with-splunk-service.md | 10 + ...spring-expression-language-spel-samples.md | 153 ++++++++++ .../spring-expression-language-tricks.md | 38 +++ kb/armory/General/ssh-keys-in-terraformer.md | 56 ++++ ...-error-appears-on-the-functions-display.md | 23 ++ ...ode-awsvpc-even-though-it-is-configured.md | 11 + ...mentation,-need-to-extend-stage-timeout.md | 10 + ...in-vault-for-use-in-spinnaker-pipeline..md | 7 + ...---orca-with-mysql-editing-the-database.md | 10 + .../support-portal-user-registration.md | 19 ++ .../swaggerui-commands-returns-4xx-errors.md | 10 + ...ommendations-for-kubernetes-deployments.md | 194 +++++++++++++ ...artifact-does-not-replace-task-iam-role.md | 23 ++ .../technical-support-case-statuses.md | 10 + ...ng-internal-cases-to-track-project-work.md | 40 +++ .../terraform-'invalid-character'-error.md | 20 ++ ...doesn't-output-the-full-list-of-changes.md | 12 + ...aform-deployment-using-armory-spinnaker.md | 15 + .../terraform-maximum-file-size-error.md | 10 + ...-environment-variables-to-store-secrets.md | 10 + ...-does-not-clone-properly-from-bitbucket.md | 10 + ...\"--executable-file-not-found-in-$path.md" | 11 + .../terraformer-named-profiles-with-gcp.md | 42 +++ ...es-input-from-a-single-file-from-github.md | 10 + ...-with-a-'timeout'-error-logging-enabled.md | 27 ++ ...sn't-render-in-pipeline-json-by-default.md | 11 + ...bject-filtering-with-agent\342\200\250.md" | 10 + .../General/the-role-of-helm-and-spinnaker.md | 33 +++ ...the-timeout-in-bake-stage-configuration.md | 14 + .../troubleshoot-baking-helm-3-charts.md | 11 + .../troubleshooting-gate-deck-cors-issues.md | 27 ++ ...ults-in-a-\"missing-client-token-error.md" | 15 + ...ecs-cluster-to-spinnaker-not-authorized.md | 174 ++++++++++++ ...tes-cloud-provider-with-a-secret-engine.md | 16 ++ ...-application-with-a-postgresql-database.md | 13 + ...-after-moving-aws-account-into-profiles.md | 12 + ...-changing-operator-and-halyard-versions.md | 34 +++ ...o-a-new-kubernetes-cluster-to-spinnaker.md | 16 ++ ...spel-variables-in-an-email-notification.md | 11 + ...ace-when-running-clouddriver-with-agent.md | 30 ++ ...-an-existing-pipeline-from-spinnaker-ui.md | 14 + ...ing-the-app-engine-plugin-in-a-pipeline.md | 28 ++ ...-warning-level-messages-in-front50-logs.md | 14 + .../General/uninstalling-armory-spinnaker.md | 44 +++ .../unlock-the-databasechangeloglock-table.md | 36 +++ ...ached-gitrepo-fails-on-artifact-fetches.md | 21 ++ ...dinghy-from-updating-existing-pipelines.md | 11 + .../upgrade-from-oss-to-armory-spinnaker.md | 14 + .../upgrade-spinnaker-using-armory-halyard.md | 7 + ...pelines-deploy-to-unavailable-namespace.md | 17 ++ ...se--request-method-'post'-not-supported.md | 19 ++ .../General/upgrade-your-kustomize-version.md | 45 +++ kb/armory/General/upgrading-an-eks-cluster.md | 103 +++++++ ...t-eks-clusters-heptio-authenticator-aws.md | 15 + .../use-case-for-aws-sticky-sessions.md | 11 + ...yard-public-and-private-docker-registry.md | 29 ++ ...-stages-with-bitbucket-hosted-artifacts.md | 11 + ...h-the-console---unlock-via-ui-unchecked.md | 12 + ...-headers-on-api-calls-to-orca,-front50-.md | 22 ++ ...manifest`-overwrites-rendered-artifacts.md | 11 + ...call-a-method-that-does-not-exist-error.md | 21 ++ ...ternal-artifacts-as-pipeline-parameters.md | 17 ++ .../General/using-openshift-with-spinnaker.md | 11 + ...o-enable-aws-auto-scaling-group-metrics.md | 32 +++ ...ttings,-suggestions,-and-considerations.md | 43 +++ ...-the-kubernetes-v2-patch-manifest-stage.md | 8 + .../General/using-travis-ci-with-spinnaker.md | 29 ++ ...-of-remote-base-in-a-different-git-repo.md | 46 +++ ...ate.halvalidator-detected-a-fatal-error.md | 15 + ...as-json-and-developing-dinghy-json-code.md | 15 + ...ulating-in-the-drop-down-menu-in-the-ui.md | 31 ++ .../vulnerability-management-policy.md | 47 ++++ ...hrinkcluster-times-out-after-30-minutes.md | 16 ++ ...rvices-can-be-affected-apply-opa-policy.md | 18 ++ ...former-with-the-provisioner,-local-exec.md | 16 ++ ...ets-and-vpc-will-not-populate-in-the-ui.md | 11 + ...s-and-failing-to-deploy-to-k8s-clusters.md | 18 ++ ...re-does-spinnaker-begin-and-jenkins-end.md | 10 + ...t-it-cannot-communicate-with-opa-server.md | 12 + ...ory-arm-cli,-cannot-refer-to-pipelineid.md | 15 + ...-add-a-custom-ca-for-operator-and-vault.md | 91 ++++++ ...ustom-smtp-server-to-send-notifications.md | 38 +++ ...add-additional-users-to-an-organization.md | 23 ++ ...r-empty-roles-in-fiat-using-redis-mysql.md | 91 ++++++ ...hange-spinnaker's-login-session-timeout.md | 21 ++ ...p-old-executions-in-orca-mysql-database.md | 0 ...kes-older-than-specific-days-from-redis.md | 0 ...how-to-debug-missing-agent-k8s-accounts.md | 47 ++++ ...sion-for-a-particular-spinnaker-service.md | 40 +++ .../how-to-deploy-encrypted-ebs-volumes.md | 103 +++++++ ...stored-in-configmaps-instead-of-secrets.md | 67 +++++ ...nt-of-clouddriver-caching-agents-to-run.md | 129 +++++++++ ...-authz-for-spinnaker-migration-testing-.md | 0 ...elm-repo-trigger-from-jfrog-artifactory.md | 52 ++++ ...neer-solutions-architect-for-an-account.md | 30 ++ ...be-presented-in-certificate_request.\".md" | 11 + ...ns-of-a-given-application-via-api-calls.md | 20 ++ ...res-and-execution-issues-with-end-users.md | 6 + ...as-code-dictionary-list-sprig-functions.md | 67 +++++ ...cific-logs-by-finding-the-absolute-path.md | 30 ++ ...d-startup-probes-http,-tcp-socket,-exec.md | 65 +++++ ...ault-signer-encryption-algorithm-for-s3.md | 27 ++ ...-code-for-your-spinnaker-pipeline-logic.md | 17 ++ ...o-a-child-pipeline-using-matchartifacts.md | 17 ++ ...how-to-provide-feature-request-feedback.md | 22 ++ ...l-variables-in-pipelines-as-code-dinghy.md | 139 +++++++++ ...n-pipelines-as-code-dinghy-using-uuidv4.md | 120 ++++++++ ...rizations-for-a-user-through-redis-fiat.md | 37 +++ .../How To's/how-to-scale-igor-replicas.md | 37 +++ ...to-set-application-features-with-dinghy.md | 4 +- ...set-up-least-privilege-access-with-fiat.md | 21 +- ...pecify-an-application-owner-with-dinghy.md | 18 ++ ...-switch-packer's-powershell-provisioner.md | 17 ++ ...spinnaker-using-launch-templates-in-aws.md | 25 ++ kb/armory/How To's/how-to-upgrade-operator.md | 15 + ...-to-aws-s3-from-armory-cdsh---spinnaker.md | 52 ++++ ...es,-migrating-from-launch-configuration.md | 28 ++ ...-to-use-dinghy-module-in-module-support.md | 13 + ...global-variables-in-dinghy-conditionals.md | 86 ++++++ ...es-with-launch-templates-for-spinnaker-.md | 59 ++++ ...-rds-clusters-in-spinnaker-ha-mode-only.md | 241 ++++++++++++++++ ...nd-split-functions-in-pipelines-as-code.md | 87 ++++++ ...s-in-other-appications-within-spinnaker.md | 46 +++ 464 files changed, 13316 insertions(+), 18 deletions(-) create mode 100644 kb/armory/General/403-and-permission-errors-when-enabling-new-services.md create mode 100644 kb/armory/General/403-errors-around-github-access-and-rate-limit-issues.md create mode 100644 kb/armory/General/access-denied-to-application-on-deploy-stage-without-application-write-permissions.md create mode 100644 kb/armory/General/accessing--armory-scale-agent-endpoints-to-help-in-troubleshooting.md create mode 100644 kb/armory/General/accessing-clouddriver-endpoints-to-help-in-troubleshooting-armory-scale-agent-issues.md create mode 100644 kb/armory/General/accessing-the-spinnaker-rest-api-using-httpie-and-jq.md create mode 100644 kb/armory/General/accounts-with-livemanifestcalls-set-to-true-have-incorrect-dynamic-lookup-results.md create mode 100644 kb/armory/General/adding-a-group-with-many-members-to-application-times-out-in-gcp.md create mode 100644 kb/armory/General/adding-additional-policies-and-access-to-an-eks-cluster.md create mode 100644 kb/armory/General/adjust-clean-up-timing-for-armory-agent.md create mode 100644 kb/armory/General/agent-deployment-error--exceptions.operationtimedoutexception--timeout-exceeded-for-operation-.md create mode 100644 kb/armory/General/applying-custom-sizes-to-services.md create mode 100644 kb/armory/General/armory-agent---deploy-stage-issues---404-not-found--unknown-kind-apiservice-in-apiservice-....md create mode 100644 kb/armory/General/armory-agent-testing,-investigation,-and-checks.md create mode 100644 kb/armory/General/armory-cloud---cannot-register-user-as-user-already-exists-status-code-409.md create mode 100644 "kb/armory/General/armory-cloud---remote-network-agent-cannot-connect-unauthenticated--provided-credentials-did-not-have-\"xxxxxx-xxxxx'-scope.md" create mode 100644 kb/armory/General/armory-continous-deployment-managed-high-availability-and-disaster-recovery-sla-information.md create mode 100644 kb/armory/General/armory-continuous-deployment-as-a-service-cdaas-support.md create mode 100644 kb/armory/General/armory-diagnostics.md create mode 100644 kb/armory/General/armory-halyard-overview.md create mode 100644 kb/armory/General/armory-product-demo-videos.md create mode 100644 kb/armory/General/armory-release-definitions.md create mode 100644 kb/armory/General/armory-support-information-hours-of-operation-slas-procedures.md create mode 100644 kb/armory/General/authenticate-against-jenkins.md create mode 100644 kb/armory/General/authentication-timeout-results-in-401-error.md create mode 100644 kb/armory/General/authorization-configuration-with-file-based-role-definitions.md create mode 100644 kb/armory/General/auto-cleanup-pipeline-executions-of-specific-applications-in-spinnaker-.md create mode 100644 kb/armory/General/autogenerated-applications-cannot-be-deleted-w--onlyspinnakermanaged-set-to-false.md create mode 100644 kb/armory/General/aws-billing-billing-budget-alerts-and-cost-explorer.md create mode 100644 kb/armory/General/aws-image-caching-issues-for-certain-accounts,-when-multiple-accounts-are-defined.md create mode 100644 kb/armory/General/aws-lambda-&-custom-webhook-stages.md create mode 100644 kb/armory/General/aws-lambda-plugin-stage-does-not-populate-the-existing-functions-.md create mode 100644 kb/armory/General/aws-rds-certificate-update.md create mode 100644 kb/armory/General/bake-cloudfoundry-manifest-stage-fails-without-an-error-message.md create mode 100644 kb/armory/General/bake-or-deployment-stage-timeouts-caused-by-external-factors.md create mode 100644 kb/armory/General/bitbucket-cloud-vs-bitbucket-on-premises.md create mode 100644 kb/armory/General/bitbucket-webhook-fails-with-unknown-bitbucket-event-type-in-dinghy.md create mode 100644 kb/armory/General/can-users'-level-of-access-in-the-armory-console-be-retricted-using-rbac.md create mode 100644 "kb/armory/General/canary-config-metric-template-displays-invalid-metricstat-json--should-not-have-additional-properties\342\200\213-example--aws-cloudwatch.md" create mode 100644 kb/armory/General/cancel-a-stuck-pipeline.md create mode 100644 kb/armory/General/cannot-add-custom-kind-to-delete-manifest-stage.md create mode 100644 kb/armory/General/cannot-define-ebs-volumes-when-deploying-baked-ami.md create mode 100644 kb/armory/General/cannot-enable-dinghy-to-proceed-with-github-pull-request-validations.md create mode 100644 kb/armory/General/cannot-overwrite-images-in-a-knowledge-articles-update-them-or-adding-an-image-with-same-name.md create mode 100644 kb/armory/General/cannot-see-pipeline-history-for-changes-made.md create mode 100644 kb/armory/General/capacity-provider-concepts.md create mode 100644 kb/armory/General/capturing-spinnaker-configurations-with-armory's-support-bundle-w--automated-script.md create mode 100644 kb/armory/General/cases-and-change-requests.md create mode 100644 kb/armory/General/changes-to-a-configmap-not-reflected-in-new-deployment.md create mode 100644 kb/armory/General/changing-timeouts-for-spinnaker-services-using-okhttp-java-services.md create mode 100644 kb/armory/General/clean-up-schedule-for-armory-scale-agent-operations-older-than-a-specific-time-period.md create mode 100644 kb/armory/General/cloud-provider-account-disappearing-from-spinnaker-ui.md create mode 100644 kb/armory/General/clouddriver-caching-errors-|-clusteredagentscheduler---unable-to-run-agents.md create mode 100644 kb/armory/General/clouddriver-fails-to-start-up-armory-agent-during-initial-install.md create mode 100644 kb/armory/General/clouddriver-is-not-caching-subnets-or-server-groups.md create mode 100644 kb/armory/General/clouddriver-logs-show-slow-sql-warnings.md create mode 100644 "kb/armory/General/clouddriver-shows-\"connection-is-not-available,-request-timed-out-after-10000ms.\".md" create mode 100644 kb/armory/General/clouddriver-unable-to-download-artifacts-from-private-github-instance-due-to-ssl-errors.md create mode 100644 kb/armory/General/clouddriver-will-not-connect-to-mysql-causing-a-stoppage,-databasechangelock-error.md create mode 100644 kb/armory/General/combined-version-and-service-module-name-is-too-long-when-deploying-with-app-engine.md create mode 100644 kb/armory/General/common-vulnerabilities-and-exposures-cve-reports---armory-continuous-delivery.md create mode 100644 kb/armory/General/common-vulnerabilities-and-exposures-cve-reports---armory-plugins.md create mode 100644 kb/armory/General/compliance-and-auditing.md create mode 100644 kb/armory/General/concurrent-pipelines-running-with-bakes-fail-packer-behavior-in-rosco.md create mode 100644 kb/armory/General/configure-elasticache-with-spinnaker.md create mode 100644 kb/armory/General/configure-persistent-storage-to-capture-heapdumps-for-spinnaker-services.md create mode 100644 kb/armory/General/configure-spinnaker-to-use-dynamic-github-authentication-tokens.md create mode 100644 kb/armory/General/configure-spinnakers-external-account-configuration-with-vault.md create mode 100644 "kb/armory/General/configuring-aws-account-using-eap---error-running-createservergrouptask-for-pipeline-credentials-not-found*\".md" create mode 100644 kb/armory/General/configuring-slack-notifications-with-sendcompactmessages-to-send-a-shorter-and-more-compact-message.md create mode 100644 kb/armory/General/configuring-spinnaker-services-to-connect-to-redis-over-mtls.md create mode 100644 kb/armory/General/configuring-spinnaker-to-use-internal-certificate-authorities.md create mode 100644 kb/armory/General/configuring-vim-editor-to-work-with-yaml.md create mode 100644 kb/armory/General/confirming-environment-logs-and-environment-debugging-id-with-armory-support.md create mode 100644 kb/armory/General/container-image-cannot-be-selected.md create mode 100644 kb/armory/General/continuous-warning-messages--http2exception$streamexception--received-data-frame.md create mode 100644 kb/armory/General/contributing-to-oss-spinnaker-plugins-and-donations-of-code.md create mode 100644 kb/armory/General/create-aws-iam-roles-from-a-terraform-script.md create mode 100644 kb/armory/General/create-kubernetes-configmaps-and-secrets-from-artifact-files.md create mode 100644 kb/armory/General/creating-a-change-request-in-servicenow.md create mode 100644 kb/armory/General/creating-cases-and-case-management-in-servicenow.md create mode 100644 kb/armory/General/creating-custom-deployment-strategies.md create mode 100644 kb/armory/General/critical-technical-notifications.md create mode 100644 kb/armory/General/custom-runjobs-intermittently-failing-cannot-access-properties-of-application.md create mode 100644 "kb/armory/General/custom-runjobs-intermittently-failing-with-\"access-denied-to-application\"-error.md" create mode 100644 kb/armory/General/customer-user-does-not-have-access-to-create-change-request.md create mode 100644 kb/armory/General/customers-unable-to-access-resources-due-to-github-sunseting-deprecated-teams-api-endpoints.md create mode 100644 kb/armory/General/cve-2021-43832---favexploit--spinnaker-rce-vulnerability-in-gate.md create mode 100644 kb/armory/General/cve-2021-44228---log4shell---log4j-vulnerability-analysis.md create mode 100644 kb/armory/General/cve-2022-22965---spring-framework-rce-w--jdk-9+.md create mode 100644 kb/armory/General/cve-2022-42889---apache-commons-text-vulnerability.md create mode 100644 kb/armory/General/debug.armory.io-is-down,-crashes-environments-w--debugging-enabled-into-a-crash-loop-restart-.md create mode 100644 kb/armory/General/debugging-deployment-errors.md create mode 100644 kb/armory/General/debugging-slow-orca-performance.md create mode 100644 kb/armory/General/define-application-attributes-using-dinghy.md create mode 100644 kb/armory/General/delayed-image-push-pipeline-trigger.md create mode 100644 kb/armory/General/delete-pipeline-executions-in-redis.md create mode 100644 kb/armory/General/deploy-manifest-performs-spel-substitution-even-if-skip-spel-evaluation-is-checked.md create mode 100644 kb/armory/General/deploy-manifest-stage-does-not-remove-previous-helm2-deployment-resources-from-kubernetes.md create mode 100644 kb/armory/General/deploy-stage-missing-from-pipeline.md create mode 100644 kb/armory/General/deploying-helm-charts-with-armory-spinnaker-and-managing-secrets-in-helm.md create mode 100644 "kb/armory/General/deploying-through-armory-agent-for-kubernetes-with-crd-results-in-\"error-occurred-when-list-customresourcedefinition-null\".md" create mode 100644 kb/armory/General/deployment-error-exception--resolve-deploy-source-manifest-.md create mode 100644 "kb/armory/General/deployment-fails-with-\"load-balancer-service--does-not-exist\"-error.md" create mode 100644 kb/armory/General/deployment-strategy-red-black-error--load-balancer-service--does-not-exist.md create mode 100644 "kb/armory/General/deployments-cluster-tab-missing-\"undo-rollback\"-feature---clusters-tab-missing-previously-available-options.md" create mode 100644 kb/armory/General/deployments-exceeding-progress-deadline.md create mode 100644 kb/armory/General/determine-or-check-the-environment's-spinnaker-version.md create mode 100644 kb/armory/General/difference-in-the-deployment-phase-takes-and-the-service-boot-time.md create mode 100644 kb/armory/General/differences-between-github-and-gitrepo-artifacts-and-choosing-the-right-artifact-type.md create mode 100644 kb/armory/General/dinghy-'roles'-entry-is-not-getting-passed-to-actual-spinnaker-pipeline-json.md create mode 100644 kb/armory/General/dinghy-cannot-self-reference-pipelines-created-in-the-same-dinghyfile-using-pipelineid.md create mode 100644 kb/armory/General/dinghy-crashing-for-customers-using-2.24+-when-customers-are-using-a-custom-endpoint.md create mode 100644 kb/armory/General/dinghy-crashloop-when-using-redis-loss-of-connectivity-loss-of-relationship.md create mode 100644 kb/armory/General/dinghy-fails-to-create-pipelines-in-armory-2.19.8-and-prior.md create mode 100644 kb/armory/General/dinghy-file-does-not-update-pipelines-in-spinnaker.md create mode 100644 kb/armory/General/dinghy-github-status-is-always-failed-when-using-yaml-parser.md create mode 100644 kb/armory/General/dinghy-in-2.26-tries-to-validate-readme.md-as-a-module.md create mode 100644 kb/armory/General/dinghy-multi-repo-strategy-with-repoconfig.md create mode 100644 kb/armory/General/dinghy-rendering-failures---forbidden--access-denied-to-application-access-rights-verified.md create mode 100644 kb/armory/General/dinghy-skips-updating-pipeline-when-local-module-is-changed.md create mode 100644 kb/armory/General/dinghy-slack-notifications-not-working---java.lang.nosuchmethoderror--void-org.jsoup.md create mode 100644 kb/armory/General/dinghy-webhook-shows-an-html-response,-rather-than-a-json-response.md create mode 100644 kb/armory/General/dinghyfile-updates-do-not-trigger-slack-notifications.md create mode 100644 kb/armory/General/dinghyfiles-with-defined-application-permissions-fails-to-create-update-pipelines-forbidden-error.md create mode 100644 kb/armory/General/disable-default-repositories-for-plugins-on-air-gapped-environments-and-specify-the-repo.md create mode 100644 "kb/armory/General/disable-deployment-registration-banner-message-\"you-configured-a-feature-that-requires-registration\".md" create mode 100644 kb/armory/General/disable-diagnostic-settings-results-in-echo-pod-unable-to-start-armory-2.21.2.md create mode 100644 kb/armory/General/disabling-tls-1.1-in-spinnaker-and-specifying-the-protocols-to-be-used.md create mode 100644 kb/armory/General/docker-binding-to-matching-repositories-breaking-pipelines.md create mode 100644 kb/armory/General/during-migration-to-iam-for-service-accounts,-clouddriver-in-ha-returns-error--access-denied-service--amazon-s3;-status-code-403.md create mode 100644 kb/armory/General/dynamics-accounts-with-github-cannot-use-token.md create mode 100644 kb/armory/General/echo-sends-duplicate-triggers.md create mode 100644 kb/armory/General/ecs--supporting-ephemeralstorage-in-fargate.md create mode 100644 "kb/armory/General/ecs-deployment-errors-with-\"all-server-group-names-for-cluster....are-taken\".md" create mode 100644 kb/armory/General/ecs-deployment-fails-when-using-external-launch-type-task.md create mode 100644 kb/armory/General/ecs-deployment-failures.md create mode 100644 "kb/armory/General/editing-application-permissions-yields-an-error-\"cannot-read-property-'label'-of-undefined\".md" create mode 100644 kb/armory/General/emoji-in-redis-database-error-when-migrating-from-redis-to-mysql.md create mode 100644 kb/armory/General/enable-armory-agent-for-kubernetes-to-only-discover-objects-that-are-discovered-by-spinnaker-or-armory-agent.md create mode 100644 kb/armory/General/enable-aws-cloudformation.md create mode 100644 kb/armory/General/enable-basic-form-authentication-for-spinnaker-via-halyard.md create mode 100644 kb/armory/General/enable-detailed-logging---debugging-mode-in-spinnaker-environment-service.md create mode 100644 kb/armory/General/enable-two-pipelines-to-run-as-mutually-exclusive.md create mode 100644 kb/armory/General/enabling-caching-on-git-repos-and-clearing-clouddriver-caching-to-resolve-caching-issues.md create mode 100644 kb/armory/General/enabling-the-managed-pipeline-templates-ui.md create mode 100644 kb/armory/General/error---credential-not-found-aws.md create mode 100644 kb/armory/General/error--you-cannot-delete-this-application-because-it-has-server-groups.md create mode 100644 kb/armory/General/error-calling-module-error-rendering-imported-module-pipeline-module.json-invalid-character.md create mode 100644 kb/armory/General/error-displayed-on-the-ui--exception-resolve-target-manifest.md create mode 100644 kb/armory/General/error-when-deploying-manifest-kind-csidriver-with-armory-agent.md create mode 100644 "kb/armory/General/errors-don't-propagate-from-child-to-parent-pipelines-if-they-occur-in-stages-configured-to-\"ignore-the-failure\".md" create mode 100644 kb/armory/General/errors-in-deploying-dinghy-while-using-aws-elastic-cache-with-encryption-in-transit.md create mode 100644 kb/armory/General/errors-initializing-and-starting-up-policy-engine-configuration-hints,-orca-clouddriver-front50.md create mode 100644 kb/armory/General/errors-when-moving-account-configuration-to-external-s3-bucket.md create mode 100644 kb/armory/General/escalating-a-case.md create mode 100644 kb/armory/General/exception-starting-plugin-in-orca.md create mode 100644 kb/armory/General/execution-parameters-shared-when-run-job-stages-are-running-in-parallel.md create mode 100644 kb/armory/General/executionid-&-stageid-are-not-unique-in-the-webhooks-sent-by-spinnaker-customwebhook-stages.md create mode 100644 kb/armory/General/executions-after-migrating-to-terraformer-are-delayed-or-take-a-lot-more-time-to-execute.md create mode 100644 kb/armory/General/exposing-spinnaker.md create mode 100644 kb/armory/General/facing-400-error-while-saving-canary-configuration.md create mode 100644 kb/armory/General/facing-throttling-issues-when-deploying-cloud-services.md create mode 100644 "kb/armory/General/failed-to-read-manifest--\"exception-wait-for-manifest-to-stabilize.-unexpected-task-failure\".md" create mode 100644 kb/armory/General/fiat-ignores-x509-cert-and-check-role-provider-for-permissions.md create mode 100644 kb/armory/General/fiat-unable-to-start-in-a-cloudfoundry-environment.md create mode 100644 kb/armory/General/flushall-on-spinnaker's-redis-cache.md create mode 100644 kb/armory/General/flushing-gate-sessions.md create mode 100644 kb/armory/General/front50-pods-failing-to-start-after-upgrading-spinnaker-version-from-2.20.1-to-2.23.5.md create mode 100644 kb/armory/General/gate-metrics-not-available-to-the-monitoring-daemon.md create mode 100644 kb/armory/General/gcp--exception--wait-for-server-group-disabled----error-while-querying-target-pool-instance-health.md create mode 100644 kb/armory/General/gcp-error--code-=-permissiondenied-desc-=-forbidden--http-status-code-403;-transport-.md create mode 100644 kb/armory/General/getting-the-ip-of-an-ec2-instance-with-pipeline-expressions.md create mode 100644 "kb/armory/General/github-changed-default-base-branch-from-\"master\"-to-\"main\".md" create mode 100644 kb/armory/General/github-dinghyfiles-referring-to-mixed-case-applications-fail.md create mode 100644 kb/armory/General/github-issues-with-certificates-triggered-failure-to-be-able-to-execute-pipelines,-due-to-plugins-missing-unavailable.md create mode 100644 kb/armory/General/github-notifications-do-not-update-github-commit-status.md create mode 100644 kb/armory/General/github-trigger-matches-multiple-artifacts-on-trigger-.tf-files.md create mode 100644 kb/armory/General/google-app-engine-gae-deploy-stage-not-showing-artifacts-that-have-been-added-and-can't-edit.md create mode 100644 kb/armory/General/google-cloud-provider---private-gke.md create mode 100644 kb/armory/General/groovy.lang.missingmethodexception-when-referencing-github-config-artifact-for-google-app-engine.md create mode 100644 kb/armory/General/hashicorp-terraform-gpg-rotation-codecov-vulnerability.md create mode 100644 kb/armory/General/high-number-of-github-emails-after-every-dinghyfile-update.md create mode 100644 kb/armory/General/hitting-annotation-hard-limit-when-deploying-a-configmap-secret-too-long.md create mode 100644 kb/armory/General/hitting-igor's-caching-thresholds.md create mode 100644 kb/armory/General/http-https-redirects-with-authentication.md create mode 100644 kb/armory/General/igor-pod-not-running-or-starting-up-when-configuring-pipeline-triggers-500-internal-server-error.md create mode 100644 kb/armory/General/impersonate-a-user-for-servicenow-troubleshooting-servicenow-admins.md create mode 100644 kb/armory/General/incorrect-oauth2.0-redirects.md create mode 100644 kb/armory/General/increase-the-agent-operation-wait-timeouts-on-agent-clouddriver-plugin.md create mode 100644 kb/armory/General/increase-timeouts-on-agent-plugin-and-agent-to-avoid-network-latency-issues.md create mode 100644 kb/armory/General/infinite-authentication-loop---user-initially-authenticate-without-issue,-but-loops-upon-timeout.md create mode 100644 "kb/armory/General/infinite-ui-loop-\"save-changes\"-stuck-attempting-to-save-a-json-canary-config-in-the-spinnaker-ui.md" create mode 100644 kb/armory/General/infrastructure-tab-not-showing-correct-deployment-details.md create mode 100644 kb/armory/General/ingesting-aws-cloudformation-templates-larger-than-51200-bytes-as-an-artifact.md create mode 100644 kb/armory/General/ingesting-metrics-from-observability-plugin-to-datadog.md create mode 100644 kb/armory/General/initiated-login-does-not-work-with-saml-2.0.md create mode 100644 kb/armory/General/instance-is-not-healthy-but-the-deployment-shows-as-succeeded-.md create mode 100644 kb/armory/General/integration-with-nexus.md create mode 100644 kb/armory/General/internal-error,-failed-calling-webhook--when-deploying-armory-continuous-deployment-after-upgrading-the-armory-operator.md create mode 100644 kb/armory/General/invalid-token-with-named-profiles-assume-role.md create mode 100644 kb/armory/General/is-it-possible-to-restrict-user-access-to-certain-envrionments-with-rbac.md create mode 100644 kb/armory/General/jason-mcintosh's-handy-information-about-setting-up-monitoring.md create mode 100644 kb/armory/General/jenkins-ci-driver-can-start-but-spinnaker-fails-to-monitor-the-jenkins-job-tomcat-server.md create mode 100644 kb/armory/General/jenkins-timeout-issues.md create mode 100644 kb/armory/General/jobs-run-by-a-service-account-in-a-different-namespace-generates-generic-errors-index--1-out-of-bounds-for-length-0.md create mode 100644 kb/armory/General/jsonpath-update-allows-constraints-on-additional-webhooks-payload-values-directory-specific.md create mode 100644 kb/armory/General/k8s-v1.21-causing-valut-intergation-outages-with-kuberenetes-auth-method.md create mode 100644 kb/armory/General/kayenta-404s-in-armory-spinnaker.md create mode 100644 kb/armory/General/kubernetes-and-docker-accounts-are-not-visible-in-spinnaker-console-ui.md create mode 100644 kb/armory/General/kubernetes-namespaces-still-appearing-after-deletion-in-spinnaker-ui.md create mode 100644 kb/armory/General/kubernetes-service-name-change-creates-a-new-service-old-service-not-deleted.md create mode 100644 kb/armory/General/kubernetes-v1-provider-removed.md create mode 100644 kb/armory/General/lambda-creation-for-aws-exception--no-message-available.md create mode 100644 kb/armory/General/large-fiat-headers-can-cause-permission-errors-due-to-timeouts-and-400-errors..md create mode 100644 kb/armory/General/load-balancer-service-svc-spinnaker-demo-does-not-exist.md create mode 100644 "kb/armory/General/locked-pipelines-cannot-be-unlocked-except-by-admin-access-users-when-using-\"run-as\".md" create mode 100644 kb/armory/General/long-force-cache-refresh-kubernetes.md create mode 100644 "kb/armory/General/manual-installation-of-project-aurora-for-spinnaker\342\204\242-no-helm.md" create mode 100644 kb/armory/General/metric-data-does-not-flow-into-splunk-despite-setting-up-hec-connection.md create mode 100644 kb/armory/General/migrate-clouddriver's-redis-to-its-own-instance.md create mode 100644 kb/armory/General/migrating-to-the-policy-engine-plugin-from-the-policy-engine-extension.md create mode 100644 kb/armory/General/missing-file-in-rosco-image-got-exception-create-bake.md create mode 100644 kb/armory/General/missing-run-as-user-field-for-triggers-in-spinnaker-ui.md create mode 100644 kb/armory/General/monitored-deploy-rollout.md create mode 100644 kb/armory/General/mysql-communication-failure-when-using-tls1.1.md create mode 100644 kb/armory/General/mysql-table-name-change-error-when-rolling-back-spinnaker-undo-renamed-values.md create mode 100644 kb/armory/General/network-latency-between-clouddriver-and-agent-causing-timeout-issues.md create mode 100644 kb/armory/General/nexus--could-not-parse-error-response-unexpected-character-'<'-code-60--expected-a-valid-value-.md create mode 100644 kb/armory/General/no-expected-artifacts-field.md create mode 100644 "kb/armory/General/no-ingress-is-supported-when-using-api-version-\"networking.k8s.io-v1\".md" create mode 100644 kb/armory/General/no-plugins-error-on-service-startup.md create mode 100644 kb/armory/General/nodeselector-toleration-settings-do-not-get-propagated-while-using-operator.md create mode 100644 kb/armory/General/nouniquebeandefinitionexception-error-with-armory-cloud-enabled.md create mode 100644 kb/armory/General/oauth-setup-for-github.md create mode 100644 kb/armory/General/objects-disappearing-under-the-cluster-tab-and-a-slow--deck-ui-cluster-page.md create mode 100644 kb/armory/General/obscuring-last-names-in-public-articles-and-cases---internal-users.md create mode 100644 kb/armory/General/opa-policies-not-working-after-spinnaker-upgrade-.md create mode 100644 kb/armory/General/opa-policy-deployment-error--failed-to-shutdown-server-gracefully.md create mode 100644 kb/armory/General/openshift--how-to-supply-the-complete-certificate-chain.md create mode 100644 kb/armory/General/operator-cannot-connect-to-halyard-due-to-a-tcp-timeout.md create mode 100644 kb/armory/General/operator-error-when-performing-a-spinnaker-upgrade-in-an-air-gapped-environment.md create mode 100644 kb/armory/General/operator-is-throwing-error-message-about-modified-object.md create mode 100644 kb/armory/General/orca-mishandling-spel-expressions.md create mode 100644 kb/armory/General/orca-stuck-in-crashloopbackoff-db--mysql.md create mode 100644 kb/armory/General/orca-with-sql-high-memory-usage---tables-information-dive.md create mode 100644 kb/armory/General/packet-for-query-is-too-large---mysql-max_allowed_packet.md create mode 100644 kb/armory/General/pagerduty---setting-up-for-new-users.md create mode 100644 kb/armory/General/pagerduty-notifications-integration-with-spinnaker.md create mode 100644 kb/armory/General/parameters-with-kubernetes-manifest-&-using-spel.md create mode 100644 kb/armory/General/password-resets.md create mode 100644 kb/armory/General/pausing-in-flight-kubernetes-deployments-in-spinnaker.md create mode 100644 "kb/armory/General/pipeline-execution-error-\"service-selector-must-have-no-label-keys-in-common-with-target-workload\".md" create mode 100644 kb/armory/General/pipeline-randomly-cancelled.md create mode 100644 "kb/armory/General/pipeline-stuck-on-\"wait-for-manifest-to-stabilize\"-task.md" create mode 100644 kb/armory/General/pipelines-are-triggered-by-anonymous-without-sufficient-permission.md create mode 100644 kb/armory/General/pipelines-as-code-dinghy-starts-with-multiplebranchesenabled--true-even-though-it-has-been-explicitly-set-to-false.md create mode 100644 kb/armory/General/pipelines-being-triggered-with-stale-artifacts.md create mode 100644 kb/armory/General/pipelines-with-gae-githttpsusername-githttpspassword-fail-with-'not-authorized'-exception.md create mode 100644 kb/armory/General/pod-details-do-not-show-for-some-pods.md create mode 100644 kb/armory/General/pods-going-to-unhealthy-state-due-to-zombie-processes.md create mode 100644 kb/armory/General/pods-running-jobs-or-scripts-remain-after-completed-state.md create mode 100644 kb/armory/General/policy-engine---no-policy-decision-detected,-missing-spinnaker.persistence.pipelines.md create mode 100644 kb/armory/General/policy-engine---opa-for-styra-das-deployments.md create mode 100644 kb/armory/General/policy-engine--pluginruntimeexception--failed-to-write-file-'plugins-armory.policyengine-policy-engine-vx.x.x'-to-plugins-folder.md create mode 100644 kb/armory/General/policy-engine-error-typeerror--e.takeuntil-is-not-a-function-when-accessing-project-tab.md create mode 100644 kb/armory/General/policy-engine-regoscript-examples.md create mode 100644 kb/armory/General/potential-outages-when-vault-integration-is-on-w--k8s-auth-method-and-updating-to-k8s-1.21.md create mode 100644 kb/armory/General/primary-and-secondary-contacts-and-customer-service-portal-administrators.md create mode 100644 kb/armory/General/producing-artifacts-in-the-webhook-stage.md create mode 100644 kb/armory/General/provide-logs-from-pods-to-armory-support-for-troubleshooting-purposes.md create mode 100644 kb/armory/General/pull-request-validation-error---file-not-found-module.md create mode 100644 kb/armory/General/quiet-the-echo-service.md create mode 100644 kb/armory/General/redis-to-sql-migration--can't-see-old-history.md create mode 100644 kb/armory/General/reduce-kubeconfig-size.md create mode 100644 kb/armory/General/renaming-a-configmap-creates-a-new-copy-and-does-not-delete-the-original.md create mode 100644 kb/armory/General/renaming-an-application-in-spinnaker.md create mode 100644 kb/armory/General/render-and-validate-dinghy-pipelines-with-armory-cli.md create mode 100644 kb/armory/General/replicaset-versioning-is-erratic,-sometimes-will-work,-sometimes-will-not.md create mode 100644 kb/armory/General/resolve-cluster-wide-dockerhub-rate-limit-warning.md create mode 100644 "kb/armory/General/resource-requests-on-custom-stages-does-not-work--error--got-\"map\",-expected-\"string\".md" create mode 100644 kb/armory/General/resources-are-duplicated-in-the-ui-when-migrating-to-armory-agent.md create mode 100644 kb/armory/General/restarting-spinnaker-unexpectedly-triggers-many-jenkins-jobs-when-using-redis.md create mode 100644 kb/armory/General/restrict-application-creation.md create mode 100644 kb/armory/General/restricting-the-user-into-specific-namespace-in-eks-cluster.md create mode 100644 "kb/armory/General/rosco's-\"bakescompleted_seconds_count\"-metric-unavailable-in-prometheus.md" create mode 100644 kb/armory/General/rosco-will-crash-causing-a-failed-pipeline-erratically.md create mode 100644 kb/armory/General/run-a-generic-shell-script-with-spinnaker.md create mode 100644 kb/armory/General/run-job---producing-artifacts.md create mode 100644 kb/armory/General/run-job-manifest-configuration-has-a-non-zero-backoff-does-not-show-results-of-multi-failure.md create mode 100644 kb/armory/General/run-pipeline-stage-picks-up-and-uses-pipeline-artifacts-from-previous-stages.md create mode 100644 kb/armory/General/runjob-does-not-execute-due-to-write-permissions-in-fiat.md create mode 100644 kb/armory/General/s3-buckets-for-front50-revision-history.md create mode 100644 kb/armory/General/salesforce-to-servicenow-service-portal-migration.md create mode 100644 kb/armory/General/searching-applications-folder-with-execution-id-times-out.md create mode 100644 kb/armory/General/securing-dinghy-deployments-occurring-via-github.md create mode 100644 kb/armory/General/serverless-support-with-spinnaker.md create mode 100644 kb/armory/General/servicenow---add-new-version-of-spinnaker.md create mode 100644 kb/armory/General/servicenow---all-about-change-requests,-navigation-and-anatomy.md create mode 100644 kb/armory/General/servicenow---all-about-support-cases,-navigation-and-anatomy.md create mode 100644 kb/armory/General/servicenow---allowing-customers-to-see-tickets-across-organization-divisions.md create mode 100644 kb/armory/General/servicenow---attaching-kb-articles.md create mode 100644 kb/armory/General/servicenow---create-edit-knowledge-base-articles.md create mode 100644 kb/armory/General/servicenow---creating-cases-and-change-requests-on-behalf-of-customers.md create mode 100644 kb/armory/General/servicenow---creating-users.md create mode 100644 kb/armory/General/servicenow---customer-account-company-creation.md create mode 100644 kb/armory/General/servicenow---escalation-response-for-support-engineers.md create mode 100644 kb/armory/General/servicenow---finding-a-resource-case,-change-request,-etc....md create mode 100644 kb/armory/General/servicenow---navigation-of-the-backend-site.md create mode 100644 kb/armory/General/servicenow---notifications-on-case-comments.md create mode 100644 kb/armory/General/servicenow---sales-engineers---opening-and-managing-support-cases.md create mode 100644 kb/armory/General/servicenow---using-lists-and-views.md create mode 100644 kb/armory/General/servicenow-service-portal-customer-administrators.md create mode 100644 kb/armory/General/set-execution-history-lookback.md create mode 100644 kb/armory/General/setting-a-github-status-check-on-pr-from-a-pipeline-status.md create mode 100644 kb/armory/General/setting-an-sql-cleanup-job-for-clouddriver.md create mode 100644 kb/armory/General/setting-up-logging-without-tokens.md create mode 100644 kb/armory/General/shared-configuration-repository.md create mode 100644 kb/armory/General/shell-script-introduce-massive-warning-in-spinnaker-ui-failed-to-evaluate-not-found.md create mode 100644 kb/armory/General/sourceami-must-be-defined-in-two-areas-if-a-template-json-is-used-in-aws-bakes.md create mode 100644 kb/armory/General/specify-a-subnet-during-bake-stage.md create mode 100644 kb/armory/General/spel-expression-failure-interpreting--with-terraform-templates-.md create mode 100644 "kb/armory/General/spel-expression-issue--must-be-of-type-string--\"integer\".md" create mode 100644 kb/armory/General/spinnaker-access-denied-error-with-armory-agent-deployed-and-fiat-enabled-exception-monitor-pipeline.md create mode 100644 kb/armory/General/spinnaker-and-helm.md create mode 100644 kb/armory/General/spinnaker-cloud-provider-limit.md create mode 100644 kb/armory/General/spinnaker-does-not-properly-clean-up-old-artifacts-and-resources-when-redeploying.md create mode 100644 kb/armory/General/spinnaker-fails-to-deploy-to-google-appengine-flex-due-to-name-being-too-long.md create mode 100644 kb/armory/General/spinnaker-improperly-errors-even-when-spel-expressions-are-commented-out.md create mode 100644 "kb/armory/General/spinnaker-is-down-with-the-error-of-\"oom-command-not-allowed-when-used-memory->-'maxmemory\"-redis.md" create mode 100644 kb/armory/General/spinnaker-kubernetes-v2-faqs.md create mode 100644 kb/armory/General/spinnaker-not-removing-the-old-auto-scaling-group-after-a-new-deployment.md create mode 100644 kb/armory/General/spinnaker-only-supports-unique-account-names-across-both-aws-and-ecs.md create mode 100644 kb/armory/General/spinnaker-pipelines-not-displaying-in-infrastructure-tab.md create mode 100644 kb/armory/General/spinnaker-supports-kustomize-up-to-v3.8.5-as-the-render-engine-in-bake-manifest-configuration,-but-not-v4.x-artifact-not-found.md create mode 100644 kb/armory/General/spinnaker-timeout-issues-with-scale-agent--increasing-grpc-message-size.md create mode 100644 kb/armory/General/spinnaker-timesout-after-60000-milliseconds-when-updating-to-spinnaker-2.19.x-from-2.18.x-or-lower.md create mode 100644 kb/armory/General/splunk-ami-issues---io-wait-warning,-connection-lost-splunk-crashed-with-splunk-service.md create mode 100644 kb/armory/General/spring-expression-language-spel-samples.md create mode 100644 kb/armory/General/spring-expression-language-tricks.md create mode 100644 kb/armory/General/ssh-keys-in-terraformer.md create mode 100644 kb/armory/General/stacktrace-error-appears-on-the-functions-display.md create mode 100644 kb/armory/General/stage-returns-error-that-fargate-only-supports-network-mode-awsvpc-even-though-it-is-configured.md create mode 100644 kb/armory/General/stage-times-out-during-implementation,-need-to-extend-stage-timeout.md create mode 100644 kb/armory/General/storing-application-secrets-in-vault-for-use-in-spinnaker-pipeline..md create mode 100644 kb/armory/General/stuck-pipeline---orca-with-mysql-editing-the-database.md create mode 100644 kb/armory/General/support-portal-user-registration.md create mode 100644 kb/armory/General/swaggerui-commands-returns-4xx-errors.md create mode 100644 kb/armory/General/system-recommendations-for-kubernetes-deployments.md create mode 100644 kb/armory/General/task-definition-from-a-github-artifact-does-not-replace-task-iam-role.md create mode 100644 kb/armory/General/technical-support-case-statuses.md create mode 100644 kb/armory/General/technical-support-engineers---creating-internal-cases-to-track-project-work.md create mode 100644 kb/armory/General/terraform-'invalid-character'-error.md create mode 100644 kb/armory/General/terraform-apply-stage-doesn't-output-the-full-list-of-changes.md create mode 100644 kb/armory/General/terraform-deployment-using-armory-spinnaker.md create mode 100644 kb/armory/General/terraform-maximum-file-size-error.md create mode 100644 kb/armory/General/terraform-named-profiles-are-unable-to-use-environment-variables-to-store-secrets.md create mode 100644 kb/armory/General/terraformer-does-not-clone-properly-from-bitbucket.md create mode 100644 "kb/armory/General/terraformer-error-exec--\"terraform\"--executable-file-not-found-in-$path.md" create mode 100644 kb/armory/General/terraformer-named-profiles-with-gcp.md create mode 100644 kb/armory/General/terraformer-only-takes-input-from-a-single-file-from-github.md create mode 100644 kb/armory/General/terraformer-stages-stop-running-jobs-with-a-'timeout'-error-logging-enabled.md create mode 100644 kb/armory/General/terraformerversion-doesn't-render-in-pipeline-json-by-default.md create mode 100644 "kb/armory/General/testing-x509-certificates-subject-filtering-with-agent\342\200\250.md" create mode 100644 kb/armory/General/the-role-of-helm-and-spinnaker.md create mode 100644 kb/armory/General/the-timeout-of-bake-stage-does-not-respect-the-timeout-in-bake-stage-configuration.md create mode 100644 kb/armory/General/troubleshoot-baking-helm-3-charts.md create mode 100644 kb/armory/General/troubleshooting-gate-deck-cors-issues.md create mode 100644 "kb/armory/General/trying-to-turn-on-vault-results-in-a-\"missing-client-token-error.md" create mode 100644 kb/armory/General/unable-to-add-ecs-cluster-to-spinnaker-not-authorized.md create mode 100644 kb/armory/General/unable-to-add-kubernetes-cloud-provider-with-a-secret-engine.md create mode 100644 kb/armory/General/unable-to-configure-permissions-on-a-spinnaker-application-with-a-postgresql-database.md create mode 100644 kb/armory/General/unable-to-deploy-after-moving-aws-account-into-profiles.md create mode 100644 kb/armory/General/unable-to-deploy-spinnaker-after-changing-operator-and-halyard-versions.md create mode 100644 kb/armory/General/unable-to-deploy-to-a-new-kubernetes-cluster-to-spinnaker.md create mode 100644 kb/armory/General/unable-to-evaluate-spel-variables-in-an-email-notification.md create mode 100644 kb/armory/General/unable-to-list-kubernetes-endpoints-in-namespace-when-running-clouddriver-with-agent.md create mode 100644 kb/armory/General/unable-to-retrigger-a-an-existing-pipeline-from-spinnaker-ui.md create mode 100644 kb/armory/General/unable-to-see-the-accounts-listed-when-using-the-app-engine-plugin-in-a-pipeline.md create mode 100644 kb/armory/General/unexpected-stale-read-warning-level-messages-in-front50-logs.md create mode 100644 kb/armory/General/uninstalling-armory-spinnaker.md create mode 100644 kb/armory/General/unlock-the-databasechangeloglock-table.md create mode 100644 kb/armory/General/updates-of-previously-cached-gitrepo-fails-on-artifact-fetches.md create mode 100644 kb/armory/General/updating-spinnaker-stops-dinghy-from-updating-existing-pipelines.md create mode 100644 kb/armory/General/upgrade-from-oss-to-armory-spinnaker.md create mode 100644 kb/armory/General/upgrade-spinnaker-using-armory-halyard.md create mode 100644 kb/armory/General/upgrade-to-spinnaker-1.20.x+-2.20.x+-causes-errors-as-pipelines-deploy-to-unavailable-namespace.md create mode 100644 kb/armory/General/upgrade-using-armory-operator-is-not-working--got-halyard-response-status-405,-response--request-method-'post'-not-supported.md create mode 100644 kb/armory/General/upgrade-your-kustomize-version.md create mode 100644 kb/armory/General/upgrading-an-eks-cluster.md create mode 100644 kb/armory/General/upgrading-from-2.19.x-to-2.20.x-error-authenticating-against-eks-clusters-heptio-authenticator-aws.md create mode 100644 kb/armory/General/use-case-for-aws-sticky-sessions.md create mode 100644 kb/armory/General/use-custom-images-with-halyard-public-and-private-docker-registry.md create mode 100644 kb/armory/General/username-read-error-when-executing-terraform-stages-with-bitbucket-hosted-artifacts.md create mode 100644 kb/armory/General/users-are-unable-to-unlock-a-spinnaker-pipeline-through-the-console---unlock-via-ui-unchecked.md create mode 100644 kb/armory/General/users-unable-to-execute-or-create-pipelines-due-to-large-headers-on-api-calls-to-orca,-front50-.md create mode 100644 kb/armory/General/using-`find-artifacts-from-resource-manifest`-overwrites-rendered-artifacts.md create mode 100644 kb/armory/General/using-custom-resource-status-plugin-and-scale-agent-leads-to-an-attempt-was-made-to-call-a-method-that-does-not-exist-error.md create mode 100644 kb/armory/General/using-external-artifacts-as-pipeline-parameters.md create mode 100644 kb/armory/General/using-openshift-with-spinnaker.md create mode 100644 kb/armory/General/using-pipelines-as-code-to-enable-aws-auto-scaling-group-metrics.md create mode 100644 kb/armory/General/using-s3-as-a-backend-for-front50-settings,-suggestions,-and-considerations.md create mode 100644 kb/armory/General/using-the-kubernetes-v2-patch-manifest-stage.md create mode 100644 kb/armory/General/using-travis-ci-with-spinnaker.md create mode 100644 kb/armory/General/utilizing-kustomize-of-remote-base-in-a-different-git-repo.md create mode 100644 kb/armory/General/validator-*validate.halvalidator-detected-a-fatal-error.md create mode 100644 kb/armory/General/viewing-pipelines-as-json-and-developing-dinghy-json-code.md create mode 100644 kb/armory/General/vpcs-and-subnets-not-populating-in-the-drop-down-menu-in-the-ui.md create mode 100644 kb/armory/General/vulnerability-management-policy.md create mode 100644 kb/armory/General/waitforclustershrinktask-of-stage-shrinkcluster-times-out-after-30-minutes.md create mode 100644 kb/armory/General/what-spinnaker-services-can-be-affected-apply-opa-policy.md create mode 100644 kb/armory/General/what-to-consider-when-using-terraformer-with-the-provisioner,-local-exec.md create mode 100644 kb/armory/General/when-deploying-ecs,-subnets-and-vpc-will-not-populate-in-the-ui.md create mode 100644 kb/armory/General/when-using-aws-iam-authenticator,-fail-to-read-namespaces-and-failing-to-deploy-to-k8s-clusters.md create mode 100644 kb/armory/General/where-does-spinnaker-begin-and-jenkins-end.md create mode 100644 kb/armory/General/while-disabling-policy-engine,-orca-continually-advises-that-it-cannot-communicate-with-opa-server.md create mode 100644 kb/armory/General/while-using-armory-arm-cli,-cannot-refer-to-pipelineid.md create mode 100644 kb/armory/How To's/how-to-add-a-custom-ca-for-operator-and-vault.md create mode 100644 kb/armory/How To's/how-to-add-a-custom-smtp-server-to-send-notifications.md create mode 100644 kb/armory/How To's/how-to-add-additional-users-to-an-organization.md create mode 100644 kb/armory/How To's/how-to-audit-for-empty-roles-in-fiat-using-redis-mysql.md create mode 100644 kb/armory/How To's/how-to-change-spinnaker's-login-session-timeout.md rename kb/armory/{General => How To's}/how-to-cleanup-old-executions-in-orca-mysql-database.md (100%) rename kb/armory/{General => How To's}/how-to-cleanup-rosco-bakes-older-than-specific-days-from-redis.md (100%) create mode 100644 kb/armory/How To's/how-to-debug-missing-agent-k8s-accounts.md create mode 100644 kb/armory/How To's/how-to-deploy-a-hotfix-version-for-a-particular-spinnaker-service.md create mode 100644 kb/armory/How To's/how-to-deploy-encrypted-ebs-volumes.md create mode 100644 kb/armory/How To's/how-to-deploy-spinnaker-with-configurations-stored-in-configmaps-instead-of-secrets.md create mode 100644 kb/armory/How To's/how-to-determine-the-correct-amount-of-clouddriver-caching-agents-to-run.md rename kb/armory/{General => How To's}/how-to-disable-security-authn-and-authz-for-spinnaker-migration-testing-.md (100%) create mode 100644 kb/armory/How To's/how-to-enable-an-artifactory-helm-repo-trigger-from-jfrog-artifactory.md create mode 100644 kb/armory/How To's/how-to-find-the-tam-account-executive-sales-engineer-solutions-architect-for-an-account.md create mode 100644 "kb/armory/How To's/how-to-fix-tls-error-\"reason--extension-5-should-not-be-presented-in-certificate_request.\".md" create mode 100644 kb/armory/How To's/how-to-get-a-list-of-all-executions-of-a-given-application-via-api-calls.md rename kb/armory/{General => How To's}/how-to-investigate-pipeline-failures-and-execution-issues-with-end-users.md (99%) create mode 100644 kb/armory/How To's/how-to-iterate-complex-structures-in-pipelines-as-code-dictionary-list-sprig-functions.md create mode 100644 kb/armory/How To's/how-to-limit-logs-to-information-to-specific-logs-by-finding-the-absolute-path.md create mode 100644 kb/armory/How To's/how-to-modify-custom-readiness,-liveness-and-startup-probes-http,-tcp-socket,-exec.md create mode 100644 kb/armory/How To's/how-to-override-spinnaker's-default-signer-encryption-algorithm-for-s3.md create mode 100644 kb/armory/How To's/how-to-parse-terraform-plan-status-code-for-your-spinnaker-pipeline-logic.md create mode 100644 kb/armory/How To's/how-to-pass-artifacts-from-a-parent-pipeline-to-a-child-pipeline-using-matchartifacts.md create mode 100644 kb/armory/How To's/how-to-provide-feature-request-feedback.md create mode 100644 kb/armory/How To's/how-to-provision-jenkins-triggers-with-global-variables-in-pipelines-as-code-dinghy.md create mode 100644 kb/armory/How To's/how-to-provision-unique-reference-ids-for-sub-module-stages-in-pipelines-as-code-dinghy-using-uuidv4.md create mode 100644 kb/armory/How To's/how-to-return-all-effective-permissions-authorizations-for-a-user-through-redis-fiat.md create mode 100644 kb/armory/How To's/how-to-scale-igor-replicas.md rename kb/armory/{General => How To's}/how-to-set-application-features-with-dinghy.md (99%) rename kb/armory/{General => How To's}/how-to-set-up-least-privilege-access-with-fiat.md (96%) create mode 100644 kb/armory/How To's/how-to-specify-an-application-owner-with-dinghy.md create mode 100644 kb/armory/How To's/how-to-switch-packer's-powershell-provisioner.md create mode 100644 kb/armory/How To's/how-to-tag-ebs-volumes-with-spinnaker-using-launch-templates-in-aws.md create mode 100644 kb/armory/How To's/how-to-upgrade-operator.md create mode 100644 kb/armory/How To's/how-to-upload-to-aws-s3-from-armory-cdsh---spinnaker.md create mode 100644 kb/armory/How To's/how-to-use-and-migrate-to-launch-templates,-migrating-from-launch-configuration.md create mode 100644 kb/armory/How To's/how-to-use-dinghy-module-in-module-support.md create mode 100644 kb/armory/How To's/how-to-use-global-variables-in-dinghy-conditionals.md create mode 100644 kb/armory/How To's/how-to-use-spot-instances-with-launch-templates-for-spinnaker-.md create mode 100644 kb/armory/How To's/how-to-use-the-reader-endpoint-for-rds-clusters-in-spinnaker-ha-mode-only.md create mode 100644 kb/armory/How To's/how-to-use-trunc-and-split-functions-in-pipelines-as-code.md create mode 100644 kb/armory/How To's/how-to-utilize-canary-analysis-in-other-appications-within-spinnaker.md diff --git a/kb/armory/General/403-and-permission-errors-when-enabling-new-services.md b/kb/armory/General/403-and-permission-errors-when-enabling-new-services.md new file mode 100644 index 0000000000..cccbb061dc --- /dev/null +++ b/kb/armory/General/403-and-permission-errors-when-enabling-new-services.md @@ -0,0 +1,12 @@ +--- +title: 403 and Permission Errors when Enabling New Services +--- + +## Issue +Armory has found that some customers enabling new services in Spinnaker may encounter various errors, including 403 access errors, when attempting to execute pipelines or perform other tasks. +This issue can usually be related to changes in customer deployments related to policies on minimum access requirements.   + +## Cause +As general guidance for account role access, the AWS Power User role should be used when granting permissions.  However, customers may find that their internal security policy requires more granularly defined access.  +  + diff --git a/kb/armory/General/403-errors-around-github-access-and-rate-limit-issues.md b/kb/armory/General/403-errors-around-github-access-and-rate-limit-issues.md new file mode 100644 index 0000000000..42540bf837 --- /dev/null +++ b/kb/armory/General/403-errors-around-github-access-and-rate-limit-issues.md @@ -0,0 +1,15 @@ +--- +title: 403 Errors around GitHub access and Rate Limit Issues +--- + +## Issue +Customers may encounter 403 errors around GitHub either as a part of their calls to retrieve Artifacts, Dinghy, or other services.  There are a variety of reasons why this may happen, but customers will often see that their CloudDriver logs will indicate the following errors: + +```com.netflix.spinnaker.clouddriver.artifacts.exceptions.FailedDownloadException: Unable to determine the download URL of artifact Artifact(type=github/file, customKind=false, name=null, version=main, location=null, reference=https://api.github.com/, metadata={}, artifactAccount=, provenance=null, uuid=null): Received 403 status code from api.github.com``` + + +or a 403 Status in their Spinnaker UI Console + +## Cause +Multiple factors may be causing a Github 403 status.   + diff --git a/kb/armory/General/access-denied-to-application-on-deploy-stage-without-application-write-permissions.md b/kb/armory/General/access-denied-to-application-on-deploy-stage-without-application-write-permissions.md new file mode 100644 index 0000000000..1c0fdd873a --- /dev/null +++ b/kb/armory/General/access-denied-to-application-on-deploy-stage-without-application-write-permissions.md @@ -0,0 +1,10 @@ +--- +title: Access denied to application on Deploy Stage without application WRITE permissions +--- + +## Issue +Currently, application ```WRITE``` permissions are required to use the Deploy Stage in Spinnaker with AWS, GCP, Titus and other deployment targets.However, when using the Kubernetes provider, only ```EXECUTE``` permissions are needed to use the Deploy Stage.This is an open issue in OSS Spinnaker: [spinnaker/spinnaker#6400](https://github.com/spinnaker/spinnaker/issues/6400)```EXECUTE``` application permissions were developed recently and, so far, have only been fully implemented on the Kubernetes V2 provider. It is possible to set ```EXECUTE``` permissions on the Deploy Stage for other targets, however the stage will also require application ```WRITE``` permissions to run successfully. + +## Cause +Deploy stage on** targets other than the Kubernetes provider** does not run successfully because it does not have application ```WRITE``` permissions set. + diff --git a/kb/armory/General/accessing--armory-scale-agent-endpoints-to-help-in-troubleshooting.md b/kb/armory/General/accessing--armory-scale-agent-endpoints-to-help-in-troubleshooting.md new file mode 100644 index 0000000000..55b440c684 --- /dev/null +++ b/kb/armory/General/accessing--armory-scale-agent-endpoints-to-help-in-troubleshooting.md @@ -0,0 +1,22 @@ +--- +title: Accessing Armory Scale Agent Endpoints to help in Troubleshooting +--- + +## Introduction +Customers using Armory Scale Agent for Kubernetes may encounter issues when running Kubernetes deployments.  The endpoints below can be used to diagnose and gain more information to aid in troubleshooting.  The following KB article explains how to access those endpoints. +Customers may also want to access the CloudDriver endpoints and dig into information that can be found there.  They can do so by following the information in this KB article:[https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010601](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010601) + +## Prerequisites +* Armory Enterprise Spinnaker with Armory Scale Agent for Kubernetes enabled.* Access to the cluster in which Agent is deployed.* Users would also require to port-forward the Agent pod to access the endpoints. The process to do so can be found below in the Instructions section. + +## Instructions +**Port Forward to the Agent ports** +Customers will first need to set up a port forward to the Agent pod.  This can be accomplished by executing the below command. After running the command, the Clouddriver service will be accessible on ```localhost:8082 ``` +```kubectl port-forward pod/armory-agent-xxx 8082``` +**Get details about the Armory Agent account** +To get the details about the accounts that are configured in Armory Agent and if they were loaded after the Agent startup, invoke the below endpoint +```curl -kv http://localhost:8082/accounts/``` +**Attain Agent goroutines ** +Armory Agent is written in Golang. For troubleshooting purposes, it might be necessary to capture the list of ```goroutines``` that are run within the Agent to see if a particular function is being executed or not. Invoking the below endpoint would return the list of ```goroutines``` within Agent. +```curl -kv http://:8082/debug/pprof/goroutine?debug=1``` + diff --git a/kb/armory/General/accessing-clouddriver-endpoints-to-help-in-troubleshooting-armory-scale-agent-issues.md b/kb/armory/General/accessing-clouddriver-endpoints-to-help-in-troubleshooting-armory-scale-agent-issues.md new file mode 100644 index 0000000000..35803b6e04 --- /dev/null +++ b/kb/armory/General/accessing-clouddriver-endpoints-to-help-in-troubleshooting-armory-scale-agent-issues.md @@ -0,0 +1,32 @@ +--- +title: Accessing Clouddriver Endpoints to help in Troubleshooting Armory Scale Agent issues +--- + +## Introduction +Customers using Armory Agent for Kubernetes may run into issues when running Kubernetes deployments.  Customers may want to access CloudDriver endpoints for the purpose of Troubleshooting Armory Scale Agent issues.  The article below touches upon accessing the endpoints in Clouddriver to help in the troubleshooting. +For details on accessing Scale Agent Endpoints, please visit: [https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010602](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010602) + +## Prerequisites +* Armory Enterprise Spinnaker with Armory Agent for Kubernetes enabled.* Access to the cluster in which Spinnaker services are deployed.* Users would also require to port-forward the ```Clouddriver port 7002``` to access Clouddriver endpoints. The process to do so can be found below in the Instructions section. + +## Instructions +### Port Forward to access Clouddriver ports +To port forward to access the Clouddriver service on port 7002, execute the below command. Post running the command, the Clouddriver service shall be accessible on ```localhost:7002 ``` +```kubectl port-forward svc/spin-clouddriver -n spinnaker-namespace 7002:7002``` +### List agents, accounts, and the Clouddriver instances they are connected to +The below endpoint returns a JSON response that contains +* The list of agents* The accounts each agent is accessing* The specific Clouddriver pods that the Agents are connected to/registered with +```curl -kv http://localhost:7002/armory/agents``` + +### Attain the steps that a deploy operation went through +When a deployment stage that contains an Agent account is triggered, an ```operation ID``` is generated by the plugin and this operation undergoes a series of steps. +* The request first gets received by a Clouddriver instance.* If there are multiple replicas of Clouddriver, the operation then gets passed on to the specific Clouddriver instance which is connected to/registered with the Armory Agent making the request using the lookup table.  It correlates it with the account where the deployment is supposed to be triggered.* The operation is then passed on to the Agent.* Once the Agent deploys the operation, the operation follows the return route where it gets passed on to the corresponding Clouddriver instances and then finally returns the response to Orca. +Below is the endpoint to track the steps that the operation went through +```curl -kv http://localhost:7002/armory/agent/operations/{opId}``` +In the case of an error, the ```operation ID``` should be displayed in the Spinnaker Console UI.  However, if you need to locate the ```operation ID``` for a successful execution, please query the backend table, ```kubesvc_ops_history```.  For more information on querying tables related to Agent, please read the following KB article: [https://support.armory.io/support?id=kb_article&sysparm_article=KB0010603](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010603). + +### List Clouddriver instances +Invoking the below endpoint would return the list of Clouddriver replicas that have the plugin enabled +```curl -kv http://localhost:7002/armory/clouddrivers``` +  + diff --git a/kb/armory/General/accessing-the-spinnaker-rest-api-using-httpie-and-jq.md b/kb/armory/General/accessing-the-spinnaker-rest-api-using-httpie-and-jq.md new file mode 100644 index 0000000000..7ce9d04d01 --- /dev/null +++ b/kb/armory/General/accessing-the-spinnaker-rest-api-using-httpie-and-jq.md @@ -0,0 +1,81 @@ +--- +title: Accessing the Spinnaker REST API using HTTPie and jq +--- + +## Introduction +This article advises about how to use ```HTTPie``` to access the ```Spinnaker REST API```. It provides guidelines about using the Spinnaker API programmatically. + +## Prerequisites +The ```Spinnaker REST API``` can be accessed using ```HTTPie```. The results can be filtered and formatted by ```jq```.The following are the prerequisites for accessing the Spinnaker API in this KB: +* The Spinnaker API (Gate) URL.* Download and install of HTTPie: [https://httpie.io/](https://httpie.io/) and jq: [https://stedolan.github.io/jq/download/](https://stedolan.github.io/jq/download/) +**Note: **For long term use, consider the use of ```x509 certificates``` for Spinnaker to make the calls. For development cases, a session cookie after logging in to Spinnaker can be used to test the ```REST API``` call. +In the examples below, a session cookie will be used. + +## Instructions +## Attain a session cookie +* Log on to the ```Deck GUI```. +* Open the **Developer Tools** of your browser. +* Go to the **Network** tab -> find the call to **user** on the left tab. +* Look for the **Request Headers** and the **cookie** parameter. This cookie will need to be set when making API calls. +* Note that this cookie does expire after some time and is not permanent. +## Using the Spinnaker REST API +In the following examples we have set the parameters as follows: +**Gate URL** +GATE=http://x.x.x.x + +# Spinnaker Gate Session Cookie, below is just an example. +SPINCOOK='Cookie:SESSION=YjdhNDg3NmQtZjE5Zi00xxxxxxxxxx' + +# Spinnaker pipeline ID, below is just an example. +PIPEID='01EWXEEZTZPX09FE86XA7S75BA' + +# Spinnaker Application Name +APP=spintest + +# Create httpie session named test. Then use the session name for all the following calls. +# Setting verify to no will skip host SSL certificate verification. +# This setting should only be used if GATE is using a self-signed certificate. +# This will eliminate having to pass the cookie which gets tricky +eval http --verify=no --session=test $GATE/applications "'"$SPINCOOK"'" + +# create/update applications and projects (see example below for min-app-create.json) +eval http --verify=no --session=test POST $GATE/tasks + +The following is an example of ```min-app-create.json``` file which will create an application ```app1``` with minimal configuration and can be used as a test. +``` + { + "job": [ + { + "type": "createApplication", + "application": { + "cloudProviders": "kubernetes", + "instancePort": 80, + "providerSettings": { + "aws": { + "useAmiBlockDeviceMappings": false + } + }, + "name": "app1", + "email": "app@example.com", + "permissions": { + "READ": [ + "superadmins", + "qa" + ], + "EXECUTE": [ + "superadmins" + ], + "WRITE": [ + "superadmins" + ] + } + }, + "user": "test_user" + } + ], + "application": "app1", + "description": "Create Application: app1" +} +``` +For more information on the Spinnaker API please refer to: [https://spinnaker.io/reference/api](https://spinnaker.io/reference/api) and [https://docs.armory.io/docs/spinnaker-user-guides/writing-scripts/](https://docs.armory.io/docs/spinnaker-user-guides/writing-scripts/). + diff --git a/kb/armory/General/accounts-with-livemanifestcalls-set-to-true-have-incorrect-dynamic-lookup-results.md b/kb/armory/General/accounts-with-livemanifestcalls-set-to-true-have-incorrect-dynamic-lookup-results.md new file mode 100644 index 0000000000..d703a55b2a --- /dev/null +++ b/kb/armory/General/accounts-with-livemanifestcalls-set-to-true-have-incorrect-dynamic-lookup-results.md @@ -0,0 +1,13 @@ +--- +title: Accounts with liveManifestCalls Set to True Have Incorrect Dynamic Lookup Results +--- + +## Issue +After enabling ```liveManifestCalls: true```, the environment begins exhibiting odd behaviors.  Resources that have been deployed/changed in previous stages are not being taken in to consideration in current stages, leading to errors and issues in the Pipeline DeploymentThis can be especially detrimental to any pipelines that use a rollout strategy along with a ```strategy.spinnaker.io/max-version-history``` annotation, causing inconsistent state of deployment targets, and also pipeline failures + +## Cause +When the flag is set to true, the 'Deploy Manifest' stage waits for the newly-deployed resource by directly polling the cluster instead of by checking in Spinnaker's cache. In general, the stage will finish more quickly, as it can complete as soon as the resource is ready instead of once the new resource is reflected in the cache.  +One significant issue that may occur though, that the stage may complete before the cache reflects these changes. Spinnaker expects that stages mutating infrastructure will not complete until the cache has been updated to reflect these mutations. +**The result is that any downstream stages that rely on the cache being up-to-date (as stages are generally allowed to do) will either fail or produce incorrect results**.  +As for how it relates to this issue, any stages that use **dynamic target selection** to patch/enable/disable a resource. Stages looking in the cache to find the oldest/newest/etc. resource, and act based on the state of the cache when they run will also be affected.  Finally, rollout strategies are another example where dynamic target selection is affected.  As a result of the cache not being up to date, this can lead to omitting a newly deployed/deleted/patched resource from a prior stage.For pipelines that have use a rollout strategy along with a ```[strategy.spinnaker.io/max-version-history](http://strategy.spinnaker.io/max-version-history)``` annotation, this can be especially painful.  When a ```max version flag``` is set, at the time of execution **Clouddriver knows only of** **N replicasets existing** and will try to disable the **N-1 older replicasets**.This means that there are situations where although Orca plans for X disable manifest tasks, the oldest one is already deleted at the time of the task execution causing a pipeline failure.Furthermore if a failed pipeline is executed 2-3 or more times until it succeeds it causing a very inconsistent state of the deployment targets depending on which Disable manifest task completes and which doesn't. + diff --git a/kb/armory/General/adding-a-group-with-many-members-to-application-times-out-in-gcp.md b/kb/armory/General/adding-a-group-with-many-members-to-application-times-out-in-gcp.md new file mode 100644 index 0000000000..cb34272421 --- /dev/null +++ b/kb/armory/General/adding-a-group-with-many-members-to-application-times-out-in-gcp.md @@ -0,0 +1,16 @@ +--- +title: Adding a group with many members to application times out in GCP +--- + +## Issue +An organization using Spinnaker and Google Cloud Platform (GCP) may run into API rate limiting issues involving Fiat Managed Service Accounts.  These rate limits can cause errors, timeouts, and general non-responsiveness when the expectation is full functionality.  +  +Example Error:  +```202X-XX-XX XX:XX:XX.453 WARN 1 — [ scheduling-1] s.f.r.g.GoogleDirectoryUserRolesProvider : [] Failed to fetch groups for user xx####x#-xx#x-##xx-x#x#-###c####x#x#@managed-service-account: Invalid Input: memberKey" ``` +  +  + +## Cause +Spinnaker uses ```Google Workspaces API``` to manage and run ```FIAT``` Managed Services, resulting in consistent, frequent API calls when calling managed service accounts.  The results are that an organization can be temporarily blocked and rate limited due to excessive usage. +Specifically, requests are made to Google APIs whenever pipelines execute using the managed service account from an automated trigger, causing multiple API calls for routine deployments.  + diff --git a/kb/armory/General/adding-additional-policies-and-access-to-an-eks-cluster.md b/kb/armory/General/adding-additional-policies-and-access-to-an-eks-cluster.md new file mode 100644 index 0000000000..f3e7e4aa96 --- /dev/null +++ b/kb/armory/General/adding-additional-policies-and-access-to-an-eks-cluster.md @@ -0,0 +1,15 @@ +--- +title: Adding Additional Policies and Access to an EKS Cluster +--- + +## Introduction +As environments become more complex and additional security requirements are necessary for the nods and cluster of the environment, some fundamental basics of policies and permissions should be considered for the environment.  +Depending on the particular security policy of the environment + +## Prerequisites +Access to the cluster, roles and policy for the AWS environment + +## Instructions +When deploying the EKS environment, IAM Roles are usually used to manage the security and access available to the environment.  The roles and permissions associated to allow Spinnaker to interact with the AWS environment are tied to the **EKS Cluster Node IAM Role** which can be found at: +* Log in to the AWS Management Console* Go to the EKS Administration portal* On the left menu select **Clusters **under **Amazon EKS** and select the appropriate cluster name* Click on the **Configuration** tab, then the **Compute** tab* Click on the appropriate **Node Group*** In the information for the Node Group, there will be an entry regarding the **Node IAM Role ARN**. Access needs to be provided to this role so that the cluster can access the appropriate resources (e.g. **Secrets Manager** or **S3 Buckets** for the storage of Secrets) + diff --git a/kb/armory/General/adjust-clean-up-timing-for-armory-agent.md b/kb/armory/General/adjust-clean-up-timing-for-armory-agent.md new file mode 100644 index 0000000000..c5461730fd --- /dev/null +++ b/kb/armory/General/adjust-clean-up-timing-for-armory-agent.md @@ -0,0 +1,13 @@ +--- +title: Adjust Clean Up Timing for Armory Agent +--- + +## Introduction +After a set amount of time, if CloudDriver is unable to reach an Agent, it is deemed to be marked for cleanup.  The cleanup timing may need to be adjusted Settings on CloudDriver can be adjusted to clean up missing / unreachable agents.  + +## Prerequisites +Armory Agent should be [installed and configured](https://docs.armory.io/docs/armory-agent/) + +## Instructions +To make adjustments on the cleanup timing, the adjustment should be made to the ```kubesvc.cache.accountCleanupFrequencySeconds``` value in the Plugin adjustments. By default, the clean up frequency is set to 600 seconds (10 minutes)For further information on adjusting the value on the plugin, please refer to: [https://docs.armory.io/docs/armory-agent/agent-plugin-options/](https://docs.armory.io/docs/armory-agent/agent-plugin-options/) + diff --git a/kb/armory/General/agent-deployment-error--exceptions.operationtimedoutexception--timeout-exceeded-for-operation-.md b/kb/armory/General/agent-deployment-error--exceptions.operationtimedoutexception--timeout-exceeded-for-operation-.md new file mode 100644 index 0000000000..b1f66a61a0 --- /dev/null +++ b/kb/armory/General/agent-deployment-error--exceptions.operationtimedoutexception--timeout-exceeded-for-operation-.md @@ -0,0 +1,22 @@ +--- +title: Agent Deployment Error- exceptions.OperationTimedOutException- Timeout exceeded for operation +--- + +## Issue +When using Agent, customers may experience that they see the following errors during a deployment execution: + +``` +message="com.netflix.spinnaker.clouddriver.exceptions.OperationTimedOutException: Timeout exceeded for operation ......., type: 6, account: .....-ingress-dev, time: 30022 ms, timeout: 30000 ms + at io.armory.kubesvc.services.ops.KubesvcOperations.performOperation(KubesvcOperations.java:96) + at io.armory.kubesvc.services.ops.cluster.ClusteredKubesvcOperations.performOperation(ClusteredKubesvcOperations.java:70) + at io.armory.kubesvc.util.OperationUtils.perform(OperationUtils.java:76) + at io.armory.kubesvc.services.ops.executor.KubesvcExecutor.deploy(KubesvcExecutor.java:301) + at jdk.internal.reflect.GeneratedMethodAccessor703.invoke(Unknown Source) + at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + at java.base/java.lang.reflect.Method.invoke(Method.java:566) +``` + +## Cause +The error indicates that the Clouddriver sent a ```deploy operation``` to the kubesvc Agent and was not able to obtain the result back from the Agent in time. +  + diff --git a/kb/armory/General/applying-custom-sizes-to-services.md b/kb/armory/General/applying-custom-sizes-to-services.md new file mode 100644 index 0000000000..550a4f6a7b --- /dev/null +++ b/kb/armory/General/applying-custom-sizes-to-services.md @@ -0,0 +1,35 @@ +--- +title: Applying Custom Sizes to Services +--- + + +*Note: This guide assumes you’re deploying Spinnaker on Kubernetes using the [Distributed](https://www.spinnaker.io/setup/install/environment/#distributed-installation) deployment type with Halyard.* +You may find the need to apply custom limits and requests to some of the Spinnaker services – doing so can provide more stable behavior of the services while also preventing extreme bursts of CPU activity or the allocation of more memory than is available (and thus being OOM Killed). +Custom Sizing in Hal Config +Custom sizing can be applied to a service through the ```deploymentEnvironment.customSizing``` section of your hal config. The following example sets requests for ```gate``` of 250 millicpu and 512 MiB with limits of 500 millicpu and 1GiB of memory. + deploymentEnvironment: + customSizing: + gate: + limits: + cpu: 500m + memory: 1Gi + requests: + cpu: 250m + memory: 512Mi +(Note: for these settings to be applied to sidecar containers as well, prefix the service name with “spin-“) +For a more complete understanding of these settings see the [Kubernetes docs](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) + +JAVA_OPTS in Service Settings +In addition to the options available through the ```deploymentEnvironment.customSizing``` settings, JVM-based services have ```JAVA_OPTS``` that can be set or overridden through their [service-settings](https://www.spinnaker.io/reference/halyard/custom/#tweakable-service-settings). This is done by creating or editing the ```$HOME/.hal/default/service-settings/.yml``` of the respective service +By default, the following is set for all JVM-based services and provides 1/2 of the container’s allocated memory to be used by the service: +```JAVA_OPTS=-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2``` +(Note: The above is currently *not* being set for Gate and should therefore be set manually by the user) + +Setting Xmx and Xms JAVA_OPTS +You may find it useful to set Xmx and Xms of a JVM-based service. The example below shows how this would be done to the ```gate``` service by editing/creating ```$HOME/.hal/default/service-settings/gate.yml```: +env: + JAVA_OPTS: "-Xms512m -Xmx1024m" + +Finally… +Make sure to ```hal deploy apply``` for your changes to take effect. + diff --git a/kb/armory/General/armory-agent---deploy-stage-issues---404-not-found--unknown-kind-apiservice-in-apiservice-....md b/kb/armory/General/armory-agent---deploy-stage-issues---404-not-found--unknown-kind-apiservice-in-apiservice-....md new file mode 100644 index 0000000000..de811323fd --- /dev/null +++ b/kb/armory/General/armory-agent---deploy-stage-issues---404-not-found--unknown-kind-apiservice-in-apiservice-....md @@ -0,0 +1,13 @@ +--- +title: Armory Agent - Deploy Stage Issues - 404 (Not Found)- unknown kind ApiService in APIService ... +--- + +## Issue +In a deployment, the following 404 error is seen pertaining to the Armory Agent and failed deployment stage: +``` level=error msg="error for operation ......, return 404 (Not Found): unknown kind ApiService in group apiregistration.k8s.io for account spinnaker" func="armory/kubesvc/pkg/ops.(*OpsEnv).perform" file="/workspace/pkg/ops/ops.go:145" error=""``` + + +## Cause +Agent is not able to get/delete/deploy SpinnakerService.spinnaker.armory.io if the target Kubernetes cluster also has OSS SpinnakerService.spinnaker.io defined. This is due to the logic used to prevent Deployment.apps and Deployment.extensions from populating the Spinnaker cache with duplicate information. (Both APIs manage the same resources).  + + diff --git a/kb/armory/General/armory-agent-testing,-investigation,-and-checks.md b/kb/armory/General/armory-agent-testing,-investigation,-and-checks.md new file mode 100644 index 0000000000..94fb68871e --- /dev/null +++ b/kb/armory/General/armory-agent-testing,-investigation,-and-checks.md @@ -0,0 +1,83 @@ +--- +title: Armory Agent Testing, Investigation, and Checks +--- + +## Introduction +When having issues connecting Armory Agent with CloudDriver, there are some checks and tests that should be run in order to get some general information about where the issues may reside.These checks are useful to perform before opening a ticket so that we can ensure a quicker resolution to the issue.  Please also consult our [docs on setting up Armory Agent](https://docs.armory.io/docs/armory-agent/) which include a quick start guide and many settings that may be applicable to the environment + +## Prerequisites +```grpcurl``` should be installed ([https://github.com/fullstorydev/grpcurl](https://github.com/fullstorydev/grpcurl))**Please note:** ```http/2``` traffic must be available between the Agent(s) and CloudDriver.  This may require adjustments on the firewall or security to allow for the traffic to be made available  + +## Instructions +Below are some common considerations to think about when investigating communication issues, and offer a starting point before opening a ticket with Support.  +### Ensuring the Correct CloudDriver Pod is being Checked for Error Messages +Within all CloudDriver pods, the repeating log entry ```Assigning accounts to Kubesvc enabled Clouddriver``` is a normally occurrence happening once every 30s to check if caching assignments need to be changed from one CloudDriver Pod to another (basically acting like a heartbeat).All CloudDriver pods should have this same message within their logs, regardless if they are connected or not as the primary pod that the Agents are interacting with.The logs that contain relevant information about the connection status may exist on another pod, and should also be checked for additional messages not in the originating pod.  As an example, this is the output in a pod which has a healthy connection. +``` +21-01-26 03:42:14.876 INFO 1 --- [ecutionAction-6] a.k.s.r.c.c.MNKubesvcAccountLoadBalancer : Account assignment done +2021-01-26 03:42:16.874 INFO 1 --- [ecutionAction-8] c.k.c.a.KubernetesV2OnDemandCachingAgent : spinnaker/KubernetesCoreCachingAgent[1/1]: agent is starting +2021-01-26 03:42:16.874 INFO 1 --- [ecutionAction-5] c.k.c.a.KubernetesV2OnDemandCachingAgent : spinnaker/KubernetesUnregisteredCustomResourceCachingAgent[1/1]: agent is starting +2021-01-26 03:42:18.513 INFO 1 --- [ecutionAction-5] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesUnregisteredCustomResourceCachingAgent[1/1]: grouping Addon.k3s.cattle.io has 12 entries and 0 relationships +2021-01-26 03:42:18.513 INFO 1 --- [ecutionAction-5] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesUnregisteredCustomResourceCachingAgent[1/1]: grouping HelmChart.helm.cattle.io has 1 entries and 0 relations +hips +2021-01-26 03:42:18.513 INFO 1 --- [ecutionAction-5] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesUnregisteredCustomResourceCachingAgent[1/1]: grouping SpinnakerService.spinnaker.armory.io has 1 entries and + 0 relationships +2021-01-26 03:42:19.429 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping role has 7 entries and 0 relationships +2021-01-26 03:42:19.429 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping pod has 23 entries and 24 relationships +2021-01-26 03:42:19.429 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping roleBinding has 8 entries and 0 relationships +2021-01-26 03:42:19.429 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping secret has 75 entries and 29 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping daemonSet has 4 entries and 20 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping artifact has 1 entries and 0 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping storageClass has 1 entries and 0 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping persistentVolume has 2 entries and 0 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping customResourceDefinition has 5 entries and 0 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping event has 271 entries and 0 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping clusters has 42 entries and 118 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping deployment has 15 entries and 103 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping clusterRole has 66 entries and 0 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping apiService has 38 entries and 0 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping serviceAccount has 40 entries and 0 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping replicaSet has 35 entries and 125 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping validatingWebhookConfiguration has 1 entries and 0 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping statefulSet has 2 entries and 10 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping service has 20 entries and 59 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping configMap has 13 entries and 1 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping namespace has 6 entries and 0 relationships +2021-01-26 03:42:19.430 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping controllerRevision has 6 entries and 6 relationships +2021-01-26 03:42:19.431 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping clusterRoleBinding has 49 entries and 0 relationships +2021-01-26 03:42:19.431 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping persistentVolumeClaim has 2 entries and 0 relationships +2021-01-26 03:42:19.431 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping job has 1 entries and 1 relationships +2021-01-26 03:42:19.431 INFO 1 --- [ecutionAction-8] n.s.c.k.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesCoreCachingAgent[1/1]: grouping applications has 13 entries and 118 relationships +``` +### Check to Ensure CloudDriver Pods Were Able to Register with the Agents +After finishing the creation of the plugin and installing agent, attempt to run the following command, replacing with the namespace with the correct namespace that CloudDriver resides in +```kubectl -n logs deploy/spin-clouddriver | grep Registering``` +The output should contain several references to the registration of the agent, e.g. +``` +ubuntu@ip-192-168-88-133:~/agent-k8s-training$ kubectl -n spinnaker logs deploy/spin-clouddriver | grep Registering +2021-01-26 03:35:55.033 INFO 1 --- [ main] i.a.k.s.r.c.KubesvcAccountProvider : Registering Kubernetes v2 account account01 +2021-01-26 03:35:55.111 INFO 1 --- [ main] i.a.k.s.r.c.KubesvcAccountProvider : Registering Kubernetes v2 account account17 +2021-01-26 03:36:46.124 INFO 1 --- [ault-executor-0] i.a.k.s.r.kubesvc.KubesvcRegistry : Registering Kubesvc instance 9fa5f8a8-b617-4ba4-af89-cedfb728 +d59b +2021-01-26 03:37:14.888 INFO 1 --- [ault-executor-1] i.a.k.s.r.kubesvc.KubesvcRegistry : Registering Kubesvc instance 72f2b5c2-2110-4982-b198-206a97d3 +b026 +``` +### Check to Ensure CloudDriver Pods Were Able to Connect with the Agents +Attempt to run the following command, replacing `````` with the namespace with the correct namespace that CloudDriver resides in +```kubectl -n logs deploy/spin-kubesvc | grep connect``` + +The output should appear references like the below example, showing the connection to the agent: +``` +time="2021-01-26T03:42:19Z" level=info msg="connecting to 3.21.240.197:9091..." +time="2021-01-26T03:42:19Z" level=info msg="connected to 3.21.240.197:9091" +time="2021-01-26T03:42:19Z" level=info msg="connecting to Spinnaker: a420f5f3-1de2-4c49-a354-d2c687c874e5" +``` + +### Check Service Account(s) Were Created in the Correct Namespace +Please also check that the service account was created in the correct namespace and provide that output by running  +```kubectl get serviceaccounts -n ``` +Please also provide the YAML output of the service account  +```kubectl get serviceaccounts/ -o yaml -n ``` + +### Perform a gRPCurl to Confirm Communication +Finally, it is a good idea to check that a ```grpcurl``` to attain the output from where the agent is installed to the IP address of the load balancer for the CloudDriver pod.There is a list of commands and what can be done to enable verbose output located in the [troubleshooting section of our Agent Docs](https://docs.armory.io/docs/armory-agent/agent-troubleshooting/#testing-grpc-endpoints) + diff --git a/kb/armory/General/armory-cloud---cannot-register-user-as-user-already-exists-status-code-409.md b/kb/armory/General/armory-cloud---cannot-register-user-as-user-already-exists-status-code-409.md new file mode 100644 index 0000000000..fd35ab3706 --- /dev/null +++ b/kb/armory/General/armory-cloud---cannot-register-user-as-user-already-exists-status-code-409.md @@ -0,0 +1,18 @@ +--- +title: Armory Cloud - Cannot Register User as User Already Exists (status code 409) +--- + +## Issue +**Early Access****As part of participating in an early access program, you will receive information from Armory that is not yet publicly available and is subject to your NDA or confidentiality obligations with Armory.** +Administrators attempting to register new users in their Armory Cloud instance may find that they receive an error when attempting to add them through the console.  The message advises: ```Request failed with status code 409: The user already exists: Error ID: [uuid of instance]``` +Customers cannot find the user in their user list, and if the user in question attempts to log in, they are able to, but do not see anything, or have a blank login.  Even after resetting passwords, they still see the same results. +  + + +## Cause +The main cause is because a user has been associated with an incorrect account.  This will mainly happen as a result of a user attempting to register an account and signed up in the [Armory Cloud signup page](https://console.cloud.armory.io/signup), and did not provide the exact spelling of the Company as registered with Armory.  +For example, the company could be registered as ```ACME``` with Armory, but the person entered ```ACME Inc``` instead +As a result, the user is registered with Armory, and cannot be re-invited, and if attempting to sign in, they will not see any data because the company does not exist within Armory records +  + + diff --git "a/kb/armory/General/armory-cloud---remote-network-agent-cannot-connect-unauthenticated--provided-credentials-did-not-have-\"xxxxxx-xxxxx'-scope.md" "b/kb/armory/General/armory-cloud---remote-network-agent-cannot-connect-unauthenticated--provided-credentials-did-not-have-\"xxxxxx-xxxxx'-scope.md" new file mode 100644 index 0000000000..856cf168ec --- /dev/null +++ "b/kb/armory/General/armory-cloud---remote-network-agent-cannot-connect-unauthenticated--provided-credentials-did-not-have-\"xxxxxx-xxxxx'-scope.md" @@ -0,0 +1,33 @@ +--- +title: Armory Cloud - Remote Network Agent cannot connect (UNAUTHENTICATED- provided credentials did not have "xxxxxx-xxxxx' scope) +--- + +## Issue +**Early Access****As part of participating in an early access program, you will receive information from Armory that is not yet publicly available and is subject to your NDA or confidentiality obligations with Armory.** +Admins may find that the Remote Network Agent pods are unable to start successfully.   +Upon investigation, the following types of errors can be found in the logs of Remote Network Agent pods.  The specific scope that is shown below, ```connect:agentHub``` may differ in the customer environment +``` +[2022-02-11 00:00:02,118] [INFO] An error occurred, attempting to re-connect with hub +[2022-02-11 00:00:07,122] [INFO] The Remote Network Agent is connected to Armory Cloud Agent Hub waiting for requests +[2022-02-11 00:00:07,134] [ERROR] Error while receiving messages from the hub +io.grpc.StatusRuntimeException: UNAUTHENTICATED: provided credentials did not have 'connect:agentHub' scope + at io.grpc.Status.asRuntimeException(Status.java:535) + at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478) + at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463) + at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:427) + at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:460) + at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:557) + at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:69) + at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:738) + at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:717) + at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) + at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) + at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) + at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) + at java.base/java.lang.Thread.run(Thread.java:833) +[2022-02-11 00:00:07,134] [WARN] Agent exited while loop without shutting down channel, telling channel to shutdown now +[2022-02-11 00:00:07,135] [INFO] An error occurred, attempting to re-connect with hub +``` +## Cause +A change in scope can be found if an environment hasn't updated Agent in a while, or the environment is using an out-of-date version of Agent.  A scope change is just one of the reasons why the following error could occur.      + diff --git a/kb/armory/General/armory-continous-deployment-managed-high-availability-and-disaster-recovery-sla-information.md b/kb/armory/General/armory-continous-deployment-managed-high-availability-and-disaster-recovery-sla-information.md new file mode 100644 index 0000000000..3016f53148 --- /dev/null +++ b/kb/armory/General/armory-continous-deployment-managed-high-availability-and-disaster-recovery-sla-information.md @@ -0,0 +1,30 @@ +--- +title: Armory Continous Deployment Managed High Availability and Disaster Recovery SLA Information +--- + + +### Support SLA's +Support SLA's are defined [here](http://go.armory.io/sla). This document does not intend to change or alter the support response or resolution times. +The goal of this document is to provide Armory Legal with the information they need to build the contract language required for ***Availability SLA's***, along with providing support with the correct information to publish on Armory's support page. +### Standard Availability SLA's +Armory offers 99.5% uptime availability for all managed or as-a-Service products unless specifically stated otherwise. + +#### **Uptime and Availability Chart** +**Availability Percentage****Accepted Downtime per Year**99.0%3.65 Days99.5%1.83 Days99.9%8.76 Hours99.99%52.56 Minutes99.999%5.26 Minutes +  +  +  +  +### Disaster Recovery SLA's +When Armory customers purchase the available CD Managed add-on known as ***Disaster Recovery (DR)*** the ```Recovery Time Objective (RTO) is <= 30 minutes```. This means Armory will restore service in thirty minutes or less beginning at the time of fail-over or recovery request. +A failover or recovery request can be automated or manual. Customers have the choice of DR configurations being a manual trigger or an automated trigger. Regardless of the choice of automated or manual an ***agreed time ***between failure and initiate failover/recovery must be established. Armory's RTO SLA will begin at the time the failover/recovery is initiated. +#### **Examples:** +* If 15 minutes of consecutive unavailable service initiate a failover/recovery request — would mean there could be a total of 45 minutes of downtime of which 30 minutes is within Armory's RTO SLA* The customer agrees to manually notify Armory when they [the customer] wants a failover/recovery initiated. Armory's response SLA's would factor into the total availability (e.g.: ```P0 response time is 15 minutes + <= 30 minutes RTO = 45 mins total SLA```) +### High-Availability SLA's +Armory offers an Active/Active(Active) High-Availability solution as a second option for CD Managed customers. When Armory customers purchase the available CD Managed add-on known as ***High-Availability (HA)*** the availability SLA is 99.999% uptime (also known as 5.26 Minutes of accepted downtime per year) across all configured regions. +Armory's Active/Active(Active) HA configuration means there is an active configuration of Armory CD running in 3 separate availability zones *(e.g.: us-east-1, us-east-2, us-west-1)* and traffic and data are in sync between the three Armory CD instances. So, if one or two instances become unavailable the remaining instance(s) service the users accordingly. +### Disclaimer +Armory's agreed SLA's are determined by the underlying cloud services and dependant 3rd party services (e.g.: Jenkins, Vault,…) being available. If there are outages beyond Armory's control *(i.e.: AWS, Azure, GCP, outages), or the customer does not comply with the recommended settings and configurations Armory recommends,* the SLA's are not guaranteed. Only when all required services are available and recommended configurations implemented will the SLA be honoured. +### Notes +Customers ***DO NOT*** need to purchase HA and DR. The add-ons are one ***OR*** the other. Because HA provides traffic and data synchronization there is no need to initiate a failover/recovery (i.e.: the service should remain up). If for whatever reason the service is unavailable in an HA configuration Armory provides a best-effort ASAP recovery. + diff --git a/kb/armory/General/armory-continuous-deployment-as-a-service-cdaas-support.md b/kb/armory/General/armory-continuous-deployment-as-a-service-cdaas-support.md new file mode 100644 index 0000000000..225efd0975 --- /dev/null +++ b/kb/armory/General/armory-continuous-deployment-as-a-service-cdaas-support.md @@ -0,0 +1,150 @@ +--- +title: Armory Continuous Deployment as a Service (CDaaS) Support +--- + + +Welcome to the initial Armory Continuous Deployment as a Service (CD-as-a-Service) Support Page!  For those starting your journey with Armory CD-as-a-Service, please refer to this page for some initial guidance on how Armory can assist you and your team. + +## Table of Contents +* [Starting with Armory CD-as-a-Service](#mcetoc_1h5omdj69c)* [Trialing CD-as-a-Service](#mcetoc_1h5on72569u)* [Purchasing CD-as-a-Service](#mcetoc_1h5on72569v)[Support Plans, SLAs, and Response Times](#mcetoc_1h5on7256a0) +* [SLA Matrix - Armory CD-as-a-Service Team-Level Support plan](#mcetoc_1h5on7256a2)* [SLA Matrix - Armory CD-as-a-Service Enterprise-Level Support plan](#mcetoc_1h5onh7v4as) + + +Starting with Armory CD-as-a-Service +Customers starting their journey with Armory CD-as-a-Service will want to begin in the following places. +* [Armory Docs CD-as-a-Service](https://docs.armory.io/cd-as-a-service/) * [Armory Knowledge Base - CD-as-a-Service](https://support.armory.io/support?id=kb_search&kb_knowledge_base=faf2a7e0db1f815079f53ec8f4961991)[Armory Support Information (Hours of Operation/SLAs/Procedures)](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010007) +* [Password Resets](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010247)* [Support Portal User Registration](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010200)* [ServiceNow Service Portal Customer Administrators](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010240)* [Technical Support Case Statuses](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010011)* [Case Escalations](https://armory.service-now.com/support?id=kb_article&sysparm_article=KB0010203)* [Critical](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010468)[ Notifications Mailing List](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010468) + +Customers with support contracts needing to submit a case to our support team should create a case in our ServiceNow Portal. +* [Creating Cases and Case Management in Service Now](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010136) +Trialing CD-as-a-Service +Customers looking to Trial Armory's CD-as-a-Service offering should begin their journey in our [CD-as-a-Service Quickstart Guide](https://docs.armory.io/cd-as-a-service/setup/quickstart/).  This comprehensive guide will direct new users on how to sign up for a free trial.  If your company is already trialing CD-as-a-Service, it is recommended that you contact your administrator to provide you with access to the company CD-as-a-Service environment. +Purchasing CD-as-a-Service +Customers looking to explore pricing for CD-as-a-Service can do so by visiting the [Armory CD-as-a-Service Pricing section](https://www.armory.io/products/continuous-deployment-as-a-service/#pricing).  This section will provide more information about the differences between the Free, Team, and Enterprise tiers and a handy comparison chart. +**Support Plans, **SLAs, and Response Times +Customers with a Support contract can find their SLAs below +SLA Matrix - Armory CD-as-a-Service Team-Level Support plan + + +**Support Case Priority** + +**Response Time SLA** + +**Schedule** + +**Target Resolution Time** + +**Definition** + +**Example Case** + +P0 - Critical (Outage) + +1 Hour +11 x 5 Business Hours (EST/PST) +ASAP +Critical - Customers cannot deploy or set parameters for an in-production environment without a workaround, and it has been confirmed that the issue exists within CD-as-a-ServiceCD-as-a-Service Remote Network Agent (RNA) is working, but no communication is occurring between the CD-as-a-Service console and RNA agent in a production environment, and no Network changes have been applied +P1 - High + +12 Hours + +11 x 5 Business Hours (EST/PST) + +1 Week + +Production instance features are functionally impaired with no acceptable workaround/rollback. + +Deployment triggers show errors and are not completed successfully. The continuous delivery cycle is severely impacted. + +P2 - Medium + +32 Hours + +11 x 5 Business Hours (EST/PST) + +Best Effort + +Any issue affecting the Production or Dev instances that is non-critical and non-time-sensitive. + +A bug impacts the CD-as-a-Service environment, but a workaround is available, and business resumes as usual. + +P3 - Low + +48 Hours + +11 x 5 Business Hours (EST/PST) + +Best Effort + +Questions or clarifications around features/docs, etc. No immediate impact on the business. + +Inquiries about configurations, How to requests, or non-production issues. + + +* Armory offers 11 Business Hours x 5 days a week Support based on the SLA agreements listed above. * SLAs for the First Response are based on the priority of the case. We will prioritize resolution based on how severe the impact is. Our focus will **always** go to the highest priority cases.* Business Hours are defined as 8 hours between ***9 am-8 pm EST**** / ****6 am-5 pm PST.**** The SLA times listed in **Response Time SLA** are when customers can expect the first human response and acknowledgment of the case.* Armory Technical Support will try to resolve any issues as soon as possible. However, the Resolution Time SLAs are *not considered an expected time-to-resolution.* +SLA Matrix - Armory CD-as-a-Service Enterprise-Level Support plan + + +**Support Case Priority** + +**Response Time SLA** + +**Schedule** + +**Target Resolution Time** + +**Definition** + +**Example Case** + +P0 - Critical (Outage) + +1 Hour + +24 x 7 + +ASAP + +Critical - Customers cannot deploy or set parameters for an in-production environment without a workaround, and it has been confirmed that the issue exists within CD-as-a-Service + +CD-as-a-Service Remote Network Agent (RNA) is working, but no communication is occurring between the CD-as-a-Service console and RNA agent in a production environment, and no Network changes have been applied + +P1 - High + +8 Hours + +11 x 5 Business Hours (EST/PST) + +3 Business Days + +Production instance features are functionally impaired with no acceptable workaround/rollback. + +Deployment triggers show errors and are not completed successfully. The continuous delivery cycle is severely impacted. + +P2 - Medium + +24 Hours + +11 x 5 Business Hours (EST/PST) + +Best Effort + +Any issue affecting the Production or Dev instances that is non-critical and non-time-sensitive. + +A bug impacts the CD-as-a-Service environment, but a workaround is available, and business resumes as usual. + +P3 - Low + +48 Hours + +11 x 5 Business Hours (EST/PST) + +Best Effort + +Questions or clarifications around features/docs, etc. No immediate impact on the business. + +Inquiries about configurations, How to requests, or non-production issues. + + +* Armory offers 11 Business Hours x 5 days a week Support (***24x7 for P0s***) based on the above mentioned SLA agreements. * Business Hours are between ***9 AM - 8 PM EST** / **6 AM - 5 PM PST.**** The SLA times listed in **Response Time SLA** are when customers can expect the first human response and acknowledgment of the case.* Armory Technical Support will try to resolve any issues as soon as possible. However, the Resolution Time SLAs are *not considered an expected time-to-resolution.* + diff --git a/kb/armory/General/armory-diagnostics.md b/kb/armory/General/armory-diagnostics.md new file mode 100644 index 0000000000..4d22e082ed --- /dev/null +++ b/kb/armory/General/armory-diagnostics.md @@ -0,0 +1,8 @@ +--- +title: Armory Diagnostics +--- + + +This page has been recategorized and moved to our Documents Library: +[https://docs.armory.io/continuous-deployment/armory-admin/observe/diagnostics-configure/](https://docs.armory.io/continuous-deployment/armory-admin/observe/diagnostics-configure/) + diff --git a/kb/armory/General/armory-halyard-overview.md b/kb/armory/General/armory-halyard-overview.md new file mode 100644 index 0000000000..6f359b5c5f --- /dev/null +++ b/kb/armory/General/armory-halyard-overview.md @@ -0,0 +1,7 @@ +--- +title: Armory Halyard Overview +--- + + +This page has been recategorized and moved to our Documents Library:[https://docs.armory.io/overview/overview-of-armory-halyard/](https://docs.armory.io/overview/overview-of-armory-halyard/) + diff --git a/kb/armory/General/armory-priority-zero-p0-case-handling-procedures.md b/kb/armory/General/armory-priority-zero-p0-case-handling-procedures.md index b519b349bf..6da3b59dd4 100644 --- a/kb/armory/General/armory-priority-zero-p0-case-handling-procedures.md +++ b/kb/armory/General/armory-priority-zero-p0-case-handling-procedures.md @@ -14,7 +14,7 @@ Armory Customers with a Support Service agreement may be required to open a case * [How can a Customer Escalate a non-P0 Case to a P0 Status?](#mcetoc_1h50f58lv57) -When should a Priority Zero case be opened? +### When should a Priority Zero case be opened? A P0 case should be opened if the environment has * A Production system critical outage for an Armory product (Armory CD Self Hosted, Armory CD-as-a-Service, Spinnaker OSS, etc.) * A security incident that requires immediate resolution diff --git a/kb/armory/General/armory-product-demo-videos.md b/kb/armory/General/armory-product-demo-videos.md new file mode 100644 index 0000000000..e11df32b84 --- /dev/null +++ b/kb/armory/General/armory-product-demo-videos.md @@ -0,0 +1,7 @@ +--- +title: Armory Product Demo Videos +--- + + +Thank you for taking an interest in Armory SpinnakerWe have a series of product demo and videos outlining Armory-specific functionality across the entire SDLC before kicking the tires.For more information, please visit the link below[https://www.armory.io/blog/armory-product-demos/](https://www.armory.io/blog/armory-product-demos/)  + diff --git a/kb/armory/General/armory-release-definitions.md b/kb/armory/General/armory-release-definitions.md new file mode 100644 index 0000000000..c2e12c138d --- /dev/null +++ b/kb/armory/General/armory-release-definitions.md @@ -0,0 +1,8 @@ +--- +title: Armory Release Definitions +--- + + +Armory Release Definitions have been moved to our docs site as a part of our re-organization.You can find the information here: +[https://docs.armory.io/docs/release-notes/release-definitions/](https://docs.armory.io/docs/release-notes/release-definitions/) + diff --git a/kb/armory/General/armory-support-information-hours-of-operation-slas-procedures.md b/kb/armory/General/armory-support-information-hours-of-operation-slas-procedures.md new file mode 100644 index 0000000000..68ad262bc8 --- /dev/null +++ b/kb/armory/General/armory-support-information-hours-of-operation-slas-procedures.md @@ -0,0 +1,264 @@ +--- +title: Armory Support Information (Hours of Operation/SLAs/Procedures) +--- + + + +Armory offers a variety of support options depending on your needs. With the resources below, customers should be able to find answers to any questions regarding their support contract and what kind of service-level agreements to be expected as a customer of Armory. There are many ways to contact support, but our first suggestion is always to check our documentation at [docs.armory.io](http://docs.armory.io/) or our [kb.armory.io](https://kb.armory.io) as we constantly seek to improve our customers’ self-service experience. +  +A self-service experience is impactful because support no longer becomes a bottleneck to resolving your issues. We will be here to assist if the need arises. Armory's Support focuses our time and attention on enhancing our documentation, knowledge base, and automation to ensure that a customer's overall experience with Armory is as streamlined and efficient as possible. +  +If you can’t find an answer to your question or concern at any given time, you can submit an issue with Armory Support.  Customers looking to register can do so by contacting their appointed [ServiceNow Service Portal Customer Administrators](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010240). Once registered, please submit tickets at [support.armory.io](https://support.armory.io/). + + +  + + +## Table of Contents +[Communication Methods](#mcetoc_1h4ot4buh10) +* [Phone Support](#mcetoc_1h4ot7pqp39)* [Armory CDaaS Support](#mcetoc_1h4ot7pqp3a)* [Product Feedback & Feature Requests](#mcetoc_1h4ot7pqp3b) +[Support Plans, Hours and SLAs](#mcetoc_1h4ot4buh11) +* [Armory offers the following Support Plans:](#mcetoc_1h4ot4buh12)* [SLA Matrix - Armory Essential Support for OSS Cases](#mcetoc_1h4ot4buh13)* [SLA Matrix - Armory CD Self-Hosted Support Cases](#mcetoc_1h4ot4buh14)* [SLA Matrix - Armory Global Premium Support Cases](#mcetoc_1h4ot4buh15)* [Armory Continous Deployment - Managed: High Availability and Disaster Recovery SLA Information](#mcetoc_1h4ot7pqp3c) +* [How is Priority Determined during Case Creation?](#mcetoc_1h4ot4buh16)* [Escalation Policies](#mcetoc_1h4ot4buh17) + + +  +Communication Methods +* Armory ServiceDesk Platform at [support.armory.io](https://support.armory.io/)* Slack for informal communications (a Slack message will not trigger our alerting system for Priority 0 cases; hence we recommend defaulting to issue submission via the Armory ServiceDesk). +Phone Support +On-demand **phone support is currently not an offering** at any Armory Support level, with one exception. If both parties deem a phone call necessary, we will provide an on-demand Zoom call or a scheduled Zoom call to ensure that the necessary context is captured in advance and necessary parties are involved.  +  +**The only exception is in specific cases for which customers require a P0 resolution**.  For more details, please refer to the following documentation available to [Armory Customers about P0 Case-handling](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010776#phone) (the document is only available to customers that have a login to our Support Portal). +  +#### Issue Creation +We highly suggest [enabling Armory Diagnostics](https://docs.armory.io/armory-enterprise/armory-admin/observe/diagnostics-configure/).  This allows our team to gather information about the current state of your environment and allows us to skip a few steps in our initial investigation, allowing for a quicker resolution.  Customers can also use and implement the [Armory Support Bundle within their environment](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010430) to provide the Customer's current configuration of their Spinnaker environment to the Support Team. +When submitting an Armory issue to the support team, please include as much detail as possible to ensure we have all the context to start our triaging. The more meaningful context provided, the quicker the path to resolution will be. Customers can also request a Zoom call from the Support Engineer assigned to their case. +  +Please visit our [document explaining the Case Submission best practices](https://kb.armory.io/s/article/Case-Submission-Best-Practices) page for more suggestions on optimizing case experiences. +  +The following are useful articles for users navigating our Service Desk Portal and a great place to start. +* [Support Portal User Registration](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010200)* [Password Resets](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010247)* [Creating Cases and Case Management in Service Now](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010136)* [Creating Change Requests in ServiceNow](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010267)* [ServiceNow Service Portal Customer Administrators](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010240)* [Technical Support Case Statuses](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010011)* [Case Escalations](https://armory.service-now.com/support?id=kb_article&sysparm_article=KB0010203)* [Critical](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010468)[ Notifications Mailing List](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010468) +Armory CD-as-a-Service Support (Hours of Operation/SLAs/Procedures) +Customers looking for Armory Continuous Delivery as a Service support can find more information [in the following KB article](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010598).   +Product Feedback & Feature Requests +We request that customers submit their feedback for product feedback and/or feature requests by following the [procedure outlined in this Knowledge Article](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010023).  We ask that customers submit this information in their own words to ensure the proper context is captured for our product teams.Populating the mandatory information will ensure that the request is prioritized properly. Customers are welcome to follow up with their Customer Success Manager on the status of the feedback/request at any time. +  +Support Plans, Hours and SLAs +**Armory offers the following Support Plans:** +**[Armory Essential Support for OSS](#SLAArmoryEssentialOSS) - ***Introduced July 2022* +**[Armory Continuous Deployment Self-Hosted Support](#SLAArmoryCDSH) - ***formerly Armory Enterprise Support* +**[Armory Global Premium Support](#SLAArmoryPremium) -** *formerly Armory Premium Support* +**Armory Continuous Deployment Managed Support - ***formerly Armory Managed Spinnaker Support*** - Based on Support Package Customer has purchased** +SLA Matrix - Armory Essential Support for OSS Cases + + +**Support Case Priority** + +**Response Time SLA** + +**Schedule** + +**Target Resolution Time** + +**Definition** + +**Example Case** + +P0 - Critical (Outage) + +1 Hour +11 x 5 Business Hours (EST/PST) +ASAP +Critical - Production instance is unavailable or unusable. No workaround/rollback. Support requires you to have dedicated resources available to work on the issue on an ongoing basis during your contractual hours.Spinnaker Production Server is showing 5xx errors for all users. +P1 - High + +12 Hours + +11 x 5 Business Hours (EST/PST) + +1 Week + +Production instance features are functionally impaired with no acceptable workaround/rollback. + +Pipelines are triggering errors and not completing successfully. The continuous delivery cycle is severely impacted. + +P2 - Medium + +32 Hours + +11 x 5 Business Hours (EST/PST) + +Best Effort + +Any issue affecting the Production or Dev instances that is non-critical and non-time-sensitive. + +A bug impacts the Spinnaker instance, but a workaround is available, and business resumes as usual. + +P3 - Low + +48 Hours + +11 x 5 Business Hours (EST/PST) + +Best Effort + +Questions or clarifications around features/docs, etc. No immediate impact to the business. + +Inquiries about configurations, How to requests, or non-production issues. + + +Please note that we can only officially support items that are shared across OSS Services **and **[Armory's ECM](https://docs.armory.io/armory-enterprise/feature-status/armory-enterprise-matrix/). +* Armory offers 11 Business Hours x 5 days a week Support based on the SLA agreements listed above. * SLAs for First Response is based on the priority of the case. We will prioritize resolution based on how severe the impact is. Our focus will **always** go to the highest priority cases.* Business Hours are defined as 8 hours between ***9 am-8 pm EST**** / ****6 am-5 pm PST.**** The SLA times listed in **Response Time SLA** are when customers can expect the first human response and acknowledgment of the case.* Armory Technical Support will make the best effort to resolve any issues as soon as possible. However, the Resolution Time SLAs are *not considered an expected time-to-resolution.** For organizations interested in Essential Support, please contact Armory and we can discuss how to customize the offering to meet your needs and future roadmap. +SLA Matrix - Armory CD Self-Hosted Support Cases + + +**Support Case Priority** + +**Response Time SLA** + +**Schedule** + +**Target Resolution Time** + +**Definition** + +**Example Case** + +P0 - Critical (Outage) + +1 Hour + +24 x 7 + +ASAP + +Critical - The Armory Production instance is unavailable or unusable. No workaround/rollback. Support requires you to have dedicated resources available to work on the issue on an ongoing basis during your contractual hours. + +Cannot log into the Armory Production instance. Armory Production is showing 5xx errors for all users. + +P1 - High + +8 Hours + +11 x 5 Business Hours (EST/PST) + +3 Business Days + +Armory Production instance features are functionally impaired with no acceptable workaround/rollback. + +Pipelines are triggering errors and not completing successfully. The continuous delivery cycle is severely impacted. + +P2 - Medium + +24 Hours + +11 x 5 Business Hours (EST/PST) + +Best Effort + +Any issue affecting the Armory instance which is non-critical and non-time-sensitive. + +A bug impacts the Armory instance, but a workaround is available, and business resumes as usual. + +P3 - Low + +48 Hours + +11 x 5 Business Hours (EST/PST) + +Best Effort + +Questions or clarifications around features/docs, etc. No immediate impact to the business. + +Customer wants to inquire about a product feature request.  + + +  +* Armory offers 11 Business Hours x 5 days a week Support (***24x7 for P0s***) based on the SLA agreements listed above. * Business Hours are defined as between ***9 AM - 8 PM EST** / **6 AM - 5 PM PST.**** The SLA times listed in **Response Time SLA** are when customers can expect the first human response and acknowledgment of the case.* Armory Technical Support will make the best effort to resolve any issues as soon as possible. However, the Resolution Time SLAs are *not considered an expected time-to-resolution.* +SLA Matrix - Armory **Global Premium **Support Cases + + +**Support Case Priority** + +**Response Time SLA** + +**Schedule** + +**Target Resolution Time** + +**Definition** + +**Example Case** + +P0 - Critical (Outage) + +30 Min + +24 x 7 + +ASAP + +Critical - The Armory Production instance is unavailable or unusable. No workaround/rollback. Support requires you to have dedicated resources available to work on the issue on an ongoing basis during your contractual hours. + +Cannot log into the Armory Production instance. Armory Production is showing 5xx errors for all users. + +P1 - High + +4 Hours + +24 x 5 Business Hours + +3 Business Days + +Armory Production instance features are functionally impaired with no acceptable workaround/rollback. + +Pipelines are triggering errors and not completing successfully. The continuous delivery cycle is severely impacted. + +P2 - Medium + +12 Hours + +24 x 5 Business Hours + +Best Effort + +Any issue affecting the Armory instance which is non-critical and non-time-sensitive. + +A bug impacts the Armory instance, but a workaround is available, and business resumes as usual. + +P3 - Low + +24 Hours + +24 x 5 Business Hours + +Best Effort + +Questions or clarifications around features/docs, etc. No immediate impact to the business. + +Customer wants to inquire about a product feature request.  + + +  +* Armory offers 24 Business Hours x 5 days a week Support (***24x7 for P0s***) based on the SLA agreements listed above. * Business Hours are defined as ***7 PM Sunday EST/4 PM Sunday PST - 8 PM Friday EST / 5 PM Friday PST**** The SLA times listed in **Response Time SLA** are when customers can expect the first human response and acknowledgment of the case.* Armory Technical Support will make the best effort to resolve any issues as soon as possible. However, the Resolution Time SLAs are *not considered an expected time-to-resolution.* +Armory Continous Deployment - Managed: High Availability and Disaster Recovery SLA Information +For information on Armory Continous Deployment Managed environments and High Availability and Disaster Recovery SLA information, please visit the following Knowledge Article:[https://support.armory.io/support?id=kb_article&sysparm_article=KB0010626](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010626) +  +How is Priority Determined during Case Creation? +The customer case form will include the following fields to determine the priority on our ServiceDesk. We prevent the change and manual priority adjustment to ensure a consistent customer experience. Our P0s are linked to a global paging system and WILL wake up any on-call team member, so we want to ensure that if a P0 is being submitted, it requires immediate attention. +Environment  +* Production* Non-Production +Impact +* Minor* Moderate* Significant* Critical (Outage) +Workaround/Rollback Available +* Yes* No +Escalation Policies + +A case's priority shall remain Read-Only to enforce a consistent customer experience. We want every interaction to be consistent, and we want to ensure that resources are distributed in a fair and equal manner.  +  +For Cases where there is a severe blocker to deploying the Armory services, or a critical timeline/deadline is approaching, the customer may request an escalation. In the case an escalation is required, please follow one of the following steps: +* For cases requiring an increase in status to a P0 (Production System, Critical outage) status, [please use the following documentation](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010776) to contact our Support Team.* For cases requiring an increase/escalated response that are not P0 incidences, please follow our C[ase Escalations Procedure](https://armory.service-now.com/support?id=kb_article&sysparm_article=KB0010203) + + +  + diff --git a/kb/armory/General/authenticate-against-jenkins.md b/kb/armory/General/authenticate-against-jenkins.md new file mode 100644 index 0000000000..958ad9f06e --- /dev/null +++ b/kb/armory/General/authenticate-against-jenkins.md @@ -0,0 +1,30 @@ +--- +title: Authenticate Against Jenkins +--- + +## Introduction +If you have are trying to authenticate Spinnaker against a Jenkins master that has some third party authentication set up, you have to create an API token for Spinnaker to authenticate against Jenkins. + +## Prerequisites +Access to Jenkins Administration + +## Instructions +* Log into Jenkins* Click on your username (in the top right)* Click on “Configure” (on the left)* Under the “API Token” section, click on “Add new Token”, and “Generate” and record the generated token* Record your username (should be in the URL for the current page - if you’re at [https://jenkins.domain.com/user/justinrlee/configure](https://jenkins.domain.com/user/justinrlee/configure), then the username is, as an example, ```janesmith```)Add the Jenkins master to Spinnaker with this: +``` +hal config ci jenkins enable + echo | hal config ci jenkins master add \ + --address https://jenkins.domain.com \ + --username \ + --password + + # Password will be read from stdin​ +``` +For example, if the generated token is ```1234567890abcdefghijklmnopqrstuvwx```, and your username is ```janesmith``` and you want to identify the Jenkins master as ```jenkins-prod```, then you’d run something like this: +``` + echo 1234567890abcdefghijklmnopqrstuvwx | hal config ci jenkins master add jenkins-prod \ + --address https://jenkins.domain.com \ + --username janesmith \ + --password​ +``` +* Also, don’t forget to configure the CSRF stuff: [https://www.spinnaker.io/setup/ci/jenkins/#configure-jenkins-and-spinnaker-for-csrf-protection](https://www.spinnaker.io/setup/ci/jenkins/#configure-jenkins-and-spinnaker-for-csrf-protection) + diff --git a/kb/armory/General/authentication-timeout-results-in-401-error.md b/kb/armory/General/authentication-timeout-results-in-401-error.md new file mode 100644 index 0000000000..185ba56872 --- /dev/null +++ b/kb/armory/General/authentication-timeout-results-in-401-error.md @@ -0,0 +1,18 @@ +--- +title: Authentication Timeout Results in 401 Error +--- + +## Issue +You may encounter the following error when a user attempts to login: +``` +{ + error: "Unauthorized", + message: "Authentication Failed: Error validating SAML message", + status: 401, + timestamp: 1553109495710 +} +``` + +## Cause +This issue occurs because of a known issue in Spring. Spinnaker does not accept SAML tokens signed by a user who authenticated more than 2 hours ago even if your authentication system allows it. + diff --git a/kb/armory/General/authorization-configuration-with-file-based-role-definitions.md b/kb/armory/General/authorization-configuration-with-file-based-role-definitions.md new file mode 100644 index 0000000000..2694792f6b --- /dev/null +++ b/kb/armory/General/authorization-configuration-with-file-based-role-definitions.md @@ -0,0 +1,48 @@ +--- +title: Authorization Configuration with File Based Role Definitions +--- + +## Introduction +One of the nice things of Spinnaker is that authentication (login, “authn”) and authorization (permissions, “authz”) are loosely coupled.Although usually you use the same provider for both, like GitHub or LDAP, it’s possible to have different providers.After you have [authentication working](https://www.spinnaker.io/setup/security/authentication/) ([hal commands](https://github.com/spinnaker/halyard/blob/master/docs/commands.md#hal-config-security-authn)), you can enable file based authorization to see Spinnaker’s permissions logic working. This is ideal for changing a user’s roles and defining different roles without actually changing your production authorization role provider, since this usually not only requires knowledge that a Spinnaker administrator may not have readily available, but also it’s usually performed by specialized security teams under strict rules.  + +## Prerequisites +N/A + +## Instructions +File based authorization is basically a yaml file that has the mappings between user names and role names: +``` +users: + - username: foo + roles: + - bar + - baz + - username: batman + roles: + - crimeFighter + - jokerJailer + - username: robin + roles: [] + - username: joker +``` + +Roles don’t need to be defined elsewhere, they’re automatically created based on these mappings. +First you need to know the actual user names provided by the authentication mechanism, for this you can take a look at the upper right corner of Spinnaker’s UI to find the exact user name of the logged in user: + +Then you need to create a file like the one above defining username-role mappings, and enable file based authorization in halyard: +``` +hal config security authz file edit --file-path ${ROLES_FILE} +hal config security authz edit --type FILE +hal config security authz enable +hal deploy apply +``` + +To verify that this configuration is working, there should be no errors in fiat pod, and going to an application configuration section under Application Attributes, should show a new field called ```Permissions```: + +The dropdown will show *the roles available to the logged in user*, it won’t show all the roles defined in the file. Also, you may need to clear cookies, logout and login or use an incognito window in order to see this new field available. + +Once you have authorization in place, you can continue with: +* [Restricting access to applications](https://www.spinnaker.io/setup/security/authorization/#applications) +* [Restricting access to cloud accounts](https://www.spinnaker.io/setup/security/authorization/#accounts) +* [Defining service accounts for automatic pipeline triggers](https://www.spinnaker.io/setup/security/authorization/service-accounts/) +Remember that if you’re configuring these restrictions, when you move to the real role provider its role names should exactly match the ones you defined in the file. + diff --git a/kb/armory/General/auto-cleanup-pipeline-executions-of-specific-applications-in-spinnaker-.md b/kb/armory/General/auto-cleanup-pipeline-executions-of-specific-applications-in-spinnaker-.md new file mode 100644 index 0000000000..a441e82015 --- /dev/null +++ b/kb/armory/General/auto-cleanup-pipeline-executions-of-specific-applications-in-spinnaker-.md @@ -0,0 +1,35 @@ +--- +title: Auto cleanup pipeline executions of specific applications in Spinnaker +--- + +## Introduction +Admins of Armory CDSH / OSS Spinnaker instances may want to clean up older pipeline executions and retain executions only for a certain period to free space on the database and provide a better user experience.  +This can help with managing database space and can also improve UI at times when loading historical information. + +## Prerequisites +An Operational Spinnaker instance, with Administration access + +## Instructions +To clean up pipeline executions after a certain period, apply the below config to Orca’s profile settings in the ```SpinnakerConfig.yml``` file``` spinnakerConfig.profiles.orca```for Operator-based installations and ```orca-local.yml ```in the case of Halyard-based installations +pollers: + +``` + oldPipelineCleanup: + enabled: true # This enables old pipeline execution cleanup (default: false) + intervalMs: 3600000 # How many milliseconds between pipeline cleanup runs (default: 1hr or 3600000) + thresholdDays: 60 # How old a pipeline execution must be to be deleted (default: 30) + minimumPipelineExecutions: 10 # How many executions to keep around (default: 5) +``` +The above setting applies to pipeline executions under all the applications. If Administrators would like to have different thresholds for different sets of applications, please include the applications under ```exceptionalApplications``` as shown below +``` + pollers: + oldPipelineCleanup: + enabled: true + thresholdDays: 30 + minimumPipelineExecutions: 30 + exceptionalApplicationsThresholdDays: 5 + exceptionalApplications: + - application1 + - application2 + - application3 +``` \ No newline at end of file diff --git a/kb/armory/General/autogenerated-applications-cannot-be-deleted-w--onlyspinnakermanaged-set-to-false.md b/kb/armory/General/autogenerated-applications-cannot-be-deleted-w--onlyspinnakermanaged-set-to-false.md new file mode 100644 index 0000000000..add300afcb --- /dev/null +++ b/kb/armory/General/autogenerated-applications-cannot-be-deleted-w--onlyspinnakermanaged-set-to-false.md @@ -0,0 +1,11 @@ +--- +title: Autogenerated Applications Cannot be Deleted w/ onlySpinnakerManaged set to false +--- + +## Issue +When enabling ```onlySpinnakerManaged: false``` applications will be generated to populate the Spinnaker console based on the Kubernetes environment and account assigned in the Accounts declaration of either the **Halyard** ([https://docs.armory.io/docs/armory-admin/kubernetes-account-add/#how-spinnaker-operates-when-deploying-to-kubernetes-targets](https://docs.armory.io/docs/armory-admin/kubernetes-account-add/#how-spinnaker-operates-when-deploying-to-kubernetes-targets)) or **Operator** ([https://docs.armory.io/docs/installation/operator-reference/providers/#account-parameters-8](https://docs.armory.io/docs/installation/operator-reference/providers/#account-parameters-8)) configuration.The ingestion and creation of these **Autogenerated Applications Placeholders** is based on the naming convention of resources in KubernetesAttempting to manage or delete the applications can sometimes lead to the following displayed error where ```The application tenant has not been configured``` is displayed and the application cannot be deleted + + +## Cause +The way this legacy item works is if set to false, is that it will then attempt configure applications for resources in Kubernetes, based on what the Spinnaker environment is able to see. However, this feature can end up picking up information incorrectly and attempt to populate the Spinnaker environment with Autogenerated Applications Placeholders that were unintended.  The application doesn't actually exist, and then the Autogenerated Applications Placeholders are misconfigured and do not correlate to resources properly.  If changes are also made to the Kubernetes cluster, this will further complicate things by causing the error message aboveMaking things more difficult is that MySQL doesn't have an automatic cleanup to remove these misconfigured Autogernerated applications, unlike in REDIS, which can have an auto expiry set that will respect the change of the ```onlySpinnakerManaged``` value back to ```true```. + diff --git a/kb/armory/General/aws-billing-billing-budget-alerts-and-cost-explorer.md b/kb/armory/General/aws-billing-billing-budget-alerts-and-cost-explorer.md new file mode 100644 index 0000000000..5176e4138a --- /dev/null +++ b/kb/armory/General/aws-billing-billing-budget-alerts-and-cost-explorer.md @@ -0,0 +1,98 @@ +--- +title: AWS Billing - Billing, Budget Alerts and Cost Explorer +--- + + +The following is quick information about work that Technical Support Engineering have done in Budgets and Cost Explorer, and how to use both to review billing in AWS + +A quick aside, in order to get billing access, the admin will need to make sure that billing access is provided to the IAM Account as a policy, Console access is provided, and most importantly, as the root in the main account, a[ selection needs to be made in the account preferences to allow IAM users to have access to the billing](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/control-access-billing.html#ControllingAccessWebsite-Activate).  If your account doesn’t have this access, you will probably need to talk to @Black Mountain. Black Mountain + + +## How to get the monthly bill + +If your account is a part of the “My Organization” for Armory, you will not be able to receive a PDF of the monthly bill/invoice for the account.  You can only receive a full bill for the entire organization (This is a feature request I have been requesting for almost a year now - Kinnon) +  + +For now, you will have to manually download the “final” invoice +* Log in to the account +* Go to **Billing → Bills** +* You can select the monthly bills from the dropdown in the upper left corner. + (1)  Please note that you can see the current month’s bill cost, but note that it will fluctuate due to changes that can happen in the month Each service charge is laid out at (2) + * You can expand each section + * Each service is broken down by a rate and how many hours/amount of rate the service was used.  For example, EC2 by number of hours, Bandwidth by GB + * Please note that regional costs are also calculated.  e.g. Our EC2 costs for N. Virginia will be separated from Oregon + * Also, because different months have different amount of days, your billing will reflect these subtleties since it is based on number of hours +* You can download a CSV, but I would recommend a PDF because it’s more legible.  Click on Print (3) → and select to save as a PDFBill location + * For Tech Support Eng, Save to our DropBox Billing location:[https://www.dropbox.com/work/Armory%20Tribe%20Team%20Folder/Customer%20Success/Technical%20Support/AWS%20-%20Bill](https://www.dropbox.com/work/Armory%20Tribe%20Team%20Folder/Customer%20Success/Technical%20Support/AWS%20-%20Bill) + +  + + +## How Budget Alerts - Monthly Estimate Cost is set up + +Budget Alerts are a good way get an estimation on whether your monthly spend is going to exceed your budget line.  Please keep in mind that the report prediction requires several weeks of data before it can start providing accurate information, and also if your account is new, you are probably making lots of changes rapidly to get the account set up + +There are several types of Budgets and Alerts besides for cost, including ones to look at compute usage, RI usage, and Savings Plans.  Below is the steps for a monthly Dollar Cost Estimate Budget Alert + +1. If you have not created an SNS notification and topic, it is recommended that you do so first.  This will allow you to send alerts to all those who subscribe to your SNS topic, and also allow you to create a Zapier if you want to have Slack notifications + * SNS Topic Creation: [https://docs.aws.amazon.com/sns/latest/dg/sns-getting-started.html](https://docs.aws.amazon.com/sns/latest/dg/sns-getting-started.html) + * Addition to SNS Access Policy for the Budget Alert - Add the following so Budget Alerts can send notification to SNS + * Go to your SNS Topic - Click Edit on the Upper Right Corner + * Click on the Access Policy section and expand it + * Add the following highlighted section, making changes to the arn section to match your Topic’s arn +``` +... +}, +{ + "Sid": "AWSBudgets-notification-1", + "Effect": "Allow", + "Principal": { + "Service": "budgets.amazonaws.com" + }, + "Action": "SNS:Publish", + "Resource": "arn:aws:sns:RegionTopicIs:AccountTopicisIn:TopicName" +}, +{ + "Sid": "__console_sub_0", +...​ +``` + + + +2. Choose your integrator with Slack: + * **Zapier** - SNS to Slack notification - [+Zapier AWS SNS to Slack Integration](https://paper.dropbox.com/doc/YuCVMrOWU9ea0Rm7HsxAP)  + * **AWS Chatbot (Beta)** - [https://aws.amazon.com/chatbot/faqs/](https://aws.amazon.com/chatbot/faqs/) +3. Go to **Billing → Budget**Click on **Create Budget** → Select the kind of Budget you will be making.  You will have four choices: + * Cost Budget - For Dollar Costs (the one we will select for this example) + * Usage Budget - For monitoring usage thresholds of instances* Reservation Budget - Track the RI Utilization usage + * Savings Plans Budget - Tracking the usage and coverage of Savings Plans +4. Click on the **Set your Budget Button** and fill the details + * Name: e.g. Monthly Tech Support Costing Budget + * Period: Monthly* Budget Effective Dates: Recurring Budget + * Start Month: Current month (cannot backdate) + * Budget Amount: Fixed (Can do Monthly budget planning, which allows you to set month by month increases to the budget) + * Budgeted amount: Provide your budget for the account (note if you are planning on being alerted when projected cost reaches a threshold, you should factor this into your budget planning) + * Budget Parameters - Leave default, but you will need to check Refunds and Credits  +6. Click on Configure Alerts → Fill in the Alert Information + * Send alert based on: Select Forecasted costs (note: at least 5 weeks are required to generate an accurate Forecast) + * Alert Threshold: Set by percentage, or fixed dollar amount* Check Mark: Notify via Amazon Simple Notification Service (SNS) topic + * Add the ARN of the SNS topic created in Step 1 + + +## How to use Cost Explorer + +Cost explorer is a powerful tool for you to filter and look at various costs and utilization on your account, including how much of your Savings Plans are being used, or your Reserved Instances + +A few items to note: +* The most recent two days in Cost Explorer are never accurate.  They will either over-report or under report.  You will need to allow for the last two days to “settle” before reading into them* Your filters may end up filtering your data, so please be careful of any that you have selected* Tagging is your friend.  If you have tagged your information sufficiently, then you will be able to sort and organize via the tagging any resources and costs to individuals + +* Log in to your AWS Account* Go to Cost Explorer → Click on Launch Cost Explorer* From here you can build your various reports + +### **Report to Show Cost Usage** **(Example**** is for Daily, MTD)** +**** +Click on the Drop Down for the Date Range (1) and select the appropriate time +* 3M, 6M, 1Y will not include partial data from the current month, all others will* You can enable forecasting by using the + values, but they are not usually useful* I usually chose MTD (Month to Day) to start with, as you are usually trying to find your current progress* You can customize the days by clicking on the Calendars +* Select Daily from the units drop down (2)* Clear all filters (3) for now (you can customize them as necessary later)* Select Sort by Service (4).  You can chose different grouping choices depending on what you will need* You will then be able to see a breakdown on all services, per day, and their costs.  (5) Please note again, that the most recent two days will be inaccurate, and you should not count on them* Feel Free to Save this report to be used later (see upper left corner to save the report) +  + + diff --git a/kb/armory/General/aws-image-caching-issues-for-certain-accounts,-when-multiple-accounts-are-defined.md b/kb/armory/General/aws-image-caching-issues-for-certain-accounts,-when-multiple-accounts-are-defined.md new file mode 100644 index 0000000000..0b58e363fd --- /dev/null +++ b/kb/armory/General/aws-image-caching-issues-for-certain-accounts,-when-multiple-accounts-are-defined.md @@ -0,0 +1,12 @@ +--- +title: AWS image caching issues for certain accounts, when multiple accounts are defined +--- + +## Issue +The stage ```Find image from Cluster``` fails to find images that that belong to certain AWS accounts on Spinnaker versions earlier than 2.23.3, when multiple accounts are defined.   + + +## Cause +There was an issue with Spinnaker versions 2.23 and earlier where Clouddriver fails to cache images that belong to the first account (alphabetically) for each region. +For example, in a region (e.g. us-east-1) with accounts named ```Edith```, ```Milton```, and ```Pixel```, ```Edith```’s images do not get cached. + diff --git a/kb/armory/General/aws-lambda-&-custom-webhook-stages.md b/kb/armory/General/aws-lambda-&-custom-webhook-stages.md new file mode 100644 index 0000000000..abf0002bf6 --- /dev/null +++ b/kb/armory/General/aws-lambda-&-custom-webhook-stages.md @@ -0,0 +1,198 @@ +--- +title: AWS Lambda & Custom Webhook Stages +--- + +## Introduction +## Note: This document is obsolete. +As of **Spinnaker 1.23 (Armory 2.23)**, there is now a **Lambda plugin** available - see the [plugin GitHub repo](https://github.com/spinnaker-plugins/aws-lambda-deployment-plugin-spinnaker) for documentation. +**Published Feb 2020**, Enabling Lambda in the UI: Refer to the AWS blogpost for [enabling Lambda with Spinnaker](https://aws.amazon.com/blogs/opensource/how-to-integrate-aws-lambda-with-spinnaker/) +  +**-- Original KB Follows --** +Back in December of 2018, AWS added supported for Lambda and it was released in Spinnaker OSS v1.12. The challenge, however, was that there was no UI (Deck) components to support the usage of Lambda. Instead, a [README.md](https://github.com/spinnaker/clouddriver/blob/master/clouddriver-lambda/README.md) was published that specified the API to make changes to Lambda. +This document will show you how to create a custom webhook stage that utilizes the Lambda API built into Clouddriver. This document also assumes that you have configured an AWS account. If you need more instructions on configuring your AWS account please refer to either [Deploying to AWS from Spinnaker (using IAM instance roles)](https://docs.armory.io/spinnaker-install-admin-guides/add-aws-account-iam/) or [Deploying to AWS from Spinnaker (using IAM credentials)](https://docs.armory.io/spinnaker-install-admin-guides/add-aws-account/) +**Note: **This is a proof of concept implementation and is not recommended for production use! + +## Prerequisites +Access to your AWS Environment + +## Instructions + +### Enable AWS Lambda in Spinnaker +First, we need to enable Lambda by adding a ```clouddriver-local.yml``` file to your hal config profiles directory e.g. ```.hal/default/profiles/clouddriver-local.yml``` (As of Halyard OSS version 1.20, there is no support for adding Lambda configurations via hal commands.)  +```` +aws: + lambda: + enabled: true + accounts: + - name: aws + lambdaEnabled: true + providerVersion: V1 + accountId: '555692138000' + regions: + - name: us-east-1 + - name: us-west-2 + assumeRole: role/awaylambda05242019-managedrole +```` +You can check your configuration by running ```hal deploy apply``` and check the clouddriver logs (e.g. ```kubectl logs -f -n spinnaker spin-clouddriver-xxxxx```) for any start-up errors. Refer to the [Debugging](https://kb.armory.io/aws/AWS-Lambda-HowTo/#debugging) section for checking your deployment.  +Adding Custom Webhook Stage +Next, we need to add a custom webhook stage. Alternatively, you could use a simple webhook stage. Following this guide on [Custom Webhook Stages](https://www.spinnaker.io/guides/operator/custom-webhook-stages/), you should add ```.hal/default/profiles/orca-local.yml``` with the following content: + +```` +webhook: + preconfigured: + - label: Lambda - Get Functions + type: lambdaGetFunctions + enabled: true + description: Get Lambda Functions + method: GET + url: http://spin-clouddriver:7002/functions + customHeaders: + Accept: + - "application/json" + - label: Lambda - Update Function Code + type: lambdaUpdateFunctionCode + enabled: true + description: Update Lambda Function Code + method: POST + url: http://spin-clouddriver:7002/aws/ops/updateLambdaFunctionCode + customHeaders: + Accept: + - "application/json" + Content-Type: + - "application/json" + payload: |- + { + "credentials": "${#root['parameterValues']['account']}", + "region": "${#root['parameterValues']['region']}", + "functionName": "${#root['parameterValues']['functionName']}", + "s3bucket": "${#root['parameterValues']['bucketname']}", + "s3key": "${#root['parameterValues']['key']}", + "publish": "${#root['parameterValues']['publish']}" + } + parameters: + - label: Spinnaker Account Name + name: account + type: string + - label: Region + name: region + type: string + - label: Function Name + name: functionName + type: string + - label: S3 Bucket Name + name: bucketname + type: string + - label: S3 Key + name: key + type: string + - label: Publish + name: publish + type: string + - label: Lambda - Update Function Configuration + type: lambdaUpdateFunctionConfig + enabled: true + description: Update Lambda Function Configuration + method: POST + url: http://spin-clouddriver:7002/aws/ops/updateLambdaFunctionConfiguration + customHeaders: + Accept: + - "application/json" + Content-Type: + - "application/json" + payload: |- + { + "region": "${#root['parameterValues']['region']}", + "functionName": "${#root['parameterValues']['functionName']}", + "description": "${#root['parameterValues']['description']}", + "credentials": "${#root['parameterValues']['account']}", + "role": "${#root['parameterValues']['roleARN']}", + "timeout": "${#root['parameterValues']['timeout']}" + } + parameters: + - label: Region + name: region + type: string + - label: Function Name + name: functionName + type: string + - label: Description + name: description + type: string + - label: Spinnaker Account Name + name: account + type: string + - label: Role ARN + name: roleARN + type: string + - label: Timeout (secs) + name: timeout + type: string +```` + +You might notice that the ```parameterValues``` are being referenced with a ```#root``` helper function. This is to help ensure that Orca can evaluate the expressions using the parameter values from within the stage. +Run ```hal deploy apply``` and you should see a new orca pod. And if you want to verify you can exec into orca, and find the ```orca-local.yml``` file under ```/opt/spinnaker/config/l``` + +### Creating your Pipeline +After you’ve deployed your changes to orca, you should now be able to see the new stages when configuring your pipeline. Simply select the stage, and provide the values: example below: +[](https://cl.ly/1d686d0f23ed) +Referencing values from ```Get Functions``` +You can use SpEL to get values from the response to the ```Lambda - Get Functions``` stage. Here is an example SpEL expression to get a function name. +```${#stage("Lambda - Get Functions").context.webhook.body[0].functionName}``` +Check the ‘source’ of the pipeline to see the contents of the webhook - or reference the Get Functions output below in the [curl](https://kb.armory.io/aws/AWS-Lambda-HowTo/#curl) section.  +Debugging +For debugging your aws lambda setup, check the clouddriver logs: e.g. ```kubectl logs -f -n spinnaker spin-clouddriver-xxxxx``` For debugging your custom webhook, check the logs for both orca and clouddriver. +Debugging Tools +You can deploy a [debugging pod](https://github.com/armory/docker-debugging-tools) into the same cluster and namespace where Spinnaker lives. This pod contains useful tools that are missing in the Spinnaker micro-services to help you debug your deployment. Check out the [debugging-tools repo](https://github.com/armory/docker-debugging-tools). +curl +With the debugging pod deployed, you can exec into the pod and run the following calls to test if Lambda is properly configured in Spinnaker. The key point to note here is that the base URL is pointing to clouddriver (e.g. [http://spin-clouddriver:7002](http://spin-clouddriver:7002/)) + + +```curl -X GET --header 'Accept: application/json' 'http://spin-clouddriver:7002/functions'; | jq .``` + + +This command requests for the lambda functions in the AWS account and pipes the output to jquery to create a nice human readable format. +You should expect an output similar to the following: +```` +{ + "account": "aws", + "codeSha256": "gxyL5FY9DLxavA+MMBAFrsnhL7SRL/CiSwciLGGWBCI=", + "codeSize": 262, + "description": "", + "eventSourceMappings": [], + "functionArn": "arn:aws:lambda:us-west-2:555692138000:function:helloArmoryfx", + "functionName": "helloArmoryfx", + "handler": "index.handler", + "lastModified": "2019-05-28T15:12:40.330+0000", + "layers": [], + "memorySize": 128, + "region": "us-west-2", + "revisionId": "4a799ab5-9cf2-491c-a07f-129247a27b4e", + "revisions": { + "4a799ab5-9cf2-491c-a07f-129247a27b4e": "$LATEST" + }, + "role": "arn:aws:iam::555692138000:role/service-role/helloArmoryfx-role-y9j7d2ll", + "runtime": "nodejs10.x", + "timeout": 10, + "tracingConfig": { + "mode": "PassThrough" + }, + "version": "$LATEST" +} +```` + +From here you can test other commands such as Lambda update code. + +```` +curl -X POST \ + http://spin-clouddriver:7002/aws/ops/updateLambdaFunctionCode \ + -H 'Accept: application/json' \ + -H 'Content-Type: application/json' \ + -d '{ + "region": "us-west-2", + "functionName": "helloArmoryfx", + "credentials": "aws", + "s3bucket": "armory-sales-away", + "s3key": "lambdacode/lambdacode-v0.1.zip", + "publish": "true" + }' +```` diff --git a/kb/armory/General/aws-lambda-plugin-stage-does-not-populate-the-existing-functions-.md b/kb/armory/General/aws-lambda-plugin-stage-does-not-populate-the-existing-functions-.md new file mode 100644 index 0000000000..a700b82900 --- /dev/null +++ b/kb/armory/General/aws-lambda-plugin-stage-does-not-populate-the-existing-functions-.md @@ -0,0 +1,10 @@ +--- +title: AWS Lambda Plugin stage does not populate the existing Functions +--- + +## Issue +Functions that had been created in Spinnaker, and cached by Clouddriver, do not get populated by the AWS Lambda Plugin in the Spinnaker UI. The UI would show the following instead: + +## Cause +This is a known issue, and impacts configuring AWS Lambda on a new AWS account. The issue stems from accounts getting merged by index and not by name. The Github issue is noted at: [https://github.com/spinnaker/spinnaker/issues/6271](https://github.com/spinnaker/spinnaker/issues/6271)The Armory documentation about the issue can be found here:[https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-26-1/#lambda-ui-issue](https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-26-1/#lambda-ui-issue) + diff --git a/kb/armory/General/aws-rds-certificate-update.md b/kb/armory/General/aws-rds-certificate-update.md new file mode 100644 index 0000000000..b05e249d9f --- /dev/null +++ b/kb/armory/General/aws-rds-certificate-update.md @@ -0,0 +1,39 @@ +--- +title: AWS RDS Certificate Update +--- + +## Introduction +Amazon announced that they are replacing their [SSL/TLS certificates for a variety of data services](https://aws.amazon.com/blogs/aws/urgent-important-rotate-your-amazon-rds-aurora-and-documentdb-certificates/). +This issue only impacts you if you have SSL/TLS enabled for Spinnaker’s connections to Aurora. Spinnaker services that can use Aurora are CloudDriver, Orca, and Front50. The update might require restarting your Aurora instance, which causes your Spinnaker deployment to be temporarily unavailable.During the downtime, any services that use Aurora will display errors. This is expected until the database is available again. +Aurora picks up the new certificate after a restart. You can update your certificates during planned downtime or immediately. + +## Prerequisites +N/A + +## Instructions +During Pre-Planned Set Maintenance Window (set within your RDS) +Run the following command via CLI: +``` +aws rds modify-db-instance --db-instance-identifier database-1 \ +--ca-certificate-identifier rds-ca-2019 +During the next maintenance window, the update occurs.  +Manually Triggering Update Immediately (SSL) +``` + +### You can update the certificates immediately, which will result in downtime.  Recommendation is to plan for an outage as a part of your maintenance. +Run the following command: +``` +aws rds modify-db-instance --db-instance-identifier database-1 \ +--ca-certificate-identifier rds-ca-2019 --apply-immediately +``` + +Manually Triggering Update Immediately (No SSL) +If you do not use SSL, you can update your certificates without restarting the database. +Run the following command: +``` +aws rds modify-db-instance --db-instance-identifier database-1 \ +--ca-certificate-identifier rds-ca-2019 --no-certificate-rotation-restart +``` + + + diff --git a/kb/armory/General/bake-cloudfoundry-manifest-stage-fails-without-an-error-message.md b/kb/armory/General/bake-cloudfoundry-manifest-stage-fails-without-an-error-message.md new file mode 100644 index 0000000000..46beb204fc --- /dev/null +++ b/kb/armory/General/bake-cloudfoundry-manifest-stage-fails-without-an-error-message.md @@ -0,0 +1,20 @@ +--- +title: Bake CloudFoundry Manifest Stage Fails without an error message +--- + +## Issue +When a pipeline fails in the Bake CloudFoundry manifest stage, there are no errors displayed in Deck. +In an example scenario, where Clouddriver is unable to download the artifact from Bitbucket, the Rosco logs show the following: + +Clouddriver logs show the following, for example: +``` +2021-04-22 14:24:00.966 WARN 1 — [0.0-7002-exec-6] c.n.s.k.w.e.GenericExceptionHandlers : Handled error in generic exception handler +com.netflix.spinnaker.clouddriver.artifacts.exceptions.FailedDownloadException: Unable to download the contents of artifact Artifact(type=......t/file, customKind=false, name=manifests/uat/vars.qa.3z.yml, version=null, location=null, reference=https://bitbucketdc-cluster07.jpmchase.net/projects...../c360-income-businessrule-service/raw/manifests/uat/v.......yml?at=refs/heads/feature/spinnaker, metadata={}, artifactAccount=bitbucket, provenance=null, uuid=null): Received 404 status code from ....... +at com.netflix.spinnaker.clouddriver.artifacts.config.SimpleHttpArtifactCredentials.download(SimpleHttpArtifactCredentials.java:51) +``` +  +The user has to obtain error information from the Rosco and Clouddriver logs, as the UI does not state the error message: + +## Cause +The error pertaining to the Bake CloudFoundry Manifest Stage Failing is not seen in the Deck UI. + diff --git a/kb/armory/General/bake-or-deployment-stage-timeouts-caused-by-external-factors.md b/kb/armory/General/bake-or-deployment-stage-timeouts-caused-by-external-factors.md new file mode 100644 index 0000000000..7b7fd408c4 --- /dev/null +++ b/kb/armory/General/bake-or-deployment-stage-timeouts-caused-by-external-factors.md @@ -0,0 +1,89 @@ +--- +title: Bake or Deployment Stage timeouts caused by external factors +--- + +## Issue +Bake and deployment stages depend on applications and environments external to Spinnaker and are therefore susceptible to externally induced timeout errors. +At time of writing, Spinnaker has no way to indicate what the problem might be and will simply fail the stage with a generic timeout error. We can, however, tell it's an external problem if error similar to the below is observed when viewing the execution details JSON, the key field being ```"kind":"NETWORK"```: +```type":"createServerGroup","name":"MyStageName","startTime":1636692612420,"endTime":1636692675134,"status":"TERMINAL","context":{"rollback":{"onFailure":false},"exception":{"exceptionType":"RetrofitError","shouldRetry":false,"details":{"kind":"NETWORK","error":"timeout","errors":[]},``` +Depending on the stage type, there will be additional pointers in the Rosco and/or Orca logs, including but not limited to the below. +#### Orca +In this exception we can see the socket closed which tends to indicate an upstream service closed the connection on Spinnaker: +``` +2021-11-12 09:41:50.276 INFO 1 --- [ handlers-20] c.n.s.orca.clouddriver.KatoRestService : java.net.SocketTimeoutException: timeout + at okio.Okio$4.newTimeoutException(Okio.java:232) + at okio.AsyncTimeout.exit(AsyncTimeout.java:286) + at okio.AsyncTimeout$2.read(AsyncTimeout.java:241) + at okio.RealBufferedSource.indexOf(RealBufferedSource.java:358) + at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:230) + at okhttp3.internal.http1.Http1ExchangeCodec.readHeaderLine(Http1ExchangeCodec.java:242) + at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.java:213) + at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.java:115) + at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:94) + at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) + at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) + +Caused by: java.net.SocketException: Socket closed + at java.base/java.net.SocketInputStream.read(SocketInputStream.java:183) + at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140) + at okio.Okio$2.read(Okio.java:140) + at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) + ... 85 more +``` +This is an example of a ServerGroup creation timeout, but again is indicating an upstream problem: + +``` +2021-11-12 09:42:11.318 ERROR 1 --- [ handlers-20] c.n.s.orca.q.handler.RunTaskHandler : Error running CreateServerGroupTask for pipeline[[pipeline_ID]] +retrofit.RetrofitError: timeout + at retrofit.RetrofitError.networkError(RetrofitError.java:27) + at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:395) + at retrofit.RestAdapter$RestHandler.invoke(RestAdapter.java:240) + at com.sun.proxy.$Proxy171.requestOperations(Unknown Source) + at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) + at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) + at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + at java.base/java.lang.reflect.Method.invoke(Method.java:566) + +Caused by: java.net.SocketTimeoutException: timeout + at okio.Okio$4.newTimeoutException(Okio.java:232) + at okio.AsyncTimeout.exit(AsyncTimeout.java:286) + at okio.AsyncTimeout$2.read(AsyncTimeout.java:241) + at okio.RealBufferedSource.indexOf(RealBufferedSource.java:358) + at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:230) + at okhttp3.internal.http1.Http1ExchangeCodec.readHeaderLine(Http1ExchangeCodec.java:242) + at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.java:213) + at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.java:115) + at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:94) + +Caused by: java.net.SocketException: Socket closed + at java.base/java.net.SocketInputStream.read(SocketInputStream.java:183) + at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140) + at okio.Okio$2.read(Okio.java:140) + at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) + ... 85 common frames omitted +``` + +#### Rosco +For bakes specifically, Rosco will log similar timeout exceptions to Orca: +``` +10:09:35,676 |-ERROR client-side exception java.net.SocketTimeoutException: Read timed out + at java.net.SocketTimeoutException: Read timed out + at at java.base/java.net.SocketInputStream.socketRead0(Native Method) + at at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115) + at at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168) + at at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140) +``` +Although the bake may ultimately complete: +``` +2021-11-08 11:09:13.406 INFO 1 --- [RxIoScheduler-2] c.n.s.rosco.jobs.local.JobExecutorLocal : Polling state for xxxx (executionId: xxxxx1)... +2021-11-08 11:09:13.407 INFO 1 --- [RxIoScheduler-2] c.n.s.rosco.jobs.local.JobExecutorLocal : State for xxxx changed with exit code 0 (executionId: xxxxx:1). +``` + +## Cause +Due the root problem being external to Spinnaker, the list of causes can be rather large. Some potential causes can include but are certainly not limited to: +* Image registry performance and/or configuration +* General network performance +* Externally enforced timeout values on upstream services +* AMI's taking longer to create due to increased size, AWS API throttling (this will be seen in Clouddriver logs) or other AWS issues +* Health checks failing due to instances taking too long to become healthy (deploying to an instance type without enough resources to come up within the check window i.e. deploying to a ***t2.micro*** when a ***t3*** may be more appropriate) + diff --git a/kb/armory/General/bitbucket-cloud-vs-bitbucket-on-premises.md b/kb/armory/General/bitbucket-cloud-vs-bitbucket-on-premises.md new file mode 100644 index 0000000000..b57299d081 --- /dev/null +++ b/kb/armory/General/bitbucket-cloud-vs-bitbucket-on-premises.md @@ -0,0 +1,10 @@ +--- +title: BitBucket Cloud vs BitBucket on Premises +--- + + +BitBucket Cloud and BitBucket on Premises APIs are divergent from each other up until certain versions.  +Some references may break due to API changes in future versions poorly communicated to the public.  For example, [https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010059](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010059) has happened previously for BitBucket On-Prem, but it also happens between On-Prem and Cloud. +When testing Bitbucket issues, it is important to match the customer's versions and types.   So, if a customer is using the cloud, the solution needs to be developed for the cloud version tested and verified with the cloud version. +If a customer uses an On-Prem version, we have to use the same On-Prem version to test and verify. + diff --git a/kb/armory/General/bitbucket-webhook-fails-with-unknown-bitbucket-event-type-in-dinghy.md b/kb/armory/General/bitbucket-webhook-fails-with-unknown-bitbucket-event-type-in-dinghy.md new file mode 100644 index 0000000000..d03d079ccc --- /dev/null +++ b/kb/armory/General/bitbucket-webhook-fails-with-unknown-bitbucket-event-type-in-dinghy.md @@ -0,0 +1,16 @@ +--- +title: BitBucket Webhook Fails with Unknown Bitbucket Event Type in Dinghy +--- + +## Issue +Upon publishing and completing the Pull Request from an on premise BitBucket, the pipeline fails.Dinghy reports an error ```Unknown bitbucket event type``` in Dinghy +Echo reports an error such as the following:  +``` +failed processing event: Event(details=Metadata(source=bitbucket, type=git, created=xxxxxxxxxxxxx, organization=null, project=null, application=null, _content_id=null, attributes=null, requestHeaders={accept=[application/json], accept-encoding=[gzip], connection=[Keep-Alive], content-length=[1301], content-type=[application/json; charset=UTF-8], host=[spin-echo.spinnaker-system:8084], user-agent=[okhttp/3.14.6], x-event-key=[repo:refs_changed], x-spinnaker-anonymous=[anonymous], x-spinnaker-application=[xxxxxxxxx-api]}), +... +eventId=deb121ab-4cff-4c23-94f3-1f8e8ae28fda)\nretrofit.RetrofitError: 500 Internal Server Error\n\tat retrofit.RetrofitError.httpError(RetrofitError.java:40)\n\tat retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:388)\n\tat retrofit.RestAdapter$RestHandler.invoke(RestAdapter.java:240)\n\tat com.sun.proxy.$Proxy187.SendPayload(Unknown Source)\n\tat io.armory.spinnaker.echo.internal.webhooks.ArmoryWebhookService.sendPayload(ArmoryWebhookService.java:19)\n\tat io.armory.spinnaker.echo.internal.webhooks.ArmoryWebhookEventListener.processEvent(ArmoryWebhookEventListener.java:47)\n\tat com.netflix.spinnaker.echo.events.EventPropagator.lambda$null$0(EventPropagator.java:47)\n\tat com.netflix.spinnaker.security.AuthenticatedRequest.lambda$propagate$0(AuthenticatedRequest.java:92)\n\tat com.netflix.spinnaker.echo.events.EventPropagator.lambda$processEvent$2(EventPropagator.java:54)\n\tat rx.internal.util.ActionSubscriber.onNext(ActionSubscriber.java:39)\n\tat rx.observers.SafeSubscriber.onNext(SafeSubscriber.java:134)\n\tat rx.internal.operators.OperatorObserveOn$ObserveOnSubscriber.call(OperatorObserveOn.java:224)\n\tat rx.internal.schedulers.CachedThreadScheduler$EventLoopWorker$1.call(CachedThreadScheduler.java:230)\n\tat rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)\n\tat java.base/java.util.concurrent.FutureTask.run(Unknown Source)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)", +``` + +## Cause +Updates to the Webhook payload format from Atlassian for Bitbucket 6.9.1 have changed the process making it also not backwards compatible with the existing Spinnaker. An update needed to be added to Spinnaker to accept the new payload format. + diff --git a/kb/armory/General/can-users'-level-of-access-in-the-armory-console-be-retricted-using-rbac.md b/kb/armory/General/can-users'-level-of-access-in-the-armory-console-be-retricted-using-rbac.md new file mode 100644 index 0000000000..d4c2dc44b4 --- /dev/null +++ b/kb/armory/General/can-users'-level-of-access-in-the-armory-console-be-retricted-using-rbac.md @@ -0,0 +1,17 @@ +--- +title: Can Users' Level of Access in the Armory Console be Retricted using RBAC +--- + +## Introduction +Customers may want to restrict the level of access certain users within their organizations have to certain tasks and functions.  These can include, but are not limited to +* Inviting additional users to the CDaaS organization +* Adding additional accounts or environments* Which environments are visible +* Amount of access a user may have in a particular environment + +## Prerequisites +Customers will need to have purchased a tier of CDaaS that includes RBAC.  This would include setting up a Trial version with our Sales team or purchase a full Enterprise license. + +## Instructions +Currently, it is not possible to make this restriction, but it is a part of our delivery of our RBAC implementation.  Please look forward to more information in mid/late Summer 2022.   +For more information, please subscribe to this knowledge article, as it will be updated once RBAC functionality has launched. + diff --git "a/kb/armory/General/canary-config-metric-template-displays-invalid-metricstat-json--should-not-have-additional-properties\342\200\213-example--aws-cloudwatch.md" "b/kb/armory/General/canary-config-metric-template-displays-invalid-metricstat-json--should-not-have-additional-properties\342\200\213-example--aws-cloudwatch.md" new file mode 100644 index 0000000000..16f0487805 --- /dev/null +++ "b/kb/armory/General/canary-config-metric-template-displays-invalid-metricstat-json--should-not-have-additional-properties\342\200\213-example--aws-cloudwatch.md" @@ -0,0 +1,14 @@ +--- +title: Canary Config Metric Template displays Invalid MetricStat JSON- should NOT have additional properties​ (Example- AWS Cloudwatch) +--- + +## Issue +When trying to configure a metric template (for example, *AWS Cloudwatch*), the UI keeps throwing the following error: +```Invalid MetricStat JSON: should NOT have additional properties​``` +  + + +## Cause +The JSON format is not valid for the metric provider ingestion.  In this example, for AWS Cloudwatch, the specific issue here is not the validity of the JSON format itself but the case of the Metric Elements portion of the Payload and the particular formatting rules of AWS Cloudwatch. +Customers should review the payload with the provider ingesting the metrics.  They should see if there are unique considerations when formatting the payload. + diff --git a/kb/armory/General/cancel-a-stuck-pipeline.md b/kb/armory/General/cancel-a-stuck-pipeline.md new file mode 100644 index 0000000000..c5d81763e5 --- /dev/null +++ b/kb/armory/General/cancel-a-stuck-pipeline.md @@ -0,0 +1,36 @@ +--- +title: Cancel a Stuck Pipeline +--- + +## Introduction +When a pipeline is stuck and cannot be cancelled via the UI, the following are some steps that can be used to cancel a pipeline. + +## Prerequisites +### Locate the Pipeline ID and Pipeline Execution ID  +#### Via Source +Both pieces of information can be found in the execution details.  Navigate to the pipeline, and go to **Execution Details** -> **Source** +The pipeline id will be labeled ```id``` and will be one of the first lines. +The execution id will be ```execution_id``` and will be towards the end. +#### Via the URL of Resources in the Console +The URL for the pipeline when editing the pipeline from the console also contains the ```Pipeline ID``````/#/applications//executions/configure/****``` +The URL for the permalink (next to the source location above) will contain the information about the ```Pipeline Execution ID``````/#/applications//executions/details/****?stage=0&step=0&details=``` +#### Via Orca Logs +The ```Pipeline Execution ID``` can also be found within the logs of the Orca instance, by running +```kubectl logs -n deploy/spin-orca -f ``` +Sometimes, locating the execution ID can be difficult unless it can be identified from any other executions that are running at the time + +## Instructions +### Via Swagger UI +* Navigate to ```/swagger-ui.html```(You can find your gate endpoint, as an example, by running ```kubectl get svc``` if you are using Spinnaker on kubernetes or kubernetes cloud environments.  For more information about exposing endpoints, please look here: [https://docs.armory.io/docs/spinnaker/exposing-spinnaker/](https://docs.armory.io/docs/spinnaker/exposing-spinnaker/))* From there you can navigate to the pipeline controller and ```/pipelines/{id}/cancel```* Once there, add the ```pipeline execution ID``` from the source of the pipeline and execute the cancel* A ```200 response``` means the pipeline was successfully cancelled* Check back in your Deck UI to confirm the cancellation +## +### If Using Redis and Need to Remove Pipeline from Execution +The following are commands that can be run in **redis-cli** that can be used to manage and remove pipeline executions.  The commands should be run in order, once the ```Pipeline ID``` and ```Pipeline Execution ID``` has been located  +**Command****Function***zrange pipeline:executions:{pipeline_id} 0 -1*this displays all executions from this pipeline for each execution that you want to remove*zrem pipeline:executions:{pipeline_id} {execution_id}*unlink the execution from pipeline_config_id*del pipeline:${execution_id}*delete the execution pipeline*del orchestration:${execution_id}*delete the execution orchestration +  +The following KB article advises about how to put all the commands together to complete the deletion:[https://kb.armory.io/s/article/Delete-Pipeline-Executions-in-Redis](https://kb.armory.io/s/article/Delete-Pipeline-Executions-in-Redis) +### If Using MySQL and Need to Remove Pipeline from Execution +The following KB article advises about how to remove a stuck pipeline from MySQL: +[https://kb.armory.io/s/article/Stuck-Pipeline-Orca-with-MySQL-Editing-the-Database](https://kb.armory.io/s/article/Stuck-Pipeline-Orca-with-MySQL-Editing-the-Database) +### ORCA Zombie +Please read the following about resolving ORCA zombie executions.  The documentation outlines how to determine if it is an ORCA zombie or not, and what to do to remediate the issue[https://spinnaker.io/guides/runbooks/orca-zombie-executions/](https://spinnaker.io/guides/runbooks/orca-zombie-executions/)  + diff --git a/kb/armory/General/cannot-add-custom-kind-to-delete-manifest-stage.md b/kb/armory/General/cannot-add-custom-kind-to-delete-manifest-stage.md new file mode 100644 index 0000000000..9e8dfe47b4 --- /dev/null +++ b/kb/armory/General/cannot-add-custom-kind-to-delete-manifest-stage.md @@ -0,0 +1,15 @@ +--- +title: Cannot add custom Kind to Delete Manifest Stage +--- + +## Issue +When adding the Delete Manifest Stage, Spinnaker will require a ```Kind``` field. For the Deploy Manifest Stage, a custom ```Kind``` can be added. However, Spinnaker is not able to get the Kubernetes custom ```Kinds``` for the ```Delete Manifest Stage```. When trying to add a custom ```Kind``` to the ```Delete Manifest Stage```, the user may see an error such as: +Exception ( Monitor Delete ) +Failed to delete none/ from cluster: error: the server doesn't have a resource type "none" + +## Cause +The ```Delete Manifest stage``` works differently within Spinnaker than the Deploy Manifest stage. +The ```KubernetesDeleteManifestOperatation``` that is used during the ```Delete Manifest stage``` uses the ```kind```'s handler to do the deletion. All unregistered ```kinds``` use the ```KubernetesUnregisteredCustomResourceHandler```, which always sets the ```kind``` to ```NONE.```  +This isn't a problem for deploy operations as it doesn't need to do a lookup to find an existing resource, it is just adding the new resource to the cluster. For the delete operations, the current ```kind``` needs to be sent as part of the delete operation, but the handler only knows the ```kind``` as ```NONE```. +  + diff --git a/kb/armory/General/cannot-define-ebs-volumes-when-deploying-baked-ami.md b/kb/armory/General/cannot-define-ebs-volumes-when-deploying-baked-ami.md new file mode 100644 index 0000000000..23f453e7ac --- /dev/null +++ b/kb/armory/General/cannot-define-ebs-volumes-when-deploying-baked-ami.md @@ -0,0 +1,10 @@ +--- +title: Cannot Define EBS Volumes When Deploying Baked AMI +--- + +## Issue +When deploying a baked AMI, it is currently not possible to define the EBS characteristics of the volumes via the UI.  The AMI as it is baked will be the AMI that is deployed. + +## Cause +Deck currently does not allow for EBS volume definitions.   + diff --git a/kb/armory/General/cannot-enable-dinghy-to-proceed-with-github-pull-request-validations.md b/kb/armory/General/cannot-enable-dinghy-to-proceed-with-github-pull-request-validations.md new file mode 100644 index 0000000000..3d0a16956d --- /dev/null +++ b/kb/armory/General/cannot-enable-dinghy-to-proceed-with-github-pull-request-validations.md @@ -0,0 +1,15 @@ +--- +title: Cannot Enable Dinghy to Proceed with GitHub Pull Request Validations +--- + +## Introduction +Customers would like for Dinghy to receive the pull request from GitHub to make a change to a pipeline, run it through a Validation Process to see if it passes or if there are any errors, and have that provided back to GitHub so that the report can be reviewed, and then approved. Once it is approved, then it can be executed. + +## Prerequisites +Armory 2.21.x + + +## Instructions +The PR Validation and Processing has been added according to the following change: +[https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-21-0/#dinghy---22062211](https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-21-0/#dinghy---22062211) +**feat(prvalidation): repository processing and PR validations (#241)**The feature is detailed in the following documentation, including steps for enabling Mandatory PR validation[https://docs.armory.io/docs/spinnaker/install-dinghy/#pull-request-validations](https://docs.armory.io/docs/spinnaker/install-dinghy/#pull-request-validations)Once Armory 2.21.x + has been installed, Dinghy will be able to process the validation workflow. + diff --git a/kb/armory/General/cannot-overwrite-images-in-a-knowledge-articles-update-them-or-adding-an-image-with-same-name.md b/kb/armory/General/cannot-overwrite-images-in-a-knowledge-articles-update-them-or-adding-an-image-with-same-name.md new file mode 100644 index 0000000000..5d5e7ed6d7 --- /dev/null +++ b/kb/armory/General/cannot-overwrite-images-in-a-knowledge-articles-update-them-or-adding-an-image-with-same-name.md @@ -0,0 +1,13 @@ +--- +title: Cannot overwrite images in a Knowledge Articles (Update them or Adding an Image with Same Name) +--- + +## Issue +If the image already exists in the Image Library, and you are attempting to reupload a change to the same image by opening an article, clicking on the image, and then clicking on the insert image icon in the WYSIWYG, then an error will come up saying that you cannot overwrite the existing image. +Whenever attempting to add an image to an article, a pop up advises that the there is already an image by this name, and to choose a different name. + + + +## Cause +An image with the name already exists within ServiceNow, and therefore, the image cannot be added with the same name.  The only way to update the image is to delete the existing image and upload it again.   + diff --git a/kb/armory/General/cannot-see-pipeline-history-for-changes-made.md b/kb/armory/General/cannot-see-pipeline-history-for-changes-made.md new file mode 100644 index 0000000000..756c730e62 --- /dev/null +++ b/kb/armory/General/cannot-see-pipeline-history-for-changes-made.md @@ -0,0 +1,10 @@ +--- +title: Cannot See Pipeline History for Changes Made +--- + +## Issue +Users can’t see pipeline history for changes made to the pipeline. After making changes to the pipeline in Spinnaker, from the **Pipeline Actions** menu users should be able to select **View Revision History**, but it says **No version history found**.The correct behavior should show as following: + +## Cause +Spinnaker leverages underlying object versioning capabilities in S3/GCS.  Check if the object versioning has been enabled for the underlying bucket.  + diff --git a/kb/armory/General/capacity-provider-concepts.md b/kb/armory/General/capacity-provider-concepts.md new file mode 100644 index 0000000000..d2e6fa6c73 --- /dev/null +++ b/kb/armory/General/capacity-provider-concepts.md @@ -0,0 +1,17 @@ +--- +title: Capacity Provider Concepts +--- + + +#### Capacity provider concepts and setup +[Capacity providers](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cluster-capacity-providers.html#capacity-providers-concepts) define the kinds of capacity available to an ECS cluster, and the [capacity provider strategy](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_CreateService.html#ECS-CreateService-request-capacityProviderStrategy) of the service determines how its tasks are spread across this capacity. +* Info on how to [set up an ECS cluster with FARGATE & FARGATE_SPOT](https://aws.amazon.com/blogs/aws/aws-fargate-spot-now-generally-available/) capacity providers.* Use the [DescribeClusters API](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DescribeClusters.html#API_DescribeClusters_RequestSyntax) to determine which capacity providers are available on a given cluster. +#### Relevant code in Spinnaker +At a high level, supporting capacity providers will involve: +* caching the providers available for each cluster within Clouddriver* Exposing capacity providers for a given cluster(s) via Gateretrieving them from Deck, and allowing the user to configure a capacity provider strategy with them for a given server group +* NOTE: An ECS service can't be created with both a capacity provider strategy *and* a launch type. Spinnaker should validate these settings are valid prior to creating the server group. +* providing those setting to ECS when deploying a new service +Similar ECS provider changes were made to implement Service Discovery: +* [spinnaker/clouddriver#3604](https://github.com/spinnaker/clouddriver/pull/3604)* [spinnaker/gate#786](https://github.com/spinnaker/gate/pull/786)* [spinnaker/deck#6899](https://github.com/spinnaker/deck/pull/6899) +#### Capacity Provider Strategies have been added to 2.24 + diff --git a/kb/armory/General/capturing-spinnaker-configurations-with-armory's-support-bundle-w--automated-script.md b/kb/armory/General/capturing-spinnaker-configurations-with-armory's-support-bundle-w--automated-script.md new file mode 100644 index 0000000000..d26db36b34 --- /dev/null +++ b/kb/armory/General/capturing-spinnaker-configurations-with-armory's-support-bundle-w--automated-script.md @@ -0,0 +1,44 @@ +--- +title: Capturing Spinnaker configurations with Armory's Support Bundle (w/ Automated Script) +--- + + +For a full detailed of the manual process this script is automating, [please refer to KB0010398](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010398) +### Introduction +Troubleshooting a Spinnaker issue may require customers to manually pull the configurations from their environment. At times, the configurations may be incomplete which would require customers pull the details again, causing a delay in actual issue resolution time. +To avoid such discrepancies, ```Armory's Support Bundle``` helps in capturing accurate and relevant details from a customer's Spinnaker installation. The Armory Support Bundle is a Kubernetes tool that pulls a standardized set of configuration details from Spinnaker manifests. The debug information is then compressed in the form of zip file, which can provided to Armory's Support team over the support case.   +### What Information can the Armory Support Bundle discover? +Armory Support Bundle retrieves all pod, service, and deployment configuration associated with an Armory Spinnaker installation. This emulates what a user does when trying to describe resources with ```kubectl```. The tool will generate a ***zip*** file with all the info it gathered so they can be easily share across teams, and attach to Support tickets. This also gives customers the opportunity to manually remove any sensitive information that is not meant to be shared with Armory. +Installation +Retrieving the Armory Support Bundle is done with a script: +*  Download ```getSupportBundle.sh``` from the following link: [https://engineering.armory.io/manifests/support-bundle/0.2.0/getSupportBundle.sh](https://engineering.armory.io/manifests/support-bundle/0.2.0/getSupportBundle.sh)*  Place it in an accessible directory, and ensure you have ```execute``` access to the file.*(optional)* Run it without any options, or with ```--help``` to view its usage: ```./getSupportBundle.sh --help```* Run it with the following options: ```./getSupportBundle.sh -n -o ``` +### Retrieving the debug information  +If after a few moments the script completes successfully, the .zip bundle will be placed in the directory specified in the step above. Once this is done, you may review the contents to redact any sensitive information, and then attach it to your Armory Support case. +### Troubleshooting +In case an issue is encountered during the script execution, an error message will be displayed.  +```Error #001: Cannot create ``` +* The provided directory path passed with the ```-o``` or ```--outputdir``` option cannot be created. Ensure you have sufficient read and write-permissions to the parent directory. +```Error #002: Provided directory is not a directory!``` +* The provided directory path passed with the ```-o``` or ```--outputdir``` option points to a file. Ensure you specify a directory instead. +```Error #003: Invalid option. Use "-h" or "--help" to see usage.``` +* One or more options passed to the script are invalid.  +```Error #004: BOTH a namespace and output directory must be specified. Use "-h" or "--help" to see usage.``` +* Either a namespace or an output directory was not specified. Both must be provided to the script. +```Error #005: Cannot copy over to directory: !``` +* You do not have write-access to provided directory. Ensure you have correct permissions, and re-run the script again. +```Error #006: Cannot fetch bundle ZIP from pod!``` +* The pod ```armory-support-bundle-visualizer``` might have either been terminated, or a stale instance is still running in the current namespace. Ensure no previous deployments of this type exist, and re-run the script again. +```Error #007: Cannot fetch visualizer.yaml from !``` +or +```Error #008: Cannot fetch deploy.yaml from !``` +* One or both of the supporting manifests could not be fetched from Armory's website with ```curl```. Ensure the environment where the script is placed has outbound access in your firewall rules. If this is not an option for you, note that support for air-gapped customers will be added soon. +```Error #009: Cannot delete visualizer pod!``` +or + +```Error #010: Cannot delete support bundle pod!``` + +* No ```kubectl delete``` privileges available within the current cluster. If you would like to manually clean-up, remove the associated deployments when sufficient privileges are available. +```Error #011: Provided namespace does not exist!``` +* Ensure the namespace provided is valid, and confirm that is where Spinnaker is installed. +`````` + diff --git a/kb/armory/General/cases-and-change-requests.md b/kb/armory/General/cases-and-change-requests.md new file mode 100644 index 0000000000..03782bbbee --- /dev/null +++ b/kb/armory/General/cases-and-change-requests.md @@ -0,0 +1,16 @@ +--- +title: Cases and Change Requests +--- + + +Customers may find themselves needing to submit either a Case or a Change Request within our portal.  The following article should assist customers with understanding when one type of submission should be used over the other. +In order to ensure issues are resolved effectively, we ask that customers please do not mix requests and issue submissions, and to please open separate tickets for separate issues/changes. +### Case Submissions +Customers should open up support cases whenever they have an inquiry or an issue that needs to be addressed.  Support will assist on these cases to either direct them to the correct resources or resolve them for our customers.  If a customer is unsure of whether to open a Change Request, or a Support Case, please default to opening a Support Case.   +Support Cases are also the only method by which a customer can initiate a P0 response.   +For detailed information about submitting cases, please read the following about [Creating Cases and Case Management in ServiceNow](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010136) +For customers who have mixed instances of both Armory Self-Hosted and Armory Managed Spinnaker environments, please note that any cases related Armory Self-Hosted environments will be resolved by the Armory Support team. +### Change Request Submissions +Customers should open change requests whenever they require Armory to make changes to their Armory Managed Services environment.  Examples include adding accounts or creating configuration changes +For detailed information about change requests, please read the following about [Creating Change Requests](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010267) + diff --git a/kb/armory/General/changes-to-a-configmap-not-reflected-in-new-deployment.md b/kb/armory/General/changes-to-a-configmap-not-reflected-in-new-deployment.md new file mode 100644 index 0000000000..67c521dffe --- /dev/null +++ b/kb/armory/General/changes-to-a-configmap-not-reflected-in-new-deployment.md @@ -0,0 +1,33 @@ +--- +title: Changes to a ConfigMap not reflected in new deployment +--- + +## Issue +Certain applications that use a ***readable hot config for dynamic configuration*** are improperly configured failing or failing when using a ConfigMap to drive the dynamic deployment strategy. +This is commonly used when teams want to make changes to configuration properties **only** (ie. the ConfigMap), without applying a complete artifact re-deployment. +Performing a rolling restart of the pods yields the same result, with the ConfigMap in question seems to not pick up any new updates to it. + +## Cause +Restarting deployments do **not** automatically take the changes of an updated ConfigMap. To demonstrate this, consider the following scenario - with a basic two-stage deployment pipeline: +```Deploy (Manifest) stage``` in which we define a ConfigMap, called ```configmaptest```, which contains a text file with the following text in it: +apiVersion: v1 +data: + some-file.txt: | + testing this file-spin-testing-12345566 +kind: ConfigMap +metadata: + name: configmaptest + namespace: spinnaker01​ +Deploy (Manifest) stage in which we define a basic nginx deployment: [https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#creating-a-deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#creating-a-deployment)...which has the above-mentioned ConfigMap "configmaptest" mounted: +volumes: + - configMap: + defaultMode: 420 + name: configmaptest + name: some-volume-name​ +...to the following path on the nginx container: +volumeMounts: + - mountPath: /testing/file.txt + name: some-volume-name + subPath: some-file.txt +*  After running this pipeline for the first time, we make a change to the ConfigMap to change the contents of the testing text file. Then, we re-run the pipeline to see if the deployment would pick up the new ConfigMap, which it didn't. We can also perform a ```rollout restart deployment``` but still doesn't take in the changes + diff --git a/kb/armory/General/changing-timeouts-for-spinnaker-services-using-okhttp-java-services.md b/kb/armory/General/changing-timeouts-for-spinnaker-services-using-okhttp-java-services.md new file mode 100644 index 0000000000..4cd8ec9b24 --- /dev/null +++ b/kb/armory/General/changing-timeouts-for-spinnaker-services-using-okhttp-java-services.md @@ -0,0 +1,11 @@ +--- +title: Changing Timeouts for Spinnaker Services Using okHTTP (Java Services) +--- + +## Issue +Java language Spinnaker Services using the ```okHTTP``` library may need to adjust timeouts using the following code.  This may be used, as an example, for the case where connections are unreliable.  By default, most of the services are set to 60000 ms (1 minute)The following explains how to change the Connect Timeout, Read Timeout, and Write Timeout, but there are additional okHTTP calls that can be set via your ```%service-local.yml``` files.Some services that use the okHTTP library include: +CloudDriverOrcaRoscoGateEchoFront50IgorFiat + +## Cause +N/A + diff --git a/kb/armory/General/clean-up-schedule-for-armory-scale-agent-operations-older-than-a-specific-time-period.md b/kb/armory/General/clean-up-schedule-for-armory-scale-agent-operations-older-than-a-specific-time-period.md new file mode 100644 index 0000000000..dd1c3439cd --- /dev/null +++ b/kb/armory/General/clean-up-schedule-for-armory-scale-agent-operations-older-than-a-specific-time-period.md @@ -0,0 +1,25 @@ +--- +title: Clean Up Schedule for Armory Scale Agent Operations Older than a Specific Time Period +--- + +## Introduction +As the number of Armory Agent deployments increases, admins will want to consider setting up a purge or clean-up schedule for older operations. +This schedule will help avoid tables growing without controls. The historical record of executed operations is kept in the ```kubesvc_ops_history``` table. As the number of deployments increases and as Agents scale, Admins will notice that the table ```kubesvc_ops_history``` will continue to grow.  A purge schedule will help maintain the size of the table. +The Armory Agent plugin automatically handles this through the configuration below + +## Prerequisites +Armory Enterprise Spinnaker with Armory Agent for Kubernetes enabled. + +## Instructions +To delete the operations older than a particular period regularly without any manual intervention, admins can use the below configurations. Please note that the below configurations are optional by default, and will not be set unless admins do so. +#### Purge the operations older than a Specific # of Weeks +Admins can set a clean-up period in ```number of weeks``` by making the following change in the configuration. Any operation older than the specified weeks in the database table ```kubesvc_ops_history``` would be purged automatically + kubesvc: + jobs.operation-history.purge.weeks: 1 +#### **Frequency of the purge process** +Admins can also set a clean-up period based in a ```spring cron expression. ``` A ```spring cron expression``` can be supplied to the configuration as shown in the example below + kubesvc: + jobs.operation-history.purge.cron: 0 0 0 * * 0 +For help defining ```cron expressions```, please visit: [https://en.wikipedia.org/wiki/Cron#CRON_expression](https://en.wikipedia.org/wiki/Cron#CRON_expression) +For the complete list of configurations on the Agent plugin, please refer [https://docs.armory.io/armory-enterprise/armory-agent/advanced-config/agent-plugin-options/](https://docs.armory.io/armory-enterprise/armory-agent/advanced-config/agent-plugin-options/) + diff --git a/kb/armory/General/cloud-provider-account-disappearing-from-spinnaker-ui.md b/kb/armory/General/cloud-provider-account-disappearing-from-spinnaker-ui.md new file mode 100644 index 0000000000..91a740bcd5 --- /dev/null +++ b/kb/armory/General/cloud-provider-account-disappearing-from-spinnaker-ui.md @@ -0,0 +1,10 @@ +--- +title: Cloud provider account disappearing from Spinnaker UI +--- + +## Issue +An organization may run into a problem in between deployments where a defined cloud provider account will disappear from the Spinnaker UI. This may lead to deployment failures and/or pipelines timeouts, due to Spinnaker being unable to discover the provider’s account.  + +## Cause +This is known bug affecting Spinnaker versions older than 1.18.0. The issue stems from how clouddriver caches resources from deleted Spinnaker accounts, which does not purge the SQL cache automatically. + diff --git a/kb/armory/General/clouddriver-caching-errors-|-clusteredagentscheduler---unable-to-run-agents.md b/kb/armory/General/clouddriver-caching-errors-|-clusteredagentscheduler---unable-to-run-agents.md new file mode 100644 index 0000000000..df72ae100f --- /dev/null +++ b/kb/armory/General/clouddriver-caching-errors-|-clusteredagentscheduler---unable-to-run-agents.md @@ -0,0 +1,13 @@ +--- +title: clouddriver-caching errors | ClusteredAgentScheduler - Unable to run agents +--- + +## Issue +When attempting to run agents, the following errors can be found in the clouddriver-caching logs: +``` +Exception: ERROR 1 --- [ClusteredAgentScheduler-0] c.n.s.c.r.c.ClusteredAgentScheduler : Unable to run agents redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool redis.clients.jedis.util.Pool.getResource(Pool.java:59) ~[jedis-3.1.0.jar:na] redis.clients.jedis.JedisPool.getResource(JedisPool.java:234) ~[jedis-3.1.0.jar:na] com.netflix.spinnaker.kork.jedis.telemetry.InstrumentedJedisPool.getResource(InstrumentedJedisPool.java:60) ~[kork-jedis-7.41.0.jar:na] com.netflix.spinnaker.kork.jedis.telemetry.InstrumentedJedisPool.getResource(InstrumentedJedisPool.java:26) ~[kork-jedis-7.41.0.jar:na] com.netflix.spinnaker.kork.jedis.JedisClientDelegate.withCommandsClient(JedisClientDelegate.java:54) ~[kork-jedis-7.41.0.jar:na] com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.acquireRunKey(ClusteredAgentScheduler.java:183) ~[cats-redis-GCSFIX.jar:na] com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.acquire(ClusteredAgentScheduler.java:136) ~[cats-redis-GCSFIX.jar:na] com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.runAgents(ClusteredAgentScheduler.java:163) ~[cats-redis-GCSFIX.jar:na] com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.run(ClusteredAgentScheduler.java:156) ~[cats-redis-GCSFIX.jar:na] java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na] java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[na:na] java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na] java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na] java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na] java.base/java.lang.Thread.run(Thread.java:834) ~[na:na] Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Failed connecting to host redis-clouddriver-master:6379 redis.clients.jedis.Connection.connect(Connection.java:204) ~[jedis-3.1.0.jar:na] redis.clients.jedis.BinaryClient.connect(BinaryClient.java:100) ~[jedis-3.1.0.jar:na] redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1866) ~[jedis-3.1.0.jar:na] redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:117) ~[jedis-3.1.0.jar:na] org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:889) ~[commons-pool2-2.7.0.jar:2.7.0] org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:424) ~[commons-pool2-2.7.0.jar:2.7.0] org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349) ~[commons-pool2-2.7.0.jar:2.7.0] redis.clients.jedis.util.Pool.getResource(Pool.java:50) ~[jedis-3.1.0.jar:na] \t... 14 common frames omitted Caused by: java.net.SocketTimeoutException: connect timed out java.base/java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:na] java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[na:na] java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[na:na] java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[na:na] java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) ~[na:na] java.base/java.net.Socket.connect(Socket.java:609) ~[na:na] redis.clients.jedis.Connection.connect(Connection.java:181) ~[jedis-3.1.0.jar:na] \t... 21 common frames omitted +``` + +## Cause +Rare Redis issue where when two pods try to start at the same time.  Because they start at the same time, they end up competing for shared resources, and only one pod ends up starting. + diff --git a/kb/armory/General/clouddriver-fails-to-start-up-armory-agent-during-initial-install.md b/kb/armory/General/clouddriver-fails-to-start-up-armory-agent-during-initial-install.md new file mode 100644 index 0000000000..c746c76b33 --- /dev/null +++ b/kb/armory/General/clouddriver-fails-to-start-up-armory-agent-during-initial-install.md @@ -0,0 +1,14 @@ +--- +title: Clouddriver fails to start up Armory Agent during initial install +--- + +## Issue +When installing Armory Agent, organizations can get the following exceptions when starting clouddriver with armory agent plugin enabled. + +``` +Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled. 2020-12-08 17:55:15.404 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'documentationPluginsBootstrapper' defined in URL [jar:file:/opt/clouddriver/lib/springfox-spring-web-2.9.2.jar!/springfox/documentation/spring/web/plugins/DocumentationPluginsBootstrapper.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'webMvcRequestHandlerProvider' defined in URL [jar:file:/opt/clouddriver/lib/springfox-spring-web-2.9.2.jar!/springfox/documentation/spring/web/plugins/WebMvcRequestHandlerProvider.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'Armory.Kubesvc.com.netflix.spinnaker.kork.plugins.api.spring.SpringLoader': Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Failed to parse configuration class [io.armory.kubesvc.agent.KubesvcCachingAgentDispatcher]; nested exception is java.io.FileNotFoundException: class path resource [com/netflix/spinnaker/clouddriver/kubernetes/caching/agent/KubernetesV2CachingAgentDispatcher.class] cannot be opened because it does not exist +``` + +## Cause +OSS Spinnaker and the Armory Extension (1.x.x / 2.x.x ) have different compatibility with the Armory Agent plugin. For incompatible versions, errors will come up as certain dependencies are not met. Typically these errors will be at the Clouddriver level.  + diff --git a/kb/armory/General/clouddriver-is-not-caching-subnets-or-server-groups.md b/kb/armory/General/clouddriver-is-not-caching-subnets-or-server-groups.md new file mode 100644 index 0000000000..3536ad4284 --- /dev/null +++ b/kb/armory/General/clouddriver-is-not-caching-subnets-or-server-groups.md @@ -0,0 +1,30 @@ +--- +title: Clouddriver is not caching subnets or server groups +--- + +## Issue +When a user tries to create a deploy stage in a cluster that deploys to ECS, the deployment is not successful. An error is seen indicating that subnets/security groups/ IAM instance profiles for ECS cannot be found. The following is seen in the Clouddriver logs: + +CD LOGS - +Pod spin-clouddriver-...............: +``` +2021-08-24 21:49:25.068 INFO 1 — [ main] n.s.c.e.s.EcsCredentialsLifeCycleHandler : ECS account, production...., was added. Scheduling caching agents +2021-08-24 21:50:06.605 INFO 1 — [cutionAction-10] c.n.s.c.e.p.a.ApplicationCachingAgent : Evicting 1 ecsapplications in ........../ApplicationCachingAgent +2021-08-24 21:50:11.614 INFO 1 — [ecutionAction-8] c.n.s.c.e.p.a.TaskHealthCachingAgent : Caching 2 task health checks in ...../TaskHealthCachingAgent +2021-08-24 21:50:11.700 INFO 1 — [cutionAction-10] c.n.s.c.e.p.a.EcsClusterCachingAgent : Caching 3 ECS clusters in .......1/EcsClusterCachingAgent +2021-08-24 21:50:11.706 INFO 1 — [cutionAction-10] c.n.s.c.e.p.a.EcsClusterCachingAgent : Evicting 1 ecsclusters in ......./EcsClusterCachingAgent +2021-08-24 21:50:11.837 INFO 1 — [ecutionAction-8] c.n.s.c.e.p.a.TaskHealthCachingAgent : Evicting 2 healths in ......./TaskHealthCachingAgent +2021-08-24 21:50:26.901 INFO 1 — [ecutionAction-9] c.n.s.c.e.p.a.TaskDefinitionCachingAgent : Caching 2 task definitions in ......./TaskDefinitionCachingAgent +2021-08-24 21:50:26.932 INFO 1 — [ecutionAction-9] c.n.s.c.e.p.a.TaskDefinitionCachingAgent : Evicting 1 taskdefinitions in .........../TaskDefinitionCachingAgent +The following is seen in the SQL Agent Logs: +SQL CLEANUP AGENT LOGS +2021-08-24 22:05:56.733 INFO 1 — [cutionAction-48] c.n.s.c.s.c.SqlUnknownAgentCleanupAgent : Scanning 'AmazonInstanceTypeCachingAgent/...... (22/38) cache records to cleanup +2021-08-24 22:05:56.744 INFO 1 — [cutionAction-48] c.n.s.c.s.c.SqlUnknownAgentCleanupAgent : Scanning 'subnets' (......) cache records to cleanup +2021-08-24 22:05:56.755 INFO 1 — [cutionAction-48] c.n.s.c.s.c.SqlUnknownAgentCleanupAgent : Scanning 'securityGroups' (24/38) cache records to cleanup +2021-08-24 22:05:56.768 INFO 1 — [cutionAction-48] c.n.s.c.s.c.SqlUnknownAgentCleanupAgent : Scanning 'vpcs' (.....) cache records to cleanup +``` + +## Cause +This issue is due to Spinnaker not caching ECS subnets and server groups in the cluster. There were naming issues within Clouddriver and Deck pertaining to servergroups and subnets:[https://github.com/spinnaker/deck/pull/9524](https://github.com/spinnaker/deck/pull/9524)[https://github.com/spinnaker/deck/pull/9032](https://github.com/spinnaker/deck/pull/9032) + + diff --git a/kb/armory/General/clouddriver-logs-show-slow-sql-warnings.md b/kb/armory/General/clouddriver-logs-show-slow-sql-warnings.md new file mode 100644 index 0000000000..1df0130895 --- /dev/null +++ b/kb/armory/General/clouddriver-logs-show-slow-sql-warnings.md @@ -0,0 +1,25 @@ +--- +title: Clouddriver Logs Show Slow SQL Warnings +--- + +## Issue +Clouddriver logs SQL calls as Slow SQL. An example of such logs can be found below. +``` +2021-09-16 12:18:14.692 WARN 1 --- [gentScheduler-0] c.n.s.k.s.telemetry.JooqSlowQueryLogger : Slow SQL (1800ms): +insert into cats_agent_locks ( + agent_name, + owner_id, + lock_acquired, + lock_expiry +) +values ( + 'some-agent-running/KubernetesUnregisteredCustomResourceCachingAgent[1/1]', + 'spin-clouddriver-123453678:1@spin-clouddriver-123453678', + 12345678, + 12345678 +) +``` + +## Cause +The exact cause of this issue is unclear as we are continuing research on this topic however, we have concluded that this seems to be an issue with the SQL agent scheduler. + diff --git "a/kb/armory/General/clouddriver-shows-\"connection-is-not-available,-request-timed-out-after-10000ms.\".md" "b/kb/armory/General/clouddriver-shows-\"connection-is-not-available,-request-timed-out-after-10000ms.\".md" new file mode 100644 index 0000000000..e38ba5174b --- /dev/null +++ "b/kb/armory/General/clouddriver-shows-\"connection-is-not-available,-request-timed-out-after-10000ms.\".md" @@ -0,0 +1,15 @@ +--- +title: Clouddriver shows "Connection is not available, request timed out after 10000ms." +--- + +## Issue +An error stating that ```Connection is not available, request timed out after 10000ms``` is observed in an environment with ```MySQL``` used for CloudDriver, along with ```HA-mode``` enabled. Clouddriver logs show the following error: +``` +2021-10-03 13:34:54.553 ERROR 1 --- [gPodsObserver-0] c.n.s.c.s.c.SqlCachingPodsObserver : Failed to manage replicas heartbeat +org.springframework.dao.TransientDataAccessResourceException: jOOQ; SQL [select * from caching_replicas where pod_id = ...?]; default - Connection is not available, request timed out after 10000ms.; nested exception is java.sql.SQLTransientConnectionException: default - Connection is not available, request timed out after 10000ms. + at org.jooq_3.13.2.MYSQL.debug(Unknown Source) ~[na:na] +``` + +## Cause +The issue pertains to Clouddriver being unable to establish connections with MySQL, with HA Mode enabled due to the amount of connections being attempted. + diff --git a/kb/armory/General/clouddriver-unable-to-download-artifacts-from-private-github-instance-due-to-ssl-errors.md b/kb/armory/General/clouddriver-unable-to-download-artifacts-from-private-github-instance-due-to-ssl-errors.md new file mode 100644 index 0000000000..e6588cdca3 --- /dev/null +++ b/kb/armory/General/clouddriver-unable-to-download-artifacts-from-private-github-instance-due-to-ssl-errors.md @@ -0,0 +1,32 @@ +--- +title: Clouddriver unable to download artifacts from private github instance due to SSL errors +--- + +## Issue +An organization may have its own private Github instance for source code management instead of using public Github instance.  They may choose to use custom SSL certificates for secure access of the private Github instance. If Spinnaker is configured to fetch artifacts from private Github instance, users may notice SSL errors in Clouddriver logs when trying to fetch artifacts similar to the one below + +``` +2021-07-26 10:02:02.326 INFO 1 --- [ MvcAsync3] c.n.s.c.a.gitRepo.GitJobExecutor : Cloning git/repo https://github.wdf.xxx.com/repo/reponame into /tmp/gitrepos/72fe7405e214a959ef3c55848d91da00d90f4f13e4eb1d1dae9361cdf0c93dd5 +2021-07-26 10:02:02.355 WARN 1 --- [0.0-7002-exec-3] c.n.s.k.w.e.GenericExceptionHandlers : Handled error in generic exception handlerjava.io.IOException: git clone --branch master --depth 1 https://token:$GIT_TOKEN@github.wdf.xxx.com/repo/reponame failed. Error: Cloning into 'reponame'... +fatal: unable to access 'https://github.wdf.xxx.com/repo/reponame/': SSL certificate problem: unable to get local issuer certificate + Output: + at com.netflix.spinnaker.clouddriver.artifacts.gitRepo.GitJobExecutor.cloneBranchOrTag(GitJobExecutor.java:156) ~[clouddriver-artifacts-8.0.4-20210625060028.jar:8.0.4-20210625060028] + at com.netflix.spinnaker.clouddriver.artifacts.gitRepo.GitJobExecutor.clone(GitJobExecutor.java:138) ~[clouddriver-artifacts-8.0.4-20210625060028.jar:8.0.4-20210625060028] + at com.netflix.spinnaker.clouddriver.artifacts.gitRepo.GitJobExecutor.cloneOrPull(GitJobExecutor.java:98) ~[clouddriver-artifacts-8.0.4-20210625060028.jar:8.0.4-20210625060028] + at com.netflix.spinnaker.clouddriver.artifacts.gitRepo.GitRepoArtifactCredentials.getInputStream(GitRepoArtifactCredentials.java:126) ~[clouddriver-artifacts-8.0.4-20210625060028.jar:8.0.4-20210625060028] + at com.netflix.spinnaker.clouddriver.artifacts.gitRepo.GitRepoArtifactCredentials.getLockedInputStream(GitRepoArtifactCredentials.java:93) ~[clouddriver-artifacts-8.0.4-20210625060028.jar:8.0.4-20210625060028] + at com.netflix.spinnaker.clouddriver.artifacts.gitRepo.GitRepoArtifactCredentials.download(GitRepoArtifactCredentials.java:69) ~[clouddriver-artifacts-8.0.4-20210625060028.jar:8.0.4-20210625060028] + at com.netflix.spinnaker.clouddriver.artifacts.ArtifactDownloader.download(ArtifactDownloader.java:37) ~[clouddriver-artifacts-8.0.4-20210625060028.jar:8.0.4-20210625060028] + at com.netflix.spinnaker.clouddriver.controllers.ArtifactController.lambda$fetch$0(ArtifactController.java:67) ~[clouddriver-web-8.0.4-20210625060028.jar:8.0.4-20210625060028] + at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:111) ~[spring-webmvc-5.2.9.RELEASE.jar:5.2.9.RELEASE] + at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:98) ~[spring-webmvc-5.2.9.RELEASE.jar:5.2.9.RELEASE] + at org.springframework.web.context.request.async.WebAsyncManager.lambda$startCallableProcessing$4(WebAsyncManager.java:337) ~[spring-web-5.2.11.RELEASE.jar:5.2.11.RELEASE] + at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na] + at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na] + at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]2021-07-26 10:02:02.355 ERROR 1 --- [0.0-7002-exec-3] c.n.s.k.w.e.GenericExceptionHandlers : Internal Server Error +``` + +## Cause +Although the certificate for private github instance may be imported into the ```java keystore``` and later mounted to Clouddriver by following the steps from the knowledge article: [https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010087](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010087), it would work only for ```http``` calls that are made by Clouddriver. +However in this particular scenario, Clouddriver pulls the artifacts through git command instead of http and the error by itself is actually from the git command. This can be replicated by running the git commands from the Clouddriver pods. + diff --git a/kb/armory/General/clouddriver-will-not-connect-to-mysql-causing-a-stoppage,-databasechangelock-error.md b/kb/armory/General/clouddriver-will-not-connect-to-mysql-causing-a-stoppage,-databasechangelock-error.md new file mode 100644 index 0000000000..0cf0e1d5e6 --- /dev/null +++ b/kb/armory/General/clouddriver-will-not-connect-to-mysql-causing-a-stoppage,-databasechangelock-error.md @@ -0,0 +1,12 @@ +--- +title: Clouddriver will not Connect to MySQL Causing a Stoppage, DatabaseChangeLock Error +--- + +## Issue +Clouddriver will not connect to MySQL. Errors can be seen throughout the logs and cause a problem where it will not allow pipelines to execute due to a ```locked``` state of MySQL with no timeframe or lockout period. +```I 2020-12-22T23:26:00.121328401Z 2020-12-22 23:26:00.121 INFO 1 --- [ main] liquibase.executor.jvm.JdbcExecutor : SELECT `LOCKED` FROM clouddriver.DATABASECHANGELOGLOCK WHERE ID=1``` + +## Cause +One of the causes of this error is a bug in MySQL where a ```locked``` status will be hit due to some other error.  When MySQL is in this state, the state doesn't reset, as there is no timeout period to remove the error state to bring MySQL back to an unlocked status. These locked records can hold indefinitely.Below is an example of an error code that references the locked log: +```Error creating bean with name 'liquibase' defined in class path resource [com/netflix/spinnaker/kork/sql/config/DefaultSqlConfiguration.class]: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock. Currently locked by spin-clouddriver-ro-deck-############## (###.######.###) since 12/22/20, 10:58 PM at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE] at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE] ... 56 common frames omitted Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'healthEndpoint' defined in class path resource [org/springframework/boot/actuate/autoconfigure/health/HealthEndpointConfiguration.class]: Unsatisfied dependency expressed through method 'healthEndpoint' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'healthContributorRegistry' defined in class path resource [org/springframework/boot/actuate/autoconfigure/health/HealthEndpointConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContribut``` + diff --git a/kb/armory/General/combined-version-and-service-module-name-is-too-long-when-deploying-with-app-engine.md b/kb/armory/General/combined-version-and-service-module-name-is-too-long-when-deploying-with-app-engine.md new file mode 100644 index 0000000000..77bb9c307c --- /dev/null +++ b/kb/armory/General/combined-version-and-service-module-name-is-too-long-when-deploying-with-app-engine.md @@ -0,0 +1,31 @@ +--- +title: Combined version and service (module) name is too long when deploying with App Engine +--- + +## Issue +An organization deploying to App Engine runs into the following error. +```Combined version and service (module) name is too long.``` +This happens in Google Cloud Platform - using App Engine Flex when deploying services. The character length maximum is **48 characters**. Names can reach more than 48 characters when environments, users, and services are all combined.  + +## Cause +Google App Engine Flex has a limitation of 48 characters when naming a service. This includes versions and the default settings.  +You can see an example ```app.yaml``` file here. + +``` +runtime: custom +env: flex +service: data-replication-service +env_variables: + SPRING_PROFILES_ACTIVE: dev +runtime_config: + jdk: openjdk8 +resources: + cpu: 1 + memory_gb: 1 + disk_size_gb: 10 +handlers: + - url: /.* + script: this field is required, but ignored +``` +Combining the **Service**, **Name**, as well as **Version** that's being committed can go over 48 characters. + diff --git a/kb/armory/General/common-vulnerabilities-and-exposures-cve-reports---armory-continuous-delivery.md b/kb/armory/General/common-vulnerabilities-and-exposures-cve-reports---armory-continuous-delivery.md new file mode 100644 index 0000000000..690f07d09f --- /dev/null +++ b/kb/armory/General/common-vulnerabilities-and-exposures-cve-reports---armory-continuous-delivery.md @@ -0,0 +1,17 @@ +--- +title: Common Vulnerabilities and Exposures (CVE) Reports - Armory Continuous Delivery +--- + + +Please note that all CVE Reports are classified as Armory Confidential and Proprietary Information, and should not be shared outside of a customer's organization.  The documents fall under the customer's MSA and NDA agreements. +As a part of our commitment to providing transparency to our customers, Armory generates a CVE report for each [Armory Continuous Deployment release](https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/).   +Armory currently uses Aqua Security to scan images for vulnerabilities. AquaSec scans are the only scans that Armory can process, as our Engineers can test the efficacy of any changes by rerunning scans to ensure the resolution of the identified vulnerabilities.   +Armory runs Aqua Security scans on code throughout the continuous integration (CI) process and runs scans for all new releases. +For more information on our Vulnerability Management Policy, please [visit our KB article on the subject](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010092). + +### CVE Reports +The most recent CVE Reports from Aqua Security are available as attachments below the article for the [supported versions of ](https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/)Armory Continuous Delivery. + +#### Armory Plugin CVE Reports +To find Armory CVE Reports for various Plugins, please visit our KB article, [Common Vulnerabilities and Exposures (CVE) Reports - Armory Plugins](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010788) + diff --git a/kb/armory/General/common-vulnerabilities-and-exposures-cve-reports---armory-plugins.md b/kb/armory/General/common-vulnerabilities-and-exposures-cve-reports---armory-plugins.md new file mode 100644 index 0000000000..0cfab66e9c --- /dev/null +++ b/kb/armory/General/common-vulnerabilities-and-exposures-cve-reports---armory-plugins.md @@ -0,0 +1,16 @@ +--- +title: Common Vulnerabilities and Exposures (CVE) Reports - Armory Plugins +--- + + +Please note that all CVE Reports are classified as Armory Confidential and Proprietary Information, and should not be shared outside of a customer's organization.  The documents fall under the customer's MSA and NDA agreements. +As a part of our commitment to providing transparency to our customers, Armory periodically provides CVE reports for Plugins.   +Armory currently uses Aqua Security to scan images for vulnerabilities. AquaSec scans are the only scans that Armory can process, as our Engineers can test the efficacy of any changes by rerunning scans to ensure the resolution of the identified vulnerabilities.   +Armory runs Aqua Security scans on code throughout the continuous integration (CI) process and runs scans for all new releases. +For more information on our Vulnerability Management Policy, please [visit our KB article on the subject](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010092). +### CVE Reports +The most recent CVE Reports from Aqua Security are available as attachments below the article. +  +#### Armory CDSH CVE Reports +To find Armory CVE Reports for the CDSH product, please visit our KB article, [Common Vulnerabilities and Exposures (CVE) Reports - Armory Continuous Delivery](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010414) + diff --git a/kb/armory/General/compliance-and-auditing.md b/kb/armory/General/compliance-and-auditing.md new file mode 100644 index 0000000000..b9d3696b26 --- /dev/null +++ b/kb/armory/General/compliance-and-auditing.md @@ -0,0 +1,17 @@ +--- +title: Compliance and Auditing +--- + + +Question: +How can Spinnaker help me with Compliance and Auditing around my deployments? +Answer: +While Spinnaker doesn’t have any *explicit* functionality around auditing and compliance it exposes a number of tools that will help you craft a solution that fits your needs and use cases. +**Authentication and Authorization** +Out of the box, Spinnaker supports a wide range of Authentication and Authorization providers. With Authentication enabled you can attach an identity to any action taken within Spinnaker. Further, with Authorization enabled, you can restrict access to Applications and Accounts to particular sets of users which means that only those with permission can have read or write access. By enabling both of these features you can ensure that only the users you have authorized have access to the systems they should. To learn about how to enable these features, be sure to check out our docs on [Authentication](https://docs.armory.io/install-guide/auth/) and [Authorization](https://docs.armory.io/install-guide/authz/). +**Event Logging** +Spinnaker provides a way to forward all events to external logging and event systems. These events fire each time an action is taken within the platform and user data is also attached to them. Once the data is collected in this external system you can use it however you see fit. This data can be used in a variety of ways such as proving that code was deployed to a staging environment before a production environment or that a specific user is the only one promoting code to production. You can find documentation on how to configure this [here](https://docs.armory.io/admin-guides/notifications/#audit-logs). +**Armory Pipelines As Code** +Armory [Pipelines as Code](https://docs.armory.io/user-guides/dinghy/) provides a way for teams to manage their pipelines alongside their application repositories. By pulling pipeline definitions into the application lifecycle they are subject to the same Pull Request/Review cycle as the rest of the application. This means that teams can make sure that there are no malicious changes to pipelines before they are put in place. Also, by having these pipelines as code, you can automate compliance checks as part of CI to ensure that any required stages are present and that any pipelines with missing stages aren’t accepted into the repository. + + diff --git a/kb/armory/General/concurrent-pipelines-running-with-bakes-fail-packer-behavior-in-rosco.md b/kb/armory/General/concurrent-pipelines-running-with-bakes-fail-packer-behavior-in-rosco.md new file mode 100644 index 0000000000..fe131b7466 --- /dev/null +++ b/kb/armory/General/concurrent-pipelines-running-with-bakes-fail-packer-behavior-in-rosco.md @@ -0,0 +1,16 @@ +--- +title: Concurrent Pipelines running with Bakes Fail (Packer behavior in Rosco) +--- + +## Issue +Attempting to run two concurrent Bake jobs (packer-based bakes only) with the same parameters in different pipelines can cause unexpected failures.  Running the same pipelines with ample time for one pipeline to complete and then the other generates no errors.   +  + +## Cause +Due to how resources are identified to prevent conflicts, bake operations can fail with little information about why a failure occurred.  Running the bakes non-concurrently (at separate times) will work fine.  This issue primarily impacts Azure bakes due to how Azure tracks unique bake operations though it can potentially affect Bakes to other cloud providers. + +The issue can be traced back to how Bakes are run.  When Bakes are run, a combination of values generates a unique key for resource and tracking purposes. +If a Bake doesn't correctly generate a unique resource for a given provider (AWS/GCP/Azure), unexpected failures can occur.  Azure is the most common place where these failures occur as Azure doesn't utilize a ```base_ami``` value, and the baking system was initially designed for AWS AMIs.  Azure bakes use a value, ```base_name```, which is not automatically added as part of the unique key (see issue: [https://github.com/spinnaker/spinnaker/issues/6738](https://github.com/spinnaker/spinnaker/issues/6738)). + +Suppose a bake occurs without unique parameters for the set of fields.  In that case, it will likely fail upon concurrent executions because there will be an overlap in the bakes since the new Bake has the same parameters as the Bake that has yet to be completed. + diff --git a/kb/armory/General/configure-elasticache-with-spinnaker.md b/kb/armory/General/configure-elasticache-with-spinnaker.md new file mode 100644 index 0000000000..b688f1ad24 --- /dev/null +++ b/kb/armory/General/configure-elasticache-with-spinnaker.md @@ -0,0 +1,60 @@ +--- +title: Configure ElastiCache with Spinnaker +--- + +## Introduction +Spinnaker stores its pipeline execution history in REDIS. The default configuration deploys a pod in the kubernetes cluster. If the pods gets restarted, all of the history gets lost.Using ElastiCache removes this possibility. + +## Prerequisites +N/A + +## Instructions +We are going to use a terraform script to create all that we need for this. The script is located in this repository: [https://github.com/armory/terraform/tree/master/elasticache](https://github.com/armory/terraform/tree/master/elasticache)This script creates an ElastiCache cluster using a Redis engine, it’s based on an already existing vpc using its subnet ids, this to allow connectivity to the Spinnaker cluster.  +### Variables +In the file ```terraform.tfvars```” you need to set some variables. +-subnet-1-cidr: the eks subnet cluster +-subnet-2-cidr = the eks subnet cluster +-availability_zones: zones where a node of the cluster will be available. This needs to be equal or less than the number of nodes in the cluster +-security_group_ids: The eks nodes sg +### Execute the Script +``` +terraform init +terraform plan -var-file=terraform.tfvars +terraform apply -var-file=terraform.tfvars +``` + +### Result +After run the script the infrastructure should be created and the output of the script is the primary endpoint of the ElastiCache cluster. +  +### Validate Spinnaker has Connection to Elasticache +To validate Spinnaker has connection to Elasticache you need to exec into the Gate pod and run +```nc -zv 6379``` +Console should should show: +bash-4.4$ nc -zv myelasticache.usw2.cache.amazonaws.com 6379 +myelasticache.usw2.cache.amazonaws.com (0.0.0.0:6379) open +If it’s not open, you should verify the subnet ids, and security groups to validate it is correct.  +## Configuring Spinnaker with the Cluster +To configure spinnaker to use this cluster follow:Using elasticache in aws eks cluster: Run terraformer scripts: Need to output the dns endpointIf terraformer is enabled, add the following to ```terraformer-local.yml``` +``` + redis: + host: + port: +``` +If dinghy is enabled, add the following to ```dinghy-local.yml``` +``` + services: + redis: + address: +``` +In ```redis.yml```, add the following code: +overrideBaseUrl: redis://: +skipLifeCycleManagement: true +Disable automatic Redis configuration in Gate by adding the following to your ```gate-local.yml``` file: +``` + redis: + configuration: + secure: true +``` +To delete the default redis pod, run +```kubectl -n spinnaker delete deployment spin-redis``` + diff --git a/kb/armory/General/configure-persistent-storage-to-capture-heapdumps-for-spinnaker-services.md b/kb/armory/General/configure-persistent-storage-to-capture-heapdumps-for-spinnaker-services.md new file mode 100644 index 0000000000..f66c81022a --- /dev/null +++ b/kb/armory/General/configure-persistent-storage-to-capture-heapdumps-for-spinnaker-services.md @@ -0,0 +1,75 @@ +--- +title: Configure persistent storage to capture heapdumps for Spinnaker Services +--- + +## Introduction +There might be a need to capture ```heapdump``` of Java-based Spinnaker Services to troubleshoot items such as memory issues. +Pods and containers are stateless and the storage on the containers are generally ephemeral which means the ***dump generated is lost when the service goes down***.  +This article explains the steps involved to configure heapdump and non-ephemeral storage on Spinnaker Services to retain these dumps. Clouddriver is taken as a reference. However, the same can be applied to other Java-based Spinnaker Services such as Orca, Gate, Front50 etc.  + +## Prerequisites +N/A + +## Instructions +There are couple steps involved to configure the heapdump to be stored on a non-ephemeral storage. +### Create a storage and mount them on the container for which heapdumps are required    +Volumes can be mounted on to the container in two ways as explained below +Generate a Kustomize patch and have the volumes and volume mounts created for the container. This involves adding the patch to the deployment of the required Spinnaker Service as shown below + +``` +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker +spec: + kustomize: + clouddriver: + deployment: + patchesStrategicMerge: + - | + spec: + template: + spec: + containers: + - name: clouddriver + volumeMounts: + - name: spinnaker-home + mountPath: /path/to/heapdump + volumes: + - name: spinnaker-home + emptyDir: {}​ +``` +Configure the volumes under the Kubernetes section of ```spec.spinnakerConfig.service-settings``` of the Spinnaker Service that the volume needs to be mounted in: +``` +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker +spec: + spinnakerConfig: + service-settings: + clouddriver: + kubernetes: + volumes: + - id: heap-dump + type: emptyDir + mountPath: /path/to/heapdump​ +``` +* After applying one of the above mentioned configurations, the volume can be found mounted in the Clouddriver pod(s). + +### **Configure JAVA_OPS parameters to generate Heapdump when JVM runs out of memory** +To generate the heapdump for the Spinnaker Service when it runs out of memory, add the below ```JAVA_OPTS``` parameters for the intended service under ```spec.spinnakerConfig.service-settings``` of Spinnaker manifest.  +``` +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker +spec: + spinnakerConfig: + service-settings: + clouddriver: + env: + JAVA_OPTS: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/heapdump +``` +This shall generate the heapdump under ```/path/to/heapdump``` when Clouddriver runs out of memory. + diff --git a/kb/armory/General/configure-spinnaker-to-use-dynamic-github-authentication-tokens.md b/kb/armory/General/configure-spinnaker-to-use-dynamic-github-authentication-tokens.md new file mode 100644 index 0000000000..b54509cded --- /dev/null +++ b/kb/armory/General/configure-spinnaker-to-use-dynamic-github-authentication-tokens.md @@ -0,0 +1,56 @@ +--- +title: Configure spinnaker to use dynamic Github authentication tokens +--- + +## Introduction +An organization may choose to have a rotation policy for the Github tokens to improve security. +In such scenarios, changes to the token are to be automatically reflected in Clouddriver without any restarts. Starting ***Armory release v2.26.2***, the GitHub, GitLab, and GitRepo artifact providers support token files that are dynamically updated. The token file is automatically reloaded when Armory Enterprise makes a request. +Please refer to the release notes for more details [https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-26-2/#artifacts](https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-26-2/#artifacts). + + +## Prerequisites +Below are the prerequisites to have the dynamic Github tokens configured with spinnaker +* Armory Spinnaker version 2.26.2 or higher.* Process to rotate the Github tokens. Rotation of tokens are assumed to be the responsibility of the customer. This can be achieved by leveraging the Github rest APIs.  + +## Instructions +Once the above prerequisites are met, follow the below instructions to configure the dynamic tokens for Github on operator based installations +Specify the path of the ```tokenFile: ```under the path ```spec.spinnakerConfig.config.artifacts.github .```This is the path where the token file shall reside under the clouddriver pod(s).  The field ```tokenFile: ```supports encrypted secret references. For instance, lets assume that the path form the token file is ```/tmp/github/github-token ``` +``` +spec: + spinnakerConfig: + config: + artifacts: + github: + enabled: true + accounts: + - name: gitrepo + username: xxx + tokenFile: encrypted:k8s!n:git-hub-dynamic-token!k:github-token #This field support "encrypted" secret references​ +``` +Create a secret containing the Github auth token which is ```base64 ```encoded. +``` +apiVersion: v1 +kind: Secret +metadata: + name: git-token +data: + github-token: XXXXXXXXX​ +Mount the secret and add the service settings for Clouddriver under ```spec.spinnakerConfig.service-settings.clouddriver.kubernetes``` as shown below +spec: + spinnakerConfig: + service-settings: + clouddriver: + kubernetes: + volumes: + - id: git-token + mountPath: /tmp/github + type: secret​ +``` +* Apply the above changes and deploy Clouddriver. +* It can be seen that the Clouddriver pod now has the Github token under ```/tmp/github/github-token```. +* If there is a change in the token, the secret in which the token resides has to be updated so that file on the Clouddriver pod automatically reflects the change. Clouddriver service ideally refers the token file before making any calls to Github and hence any changes to the token shall immediately take effect and would not require any Clouddriver restart.  + + + + + diff --git a/kb/armory/General/configure-spinnakers-external-account-configuration-with-vault.md b/kb/armory/General/configure-spinnakers-external-account-configuration-with-vault.md new file mode 100644 index 0000000000..c46ed45693 --- /dev/null +++ b/kb/armory/General/configure-spinnakers-external-account-configuration-with-vault.md @@ -0,0 +1,141 @@ +--- +title: Configure Spinnakers External Account Configuration with Vault +--- + +## Introduction +##### WARNING: Spring Cloud Config Server is not recommended for use with Armory Enterprise. As an alternative, Armory has built the External Accounts plugin to provide our customers with a reliable replacement. +Customers may find that their environment requires a way to add, delete, or modify Kubernetes deployment targets on a regular basis without redeploying Clouddriver to pull in the new account configuration.  This is because redeploying Clouddriver can impact their teams' productivity. +Spinnaker's [External Account Configuration](https://spinnaker.io/docs/setup/other_config/configuration/#external-account-configuration) feature allows customers to manage account configuration externally from Spinnaker and then read that configuration change without requiring a redeployment of Clouddriver. + +```External Account Configuration``` is only supported for ***Kubernetes and CloudFoundry provider accounts***. This document describes how this works with Kubernetes accounts. + +```External Account Configuration``` uses Spring Cloud Config Server to allow portions of Clouddriver's configuration to be hosted in an external source. See [Enabling external configuration](https://www.spinnaker.io/setup/configuration/#enabling-external-configuration) for details on the implementation and its limitations. +The general, high-level steps involved in setting up dynamic Kubernetes accounts are: +* Create a JSON type secret in Vault. This secret stores the entire contents of the Kubernetes account portion of the Clouddriver configuration.* Create or update the ```spinnakerconfig.yml``` file to enable Spring Cloud Config Server and to connect it to Vault.* Redeploy Spinnaker. + +## Prerequisites +This document assumes the following: +* A running Spinnaker cluster* A Vault instance is accessible from the Spinnaker cluster.* Vault tokens are available that Spinnaker can use to access the Vault instance.* The ```kubeconfig``` is valid for the target Kubernetes cluster. + +## Instructions +### 1. Create the secret in Vault +The secret in Vault contains the ```accounts``` section that was previously in the Halyard or Operator configuration. Note that the configuration should be left in Halyard or Operator for the Kubernetes account where Spinnaker is deployed. Clouddriver *replaces* all of its account information with what it finds in the Vault token. The configuration for the Spinnaker cluster should be added in the case that the cluster will be used as a deployment target for Clouddriver. +The ```kubeconfig``` file for each cluster is stored inline as a single line string in the ```kubeconfigContents``` element of the JSON. A ```sed``` command can be used to convert a ```kubeconfig``` file to a string: + +```sed -E ':a;N;$!ba;s/\r{0,1}\n/\\n/g' kubeconfig.yml``` + +Create a secret in Vault of type JSON with contents specific for the accounts: +``` +{ +"kubernetes": { + "accounts": [ + { + "cacheThreads": 1, + "cachingPolicies": [], + "configureImagePullSecrets": true, + "customResources": [], + "dockerRegistries": [], + "kinds": [], + "kubeconfigContents": "", + "name": "", + "namespaces": [], + "oAuthScopes": [], + "omitKinds": [], + "omitNamespaces": [], + "onlySpinnakerManaged": true, + "permissions": {}, + "providerVersion": "V2", + "requiredGroupMembership": [] + }, + { + "cacheThreads": 1, + "cachingPolicies": [], + "configureImagePullSecrets": true, + "customResources": [], + "dockerRegistries": [], + "kinds": [], + "kubeconfigContents": "", + "name": "", + "namespaces": [], + "oAuthScopes": [], + "omitKinds": [], + "omitNamespaces": [], + "onlySpinnakerManaged": true, + "permissions": {}, + "providerVersion": "V2", + "requiredGroupMembership": [] + }, + ] +} +} +``` + +### 2. Update spinnakerconfig.yml and redeploy Spinnaker +In order to complete the configuration of Spinnaker to use dynamic accounts, the backend and default-key fields must be defined. This is directly related to how the secret was created in Vault.  For example, if the secret is set to ```spinnaker```/```clouddriver```: +* ```backend``` is ```spinnaker```* ```default-key``` is ```clouddriver```. +The Spring Cloud Config Server that is internal to Clouddriver only supports the ```Vault token authentication``` type. An access token must be created in order to configure Clouddriver. +#### Operator +In the ```spinnakerService``` manifest: +``` +apiVersion: spinnaker.armory.io/{{}} +kind: SpinnakerService +metadata: + name: spinnaker +spec: + spinnakerConfig: + files: + profiles__spinnakerconfig.yml: | + spring: + profiles: + include: vault + cloud: + config: + server: + vault: + host: + port: 8200 + backend: + kvVersion: 2 + scheme: http + default-key: + token: +``` +Then, run the following command to deploy the changes: + +```kubectl -n spinnaker apply -f spinnakerservice.yml``` + +#### Halyard +Create or update the ```spinnakerconfig.yml``` file (located in ```.hal/default/profiles``` by default), with the following content: +``` +spring: + profiles: + include: vault + cloud: + config: + server: + vault: + host: + port: 8200 + backend: + kvVersion: 2 + scheme: http + default-key: + token: +``` +Then, run ```hal deploy apply``` to deploy the changes. +### 3. Check Spinnaker for new accounts +When all of the pods are running and ready, perform a hard refresh of the Spinnaker web interface. The accounts added in the Vault secret should now be available in the web interface to use. + +### Troubleshooting +If the configuration in the ```spinnakerconfig.yml``` is incorrect, Clouddriver may not start. Because External Account Configuration is also available for the Echo and Igor services, some issues may appear within those pods as well. Check the Clouddriver logs for errors related to the Vault profile. Use the ```kubectl logs ``` command to view the logs. +If External Account Configuration is working properly, the messages similar to the following can be found in the Clouddriver logs as long as the accounts were loaded and there were no changes: +```` +2020-04-23 13:49:29.921 INFO 1 --- [reshScheduler-0] o.s.boot.SpringApplication : The following profiles are active: composite,vault,local +2020-04-23 13:49:29.951 INFO 1 --- [reshScheduler-0] o.s.boot.SpringApplication : Started application in 1.55 seconds (JVM running for 63660.417) +2020-04-23 13:49:30.180 INFO 1 --- [reshScheduler-0] k.v.c.KubernetesV2ProviderSynchronizable : No changes detected to V2 Kubernetes accounts. Skipping caching agent synchronization. +If a change to an account were made, the following similar messages can be found: +2020-04-23 14:07:00.905 INFO 1 --- [reshScheduler-0] o.s.boot.SpringApplication : The following profiles are active: composite,vault,local +2020-04-23 14:07:00.921 INFO 1 --- [reshScheduler-0] o.s.boot.SpringApplication : Started application in 1.602 seconds (JVM running for 64711.387) +2020-04-23 14:07:01.178 INFO 1 --- [reshScheduler-0] k.v.c.KubernetesV2ProviderSynchronizable : Synchronizing 1 caching agents for V2 Kubernetes accounts. +2020-04-23 14:07:01.181 INFO 1 --- [reshScheduler-0] k.v.c.KubernetesV2ProviderSynchronizable : Adding 3 agents for account newaccount +```` \ No newline at end of file diff --git "a/kb/armory/General/configuring-aws-account-using-eap---error-running-createservergrouptask-for-pipeline-credentials-not-found*\".md" "b/kb/armory/General/configuring-aws-account-using-eap---error-running-createservergrouptask-for-pipeline-credentials-not-found*\".md" new file mode 100644 index 0000000000..df09c3d967 --- /dev/null +++ "b/kb/armory/General/configuring-aws-account-using-eap---error-running-createservergrouptask-for-pipeline-credentials-not-found*\".md" @@ -0,0 +1,14 @@ +--- +title: Configuring AWS account using EAP - error running CreateServerGroupTask for pipeline Credentials not found*" +--- + +## Issue +When configuring AWS account using **EAP(External Account Plugin)** and an account with the name `````` is not configured the following error can appear in the logs: ```error running CreateServerGroupTask for pipeline`````` **Credentials not found*” error.**``` + +## Cause +Orca uses a property called: ```default.bake.account```. +The expectation is that AWS AMI images are registered in the defined default account using Rosco, and from there, we share those AMIs with the ```deploy``` account upon deployment. +The default value for ```default.bake.account``` is the named ```default```if it is not defined within Orca. +If the default value is called, and there isn't an ```account``` named ```default```, administrators will need to set one in the ```~/.hal/default/profiles/orca.yml``` file in Halyard or in the ```spec.spinnakerConfig.profiles.orca```section in Operator, with the account that is the source of the custom AWS AMIs. +If the default account is missing, the value further falls back to the information defined in the ```spec.spinnakerConfig.config.providers.aws.primaryAccount``` section which may not be the same and may not have the AMI images. + diff --git a/kb/armory/General/configuring-slack-notifications-with-sendcompactmessages-to-send-a-shorter-and-more-compact-message.md b/kb/armory/General/configuring-slack-notifications-with-sendcompactmessages-to-send-a-shorter-and-more-compact-message.md new file mode 100644 index 0000000000..5fb5bdedb6 --- /dev/null +++ b/kb/armory/General/configuring-slack-notifications-with-sendcompactmessages-to-send-a-shorter-and-more-compact-message.md @@ -0,0 +1,30 @@ +--- +title: Configuring Slack notifications with sendCompactMessages to send a shorter and more compact message +--- + +## Introduction +By default, Slack notifications are sent with a verbose message including details about the stage it was triggered from. This message can be unnecessarily detailed, especially if using a custom message. +By enabling ```sendCompactMessages``` , a less verbose Slack message will be sent with only necessary information. +### Without ```sendCompactMessages``` enabled: +** +** +### with ```sendCompactMessages``` enabled: + + + +## Prerequisites +Slack notifications enabled: [https://docs.armory.io/armory-enterprise/armory-admin/notifications-slack-configure/](https://docs.armory.io/armory-enterprise/armory-admin/notifications-slack-configure/) + +## Instructions +In addition to the above Slack notifications configuration in ```spec.spinnakerConfig.config.notifications.slack```, add the following to the Echo profile: +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker +spec: + spinnakerConfig: + profiles: + echo: # equivalent of echo-local.yml + slack: + sendCompactMessages: true + diff --git a/kb/armory/General/configuring-spinnaker-services-to-connect-to-redis-over-mtls.md b/kb/armory/General/configuring-spinnaker-services-to-connect-to-redis-over-mtls.md new file mode 100644 index 0000000000..b94ce5774d --- /dev/null +++ b/kb/armory/General/configuring-spinnaker-services-to-connect-to-redis-over-mtls.md @@ -0,0 +1,215 @@ +--- +title: Configuring Spinnaker services to connect to Redis over mTLS +--- + +## Introduction + + + + +Mutual TLS, or mTLS, is a method used for mutual authentication and ensures that both the server and the client authenticate each other simultaneously. This is achieved by the server providing a server certificate to the client and the client providing a client certificate to the server. These certificates are signed by the same Certificate Authority (CA). This article explains how to enable mTLS on Redis and configure Spinnaker services to connect to Redis over mTLS. + + + + + +## Prerequisites + + + + +Below are the prerequisites to enable mTLS between Spinnaker and Redis instance +* Redis server with version 6 and above. Older versions of Redis do not support mTLS +* Recent Spinnaker distribution +* Tools like OpenSSL and Keytool for generating certificates, signing them, and importing them to Java keystore. + + + + + +## Instructions + +## Table of Contents +* [Generating certificates and enabling mTLS on Redis](#mcetoc_1h2qfsbe8u)* [Enable Redis to Accept mTLS Requests](#mcetoc_1h2qfsbe8v)* [Enable Spinnaker to use mTLS to connect to Redis](#mcetoc_1h2qfsbe813)* [Considerations before enabling mTLS for Redis in Spinnaker](#mcetoc_1h2qfsbe814) + +Generating certificates and enabling mTLS on Redis +The CA, server, and client certs should be created before proceeding with the documentation below.  While there are various ways to generate this, the script [https://github.com/redis/redis/blob/unstable/utils/gen-test-certs.sh](https://github.com/redis/redis/blob/unstable/utils/gen-test-certs.sh) should help get started and generate the certs. +Enable Redis to Accept mTLS Requests +The next step is to enable Redis with mTLS enabled. This can be enabled on the redis.conf or when starting the Redis process. +Enabling TLS in redis.conf +Adding the below config to the ```redis.conf``` file will allow TLS on Redis. +port 0 +tls-port 6379 + +tls-cert-file /usr/local/etc/redis/tls/redis.crt +tls-key-file /usr/local/etc/redis/tls/redis.key + +tls-ca-cert-file /usr/local/etc/redis/tls/ca.crt +Enabling TLS through start-up command +``` + containers: + - name: redis-ssl + image: redis:6.2 + command: + - redis-server + - "/redis-master/redis.conf" + args: + - --TLS-port 6379 + - --port 0 + - --TLS-cert-file /certs/redis.crt + - --TLS-key-file /certs/redis.key + - --TLS-ca-cert-file /certs/ca.crt + - --TLS-auth-clients yes +``` +If running Redis as a container on the kubernetes cluster, the certificates must be mounted on the container through a secret. +Validate login using redis-cli +Connect to the Redis instance over mTLS using the certificates and keys created above to validate the login: +```$redis-cli -u rediss://redis-ssl.instance.endpoint --tls --cert redis_client.crt --key redis_client.key --cacert ca.crt``` +In case of issues with certs, the login shall fail with errors similar to the one below, and the prompt shall default to ```not connected```. +Could not connect to Redis at redis-ssl.instance.endpoint:6379: SSL_connect failed: certificate verify failed +not connected> exit +Enable Spinnaker to use mTLS to connect to Redis +All the services in the core Spinnaker are built on Java, while a couple of services in Armory CDSH are built on goLang (Namely Dinghy and Terraformer). Although the GoLang services support Redis connections with TLS enabled, they do not currently support mTLS. To have Spinnaker Java-based services connect to Redis through mTLS, the below steps are to be followed: +To begin with, Spinnaker has to point to the Redis with mTLS enabled. To start this, the following content has to be added under ```~/.hal/$DEPLOYMENT/service-settings/redis.yml```for Halyard installations +overrideBaseUrl: rediss://redis-ssl.instance.endpoint:6379 +skipLifeCycleManagement: true +A similar config should be added in the ```spec.spinnakerConfig.service-settings``` section for Operator  +``` +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker +spec: + spinnakerConfig: + service-settings: + redis: + overrideBaseUrl: rediss://redis-ssl.instance.endpoint:6379 + skipLifeCycleManagement: true +``` +The above setting would override the Redis URL for all services. If the Redis endpoints are to be overridden for specific services, the below config should be added under the ```profiles section``` for each serviceFor Halyard installations, for the example of the Gate service, add the configuration in the file ```~/.hal/$DEPLOYMENT/profiles/gate-local.yml```. Each service would have its corresponding YAML file and should be created if they do not exist. +``` +redis: + baseUrl: rediss://redis-ssl.instance.endpoint:6379 + enabled: true +``` +In Operator installations, administrators will need to add the following changes for each service in the ```spec.spinnakerConfig.profiles``` section. In the example below, it is being added to the Gate service: +``` +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker # name doesn't matter since this is a patch +spec: + # spec.spinnakerConfig - This section is how to specify configuration spinnaker + spinnakerConfig: + profiles: + gate: + redis: + baseUrl: rediss://redis-ssl.instance.endpoint:6379 + enabled: true +``` +Administrators will also need to make the following modification to  ```Gate```.  In Halyard, add the following configuration to the ```~/.hal/$DEPLOYMENT/profiles/gate-local.yml``` file +``` +redis: + configuration: + secure: + true +``` +In Operator installations, administrators will need to add the following changes for each service in the ```spec.spinnakerConfig.profiles.gate``` section.   +``` +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker +spec: + spinnakerConfig: + profiles: + gate: + redis: + configuration: + secure: + true +``` +Create a **Keystore** file that stores the key and certificate created earlier. The certs and the key have to be converted to ```pkcs12 format```: +``` +openssl pkcs12 -export \\ + -in redis_client.crt \\ + -inkey redis_client.key \\ + -out redis-keystore.p12 \\ + -name "redis-keystore" +``` +###Import the pkcs12 file into Java keystore +``` +keytool -genkey \\ + -dname "cn=Generic-cert" \\ + -alias truststorekey \\ + -keyalg RSA \\ + -keystore ./redis-truststore.p12 \\ + -keypass "Somepassword" \\ + -storepass "Somepassword" \\ + -storetype pkcs12 +Create a **Trust Store** file and add the Redis cluster certificate to it +keytool -import \\ + -keystore cacerts \\ + -file ./ca.crt \\ + -alias redis-truststore +``` +Create secrets with the Keystore and the Trust Store. +``` +kubectl create secret generic internal-trust-store --from-file cacerts +kubectl create secret generic redis-key-store --from-file redis-keystore.p12 +``` + +Mount the secrets to Spinnaker services. Mount the Keystore and Trust Store from the secret onto the Spinnaker service(s).For Halyard installations, for the example of the Gate service, add the configuration in the file ```~/.hal/$DEPLOYMENT/service-settings/$SERVICE.yml```. Each Spinnaker service would have its own corresponding YAML file and should be created if they do not exist. +``` + kubernetes: + volumes: + - id: internal-trust-store + mountPath: /etc/ssl/certs/java + type: secret + - id: redis-key-store + mountPath: /tmp/certs + type: secret +``` +For Operator installations, the changes should be added in the ```spec.spinnakerConfig.service-settings``` section for each service. For example, for Gate, customers should make the following change: +``` +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker +spec: + spinnakerConfig: + service-settings: + gate: + kubernetes: + volumes: + - id: internal-trust-store + mountPath: /etc/ssl/certs/java + type: secret + - id: redis-key-store + mountPath: /tmp/certs + type: secret +``` +Supply JAVA_OPTS with the keystore cert and the password in the environment variable section.For Halyard installations, for the example of the Gate service, add the configuration in the file ```~/.hal/$DEPLOYMENT/service-settings/$SERVICE.yml```. Each Spinnaker service would have its own corresponding YAML file and should be created if they do not exist. +``` +env: + JAVA_OPTS: -Djavax.net.ssl.keyStore=/tmp/certs/redis-keystore.p12 -Djavax.net.ssl.keyStorePassword=somepassword +``` +For Operator installations, the changes should be added in the ```spec.spinnakerConfig.service-settings``` section for each service. For example, for Gate, customers should make the following change: +``` +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker +spec: + spinnakerConfig: + service-settings: + gate: + env: + JAVA_OPTS: -Djavax.net.ssl.keyStore=/tmp/certs/redis-keystore.p12 -Djavax.net.ssl.keyStorePassword=somepassword +``` +* Apply all the above changes, and Spinnaker should be able to connect to Redis over mTLS. +  +Considerations before enabling mTLS for Redis in Spinnaker +Services such as Clouddriver, Front50, Orca, and Fiat support SQL and Redis. The features in these services can be split to use either Redis or SQL. For those services, it is recommended not to use mTLS as the service would try to connect to the SQL instance with the mTLS certificate, which shall be rejected by the SQL server, causing the service not to start. +For such cases, overriding the Redis endpoint at the service level is recommended, as discussed above. + diff --git a/kb/armory/General/configuring-spinnaker-to-use-internal-certificate-authorities.md b/kb/armory/General/configuring-spinnaker-to-use-internal-certificate-authorities.md new file mode 100644 index 0000000000..7f8e433865 --- /dev/null +++ b/kb/armory/General/configuring-spinnaker-to-use-internal-certificate-authorities.md @@ -0,0 +1,72 @@ +--- +title: Configuring Spinnaker to Use Internal Certificate Authorities +--- + +## Introduction +There are a lot of places in Spinnaker which support the ability to configure custom trust/key stores for organizations who use internally signed certificates. In some cases, however, it isn’t supported yet but the environment will still need to talk to a service which serves one of these certificates. +This article shows how to import certificates into the trust/key store and configure a Spinnaker service with it.Most services in Spinnaker are written in Java. There are a few Armory specific services written in Golang, such as Dinghy and Terraformer. +If the particular certificate is not configured properly, it is possible that errors such as the one below will be found in the logs of the service +```x509: certificate signed by unknown authority``` + +## Prerequisites +This document assumes that ```kubectl``` access is available to the Kubernetes cluster and the latest version of Halyard (either OSS or Armory’s extension) is being used. + +## Instructions +Create a temporary copy of your system’s trust/key store and import your internal certificate. For a Java truststore, on a Mac, ruststore will be located at ```$(/usr/libexec/java_home/)/jre/lib/security/cacerts```. It will be different on Linux. +``` +$ mkdir /tmp/custom-trust-store +$ cp {path-to-cacerts} /tmp/custom-trust-store +$ keytool import -alias custom-ca -keystore /tmp/custom-trust-store/cacerts -file {your-internal-certificate} +``` +For Golang services, make a copy of ```/etc/ssl/cert.pem``` and append the internal certificate to your copy of ```cert.pem```. *Note: If you do not have a copy of the internal certificate - you can run the following command to retrieve it manually:* +```openssl s_client -showcerts -connect host:port |openssl x509 -outform PEM >mycertfile.pem``` +Use ```kubectl``` to create a Secret from your copy of ```cacerts``` or ```cert.pem```. +``` +$ kubectl create secret generic -n {your-spinnaker-namespace} internal-trust-store --from-file /tmp/custom-trust-store/cacerts +``` +Configure a Spinnaker service with the new trust/key store using volume mount. In this example we’ll be configuring Front50 with this new store. +**In Halyard** *Disclaimer* The ```mountPath``` suggested should work for most cases but we did notice that this could be specific to the image used. (eg. when using Ubuntu, users will need to use ```/etc/ssl/certs``` for Golang services) +```# ~/.hal/default/service-settings/front50.yml``` +``` +kubernetes: + volumes: + - id: internal-trust-store + mountPath: /etc/ssl/certs/java + type: secret +``` +```# ~/.hal/default/service-settings/dinghy.yml``` or other Golang based services kubernetes:  +``` +kubernetes: + volumes: + - id: internal-trust-store + mountPath: /etc/ssl/ + type: secret +``` +**In Operator***Disclaimer* The ```mountPath``` suggested should work for most cases but we did notice that this could be specific to the image used. (eg. when using Ubuntu, users will need to use ```/etc/ssl/certs``` for Golang services) +In the ```SpinnakerService.yml```, in the ```spec.spinnakerConfig.service-settings``` section for each service based in Java (e.g. Front50) +spec: +``` + spinnakerConfig: + service-settings: + front50: + kubernetes: + volumes: + - id: internal-trust-store + mountPath: /etc/ssl/certs/java + type: secret +``` +In the ```SpinnakerService.yml```, in the ```spec.spinnakerConfig.service-settings``` section for each service in Golang (e.g. Dinghy):  +``` +spec: + spinnakerConfig: + service-settings: + dinghy: + kubernetes: + volumes: + - id: internal-trust-store + mountPath: /etc/ssl/ + type: secret +``` +* Redeploy Spinnaker using ```hal deploy apply```, or deploying the Operator Manifest +The Spinnaker component for which you configured the volume mount should now be using the new trust/key store by default. + diff --git a/kb/armory/General/configuring-vim-editor-to-work-with-yaml.md b/kb/armory/General/configuring-vim-editor-to-work-with-yaml.md new file mode 100644 index 0000000000..b5ebd65812 --- /dev/null +++ b/kb/armory/General/configuring-vim-editor-to-work-with-yaml.md @@ -0,0 +1,25 @@ +--- +title: Configuring VIM editor to work with YAML +--- + +## Introduction +Spinnaker configs are in YAML. It is best to set up VIM to auto-indent properly so that there are less syntax errors due to tab insertion and/or spacing problems. + +## Prerequisites +VIM + +## Instructions +## For files that end with .yaml or .yml +Create or update ```~/.vimrc``` file with the following: +syntax enable +filetype on +autocmd FileType yaml,yml setlocal ts=2 sts=2 sw=2 expandtab indentkeys-=0# indentkeys-= +## For arbitrary files +The following comment can be added to the top of any file to let VIM know what configuration it should use +```# vim: set shiftwidth=2 tabstop=2 softtabstop=2 expandtab:``` +## Fix files that have a mix of tabs and spaces to all spaces +In VIM's command mode enter the following +# to go to command mode hit the Esc key +:set shiftwidth=2 tabstop=2 softtabstop=2 expandtab +:retab + diff --git a/kb/armory/General/confirming-environment-logs-and-environment-debugging-id-with-armory-support.md b/kb/armory/General/confirming-environment-logs-and-environment-debugging-id-with-armory-support.md new file mode 100644 index 0000000000..f945595732 --- /dev/null +++ b/kb/armory/General/confirming-environment-logs-and-environment-debugging-id-with-armory-support.md @@ -0,0 +1,14 @@ +--- +title: Confirming Environment Logs and Environment Debugging ID with Armory Support +--- + +## Introduction +To assist with troubleshooting, Armory Support asks our customers to please set up their environment with [Armory Diagnostics and Monitoring](https://docs.armory.io/docs/spinnaker-install-admin-guides/admin-diagnostics/) so that logs are provided to Armory support.  This will allow Armory to assist in a quicker fashion because Armory Support can investigate logs without a rigorous search and [submission by our customers](https://kb.armory.io/s/article/Provide-Logs-from-Pods-to-Armory-Support-for-Troubleshooting-Purposes) .However, there are times even when customers have been sending debugging logs, Support may still ask to confirm the environment to avoid diagnosis on the incorrect environment.  Support may ask to confirm the debug id of the environment to ensure we are observing the correct logs.We also understand that some of our customers may have security requirements and preferences that prevent diagnostic information from being sent to Armory.  In these cases, [we ask that logs be attached to your tickets to help assist with the process](https://kb.armory.io/s/article/Provide-Logs-from-Pods-to-Armory-Support-for-Troubleshooting-Purposes). + +## Prerequisites +Customer environments that are being monitored by Armory have already run through this set up first[https://docs.armory.io/docs/spinnaker-install-admin-guides/admin-diagnostics/](https://docs.armory.io/docs/spinnaker-install-admin-guides/admin-diagnostics/) + +## Instructions +After diagnostics have been enabled, your Hal Config file or Operator Spinnaker Service file will have new information and a section called ```diagnostics``` +Within this section, there is a ```uuid``` that is generated that can be used to confirm the environment matches the one we have on file.  Please provide this uuid to support if they require confirmation + diff --git a/kb/armory/General/container-image-cannot-be-selected.md b/kb/armory/General/container-image-cannot-be-selected.md new file mode 100644 index 0000000000..0b85613ed8 --- /dev/null +++ b/kb/armory/General/container-image-cannot-be-selected.md @@ -0,0 +1,40 @@ +--- +title: Container Image cannot be selected +--- + +## Issue +A pipeline has been configured to deploy a service to an ECS cluster within an AWS account. Upon defining a server group in the pipeline using ```inputs```, the Configure Deployment Cluster form is not able to retrieve any of the container images available as per the screenshot belowThe Spinnaker Managed Role has an ECR Read Only policy attached, and supports the ECR repositories on the same account. +The Clouddriver logs show that the aws, ecs and ecr account are caching but the images do not populate in the UI. +2021-08-24 16:22:55.235 INFO 1 --- [ main] c.n.s.c.security.ProviderUtils : Adding accounts [........, ........,......... ]of type NetflixAmazonCredentials... +2021-08-24 16:22:56.498 INFO 1 --- [ main] c.n.s.c.security.ProviderUtils : Adding accounts [......-ecr] of type DockerRegistryNamedAccountCredentials... +2021-08-24 16:24:02.186 INFO 1 --- [ecutionAction-0] .d.r.p.a.DockerRegistryImageCachingAgent : Describing items in .......-ecr/DockerRegistryImageCachingAgent[2/3] +2021-08-24 16:24:02.219 INFO 1 --- [ecutionAction-0] .d.r.p.a.DockerRegistryImageCachingAgent : Caching 3 tagged images in ........ecr/DockerRegistryImageCachingAgent[2/3] +2021-08-24 16:24:02.219 INFO 1 --- [ecutionAction-0] .d.r.p.a.DockerRegistryImageCachingAgent : Caching 3 image ids in .......-ecr/DockerRegistryImageCachingAgent[2/3] +.... + +## Cause +The issue can be traced back to the following exceptions that can be found in the CloudDriver and Gate logs, indicating the issue has to do with Hystrix.  Hystrix has been removed from Spinnaker OSS as of 2.24.x +Clouddriver: +``` +2021-08-24 11:42:56.529 ERROR 1 --- [.0-7002-exec-10] c.n.s.k.w.e.GenericExceptionHandlers : Internal Server Errororg.codehaus.groovy.runtime.typehandling.GroovyCastException: Cannot cast object 'com.netflix.spinnaker.clouddriver.aws.security.NetflixAssumeRoleAmazonCredentials@.... with class 'com.netflix.spinnaker.clouddriver.aws.security.NetflixAssumeRoleAmazonCredentials' to class 'com.netflix.spinnaker.clouddriver.docker.registry.security.DockerRegistryNamedAccountCredentials' + at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.continueCastOnSAM(DefaultTypeTransformation.java:415) ~[groovy-2.5.10.jar:2.5.10] + at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.continueCastOnNumber(DefaultTypeTransformation.java:329) ~[groovy-2.5.10.jar:2.5.10] + at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.castToType(DefaultTypeTransformation.java:243) ~[groovy-2.5.10.jar:2.5.10] + at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.castToType(ScriptBytecodeAdapter.java:615) ~[groovy-2.5.10.jar:2.5.10] + at com.netflix.spinnaker.clouddriver.docker.registry.controllers.DockerRegistryImageLookupController$_find_closure5.doCall(DockerRegistryImageLookupController.groovy:104) ~[clouddriver-docker-GCSFIX.jar:na] +****** +``` +Gate: +``` +2021-07-24 11:42:42.213 INFO 1 --- [etricService-10] c.n.s.g.s.internal.ClouddriverService : HTTP GET +2021-07-24 11:42:42.224 INFO 1 --- [overyService-10] c.n.s.g.s.internal.ClouddriverService : ---> HTTP GET +2021-07-24 11:42:42.227 INFO 1 --- [ecretService-10] c.n.s.g.s.internal.ClouddriverService : HTTP GET +2021-07-24 11:42:42.586 INFO 1 --- [loadBalancers-8] c.n.s.g.s.internal.ClouddriverService : ---> HTTP GET +2021-07-24 11:42:42.603 INFO 1 --- [.0-8084-exec-17] c.n.s.g.s.internal.ClouddriverService : ---> HTTP GET +2021-07-24 11:42:42.640 INFO 1 --- [.0-8084-exec-17] c.n.s.g.s.internal.ClouddriverService : HTTP GET +2021-07-24 11:42:42.662 INFO 1 --- [-roleService-10] c.n.s.g.s.internal.ClouddriverService : ---> HTTP GET +2021-07-24 11:42:42.667 INFO 1 --- [etricService-10] c.n.s.g.s.internal.ClouddriverService : ---> HTTP GET +2021-07-24 11:42:42.669 INFO 1 --- [ecretService-10] c.n.s.g.s.internal.ClouddriverService : ---> HTTP GET +2021-07-24 11:42:42.673 INFO 1 --- [overyService-10] c.n.s.g.s.internal.ClouddriverService : ---> HTTP GET +2021-07-24 11:42:42.680 INFO 1 --- [-roleService-10] c.n.s.g.s.internal.ClouddriverService : +``` diff --git a/kb/armory/General/continuous-warning-messages--http2exception$streamexception--received-data-frame.md b/kb/armory/General/continuous-warning-messages--http2exception$streamexception--received-data-frame.md new file mode 100644 index 0000000000..aecfd1b30e --- /dev/null +++ b/kb/armory/General/continuous-warning-messages--http2exception$streamexception--received-data-frame.md @@ -0,0 +1,16 @@ +--- +title: Continuous warning messages -Http2Exception$StreamException- Received DATA frame +--- + +## Issue +An organization may encounter this warning upon upgrading the Armory Agent + Plugin. This article is based on an upgrade to Armory Agent 0.8.25 and Plugin version to 0.5.30. +``` +WARN --- [-worker-ELG-3-1] i.g.n.s.io.grpc.netty.NettyServerStream : [] Exception processing message +io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 4194304: 6027014 +``` +This warning will continue and fill logs. This noise can be detrimental to other attempts to triage and resolve issues, as well as drain resources. + + +## Cause +Armory is still determining the exact cause of this warning, but our research leads us to believe it is the verbosity from a ```gRPC dependency```. This is being looked at from a deeper code level, and the work to prevent this warning from happening is in our backlog of work. + diff --git a/kb/armory/General/contributing-to-oss-spinnaker-plugins-and-donations-of-code.md b/kb/armory/General/contributing-to-oss-spinnaker-plugins-and-donations-of-code.md new file mode 100644 index 0000000000..067f5ad942 --- /dev/null +++ b/kb/armory/General/contributing-to-oss-spinnaker-plugins-and-donations-of-code.md @@ -0,0 +1,31 @@ +--- +title: Contributing to OSS Spinnaker (Plugins and Donations of Code) +--- + + + +## Table of Contents +* [Overview of OSS](#mcetoc_1h293plrv2s)* [Proposing Changes](#mcetoc_1h293plrv2t)* [Pull Requests](#mcetoc_1h293plrv2u)* [Testing](#mcetoc_1h293plrv2v)* [Releasing](#mcetoc_1h293plrv30)* [Documentation](#mcetoc_1h293plrv31) + +**Overview of OSS** +The Spinnaker open-source project operates under a well-defined governance structure. The Technical Oversight Committee (TOC) oversees the project and its overall direction. Special Interest Groups (SIGs) have been established to focus on specific areas of the project, such as Platform, Security, and Cloud. These SIGs hold regular meetings, and their schedules can be found on the [SIG Meeting Calendar](https://github.com/spinnaker/governance/blob/master/sig-index.md#sig-meeting-calendar). +**Proposing Changes** +The Spinnaker project consists of multiple repositories, with each service having its own repository. Additionally, there is a common library called Kork. However, this will change soon, as there is an ongoing effort to move towards a [monorepo structure.](https://github.com/spinnaker/governance/pull/336) +When [proposing a new feature](https://spinnaker.io/docs/community/contributing/code/submitting/#feature-proposals), [opening a GitHub issue](https://github.com/spinnaker/spinnaker/issues/new) is the recommended approach. This initiates a discussion where the community can contribute ideas, evaluate the feasibility, and assess the potential impact. The opened GitHub Issue is a central hub for collaboration, allowing contributors to align the feature's goals with the project's vision. +For large changes, such as transitioning to a monorepo or introducing a plugin framework, the [Request for Comments (RFC) p](https://spinnaker.io/docs/community/contributing/code/submitting/#requests-for-change)**[r](https://spinnaker.io/docs/community/contributing/code/submitting/#requests-for-change)**[ocess](https://spinnaker.io/docs/community/contributing/code/submitting/#requests-for-change) should be followed. This involves creating an RFC document that outlines the proposed change, its motivations, and its potential impact on the project. By sharing the RFC with the community, contributors can gather feedback and work towards consensus before implementing. +Smaller changes can be made through Pull Requests (PRs), see the Pull Requests section for more info. +By following these collaborative processes, the Spinnaker project ensures that proposed changes are thoroughly discussed, evaluated, and aligned with the community. This inclusive approach fosters a robust development environment where contributors can actively shape the project's direction and enhance its capabilities. +**Pull Requests** +When creating a Pull Request, it is important to include the purpose of the change(s) being made (see this [Deck PR as an example](https://github.com/spinnaker/deck/pull/6756)). This helps reviewers understand the context and intent of the PR. Furthermore, it is essential to describe the testing conducted to ensure the correctness and stability of the changes. +Pull Requests go through a review process, where other contributors review the proposed changes and provide feedback. The feedback can range from code style suggestions to functional and architectural considerations. It is crucial to address the feedback and iterate on the changes until they are deemed acceptable by the reviewers. +The Spinnaker project utilizes the [conventional commit format](https://spinnaker.io/docs/community/contributing/code/submitting/#commit-and-pr-message-conventions) for commit messages. This format follows a structured approach that includes a type, an optional scope, and a subject. The type indicates the nature of the changes made in the commit, while the scope provides additional context if necessary. The subject briefly describes the nature of the changes introduced by the commit. By following this format, commit messages become standardized and easier to understand, aiding in comprehending the changes made throughout the project's development. +**Testing** +Testing is a vital aspect of the Spinnaker project to maintain the quality and reliability of the software. Various tests are conducted, including unit, integration, end-to-end, and real-world tests. +Unit tests test individual components or functions in isolation to verify their behavior. Integration tests ensure that different components work together correctly. End-to-end tests simulate real-world scenarios and validate the overall functionality of the system. Real-world testing involves deploying and running Spinnaker in actual environments to assess its performance and behavior in production-like conditions. +**Releasing** +Spinnaker supports multiple releases simultaneously to accommodate different user needs and preferences. The project maintains a list of supported releases at the following link: [https://spinnaker.io/docs/releases/versions/](https://spinnaker.io/docs/releases/versions/**). +New feature releases are made [approximately every two months](https://spinnaker.io/docs/releases/release-cadence/). One of the project's goals is to minimize breaking changes between releases, ensuring a smoother upgrade process for users. Backported features, where new features are added to existing release branches, may be introduced using feature flagging, allowing users to opt into using new features while running older versions of Spinnaker. +**Documentation** +Updating documentation is an essential part of contributing to the Spinnaker project, and it follows a similar process to updating the codebase. The project's documentation is stored in a Git repository, which can be found at [https://github.com/spinnaker/spinnaker.io](https://github.com/spinnaker/spinnaker.io**). +The documentation repository utilizes Netlify, a platform for continuous deployment, to preview the updates before they are merged into the main documentation site. This allows contributors to see how the changes will be rendered and make any necessary adjustments. + diff --git a/kb/armory/General/create-aws-iam-roles-from-a-terraform-script.md b/kb/armory/General/create-aws-iam-roles-from-a-terraform-script.md new file mode 100644 index 0000000000..fc6ff3f1ec --- /dev/null +++ b/kb/armory/General/create-aws-iam-roles-from-a-terraform-script.md @@ -0,0 +1,40 @@ +--- +title: Create AWS IAM Roles from a Terraform Script +--- + +## Introduction +This document will show you how to create the AWS IAM roles from a Terraform Script. This document is based on the idea to automatize this manual process [Deploying to AWS from Spinnaker (using IAM instance roles)](https://docs.armory.io/spinnaker-install-admin-guides/add-aws-account-iam/) using a terraform script. That means you have two options to create the AWS IAM roles. + +## Prerequisites +Terraform should be installed: +[https://learn.hashicorp.com/tutorials/terraform/install-cli](https://learn.hashicorp.com/tutorials/terraform/install-cli) +You should have a profile configured in your ```~/.aws/credentials``` file. + +## Instructions +Variables +In the file ```terraform.tfvars``` you need to set some variables in order to execute properly. +* provider-region = The region* provider-profile = the name of the profile in ```~/.aws/credentials```* node-name = Find one of the nodes which is part of your EKS or other Kubernetes cluster and get the name Role i.e ```arn:aws:iam::0123456789:role/armory-spin-hal-aws-dev-node``` and copy just the name ```armory-spin-hal-aws-dev-node```* managed-role-name = The name of the managed role, whatever you want e.g. ```TestSpinnakerManagedRole```* managing-policy = The name of the managed policy, whatever you want e.g. ```TestDevSpinnakerManagingPolicy```* pass-role-name = The name of the pass-role, whatever you want e.g. ```TestPassRole```* base-role-name = The name of the base-role, whatever you want e.g. ```TestBaseIAMRole```* role-policy = The name of the role-policy, whatever you want e.g. ```TestEc2CloudForm```* managing-role-instance-profile = the name of the Managing Role Instance Profile, whatever you want e.g. ```TestInstanceProfile``` +How to Use +* Download the terraform script from: [https://github.com/armory/terraform.git](https://github.com/armory/terraform.git) and open the ```aws-roles``` folder* Edit the ```terraform.tfvars``` and fill with the corresponding values as explained above. +Run the follow commands: +``` +terraform init +terraform plan -var-file=terraform.tfvars +terraform apply -var-file=terraform.tfvars​ +``` + +Result +After running the script the roles and policies should be created and the output of the script is the instructions that you will need to execute in order to enable the aws account in halyard. +Outputs: + +``` +commands = Run this commands in order: +export AWS_ACCOUNT_NAME=aws-dev-1 +export ACCOUNT_ID=569630529054 +export ROLE_NAME=role/SpinnakerManagedRoleTerraform +hal config provider aws account add $AWS_ACCOUNT_NAME --account-id $ACCOUNT_ID --assume-role $ROLE_NAME +hal config provider aws enable +hal config provider aws account edit $AWS_ACCOUNT_NAME --regions us-east-1,us-west-2 +hal deploy apply +``` + diff --git a/kb/armory/General/create-kubernetes-configmaps-and-secrets-from-artifact-files.md b/kb/armory/General/create-kubernetes-configmaps-and-secrets-from-artifact-files.md new file mode 100644 index 0000000000..744777d353 --- /dev/null +++ b/kb/armory/General/create-kubernetes-configmaps-and-secrets-from-artifact-files.md @@ -0,0 +1,82 @@ +--- +title: Create Kubernetes Configmaps and Secrets from Artifact Files +--- + + +Inject Artifact File into Pipeline Context +Spinnaker has the concept of [artifacts](https://www.spinnaker.io/reference/artifacts/), making it possible to reference files stored in GitHub, Bitbucket, S3, GCS and others in the pipeline. You need to have [enabled and configured](https://www.spinnaker.io/setup/artifacts/) first the artifact type you need. For example for fetching files from GitHub you need to add a GitHub account to Spinnaker configuration, where you provide access credentials. + +We can use a Webhook stage to make generic HTTP requests and inject the response into the pipeline context to be referenced later, and we can leverage internal Spinnaker endpoints to make these requests for downloading artifact contents. Clouddriver is in this case the service used to download artifacts in Spinnaker, so if we want to download a file from GitHub this would be the example configuration in Webhook stage: +``` +Webhook URL: http://spin-clouddriver:7002/artifacts/fetch +Method: PUT +Payload: { + "artifactAccount": "", + "reference": "https://api.github.com/repos///contents/", + "type": "github/file", + "version": "" + } +``` + +```artifactAccount```: This is the name of the account that was configured in Spinnaker through Halyard, that has the credentials needed to connect to the artifact source. +```reference```: Location of the artifact. The example shows the url format for a file in GitHub, for S3 artifacts this would be the S3 url location. +```type```: One of the supported artifact types. If unsure of the exact value, you can always go to ```Expected Artifacts``` section, add one artifact of the desired type and then go to ```Edit as JSON``` to see the type values for different artifact types. +```version```: This also depends on the artifact type. In the case of GitHub files this is the branch name. As in the previous case you can add the artifact you want to ```Expected Artifacts``` and then see in ```Edit as JSON``` the actual value for the version field, depending on the artifact type you want to use. +We can see that most of the configuration is static. We can create a [custom webhook stage](https://www.spinnaker.io/guides/operator/custom-webhook-stages/), which is a way to abstract away common configuration and only expose the fields that need to be changed, and this would be a far more user friendly experience. For the sample webhook configuration above to be converted into a stage ```Read GitHub file``` this would be the custom webhook configuration to add to ```profiles/orca-local.yml``` file: +``` +webhook: + preconfigured: + - label: Read GitHub file + type: readGithubFile + enabled: true + description: Reads a file from GitHub and stores its content in pipeline context under ${#stage("Stage Name")["context"]["webhook"]["body"]} + method: PUT + url: http://spin-clouddriver:7002/artifacts/fetch + customHeaders: + Content-Type: + - application/json + payload: |- + { + "artifactAccount": "", + "reference": "https://api.github.com/repos/${parameterValues['org']}/${parameterValues['repo']}/contents/${parameterValues['filePath']}", + "type": "github/file", + "version": "${parameterValues['branch']}" + } + parameters: + - label: Organization + name: org + description: 'GitHub organization name' + defaultValue: '' + type: string + - label: Repository + name: repo + description: 'GitHub repository name' + defaultValue: '' + type: string + - label: File + name: filePath + description: 'Path to the file, relative to repository root' + defaultValue: '' + type: string + - label: Branch + name: branch + description: 'Branch to use' + defaultValue: '' + type: string +``` + +Create and Deploy ConfigMap or Secret +Once we have the file contents in the pipeline context, we can use a ```Deploy (Manifest)``` stage to deploy the Kubernetes ConfigMap or Secret. This would be an example yaml for a Secret: +``` +apiVersion: v1 +data: + secretFile.txt: '${#toBase64(#stage("Read GitHub file")["context"]["webhook"]["body"])}' +kind: Secret +metadata: + name: mysecret + namespace: staging +type: Opaque +``` + +Remember that [Spinnaker automatically versions](https://www.spinnaker.io/reference/artifacts/in-kubernetes-v2/#versioned-kubernetes-objects) all ConfigMaps and Secrets it deploys to be able to perform rollbacks, so this example will create a secret named ```mysecret-v000``` the first time it runs. Other deployment manifest stages down the pipeline that reference the secret, will get the correct version automatically. + diff --git a/kb/armory/General/creating-a-change-request-in-servicenow.md b/kb/armory/General/creating-a-change-request-in-servicenow.md new file mode 100644 index 0000000000..502c1f6941 --- /dev/null +++ b/kb/armory/General/creating-a-change-request-in-servicenow.md @@ -0,0 +1,20 @@ +--- +title: Creating a Change Request in ServiceNow +--- + + +At Armory, we aim to provide customers with a secure method to request changes to their environments.  Authorized users can be added and managed by customers' self-designated [ServiceNow Service Portal Customer Administrators](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010240). +Users with access to the support portal can submit requests to make changes to their environment such as adding accounts, or adjusting configuration changes to their Armory-Managed Spinnaker environment by following the procedure below +In order to ensure issues are resolved effectively, we ask that customers please do not mix requests and issue submissions, and to please open separate tickets for separate issues/changes. +### Submitting a Change Request to Armory +* Head over to [https://support.armory.io](https://support.armory.io/) and click on the **log in prompt** in the upper right corner* In the Navigation Bar, select **Change Request** -> **Create a Change Request**Fill in the information about the changes that need to be implemented.  Please refer to the Priorities defined in our [Armory Support explanation page](https://kb.armory.io/s/article/Support) which go into further details about prioritization +* The account and your name should be be displayed* **Impact: **Advise Armory about how much impact the change will have to the customer's team* **Affected Environment****:** Production, or non-production (Preprod/UAT/Dev, etc)* **Subject:** Please describe the change so that it can be referenced.  In the **Description** please also include the following information to help us complete changes as soon as possible: +* A list of changes if there are multiples of the same type.  Please attempt to provide different change requests for different types of changes so that impact and changes can be tracked and referenced +* Specifics on service in the environment the change is for +* Details about the new changes and (if applicable) what the previous settings were to allow the team to confirm the change is correct. +* The specific environment the changes should be applied to + +* Once changes are ready to submit, click on the have been completed, click on the ```save``` button + + + diff --git a/kb/armory/General/creating-cases-and-case-management-in-servicenow.md b/kb/armory/General/creating-cases-and-case-management-in-servicenow.md new file mode 100644 index 0000000000..f3fa30af3a --- /dev/null +++ b/kb/armory/General/creating-cases-and-case-management-in-servicenow.md @@ -0,0 +1,37 @@ +--- +title: Creating Cases and Case Management in ServiceNow +--- + + +At Armory, we aim to provide world-class support by giving our customers choices for different support options.  To optimize our customer's support experience, please read our guidelines at our [Armory Support explanation page](https://kb.armory.io/s/article/Support).  +The following article explains the process to create and manage cases.  The information should assist with how to set up cases to allow support to assist users as quickly as possible, and provide the best experience from beginning to end about support cases. +Before beginning, [users will need to have an active and enabled user account](https://support.armory.io/support/?id=kb_article_view&sysparm_article=KB0010200).  If an account has not yet been activated, please contact the designated Service  Portal Customer Administrators for the account. +In order to ensure issues are resolved effectively and efficiently, we ask that customers please do not mix requests and issue submissions, and to please open separate tickets for separate issues/changes. +## Preparing for Case Submissions - Armory Diagnostics +We highly suggest [enabling Armory Diagnostics](https://docs.armory.io/spinnaker-install-admin-guides/admin-diagnostics/) if the environment allows for it.  This allows our team to gather information about the current state of the environment and allows us to skip a few steps in our initial investigation, allowing for a quicker resolution. +Otherwise, we ask that any relevant logs be attached to the cases that are created.  Full pod logs often will help with our analysis.   +Additionally, in an effort to ensure faster issue triage time - you may also provide a snapshot of your current Spinnaker configuration, referred to as Armory's Support Bundle, with the help of the following automated process: [Capturing Spinnaker configurations with Armory's Support Bundle (w/ Automated Script)](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010430&sys_kb_id=c131fd831baaf050ec88b8c2cc4bcb01&spa=1). +  +## Submitting a Case to Armory Support +* Head over to [https://support.armory.io](https://support.armory.io/) and click on the **log in prompt** in the upper right corner* In the Navigation Bar, select **Cases** -> **Create a Case**Fill in the information about the issue that is being experienced.  Please refer to the Priorities defined in our [Armory Support explanation page](https://kb.armory.io/s/article/Support) which go into further details about prioritization +* The account and your name should be be displayed* **Version**: The Spinnaker version of the affected environment. Please advise about the Armory version, or if it is OSS.* **Priority: **Advise Armory Support how much priority is being placed on the issue by the customer's team* **Workaround or Rollback Available:** Advise if the issue is currently blocking your team* **Affected Environment****:** Production, or non-production (Preprod/UAT/Dev, etc)* **Feature:** Advise us about the particular feature and compatibilities that Armory supports which pertains to the issue.  For reference, please visit the [Armory Enterprise Compatibility Matrix](https://docs.armory.io/docs/armory-platform-matrix/)* **Service:** Advise about the particular service that Armory supports which pertains to the issue.  For reference, please visit the [Armory Enterprise Compatibility Matrix](https://docs.armory.io/docs/armory-platform-matrix/)* **Customer Availability: **Please provide to Armory the best available times to attempt to schedule contact* **Subject:** Please describe the issue.  As you describe the issue, suggested topics from our KB will populate that may help resolve the issue more quickly.  We ask that customers please submit one case per issue to ensure that cases and solutions are tracked properly and that multiple people can assist with your is* In the **Description** please also include the following information to help us resolve items as possible: + + +* Issue summary, and business impact.* Log files relevant to the issue (from that particular date/time window, and environment).* Steps that have already been taken by the customer.* Steps to reproduce.* Please also **attach** any relevant files, pictures, or screenshots which may aid in the investigation. + +  +## Understanding the Different Case Statuses +Throughout the lifecycle of the case, the status of the case will change [through several different stages](https://armory.service-now.com/support?id=kb_article_view&sysparm_article=KB0010011).  Some cases will go back and forth between different stages, depending on the interactions with the engineer. +##   +## Checking the Status of your Case and Updating your Case +* **Log in** to the portal at [https://support.armory.io](https://support.armory.io/).  * In the Navigation Bar, select **Cases** -> **All Cases*** From this screen, there are options to view open and closed cases created by the user, and access to view the overall account's open and closed cases* Click on the case that needs to be reviewedOnce in the case, there are options available to update the case +* **Contact**: The main contact can be updated here to any registered user on the same account.  Note that users that have deactivated logins can also be chosen, so users should be careful when making a selection* **Watch list**: Allows the addition of more users to view a case.  Note that external users to the account can also be set in the watch list so that they can observe the case* **Activity**: Update the information about the case.  A WYSIWYG option is available when posting.  Attachments can also be added. Once ready, click **Post **to commit the information to the activity feed* Once changes are completed, click on **Save** in the bottom right to update the case +* Customers can also ***Update their Case*** by responding to the emails that they have received +  +## Self Closing a Case +* **Log in** to the portal at [https://support.armory.io](https://support.armory.io/).  * In the Navigation Bar, select **Cases** -> **All Cases*** From this screen, there are options to view open and closed cases created by the user, and if the privilege is provided, access to view the overall account's open and closed cases.  * Click on the case that needs to be closed* In the bottom left corner, there is button to **Close Case.  **Click on this button to resolve and close a case without input from support* Please note that cases, once closed, cannot be opened again and will be in a **read only** state.  To open up a case about the issue again, please open a new case and provide any update information that can assist our team with a new resolution.  +  +## Reopening a Closed Case +Customers may find that an issue has returned, or a case was closed due to inactivity, but would like it to be re-opened.  In this case, we invite customers to re-open the closed case +* **Log in** to the portal at [https://support.armory.io](https://support.armory.io/).  * In the Navigation Bar, select **Cases** -> **All Cases*** Click on **My Closed Cases** or **My Company's Closed Cases** and locate the closed case to be re-opened* Click on the case that needs to be re-opened.  At the bottom of the case, there will be a button to **Open Repeat Case*** A case will be immediately opened.  Please note that this case will be opened with the same exact priority and information as the original case.  It is highly advisable that customers provide new context on the newly opened case, including new information that may help the Support Engineer to re-evaluate the root cause. + diff --git a/kb/armory/General/creating-custom-deployment-strategies.md b/kb/armory/General/creating-custom-deployment-strategies.md new file mode 100644 index 0000000000..73ae287de2 --- /dev/null +++ b/kb/armory/General/creating-custom-deployment-strategies.md @@ -0,0 +1,18 @@ +--- +title: Creating Custom Deployment Strategies +--- + +## Introduction +Custom deployment strategies allow users to create their own set of strategies with regards to deploying their systems. By creating a strategy, it can be re used for other deployments when necessary.   +Deployment strategies can be thought of as a set of "mini-stages" that are run to execute the strategy + +## Prerequisites +N/A + +## Instructions +The principle behind a deployment strategy is to create series of "mini-stages" to perform a particular roll out. In an example such as a **blue/green** or **red/black strategy**, it would potentially involve +* standing up a new AWS Auto Scaling Group* waiting for the ASG to become healthy* update the target groups/load balancer and other functions* potentially scale down the old ASG* potentially delete the old ASG +In essence, each of these individual stages are available in Spinnaker, so the deployment strategy would be to implement and orchestrate every one of these stages.To create the strategy, in the Pipelines section of Spinnaker, choose **Strategy** instead of **Pipeline** from the dropdown list when creating a new pipeline.  + +From there, lay out the stages of the strategy, including a needed **deploy** stage and then save.  The strategy will then be available for use + diff --git a/kb/armory/General/critical-technical-notifications.md b/kb/armory/General/critical-technical-notifications.md new file mode 100644 index 0000000000..71deaff5ea --- /dev/null +++ b/kb/armory/General/critical-technical-notifications.md @@ -0,0 +1,17 @@ +--- +title: Critical Technical Notifications +--- + + +As a part of being an Armory customer, our crew would like to make sure our customers are well informed of any issues discovered about Armory's products and services. +At Armory, we have implemented a **Critical Technical Notifications** mailing list. +This mailing list will provide a direct line of communication with your **Armory Enterprise Environment Administrative team(s)** and advise them about **future discovered critical issues.** +#### Topics expected to be covered by the mailing list +* Security notifications* Emergency updates* Configuration changes that should be planned in advance +This is **NOT a marketing list**. We will only use this distribution list to send critical updates relevant to your Armory Admin Team. +#### What Armory needs from your team +We ask that arrangements are made to create a Distribution List and to then provide your Account Manager or the Support Team the email address of the Distribution List.  We recommend that this be prioritized so that your team does not miss any notifications. +Customers may also receive these notifications within their Armory Slack channel or customer Slack channel.  +#### Benefits of a distribution list that is managed by your company +* Full control of who gets critical Armory notifications (subscribing and unsubscribing)* Flexibility to integrate notifications with the communication system of choice (Teams, Slack, etc)* Building the management of the distribution list into the customer's own internal processes for onboarding and offboarding of employees + diff --git a/kb/armory/General/custom-runjobs-intermittently-failing-cannot-access-properties-of-application.md b/kb/armory/General/custom-runjobs-intermittently-failing-cannot-access-properties-of-application.md new file mode 100644 index 0000000000..0038a59f3f --- /dev/null +++ b/kb/armory/General/custom-runjobs-intermittently-failing-cannot-access-properties-of-application.md @@ -0,0 +1,12 @@ +--- +title: Custom RunJobs intermittently failing (Cannot Access Properties of Application) +--- + +## Issue +On an intermittent basis, pipelines that are deploying on Kubernetes targets fail on different custom run job stages. +This could manifest in a few different ways, mainly with failure errors such as:```Cannot access properties of application , Required permissions READ. ``` + +## Cause +It may be that the configuration has defined two different Clouddriver accounts that are associated with the same namespace, so when the stage completes versus when it doesn't - it depends on which Clouddriver account was picked at ***runtime***. +Another reason may be is that the permissions given on the two different Clouddriver accounts are different. + diff --git "a/kb/armory/General/custom-runjobs-intermittently-failing-with-\"access-denied-to-application\"-error.md" "b/kb/armory/General/custom-runjobs-intermittently-failing-with-\"access-denied-to-application\"-error.md" new file mode 100644 index 0000000000..f0478a0fa6 --- /dev/null +++ "b/kb/armory/General/custom-runjobs-intermittently-failing-with-\"access-denied-to-application\"-error.md" @@ -0,0 +1,16 @@ +--- +title: Custom runjobs intermittently failing with "Access denied to application" error +--- + +## Issue +A custom runjob stage in Armory Spinnaker 2.26.0 or earlier fails with the following error displayed in the UI: +```Exception - Access denied to application - required authorization: READ``` +No permission or configuration changes have been applied prior to this failure, and the same stage successfully completes at an intermittent basis.  + +## Cause +This is a known issue that has been addressed in Armory Spinnaker 2.26.1, specifically in Orca:[https://github.com/spinnaker/orca/pull/4134](https://github.com/spinnaker/orca/pull/4134) +The issue stems from Orca passing an application name to Clouddriver to ask for permissions, however no such application actually exists. +In order to confirm this indeed is the case for a particular execution failure, look at the Clouddriver log for the following error: +```FiatAccessDeniedExceptionHandler : Encountered exception while processing request GET:/{}applications/``` +And confirm `````` is not an application that exists in your environment. + diff --git a/kb/armory/General/customer-user-does-not-have-access-to-create-change-request.md b/kb/armory/General/customer-user-does-not-have-access-to-create-change-request.md new file mode 100644 index 0000000000..8ba1145c5d --- /dev/null +++ b/kb/armory/General/customer-user-does-not-have-access-to-create-change-request.md @@ -0,0 +1,9 @@ +--- +title: Customer User does not have access to create Change Request +--- + + +A Managed Customer user will need to have a different group assigned to them in order to have the functionality to [file a change request.](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010267) + +* Log in to the ServiceNow Admin Portal* In the Navigation bar, Search for ```User```, and either look for the ```User List``` option under the Armory heading, or the ```Users``` option under the ```Organization``` heading.* Search for the user and click on them* Go to the ```Groups``` tab near the bottom and click on ```Edit…```* Remove ```Armory - Customers``` and instead add ```Armory - Managed Customers```* Click Save + diff --git a/kb/armory/General/customers-unable-to-access-resources-due-to-github-sunseting-deprecated-teams-api-endpoints.md b/kb/armory/General/customers-unable-to-access-resources-due-to-github-sunseting-deprecated-teams-api-endpoints.md new file mode 100644 index 0000000000..45a0597ce2 --- /dev/null +++ b/kb/armory/General/customers-unable-to-access-resources-due-to-github-sunseting-deprecated-teams-api-endpoints.md @@ -0,0 +1,19 @@ +--- +title: Customers Unable to Access Resources due to GitHub sunseting Deprecated Teams API Endpoints +--- + +## Issue +Customers may find a lack of access to resources in Spinnaker if they have been using the Teams API.  This is due to a change in the Teams API from a top-level path under ```/teams/:team_id```.  Customers not using the Team API will not be affected.  +Those who are using the API, will find that resources will not show up in Spinnaker UI and FIAT may display the following in logs: + +``` +ERROR 1 — [0.0-8083-exec-3] n.s.f.s.FiatAccessDeniedExceptionHandler : Encountered exception while processing request GET:/applications/dqcloud/tasks with headers={x-spinnaker-application=tempTest, host=spin-orca.spinnaker:8083, connection=Keep-Alive, x-spinnaker-user=ArmoryUser, x-spinnaker-accounts=prod,build, accept-encoding=gzip, accept=application/json, user-agent=okhttp/3.14.7} +``` +  +For more information about these changes, please take a look at the following Blog:[https://github.blog/changelog/2022-02-22-sunset-notice-deprecated-teams-api-endpoints/](https://github.blog/changelog/2022-02-22-sunset-notice-deprecated-teams-api-endpoints/) +  + +## Cause +Due to deprecation, code in FIAT must be updated to account for changes.  The following PR contains the changes to FIAT to be rolled out in 1.28.x +[https://github.com/spinnaker/fiat/pull/911](https://github.com/spinnaker/fiat/pull/911) + diff --git a/kb/armory/General/cve-2021-43832---favexploit--spinnaker-rce-vulnerability-in-gate.md b/kb/armory/General/cve-2021-43832---favexploit--spinnaker-rce-vulnerability-in-gate.md new file mode 100644 index 0000000000..600dd2ef24 --- /dev/null +++ b/kb/armory/General/cve-2021-43832---favexploit--spinnaker-rce-vulnerability-in-gate.md @@ -0,0 +1,16 @@ +--- +title: CVE-2021-43832 - FavExploit- Spinnaker RCE Vulnerability in Gate +--- + +## Issue +Armory has since published updates to the code for OSS and Armory Enterprise. +The [Spinnaker Security SIG](https://spinnaker.io/docs/community/security/) received a report of a previously undisclosed RCE attack vector that bypasses authentication in Spinnaker. +This exploit allows an actor to make any resourced API call through Gate without authentication. The documented exploit affects any ***Spinnaker version within the last four years, ***but was only discovered on Dec 14th, 2021 +Armory has created a placeholder CVE that has not been made public yet.  We ask that customers*** upgrade to a version with the fix as soon as possible.*** +***Update Jan 3, 2022: ***The following CVE was published and made available today to the general public [https://cve.report/CVE-2021-43832](https://cve.report/CVE-2021-43832) +***Update Oct 28, 2022: ***Added CVE# to title for formatting purposes.   + +## Cause +RCE attack vector discovered  +CVE-2021-43832 will be tracking this issue.  This article will be updated with a link to the CVE once it has been made public + diff --git a/kb/armory/General/cve-2021-44228---log4shell---log4j-vulnerability-analysis.md b/kb/armory/General/cve-2021-44228---log4shell---log4j-vulnerability-analysis.md new file mode 100644 index 0000000000..93a0713801 --- /dev/null +++ b/kb/armory/General/cve-2021-44228---log4shell---log4j-vulnerability-analysis.md @@ -0,0 +1,16 @@ +--- +title: CVE-2021-44228 - log4shell / log4j Vulnerability Analysis +--- + +## Issue +A potentially critical 0-day exploit CVE was identified on ***Dec 10, 2021***.  +[https://nvd.nist.gov/vuln/detail/CVE-2021-44228](https://nvd.nist.gov/vuln/detail/CVE-2021-44228) +Armory has investigated this 0-day critical issue, and has performed analysis on the vulnerability and its potential for harm to Armory Enterprise customers. +  + +## Cause +The vulnerability exposes a ***remote-execution vulnerability*** in services that use ```log4j```. Spinnaker services use ```logback```, a different logging implementation. +Here are some examples of how the vulnerability might be exploited:[https://isc.sans.edu/forums/diary/RCE+in+log4j+Log4Shell+or+how+things+can+get+bad+quickly/28120/](https://isc.sans.edu/forums/diary/RCE+in+log4j+Log4Shell+or+how+things+can+get+bad+quickly/28120/)[https://www.lunasec.io/docs/blog/log4j-zero-day/#example-vulnerable-code](https://www.lunasec.io/docs/blog/log4j-zero-day/#example-vulnerable-code) +The affected class ```org.apache.logging.log4j.core.lookup.JndiLookup``` is not bundled with Armory Enterprise. +This was validated by inspecting service dependencies, logs from active services and thread profiling services to ensure the affected class is neither packaged or used. + diff --git a/kb/armory/General/cve-2022-22965---spring-framework-rce-w--jdk-9+.md b/kb/armory/General/cve-2022-22965---spring-framework-rce-w--jdk-9+.md new file mode 100644 index 0000000000..04019fff79 --- /dev/null +++ b/kb/armory/General/cve-2022-22965---spring-framework-rce-w--jdk-9+.md @@ -0,0 +1,14 @@ +--- +title: CVE-2022-22965 - Spring Framework RCE w/ JDK 9+ +--- + +## Issue +Spring.io has published a notification about a recently discovered CVE ([https://tanzu.vmware.com/security/cve-2022-22965](https://tanzu.vmware.com/security/cve-2022-22965)) +They have posted a blog about their observations ([https://spring.io/blog/2022/03/31/spring-framework-rce-early-announcement](https://spring.io/blog/2022/03/31/spring-framework-rce-early-announcement)) which they are continuing to update. +Armory has been following the developments so far and is keeping up to date about the situation. +Based on our investigations, we do not appear to be susceptible to exploitation at this time, and therefore we are only observing developments. We will continue looking at updates and measuring the potential impact on customers. +To understand the Armory process in evaluating and determining the scope of effect of the vulnerability, Armory has posted the following Blog Article:[https://www.armory.io/blog/cve-2022-22965-spring-rce-which-does-not-impact-spinnaker/](https://www.armory.io/blog/cve-2022-22965-spring-rce-which-does-not-impact-spinnaker/) + +## Cause +Please refer to Spring's post with regard to this vulnerability:[https://spring.io/blog/2022/03/31/spring-framework-rce-early-announcement](https://spring.io/blog/2022/03/31/spring-framework-rce-early-announcement) + diff --git a/kb/armory/General/cve-2022-42889---apache-commons-text-vulnerability.md b/kb/armory/General/cve-2022-42889---apache-commons-text-vulnerability.md new file mode 100644 index 0000000000..04366feaa6 --- /dev/null +++ b/kb/armory/General/cve-2022-42889---apache-commons-text-vulnerability.md @@ -0,0 +1,17 @@ +--- +title: CVE-2022-42889 - Apache Commons Text Vulnerability +--- + +## Issue +Apache has discovered a vulnerability with the Apache Common Text function.  A variable interpolation can be performed as part of the functions, allowing properties to be dynamically evaluated and expanded.  For more information and the latest information, we ask that customers continue to monitor the vulnerability for updates: +[https://nvd.nist.gov/vuln/detail/CVE-2022-42889](https://nvd.nist.gov/vuln/detail/CVE-2022-42889) +Armory has investigated this critical issue.  As of October 21st, 2022, we have analyzed the vulnerability and its potential for harm to Armory CD customers. +Based on our investigations, we do not appear to be susceptible to exploitation at this time, and therefore we are only observing developments.  We will continue looking at updates and measuring the potential impact on customers. +  +**Update, Oct 22, 2022:** +Halyard and Operator images both contain the library, but the class is not used.  The article has been updated with the additional relevant information. + +## Cause +The vulnerability exposes a ***script execution vulnerability***.  +An example of how the vulnerability might be exploited can be found here:[https://www.rapid7.com/blog/post/2022/10/17/cve-2022-42889-keep-calm-and-stop-saying-4shell/](https://www.rapid7.com/blog/post/2022/10/17/cve-2022-42889-keep-calm-and-stop-saying-4shell/) + diff --git a/kb/armory/General/database-recommendations-for-running-spinnaker.md b/kb/armory/General/database-recommendations-for-running-spinnaker.md index 906412cd43..f6911ea007 100644 --- a/kb/armory/General/database-recommendations-for-running-spinnaker.md +++ b/kb/armory/General/database-recommendations-for-running-spinnaker.md @@ -8,18 +8,30 @@ Services within spinnaker such as Clouddriver, Front50 and Orca can be configure The recommendations around using MySQL and Redis for better scalability are outlined below ## MySQL Considerations   Below is the kind of data persisted by Front50, Orca and Clouddriver in their databases. -* Front50’s database contains pipeline definitions and needs to be properly backed up.* Orca’s database contains pipeline execution history that is displayed in Spinnaker’s UI.* Clouddriver’s database contains the infrastructure cache. If lost, it will need to be re-cached which depending on the size of the infrastructure may take a while.  -* If available, use ***cross-region replication*** to ensure durability of the data stored.* Make sure the network latency between Spinnaker and the database cluster is reasonable. It often just means located in the same datacenter region.* Clouddriver, Orca, and Front50 services must each use ***a different database for each service***. The databases can be in different database clusters or in the same one. A single cluster is easier to manage and more cost effective but the number of connections used by ***Spinnaker will be added across all services***.* The database cluster must support the number of open connections from Spinnaker and any other tools in need. For numbers refer to the database connections chart in the profiles below. +* Front50’s database contains pipeline definitions and needs to be properly backed up. +* Orca’s database contains pipeline execution history that is displayed in Spinnaker’s UI. +* Clouddriver’s database contains the infrastructure cache. If lost, it will need to be re-cached which depending on the size of the infrastructure may take a while.  +* If available, use ***cross-region replication*** to ensure durability of the data stored.* Make sure the network latency between Spinnaker and the database cluster is reasonable. It often just means located in the same datacenter region.* Clouddriver, Orca, and Front50 services must each use ***a different database for each service***. The databases can be in different database clusters or in the same one. A single cluster is easier to manage and more cost effective but the number of connections used by ***Spinnaker will be added across all services***. +* The database cluster must support the number of open connections from Spinnaker and any other tools in need. For numbers refer to the database connections chart in the profiles below. + **Service****DB Metric****Description****Value**MySQL DBDatabaseConnections_MaximumActive Database connection in DB (The connection pool value in MySQL is to be scaled as the number of pods increase)50ClouddriverTotal SQL Connection PoolTotal number of connections from the service20Front50Total SQL Connection PoolTotal number of connections from the service10OrcaTotal SQL Connection PoolTotal number of connections from the service10 + * Clouddriver connection pools can be tuned via ```sql.connectionPools.cacheWriter.maxPoolSize``` and ```sql.connectionPools.default.maxPoolSize```. Both values default to 20 and need to be increased to handle more tasks per Clouddriver.MySQL v5.7+ is recommended. Below are the sizing requirements if running MySQL or Aurora -* db.r3.large or larger for node size* 2 or more nodes* Span availability zones* MySQL compatible, v5.7+ +* db.r3.large or larger for node size* 2 or more nodes* Span availability zones +* MySQL compatible, v5.7+ The above recommendations are meant for guidance and not considered to be a complete checklist of items for using Spinnaker services with MySQL or Redis.  Depending on a customer's environment and needs, a team may need to implement additional considerations and checks.   Please also consider cleanup jobs for the MySQL database.  An example would be to look at cron jobs to maintain the [Orca Database for old pipeline executions](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010114). + ## Redis Considerations Most services rely on Redis for lightweight storage and/or task coordination. As the number of applications scale, it is recommended to switch Clouddriver, Orca to MySQL or Postgres databases.  Redis being single threaded doesn’t need more than one CPU. When available, a managed Redis (like ElastiCache) can be used. A shared Redis can be used for ease of management. It is highly recommended to use an external Redis instance specifically for Dinghy separately.  Below are the sizing required if redis is hosted on AWS elastic cache. -* 1 Redis Elasticache of size cache.r5.large * Number of replicas should be set to 3* Make sure ```Cluster Mode``` is ```not enabled``` * Make sure ```in-transit encryption``` is ```not enabled```* Manually set the configuration parameter ```notify-keyspace-events``` to ```gxE``` for the redis instance +* 1 Redis Elasticache of size cache.r5.large  +* Number of replicas should be set to 3 +* Make sure ```Cluster Mode``` is ```not enabled```  +* Make sure ```in-transit encryption``` is ```not enabled``` +* Manually set the configuration parameter ```notify-keyspace-events``` to ```gxE``` for the redis instance + The above recommendations are meant for guidance and not considered to be a complete checklist of items for using Spinnaker services with MySQL or Redis.  Depending on a customer's environment and needs, a team may need to implement additional considerations and checks.   diff --git a/kb/armory/General/debug.armory.io-is-down,-crashes-environments-w--debugging-enabled-into-a-crash-loop-restart-.md b/kb/armory/General/debug.armory.io-is-down,-crashes-environments-w--debugging-enabled-into-a-crash-loop-restart-.md new file mode 100644 index 0000000000..2fc48e36e7 --- /dev/null +++ b/kb/armory/General/debug.armory.io-is-down,-crashes-environments-w--debugging-enabled-into-a-crash-loop-restart-.md @@ -0,0 +1,14 @@ +--- +title: debug.armory.io is Down, Crashes Environments w/ Debugging Enabled into a crash loop restart +--- + +## Issue +Note: This article is for internal use only +debug.armory.io becomes unavailable and as a result, Crashloop Restarts will happen for Terraformer and Dinghy.  As an example, Dinghy was failing to startup after a restart hence not usable +The Full RCA for the incident is found here: [https://armory.slab.com/posts/5-16-2021-debug-armory-io-rca-internal-777v0u67](https://armory.slab.com/posts/5-16-2021-debug-armory-io-rca-internal-777v0u67) + +## Cause +armory-kube GKE cluster health for debug.armory.io was not in a good place.  As per Christos' investigation +* Node pool workers are not able to join the Kubernetes control plane. Both the control plane and the node pool is running in 1.17.17-gke.4900 +* Unable to issue any kubectl commands; Failing with x509 cert error for the Control plane certificates. + diff --git a/kb/armory/General/debugging-deployment-errors.md b/kb/armory/General/debugging-deployment-errors.md new file mode 100644 index 0000000000..2c22c0b694 --- /dev/null +++ b/kb/armory/General/debugging-deployment-errors.md @@ -0,0 +1,12 @@ +--- +title: Debugging Deployment Errors +--- + +## Issue +When deploying to a new Kubernetes cluster for the first time, you might receive this error:  +```deployKubernetesManifest.deployKubernetesManifst.deployment.Error reading kind [deployment].``` + +## Cause +Typically this is caused by one of the following issues: +* Your kubeconfig for the cluster is not properly configured.* The namespace used in the deployment manifest does not match the namespaces allowed as defined in your .hal/config.  + diff --git a/kb/armory/General/debugging-slow-orca-performance.md b/kb/armory/General/debugging-slow-orca-performance.md new file mode 100644 index 0000000000..bf81de45ff --- /dev/null +++ b/kb/armory/General/debugging-slow-orca-performance.md @@ -0,0 +1,12 @@ +--- +title: Debugging Slow Orca Performance +--- + +## Issue +Orca is displaying unresponsiveness and the following behavior is being observed: +* Tasks such as **Resize** or **Deploy** take much longer to complete. For example, as task that typically takes 1-3 minutes now takes 6-10 minutes. +* 30 second **Wait stages** take longer than 30 seconds to complete. + +## Cause +Under the hood, Orca uses Redis to store a queue of tasks to perform. During each polling cycle, Orca it will attempt to read any messages that haven’t been delivered and are past due. In some cases, these messsages may be deemed “undelieverable” and requeued. If enough of these messages build up, you may experience a slow down in your Pipeline executions. + diff --git a/kb/armory/General/define-application-attributes-using-dinghy.md b/kb/armory/General/define-application-attributes-using-dinghy.md new file mode 100644 index 0000000000..686bf4cf3e --- /dev/null +++ b/kb/armory/General/define-application-attributes-using-dinghy.md @@ -0,0 +1,71 @@ +--- +title: Define Application attributes using Dinghy +--- + +## Introduction +Uses may want to define the application attributes as a part of their Dinghy process.   + + +## Prerequisites +Some attributes can only be defined if available.  For example, if the cloud provider doesn't exist, it cannot be defined otherwise an error may occur.  Likewise, for Permissions, customers must [follow group definitions defined and created in FIAT](https://docs.armory.io/armory-enterprise/armory-admin/fiat-create-permissions/).  Please note that these need to be created before adding as an attribute, or an error will occur. +It is recommended that customers use the Spinnaker Console UI to guide themselves on their available options and then to define based on what can be chosen through the UI. + +## Instructions +Below is a sample of Application Attributes available +```` +{ + "name": "AppExample-APP", + "description": null, + "email": "", + "aliases": "", + "repoType": "", + "cloudProviders": [], + "platformHealthOnly": "", + "platformHealthOnlyShowOverride": "", + "trafficGuards": [], + "instancePort": "", + "enableRestartRunningExecutions": "", + "enableRerunActiveExecutions": "", + "permissions": { + "READ": [ + ], + "WRITE": [ + ], + "EXECUTE": [ + ] + } +} +```` +Customers can then define some of these attributes in the following method: +```` +{ + "name": "AppExample-APP", + "description": null, + "email": "test@abc.com", + "aliases": "", + "repoType": "", + "cloudProviders": [], + "platformHealthOnly": "false", + "platformHealthOnlyShowOverride": "false", + "trafficGuards": [], + "instancePort": "443", + "enableRestartRunningExecutions": "false", + "enableRerunActiveExecutions": "false", + "permissions": { + "READ": [ + "armory-devs", + "armory-eng", + "testexample2@armory.io", + "testexample@armory.io" + ], + "WRITE": [ + "armory-eng", + "testexample@armory.io" + ], + "EXECUTE": [ + "armory-eng", + "testexample@armory.io" + ] + } + } +```` diff --git a/kb/armory/General/delayed-image-push-pipeline-trigger.md b/kb/armory/General/delayed-image-push-pipeline-trigger.md new file mode 100644 index 0000000000..8c4cc70eb7 --- /dev/null +++ b/kb/armory/General/delayed-image-push-pipeline-trigger.md @@ -0,0 +1,10 @@ +--- +title: Delayed Image Push Pipeline Trigger +--- + +## Issue +When pushing an image to a repository (Artifactory, Docker, etc,) to trigger a pipeline, the pipeline doesn't start until some time later. + +## Cause +There are too many images in the registry that Spinnaker is not able to cache.Essentially what happens is Spinnaker will need to take time to cache all of the images before determining if a particular image was pushed for the pipeline to trigger. + diff --git a/kb/armory/General/delete-pipeline-executions-in-redis.md b/kb/armory/General/delete-pipeline-executions-in-redis.md new file mode 100644 index 0000000000..da3e811e4c --- /dev/null +++ b/kb/armory/General/delete-pipeline-executions-in-redis.md @@ -0,0 +1,31 @@ +--- +title: Delete Pipeline Executions in Redis +--- + +## Introduction +Redis can be used as a backend storage that will store the pipeline executions. In the case that there is a stuck pipeline or the pipeline execution needs to be deleted the following steps an be taken to manually delete the executions. + +## Prerequisites +### Locate the Pipeline ID and Pipeline Execution ID  +#### Via Source +Both pieces of information can be found in the execution details.  Navigate to the pipeline, and go to **Execution Details** -> **Source** +The pipeline id will be labeled ```id``` and will be one of the first lines. +The execution id will be ```execution_id``` and will be towards the end. +#### Via the URL of Resources in the Console +The URL for the pipeline when editing the pipeline from the console also contains the ```Pipeline ID``````/#/applications//executions/configure/****``` +The URL for the permalink (next to the source location above) will contain the information about the ```Pipeline Execution ID``````/#/applications//executions/details/****?stage=0&step=0&details=``` +#### Via Orca Logs +The ```Pipeline Execution ID``` can also be found within the logs of the Orca instance, by running +``````kubectl logs -n deploy/spin-orca -f `````` +Sometimes, locating the execution ID can be difficult unless it can be identified from any other executions that are running at the time + +## Instructions +Display all executions from this pipeline: +```zrange pipeline:executions:{pipeline_id} 0 -1 ​``` +2. For each execution that you want to remove: +```zrem pipeline:executions:{pipeline_id} {execution_id}​``` +3. Delete the execution: +del pipeline:${execution_id} +del orchestration:${execution_id} ​ + + diff --git a/kb/armory/General/deploy-manifest-performs-spel-substitution-even-if-skip-spel-evaluation-is-checked.md b/kb/armory/General/deploy-manifest-performs-spel-substitution-even-if-skip-spel-evaluation-is-checked.md new file mode 100644 index 0000000000..d22ec1ca1d --- /dev/null +++ b/kb/armory/General/deploy-manifest-performs-spel-substitution-even-if-skip-spel-evaluation-is-checked.md @@ -0,0 +1,12 @@ +--- +title: Deploy manifest performs SpEL substitution even if Skip SpEL Evaluation is checked +--- + +## Issue +A pipeline can be created that ingests a manifest and deploys it using the ```Deploy (Manifest)``` stage. As a part of the process, the manifest has some dynamic pieces that will need to be evaluated at runtime, with the syntax ```${variable}```. However, even with having checked the ```Skip SpEL evaluation``` option, the variables still end up getting evaluated and replaced before being sent to Kubernetes. +Using a known workaround to escape the evaluator with syntax like ```${"${variable}"}``` yields the same results - the variable is still evaluated. + +## Cause +This is a known issue, raised in:[https://github.com/spinnaker/spinnaker/issues/5910](https://github.com/spinnaker/spinnaker/issues/5910)...with target release of OSS 1.27 (mapping to Armory 2.27.2) +If SpEL expressions are instead evaluated for lines that have been commented out, see [Spinnaker Improperly Errors Even When SPEL Expressions Are Commented Out](https://support.armory.io/support?sys_kb_id=4ff1857d1b9dfc1013d4fe6fdc4bcb0b&id=kb_article_view&sysparm_rank=1&sysparm_tsqueryId=a67775581bb1c110ec88b8c2cc4bcb58). + diff --git a/kb/armory/General/deploy-manifest-stage-does-not-remove-previous-helm2-deployment-resources-from-kubernetes.md b/kb/armory/General/deploy-manifest-stage-does-not-remove-previous-helm2-deployment-resources-from-kubernetes.md new file mode 100644 index 0000000000..a850869695 --- /dev/null +++ b/kb/armory/General/deploy-manifest-stage-does-not-remove-previous-helm2-deployment-resources-from-kubernetes.md @@ -0,0 +1,12 @@ +--- +title: Deploy (Manifest) stage does not remove previous Helm2 deployment resources from Kubernetes +--- + +## Issue +Sometimes when migrating an application to Spinnaker that was previously deployed using Helm2 or ```kubectl create```, the deployment's old resources are not removed or modified as expected.Example workflow: +* Helm2 chart was previously deployed in target Kubernetes namespace* This chart was migrated into Spinnaker and modified to include a new container, deployed in same namespace +* Deployment object in Kubernetes contained both the old container (installed in previous helm chart) and the new container. + +## Cause +If the Kubernetes deployment resource does not contain a ```last-applied-configuration``` annotation, when Spinnaker uses ```kubectl apply``` to deploy the new manifest, Kubernetes may not know to remove the old container or other deployment resources. **Related issues:***[spinnaker/spinnaker#2877](https://github.com/spinnaker/spinnaker/issues/2877#issuecomment-429071589): Deploy (Manifest) stage merges attributes of existing object into deployed manifest*This user explains that they were running up against how kubectl apply was designed.```kubectl apply``` is known to create states similar to this when, for example, the previous deployment was generated with ```kubectl create```, which does not leave a ```last-applied-configuration``` annotation in the deployment. This annotation is used by ```kubectl apply``` to figure out what resources need to be removed or updated. + diff --git a/kb/armory/General/deploy-stage-missing-from-pipeline.md b/kb/armory/General/deploy-stage-missing-from-pipeline.md new file mode 100644 index 0000000000..a2df3bd033 --- /dev/null +++ b/kb/armory/General/deploy-stage-missing-from-pipeline.md @@ -0,0 +1,10 @@ +--- +title: Deploy Stage Missing From Pipeline +--- + +## Issue +The AWS/ECS Deploy stage is not available after configuring an AWS, ECS, etc. account and adding it to the pipeline's application. + +## Cause +The user attempting to create the pipeline may not have sufficient permissions. + diff --git a/kb/armory/General/deploying-helm-charts-with-armory-spinnaker-and-managing-secrets-in-helm.md b/kb/armory/General/deploying-helm-charts-with-armory-spinnaker-and-managing-secrets-in-helm.md new file mode 100644 index 0000000000..ad3ba41e0a --- /dev/null +++ b/kb/armory/General/deploying-helm-charts-with-armory-spinnaker-and-managing-secrets-in-helm.md @@ -0,0 +1,21 @@ +--- +title: Deploying Helm Charts with Armory Spinnaker and Managing Secrets in Helm +--- + +## Introduction +Discover how to deploy Helm Charts with Armory Spinnaker.  Learn how to use the stages to perform your deployments and manage secrets. + +## Prerequisites +Make sure that you have configured [artifact support in Spinnaker first](https://docs.armory.io/operator_reference/artifact/) . All Helm charts are fetched/stored as artifacts in Spinnaker.  + +## Instructions +#### Armory Video +In this video, Ethan Rogers demonstrates a workflow for deploying Helm Charts with Armory Spinnaker. +[https://www.youtube.com/watch?v=u7QF2X4WzE8](https://www.youtube.com/watch?v=u7QF2X4WzE8) + +#### EKS Workshop Rundown and Example +[https://www.eksworkshop.com/intermediate/265_spinnaker_eks/testing-helm/](https://www.eksworkshop.com/intermediate/265_spinnaker_eks/testing-helm/) +  +#### Deploying in Helm (OSS Spinnaker Guide) +[https://spinnaker.io/docs/guides/user/kubernetes-v2/deploy-helm/](https://spinnaker.io/docs/guides/user/kubernetes-v2/deploy-helm/) + diff --git "a/kb/armory/General/deploying-through-armory-agent-for-kubernetes-with-crd-results-in-\"error-occurred-when-list-customresourcedefinition-null\".md" "b/kb/armory/General/deploying-through-armory-agent-for-kubernetes-with-crd-results-in-\"error-occurred-when-list-customresourcedefinition-null\".md" new file mode 100644 index 0000000000..950d57c0e6 --- /dev/null +++ "b/kb/armory/General/deploying-through-armory-agent-for-kubernetes-with-crd-results-in-\"error-occurred-when-list-customresourcedefinition-null\".md" @@ -0,0 +1,140 @@ +--- +title: Deploying through Armory Agent for Kubernetes with CRD results in "Error occurred when List customResourceDefinition/null" +--- + +## Issue +Customers may find that Kubernetes custom resource definition (CRD) deployments fail with the below error when deployed through the Armory Agent for Kubernetes +``` +exception="io.armory.kubesvc.services.ops.executor.KubesvcRuntimeException: Error occurred when List customResourceDefinition/null in : (401: namespace default has not been authorized in the configuration of account xxxxxxxxxxx) + at io.armory.kubesvc.util.OperationUtils.perform(OperationUtils.java:79) + at io.armory.kubesvc.services.ops.executor.KubesvcExecutor.list(KubesvcExecutor.java:254) + at jdk.internal.reflect.GeneratedMethodAccessor411.invoke(Unknown Source) + at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + at java.base/java.lang.reflect.Method.invoke(Method.java:566) + at io.armory.kubesvc.services.registration.clouddriver.KubesvcCredentialsParser.lambda$makeKubesvcJobExecutor$0(KubesvcCredentialsParser.java:64) + at com.netflix.spinnaker.clouddriver.kubernetes.op.job.KubectlJobExecutor$$EnhancerByCGLIB$$4f088312.list() + at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.lambda$list$5(KubernetesCredentials.java:402) + at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.runAndRecordMetrics(KubernetesCredentials.java:618) + at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.runAndRecordMetrics(KubernetesCredentials.java:603) + at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.list(KubernetesCredentials.java:397) + at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.crdSupplier(KubernetesCredentials.java:316) + at com.google.common.base.Suppliers$ExpiringMemoizingSupplier.get(Suppliers.java:243) + at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.getCrdProperties(KubernetesCredentials.java:289) + at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesKindRegistry.getKindProperties(KubernetesKindRegistry.java:89) + at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesKindRegistry.getKindPropertiesOrDefault(KubernetesKindRegistry.java:74) + at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.getKindProperties(KubernetesCredentials.java:301) + at com.netflix.spinnaker.clouddriver.kubernetes.converter.manifest.KubernetesDeployManifestConverter.updateNamespace(KubernetesDeployManifestConverter.java:129) + at com.netflix.spinnaker.clouddriver.kubernetes.converter.manifest.KubernetesDeployManifestConverter.lambda$convertListDescription$1(KubernetesDeployManifestConverter.java:100) + at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) + at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) + at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) + at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) + at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) + at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) + at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) + at com.netflix.spinnaker.clouddriver.kubernetes.converter.manifest.KubernetesDeployManifestConverter.convertListDescription(KubernetesDeployManifestConverter.java:118) + at com.netflix.spinnaker.clouddriver.kubernetes.converter.manifest.KubernetesDeployManifestConverter.convertDescription(KubernetesDeployManifestConverter.java:72) + at com.netflix.spinnaker.clouddriver.kubernetes.converter.manifest.KubernetesDeployManifestConverter.convertDescription(KubernetesDeployManifestConverter.java:44) + at com.netflix.spinnaker.clouddriver.orchestration.OperationsService.lambda$convert$5(OperationsService.java:155) + at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) + at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) + at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) + at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) + at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) + at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) + at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) + at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) + at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) + at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274) + at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) + at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) + at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) + at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) + at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) + at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) + at com.netflix.spinnaker.clouddriver.orchestration.OperationsService.convert(OperationsService.java:223) + at com.netflix.spinnaker.clouddriver.orchestration.OperationsService.collectAtomicOperations(OperationsService.java:107) + at com.netflix.spinnaker.clouddriver.orchestration.OperationsService$collectAtomicOperations.call(Unknown Source) + at com.netflix.spinnaker.clouddriver.controllers.OperationsController.cloudProviderOperations(OperationsController.groovy:96) + at jdk.internal.reflect.GeneratedMethodAccessor550.invoke(Unknown Source) + at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + at java.base/java.lang.reflect.Method.invoke(Method.java:566) + at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) + at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) + at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) + at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878) + at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792) + at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) + at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) + at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) + at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) + at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) + at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) + at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) + at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) + at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228) + at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) + at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) + at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) + at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) + at org.springframework.web.filter.ShallowEtagHeaderFilter.doFilterInternal(ShallowEtagHeaderFilter.java:106) + at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) + at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) + at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) + at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) + at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118) + at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) + at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) + at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) + at com.netflix.spinnaker.fiat.shared.FiatAuthenticationFilter.doFilter(FiatAuthenticationFilter.java:65) + at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) + at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:158) + at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) + at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) + at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) + at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) + at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) + at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) + at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) + at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) + at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) + at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) + at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) + at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) + at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) + at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) + at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) + at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:109) + at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) + at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) + at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) + at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) + at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) + at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) + at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) + at javax.servlet.FilterChain$doFilter.call(Unknown Source) + at com.netflix.spinnaker.filters.AuthenticatedRequestFilter.doFilter(AuthenticatedRequestFilter.groovy:147) + at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) + at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) + at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) + at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) + at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) + at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) + at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) + at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) + at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:764) + at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) + at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382) + at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) + at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) + at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723) + at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) + at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) + at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) + at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) + at java.base/java.lang.Thread.run(Thread.java:829)" +``` +## Cause +This is identified as an issue with the ```Armory Agent plugin ```if the version used is below ```0.8.26```, ```0.9.18```, or ```0.10.2```.  Below is the plugin matrix for corresponding Spinnaker versions that had the issue +Armory Enterprise (Spinnaker) VersionAgent plugin Version2.25.x (1.25.x)0.8.262.26.x (1.26.x)0.9.182.27.x (1.27.x)0.10.2 + diff --git a/kb/armory/General/deployment-error-exception--resolve-deploy-source-manifest-.md b/kb/armory/General/deployment-error-exception--resolve-deploy-source-manifest-.md new file mode 100644 index 0000000000..88fb735b36 --- /dev/null +++ b/kb/armory/General/deployment-error-exception--resolve-deploy-source-manifest-.md @@ -0,0 +1,36 @@ +--- +title: Deployment error Exception ( Resolve Deploy Source Manifest ) +--- + +## Issue +The following error may be observed on a particular single application in a deployment. This error may not appear in all applications in an environment. +  +The error indicates: + +``` +Exception (Resolve Deploy Source Manifest) +Failure evaluating manifest expressions: +{CONSUL_FULLNAME="" +exec /bin/consul agent +-node="NODE" +-advertise="ADVERTISE_IP" +-bind=0.0.0.0 +-client=0.0.0.0 +-node-meta=pod.name:HOSTNAME +-hcl='leave_on_terminate=true' +-hcl='ports{grpc=}' +-config-dir=/consul/config +-config-dir=/consul/aclconfig +-datacenter= +-data-dir=/ +-encrypt="....." +-retry-join="" +``` +The following screen shows the error in the UI: + +## Cause +Spinnaker ships with a template parser, and so it needs to escape template expressions of the form ```$(..}```. +After ***OSS 1.13***, there was an addition the Manifest expression evaluation feature. This addition prevents Spinnaker from evaluating ```${}```-wrapped expressions within the ```Deploy (Manifest)``` stage by default.  +Further details are available in: [https://github.com/spinnaker/deck/pull/6696](https://github.com/spinnaker/deck/pull/6696)```````````` + + diff --git "a/kb/armory/General/deployment-fails-with-\"load-balancer-service--does-not-exist\"-error.md" "b/kb/armory/General/deployment-fails-with-\"load-balancer-service--does-not-exist\"-error.md" new file mode 100644 index 0000000000..b5648b299c --- /dev/null +++ "b/kb/armory/General/deployment-fails-with-\"load-balancer-service--does-not-exist\"-error.md" @@ -0,0 +1,18 @@ +--- +title: Deployment fails with "Load balancer service does not exist" error +--- + +## Issue +An organization deploying to Kubernetes using Kustomize may run into an issue where the bake stage of a deployment is successful and can see the service section in the baked manifest as needed. However the pipeline fails with the error, +“Load balancer service does not exist" + +Unique variables for this bug include having the Deployment Strategy set to Red/Black (Blue/Green)  + +## Cause +This is a known bug of Kubernetes, Kustomize, and even Helm. There is another Knowledge Article that goes over some of the issues and workarounds.  +[https://kb.armory.io/s/article/Load-balancer-service-svc-spinnaker-demo-does-not-exist](https://kb.armory.io/s/article/Load-balancer-service-svc-spinnaker-demo-does-not-exist) +Additionally, there is a very similar issue on Github. +[https://github.com/spinnaker/spinnaker/issues/5040](https://github.com/spinnaker/spinnaker/issues/5040) +Kubernetes has trouble viewing manifests when deployment strategies are used, thus causing errors.  + + diff --git a/kb/armory/General/deployment-strategy-red-black-error--load-balancer-service--does-not-exist.md b/kb/armory/General/deployment-strategy-red-black-error--load-balancer-service--does-not-exist.md new file mode 100644 index 0000000000..7f613e58ad --- /dev/null +++ b/kb/armory/General/deployment-strategy-red-black-error--load-balancer-service--does-not-exist.md @@ -0,0 +1,14 @@ +--- +title: Deployment Strategy Red/Black error- Load balancer service does not exist +--- + +## Issue +With the Deploy Strategy set to ***Red/Black***, and when deploying to Kubernetes using ***Kustomize***, the bake stage is successful and the service section is visible in the baked manifest.  However the pipeline fails with the following error: +``` Load balancer service does not exist``` +The UI shows the following, for example: + + +## Cause +This issue is related to the following Github issues:[https://github.com/spinnaker/spinnaker/issues/5040](https://github.com/spinnaker/spinnaker/issues/5040)[https://github.com/spinnaker/clouddriver/pull/5248](https://github.com/spinnaker/clouddriver/pull/5248) +As reported, Kubernetes has a bug where it will not see the manifest properly and will therefore error out. When utilizing the ```Deploy (Manifest)``` stage with ***Red/Black rollout strategy***, a requirement of using this stage was for a ```Service``` to exist before running the stage in order to read its ```selector``` and apply it to the newly deployed replicaset.  This requirement has been updated, so that now the ```Service``` will first be read from the stage manifest list, and if not found it will proceed with the same behaviour as before, reading it live from the cluster.  + diff --git "a/kb/armory/General/deployments-cluster-tab-missing-\"undo-rollback\"-feature---clusters-tab-missing-previously-available-options.md" "b/kb/armory/General/deployments-cluster-tab-missing-\"undo-rollback\"-feature---clusters-tab-missing-previously-available-options.md" new file mode 100644 index 0000000000..862d136d0d --- /dev/null +++ "b/kb/armory/General/deployments-cluster-tab-missing-\"undo-rollback\"-feature---clusters-tab-missing-previously-available-options.md" @@ -0,0 +1,14 @@ +--- +title: Deployments Cluster Tab missing "Undo Rollback" feature / Clusters Tab Missing Previously Available Options +--- + +## Issue +The Clusters Tab does not show the "Undo Rollout" feature in the console UI.  It was once available but is no longer seen after an upgrade.  +It is also missing options to scale deployments and possibly other features.   +  + +## Cause +This issue is a result of the Infrastructure UI change that was added in 2.26. +The following release notes correspond with the change:[https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-26-0/#kubernetes-infrastructure-in-the-ui](https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-26-0/#kubernetes-infrastructure-in-the-ui) +  + diff --git a/kb/armory/General/deployments-exceeding-progress-deadline.md b/kb/armory/General/deployments-exceeding-progress-deadline.md new file mode 100644 index 0000000000..f7ee0db59a --- /dev/null +++ b/kb/armory/General/deployments-exceeding-progress-deadline.md @@ -0,0 +1,10 @@ +--- +title: Deployments Exceeding Progress Deadline +--- + +## Issue +During long deployments, it's possible for Spinnaker to error out and fail without a major reason. Error codes may relate to timeouts.  + +## Cause +This can be due to a Kubernetes error that deals with the ```manifest.yml``` that is being deployed.  This error directly correlates to the ```progressDeadlineSeconds``` variable in the ```manifest.yml ```file + diff --git a/kb/armory/General/determine-or-check-the-environment's-spinnaker-version.md b/kb/armory/General/determine-or-check-the-environment's-spinnaker-version.md new file mode 100644 index 0000000000..9d3aee79f9 --- /dev/null +++ b/kb/armory/General/determine-or-check-the-environment's-spinnaker-version.md @@ -0,0 +1,12 @@ +--- +title: Determine or Check the Environment's Spinnaker Version +--- + + +### Question: +How to find out the version of Spinnaker that the environment is running at?  +Answer: +Besides checking the deployment configuration in Halyard or Operator, it is also possible to just check from either Gate, or the Spinnaker console.   +When running on AWS go to the Gate endpoint, (e.g. ```https://spinnaker.armory.io:8084```) and head to the ```/armory/versions``` subdirectory (e.g. ```https://spinnaker.armory.io:8084/armory/versions```) +It can also be determined via the Spinnaker environment console.  Navigate to your Deck endpoint, and it can be verified by clicking on the upper right corner ```Help``` icon and then viewing the version shown, as per the example below: + diff --git a/kb/armory/General/difference-in-the-deployment-phase-takes-and-the-service-boot-time.md b/kb/armory/General/difference-in-the-deployment-phase-takes-and-the-service-boot-time.md new file mode 100644 index 0000000000..37770cac92 --- /dev/null +++ b/kb/armory/General/difference-in-the-deployment-phase-takes-and-the-service-boot-time.md @@ -0,0 +1,14 @@ +--- +title: Difference in the deployment phase takes and the service boot time +--- + +## Issue +Users may notice that the ***deployment phase*** of a pipeline takes a long time, in comparison with ***service boot time***. +For example, the deployment phase may take around 15 minutes, but the service boot time is significantly lower (e.g. less than 1 minute). +The following screen shows an example: + +This may impact the environment as the user would need to wait until the entire pipeline completes before a rollback can be executed.  + +## Cause +This issue can be due to cache and cleanup not being configured. + diff --git a/kb/armory/General/differences-between-github-and-gitrepo-artifacts-and-choosing-the-right-artifact-type.md b/kb/armory/General/differences-between-github-and-gitrepo-artifacts-and-choosing-the-right-artifact-type.md new file mode 100644 index 0000000000..29bed77046 --- /dev/null +++ b/kb/armory/General/differences-between-github-and-gitrepo-artifacts-and-choosing-the-right-artifact-type.md @@ -0,0 +1,14 @@ +--- +title: Differences Between GitHub and GitRepo Artifacts and Choosing the Right Artifact Type +--- + +## Introduction +Some clarifications about general use cases of GitRepo, instead of GitHub/GitLab or other artifacts. + +## Prerequisites +A GitRepo should be available and accessible to the user + +## Instructions +In general, in cases where an artifact that must be passed through that contains multiple files, users should be looking to engage with [GitRepo artifacts](https://spinnaker.io/reference/artifacts-with-artifactsrewrite/types/git-repo/) instead of [other types](https://spinnaker.io/reference/artifacts-with-artifactsrewrite/types/overview/). For example, the GitRepo type must be used when using the Bake (Manifest) stage to bake a manifest using Kustomize since the Kustomize template render engine requires multiple files.   +Likewise, the same can be said for Terraform deployments, as there are often multiple modules and other files involved with the deployment + diff --git a/kb/armory/General/dinghy-'roles'-entry-is-not-getting-passed-to-actual-spinnaker-pipeline-json.md b/kb/armory/General/dinghy-'roles'-entry-is-not-getting-passed-to-actual-spinnaker-pipeline-json.md new file mode 100644 index 0000000000..05451d35b5 --- /dev/null +++ b/kb/armory/General/dinghy-'roles'-entry-is-not-getting-passed-to-actual-spinnaker-pipeline-json.md @@ -0,0 +1,12 @@ +--- +title: Dinghy 'roles' entry is not getting passed to actual Spinnaker Pipeline JSON +--- + +## Issue +An organization may find that that ```roles``` entry in a Dinghy file will not get injected to the Pipeline JSON which is a requirement for triggering Pipelines with permissions. +This can cause blockages, delays, and database errors from so many failed pipelines. + +## Cause +This is a known bug with Dinghy, and how it parses app metadata and reads longer logs. This bug will prevent the roles entry from being read correctly which then leads to permissions failures.  +This bug affects all Dinghy versions + diff --git a/kb/armory/General/dinghy-cannot-self-reference-pipelines-created-in-the-same-dinghyfile-using-pipelineid.md b/kb/armory/General/dinghy-cannot-self-reference-pipelines-created-in-the-same-dinghyfile-using-pipelineid.md new file mode 100644 index 0000000000..530c1dc0d3 --- /dev/null +++ b/kb/armory/General/dinghy-cannot-self-reference-pipelines-created-in-the-same-dinghyfile-using-pipelineid.md @@ -0,0 +1,26 @@ +--- +title: Dinghy cannot self-reference Pipelines created in the same Dinghyfile using PipelineID +--- + +## Issue +An organization may want to create a Dinghyfile pipeline that self-references the application that will be created upon the completion of the Pull Request.  It may be useful in order to automate the workflow of their pipelines. + +However, upon the initial Pull Request when the Pipeline and Applications are created, users will be notified of a ```403 Permissions``` error.  However, the Application and Pipelines were successfully created.   + +Any subsequent attempts to update the Dinghyfile and execute a new Pull Request, or redeploy the pipeline ***will not generate*** an error and the Application can be referenced without issues, so long as the Application and Pipeline are not changed or deleted.   +  + +## Cause +The cause of this issue is due to the order of executions and a race condition.  When initially creating the pipeline and references in the first Pull Request, the Dinghyfile creates the application and pipelines and then syncs permissions as defined in the Dinghyfile. +When doing so, users and admins can see in debug logs that during a deployment the ```PipelineID``` definition is looking for a yet non-existent Application, since the Pipeline and Application creation steps have yet to be created.   +``` +time="2022-05-04T20:17:37Z" level=info msg="Substituting pipeline triggerPipeline: test-dinghyfile" +time="2022-05-04T20:17:37Z" level=info msg="Looking up existing pipelines" +time="2022-05-04T20:17:38Z" level=error msg="Failed to GetPipelines for test-app: could not get pipelines for test-app - Forbidden: {\"timestamp\":\"2022-05-04T20:17:38.217+00:00\",\"status\":403,\"error\":\"Forbidden\",\"message\":\"Access denied to application test-app - required authorization: READ\"}" +time="2022-05-04T20:17:38Z" level=info msg="Creating application 'test-app'..." +``` + +Before the Application can sync permissions, no one has access to the Application in order to keep it secure.  This is why a 403 error is displayed, even if the Application has open permissions. Until the role sync happens, no one can access the application or pipelines. As a result, the PipelineID cannot be referred to successfully. + +Once the Application has been created and the initial pipeline has permissions synced, redeploying the Dinghyfile will work as the permissions, application, and pipelines have been created.  However, if the application or pipelines are removed, deleted, or renamed, the issue will re-occur. + diff --git a/kb/armory/General/dinghy-crashing-for-customers-using-2.24+-when-customers-are-using-a-custom-endpoint.md b/kb/armory/General/dinghy-crashing-for-customers-using-2.24+-when-customers-are-using-a-custom-endpoint.md new file mode 100644 index 0000000000..6f006fd80d --- /dev/null +++ b/kb/armory/General/dinghy-crashing-for-customers-using-2.24+-when-customers-are-using-a-custom-endpoint.md @@ -0,0 +1,12 @@ +--- +title: Dinghy Crashing for Customers using 2.24+ when customers are using a Custom Endpoint +--- + +## Issue +Customers using a Custom Endpoint are seeing crashes occur using 2.24+.  Upon observing the Dinghy logs, there is a notification that the service is attempting to reach out to ```api.github.com``` instead of the set custom endpoint.   +Below is an example of such an error +```2021/05/09 04:51:01 POST https://api.github.com/repos/armory-test/dinghy-test/commits/9999xx999x9xx9xxx9xx99x99x99xx99x9x99xx9/comments: 401 Bad credentials []``` + +## Cause +As of 2.24.x, a change was made to push comments on Dinghy outputs back into the comment of the PR.  This is a part of the notifications setting that was added to Dinghy to allow for more robust and detailed information about the results from a PR commit.  Currently, the notification only supports using Github Cloud, using the ```api.github.com``` endpoint, and does not account for custom endpoints + diff --git a/kb/armory/General/dinghy-crashloop-when-using-redis-loss-of-connectivity-loss-of-relationship.md b/kb/armory/General/dinghy-crashloop-when-using-redis-loss-of-connectivity-loss-of-relationship.md new file mode 100644 index 0000000000..43cd787036 --- /dev/null +++ b/kb/armory/General/dinghy-crashloop-when-using-redis-loss-of-connectivity-loss-of-relationship.md @@ -0,0 +1,17 @@ +--- +title: Dinghy Crashloop When Using Redis (Loss of connectivity/Loss of relationship) +--- + +## Issue +As per Armory Documentation, Dinghy can use Redis to store relationships, and it is recommended that Admins consider a separate, external Redis with appropriate backups and redundancy.  +Source: [https://docs.armory.io/armory-enterprise/armory-admin/dinghy-enable/#configuring-redis](https://docs.armory.io/armory-enterprise/armory-admin/dinghy-enable/#configuring-redis) +  +If Redis were to go down, Dinghy would go into CrashLoop, being unable to connect to the Redis server. +```time="2022-07-27T23:38:48Z" level=fatal msg="Redis Server at redis://armory-test-001.iwdfo7.0001.usw1.cache.amazonaws.com:6379 could not be contacted: dial tcp: lookup armory-test-redis-001.iwdfo7.0001.usw1.cache.amazonaws.com on 10.100.0.10:53: no such host"``` +  +  +  + +## Cause +As Dinghy uses Redis to provide caching of its services, connection to Redis is essential to its functionality and health.  Should the connection break, Dinghy will throw errors resulting from the lack of connectivity. + diff --git a/kb/armory/General/dinghy-fails-to-create-pipelines-in-armory-2.19.8-and-prior.md b/kb/armory/General/dinghy-fails-to-create-pipelines-in-armory-2.19.8-and-prior.md new file mode 100644 index 0000000000..fe2b029efa --- /dev/null +++ b/kb/armory/General/dinghy-fails-to-create-pipelines-in-armory-2.19.8-and-prior.md @@ -0,0 +1,13 @@ +--- +title: Dinghy Fails to Create Pipelines in Armory 2.19.8 and Prior +--- + +## Issue +Dinghy is failing to create pipelines in Spinnaker version 2.19.8, but the same Dinghyfile is creating applications on Spinnaker version 2.20.5 or above. The organization has restrictions to stay on 2.19.x for the near future and needs Dinghy to work. +Errors in logs will look something like this +```time=“******************” level=error msg="Failed to parse dinghyfile Dinghyfile: invalid character '{' looking for beginning of object key string"``` + +## Cause +Local modules were added into dinghy on Armory version 2.19.9.2.19.8 and below will fail to properly parse the dinghy file leading to failure. Dinghy modules can be updated individually as a stopgap measure before upgrading beyond 2.19.x +Dinghy image for 2.19.9 is *[docker.io/armory/dinghy:2.19.13](http://docker.io/armory/dinghy:2.19.13) * + diff --git a/kb/armory/General/dinghy-file-does-not-update-pipelines-in-spinnaker.md b/kb/armory/General/dinghy-file-does-not-update-pipelines-in-spinnaker.md new file mode 100644 index 0000000000..f0cb2adda7 --- /dev/null +++ b/kb/armory/General/dinghy-file-does-not-update-pipelines-in-spinnaker.md @@ -0,0 +1,11 @@ +--- +title: Dinghy file does not update pipelines in Spinnaker +--- + +## Issue +This issue is seen when the application and pipeline are created by the same ```dinghyfile```, and need to be imported twice so the reference will be ingested into the process properly as upon execution, and at the time of initial evaluation, the pipeline ID was unavailable.  Therefore, the ```dingyfile``` needs to be executed twice to see the pipeline ID.  + +## Cause +This issue is observed as in the first commit, the auto-gen ID is present but converted into null within the trigger and thus not assigned as it doesn't exist. At the 2nd commit, the ```pipelineID``` function is able to locate the ID. +  + diff --git a/kb/armory/General/dinghy-github-status-is-always-failed-when-using-yaml-parser.md b/kb/armory/General/dinghy-github-status-is-always-failed-when-using-yaml-parser.md new file mode 100644 index 0000000000..4046d03af4 --- /dev/null +++ b/kb/armory/General/dinghy-github-status-is-always-failed-when-using-yaml-parser.md @@ -0,0 +1,11 @@ +--- +title: Dinghy github status is always failed when using YAML parser +--- + +## Issue +When utilizing YAML parser for Dinghyfile, any pull request on a feature branch in Github gets a status update by Dinghy which is always failed with message: +```UpdateDinghyfile malformed syntax: [Error in line 1, char 0: invalid character 'a' looking for beginning of value\napplication: core-accounts-spinnakeryamltest-sys # no spaces here, must be alphanumeric]``` + +## Cause +This happens because Dinghy code first tries to parse Dinghyfile using JSON parser which fails and then does parsing with YAML parser. This successfully renders the pipeline but gives a false failure on Github pull request status. + diff --git a/kb/armory/General/dinghy-in-2.26-tries-to-validate-readme.md-as-a-module.md b/kb/armory/General/dinghy-in-2.26-tries-to-validate-readme.md-as-a-module.md new file mode 100644 index 0000000000..eb90f8acbc --- /dev/null +++ b/kb/armory/General/dinghy-in-2.26-tries-to-validate-readme.md-as-a-module.md @@ -0,0 +1,16 @@ +--- +title: Dinghy in 2.26 tries to validate README.md as a module +--- + +## Issue +An organization may run into an issue where Dinghy will run excessively slow on pipelines, and may even fail with errors about failing to validate a non ```dinghyfile``` like ```readme.md``` +```Logfiles``` can include snippets such as the following.  +time="2021-05-24T21:50:08Z" level=info msg="[Processing Push]" +time="2021-05-24T21:50:08Z" level=info msg="Push does not include [dinghyfile], skipping." +time="2021-05-24T21:50:09Z" level=error msg="Failed to parse module:\n ($CLIENTDINGHTCOMMENTS)" +time="2021-05-24T21:50:09Z" level=error msg="module parse failed: [invalid character '#' looking for beginning of value]" + + +## Cause +This is a known bug with Dinghy in Armory version 2.26. It could affect other versions too but Armory has tested with 2.23 and had no issues. Other versions below 2.26.1 have not been tested.The cause is that Dinghy will fail to validate non-dinghy files and as such, treat them improperly causing failed pipelines and delays. In the above example, the ```readme.md``` file causes a failure in the process.  + diff --git a/kb/armory/General/dinghy-multi-repo-strategy-with-repoconfig.md b/kb/armory/General/dinghy-multi-repo-strategy-with-repoconfig.md new file mode 100644 index 0000000000..d8e65edfae --- /dev/null +++ b/kb/armory/General/dinghy-multi-repo-strategy-with-repoconfig.md @@ -0,0 +1,54 @@ +--- +title: Dinghy Multi-Repo Strategy with RepoConfig +--- + +## Introduction +```RepoConfig``` is intended to allow users to configure repositories to use a branch other than ```master``` or ```main```. A sample is provided below: + +# ... other config ... +repoConfig: + - provider: github + repo: my-repository + branch: my-branch + - provider: github + repo: another-repository + branch: another-branch +# ... other config ... + + +**This feature assumes that elements of **```**repoConfig**```** are unique over the union of **```**provider**```** and **```**repository**```** keys**. In other words, if duplicate pairs of ```provider``` and ```repo``` keys are defined, the user will see unexpected behavior. Let's consider the following config: +# ... other config ...repoConfig:  - provider: github    repo: my-repository    branch: my-branch  - provider: github    repo: my-repository    branch: another-branch# ... other config ... +In this example, Dinghy would read ```repoConfig``` and *always choose the first definition* because it will match ```github``` and ```my-repository``` in the first element. Since YAML lists are order preserving, the user will never see ```another-branch``` events processed, only validated. + +## Prerequisites +Users have Dinghy configured and are trying to make use of RepoConfig correctly. + +## Instructions +As most organizations would be utilizing different applications for different environments, the solution might look like the following: + +└── my-github-organization + # the webhook on this repository points to the users dev Spinnaker instance + ├── dinghy-dev + │   ├── app4 # this app passes "dev" and "dev-account" variables to modules + │   ├── app5 + │   └── app6 + # modules are parameterized over environment + # and Spinnaker account + ├── dinghy-modules + │   ├── bake.stage + │   ├── canary.stage + │   └── deploy.stage + # the webhook on this repo points to the users nonprod Spinnaker instance + ├── dinghy-nonprod + │   ├── app1 # this app passes "nonprod" and "nonprod-account" variables to modules + │   ├── app2 + │   └── app3 + # the webhook on this repo points to the users prod Spinnaker instance + └── dinghy-prod + ├── app7 # this app passes "prod" and "prod-account" variables to modules + ├── app8 + └── app9 + + +With this setup the organization can break up their instances but still point to the same dinghy-modules within the organization. + diff --git a/kb/armory/General/dinghy-rendering-failures---forbidden--access-denied-to-application-access-rights-verified.md b/kb/armory/General/dinghy-rendering-failures---forbidden--access-denied-to-application-access-rights-verified.md new file mode 100644 index 0000000000..12f3a6c5ec --- /dev/null +++ b/kb/armory/General/dinghy-rendering-failures---forbidden--access-denied-to-application-access-rights-verified.md @@ -0,0 +1,19 @@ +--- +title: Dinghy rendering failures - Forbidden- Access denied to application (Access Rights Verified) +--- + +## Issue +A pipeline shows as blank, and the following errors are observed on Dinghy events: +``` +time="2021-07-08T15:46:56Z" level=error msg="Upsert failed: [Forbidden: Access denied to application - required authorization: WRITE]" +time="2021-07-08T15:46:56Z" level=error msg="Failed to update Pipelines for [deploy_cp/infrastructure/live/us-east-1/sandboxes/spinnaker-pipelines/infra-services/find-last-commit/dinghyfile Forbidden: Access denied to application - required authorization: WRITE]: %!s(MISSING)" +time="2021-07-08T15:46:56Z" level=error msg="Error processing Dinghyfile: [Forbidden: Access denied to application - required authorization: WRITE]" +time="2021-07-08T15:46:57Z" level=error msg="ProcessPush Failed (other): [Forbidden: Access denied to application - required authorization: WRITE]" +``` +However, access for Dinghy was confirmed to be functional for other pipelines to the same deployment targets, and therefore, the credentials being used have confirmed to be functional and access rights have been verified. + +## Cause +This issue may be caused by a missing template. For example: +```"application": "{{ var "app" }}",``` +As such, when Spinnaker tries to update the pipeline, the application is empty/cannot be found. + diff --git a/kb/armory/General/dinghy-skips-updating-pipeline-when-local-module-is-changed.md b/kb/armory/General/dinghy-skips-updating-pipeline-when-local-module-is-changed.md new file mode 100644 index 0000000000..c2f743c713 --- /dev/null +++ b/kb/armory/General/dinghy-skips-updating-pipeline-when-local-module-is-changed.md @@ -0,0 +1,10 @@ +--- +title: Dinghy skips updating pipeline when local module is changed +--- + +## Issue +When making a change to a local module on a repository, *but not the dingyfile itself* - Dinghy skips updating the pipeline because the dinghyfile was not changed. + +## Cause +This functionality is working as intended, as local_modules are not expected to have parity with first-class modules. + diff --git a/kb/armory/General/dinghy-slack-notifications-not-working---java.lang.nosuchmethoderror--void-org.jsoup.md b/kb/armory/General/dinghy-slack-notifications-not-working---java.lang.nosuchmethoderror--void-org.jsoup.md new file mode 100644 index 0000000000..f21d521b5e --- /dev/null +++ b/kb/armory/General/dinghy-slack-notifications-not-working---java.lang.nosuchmethoderror--void-org.jsoup.md @@ -0,0 +1,14 @@ +--- +title: Dinghy Slack Notifications not working - java.lang.NoSuchMethodError- void org.jsoup +--- + +## Issue +Customers may find that after upgrading to 2.27.x cannot get Slack Notifications even though it was working previously.  This can occur, for example, after setting up [Slack notifications in Dinghy](https://docs.armory.io/armory-enterprise/armory-admin/dinghy-enable/#slack-notifications) +Upon investigating Echo logs, customers may see the following error in Echo: +``````org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.NoSuchMethodError: 'void org.jsoup.select.NodeTraversor.(org.jsoup.select.NodeVisitor)'`````` +with a further reference to: +``````Caused by: java.lang.NoSuchMethodError: 'void org.jsoup.select.NodeTraversor.(org.jsoup.select.NodeVisitor)'`````` + +## Cause +The issue was caused by recent upgrades to the ```jinjava``` package in Spinnaker, which caused an issue with a dependency in jsoup.   + diff --git a/kb/armory/General/dinghy-webhook-shows-an-html-response,-rather-than-a-json-response.md b/kb/armory/General/dinghy-webhook-shows-an-html-response,-rather-than-a-json-response.md new file mode 100644 index 0000000000..1a1eae4ad7 --- /dev/null +++ b/kb/armory/General/dinghy-webhook-shows-an-html-response,-rather-than-a-json-response.md @@ -0,0 +1,13 @@ +--- +title: Dinghy webhook shows an HTML response, rather than a JSON response +--- + +## Issue +A webhook set up in an organization to push events. A completion on a pull request for Github does not result in a Dinghy Event in the Armory Spinnaker UI as seen below: + +It is expected that a JSON response would be provided, but instead, an HTML response is provided, like the example below: + +## Cause +This can be due to ```webhook-auth``` being active. +Further information on utilizing Webhooks in Spinnaker Pipelines is available at: [https://docs.armory.io/docs/spinnaker-user-guides/webhooks/](https://docs.armory.io/docs/spinnaker-user-guides/webhooks/) + diff --git a/kb/armory/General/dinghyfile-updates-do-not-trigger-slack-notifications.md b/kb/armory/General/dinghyfile-updates-do-not-trigger-slack-notifications.md new file mode 100644 index 0000000000..961cbaba63 --- /dev/null +++ b/kb/armory/General/dinghyfile-updates-do-not-trigger-slack-notifications.md @@ -0,0 +1,11 @@ +--- +title: Dinghyfile updates do not trigger Slack notifications +--- + +## Issue +Dinghy has been configured to send pipeline update results to a Slack channel, but while updating the ```dinghyfile``` successfully creates pipeline changes - no notifications are being sent to the Slack channel. + +## Cause +Dinghy's Slack notifications do not send if Dinghy's notifications are not explicitly enabled.  This was a result of some changes made to code once Github Notifications became an option for Dinghy.  +As a result, customers will need to ensure that Dinghy notifications are enabled before setting Slack notifications. Once enabled, customers will want to also consider if they want to explicitly turn off GitHub Notifications as by default, they will be enabled.  For some customers, depending on the size of Dinghy's repository, it may be desirable to keep Github notifications turned off, as per [the possible High number of GitHub emails generated after every Dinghyfile update](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010558). + diff --git a/kb/armory/General/dinghyfiles-with-defined-application-permissions-fails-to-create-update-pipelines-forbidden-error.md b/kb/armory/General/dinghyfiles-with-defined-application-permissions-fails-to-create-update-pipelines-forbidden-error.md new file mode 100644 index 0000000000..d784864d2a --- /dev/null +++ b/kb/armory/General/dinghyfiles-with-defined-application-permissions-fails-to-create-update-pipelines-forbidden-error.md @@ -0,0 +1,58 @@ +--- +title: Dinghyfiles with defined Application Permissions fails to create/update pipelines (Forbidden Error) +--- + +## Issue +Customers that define [application permissions](https://docs.armory.io/armory-enterprise/spinnaker-user-guides/using-dinghy/#application-permissions) via Dinghy may encounter an issue where it fails to create and update applications and pipelines. +``` +{ + "application": "fiat-dinghytest", + "spec": { + "appmetadata": { + "permissions": { + "READ": [ + "spin-admin-test" + ], + "WRITE": [ + "spin-admin-test" + ], + "EXECUTE": [ + "spin-admin-test" + ] + } + } + }, + "pipelines": [ + { + "application": "fiat-dinghytest”, + "name": "DinghyValidation-Tests", + "keepWaitingPipelines": false, + "limitConcurrent": true, + "stages": [ + { + "name": "Wait", + "refId": "1", + "requisiteStageRefIds": [], + "type": "wait", + "waitTime": 30 + } + ], + "triggers": [] + } + ] +} +``` +  +  +  +The following errors may be found in the Dinghy logs. +``` +time="2022-10-25T11:01:35Z" level=error msg="Failed to create application (Forbidden: {\"timestamp\":\"2022-10-25T11:01:35.821+00:00\",\"status\":403,\"error\":\"Forbidden\",\"message\":\"Access denied to application fiat-dinghytest - required authorization: READ\"})" +time="2022-10-25T11:01:35Z" level=error msg="Failed to update Pipelines for dinghyfile: Forbidden: {\"timestamp\":\"2022-10-25T11:01:35.821+00:00\",\"status\":403,\"error\":\"Forbidden\",\"message\":\"Access denied to application fiat-dinghytest - required authorization: READ\"}" +time="2022-10-25T11:01:36Z" level=error msg="Error processing Dinghyfile: Forbidden: {\"timestamp\":\"2022-10-25T11:01:35.821+00:00\",\"status\":403,\"error\":\"Forbidden\",\"message\":\"Access denied to application fiat-dinghytest - required authorization: READ\"}" +time="2022-10-25T11:01:36Z" level=error msg="ProcessPush Failed (other): Forbidden: {\"timestamp\":\"2022-10-25T11:01:35.821+00:00\",\"status\":403,\"error\":\"Forbidden\",\"message\":\"Access denied to application fiat-dinghytest - required authorization: READ\"}" +time="2022-10-25T11:01:36Z" level=error msg="Forbidden: {\"timestamp\":\"2022-10-25T11:01:35.821+00:00\",\"status\":403,\"error\":\"Forbidden\",\"message\":\"Access denied to application fiat-dinghytest - required authorization: READ\"}" +``` +## Cause +The cause of this issue is due to Dinghy not having a Service Account/Fiat User assigned to it with adequate permissions.  As a result, after the application and pipeline are created initially, an update fails because there are inadequate permissions to do so + diff --git a/kb/armory/General/disable-default-repositories-for-plugins-on-air-gapped-environments-and-specify-the-repo.md b/kb/armory/General/disable-default-repositories-for-plugins-on-air-gapped-environments-and-specify-the-repo.md new file mode 100644 index 0000000000..c7012f3731 --- /dev/null +++ b/kb/armory/General/disable-default-repositories-for-plugins-on-air-gapped-environments-and-specify-the-repo.md @@ -0,0 +1,66 @@ +--- +title: Disable default repositories for Plugins on air-gapped environments and Specify the Repo +--- + +## Introduction +In air-gapped environments, access to public internet is generally blocked and in such cases, the Spinnaker services would not be able to download the Plugin from default public repositories. +In this example, we will look at the Observability plugin, which is responsible for measuring the performance and monitoring the Spinnaker micro services.  This same method can be applied to the Policy Engine Plugin, and any other plugins that are being used. +Because it is trying to get the public repository, this delays the start time for Spinnaker micro services as applications start after the timeouts. Below is the example startup log from ```echo``` service where the service is seen to timeout when attempting to connect to the public repository for fetching the Observability plugin.  +``` +2021-06-16 14:07:29.259 INFO 1 --- [ main] org.pf4j.AbstractPluginManager : Plugin '/opt/echo/plugins/armory-observability-plugin-v1.2.0/echo' is disabled +2021-06-16 14:07:29.277 INFO 1 --- [ main] org.pf4j.AbstractPluginManager : Plugin 'Armory.ObservabilityPlugin@1.2.0' resolved +2021-06-16 14:09:40.147 ERROR 1 --- [ main] org.pf4j.update.DefaultUpdateRepository : Connection timed out (Connection timed out) +2021-06-16 14:11:51.215 ERROR 1 --- [ main] org.pf4j.update.DefaultUpdateRepository : Connection timed out (Connection timed out) +``` +The steps to disable the default repository are described below + +## Prerequisites +Spinnaker version 2.26.0 or later. There is an issue with this not working with prior versions. + +## Instructions +**Disable the Plugin for air-gapped environments:** +Identify the plugins configuration under the config files and change the ```enableDefaultRepositories:``` to "false".  After doing so, all Plugins will no longer access the cloud repository for the plugin. +``` +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker + namespace: spinnaker +spec: + spinnakerConfig: + profiles: + spinnaker: + spinnaker: + extensibility: + enableDefaultRepositories: false + plugins: + Armory.ObservabilityPlugin: + ..... +``` +However, plugins could be maintained under repositories that are internal to the organization. In such situations, adding the custom repository details under the path  spinnakerConfig.profiles.spinnaker.spinnaker.extensibility.repositories:  should fetch the plugins from the custom repository. +``` +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker + namespace: spinnaker +spec: + spinnakerConfig: + profiles: + spinnaker: + spinnaker: + extensibility: + enableDefaultRepositories: false + plugins: + Armory.ObservabilityPlugin: + ..... + enabled: true + version: 1.0.0 + repositories: + armory-observability-plugin-releases: + url: https://custom.repository/armory-plugins/armory-observability-plugin-releases/master/repositories.json + armory-plugins: + url: https://dl.bintray.com/armory/armory-plugins/repositories.json +```                 + + diff --git "a/kb/armory/General/disable-deployment-registration-banner-message-\"you-configured-a-feature-that-requires-registration\".md" "b/kb/armory/General/disable-deployment-registration-banner-message-\"you-configured-a-feature-that-requires-registration\".md" new file mode 100644 index 0000000000..a41ff6309d --- /dev/null +++ "b/kb/armory/General/disable-deployment-registration-banner-message-\"you-configured-a-feature-that-requires-registration\".md" @@ -0,0 +1,29 @@ +--- +title: Disable Deployment Registration banner message "You configured a feature that requires registration" +--- + +## Introduction +Starting in Armory Enterprise 2.27, users will be prompted with a banner message in the Spinnaker UI to register their instance with Armory Cloud. This feature will attempt to make API calls to Armory in order to check whether the instance has been registered with Armory, which may be considered usage data and therefore not be permitted with some customers depending on their contract or situation.The text of the banner message reads: +```You configured a feature that requires registration. Registering shares your unique instance ID, which is required for certain features to work. Register here: ``` +This is a non-optional feature for most users of Armory, however **air-gapped customers may need this feature disabled** as it will attempt to make API calls toward Armory to check registration status.The banner should not be removed, and will not interfere with Spinnaker Usage.  Please note customers should be sending this data to ArmoryImplementation: [RegistrationVerificationMessenger.kt](https://github.com/armory-plugins/instance-registration/blob/master/instance-registration-gate/src/main/kotlin/io/armory/spinnaker/gate/registration/RegistrationVerificationMessenger.kt) + +## Prerequisites +Air-gapped Armory 2.27+ + +## Instructions +To disable the deployment registration message from appearing, add the following patch to your Spinnaker configuration: +``` +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker +spec: + spinnakerConfig: + profiles: + spinnaker: + spinnaker: # This second `spinnaker` is required + extensibility: + plugins: + Armory.InstanceRegistration: + enabled: false +``` \ No newline at end of file diff --git a/kb/armory/General/disable-diagnostic-settings-results-in-echo-pod-unable-to-start-armory-2.21.2.md b/kb/armory/General/disable-diagnostic-settings-results-in-echo-pod-unable-to-start-armory-2.21.2.md new file mode 100644 index 0000000000..1c4edc3f3c --- /dev/null +++ b/kb/armory/General/disable-diagnostic-settings-results-in-echo-pod-unable-to-start-armory-2.21.2.md @@ -0,0 +1,31 @@ +--- +title: Disable diagnostic settings results in echo pod unable to start (Armory 2.21.2) +--- + +## Issue +Under Armory 2.21.2, when diagnostics are disabled in the configuration: +``` + diagnostics: + enabled: false + uuid: xxxxxx-0d0f-40c2-aa78-xxxxxxx + logging: + enabled: false + endpoint: https://debug.armory.io/v1/logs +``` + +And then redeployed to Spinnaker, the Echo pod unable to boot up.   +The following error can be found in the logs: +APPLICATION FAILED TO START +*************************** + +Description: + +Parameter 0 of constructor in io.armory.spinnaker.echo.plugins.events.PluginEventsController required a bean of type 'io.armory.spinnaker.echo.plugins.events.DebugService' that could not be found. + +The following candidates were found but could not be injected: +- Bean method 'diagnosticsService' in 'PluginEventsConfig' not loaded because @ConditionalOnProperty (diagnostics.enabled) found different value in property 'diagnostics.enabled' +Only if the settings are reversed again so that ```armory.diagnostic.enabled``` flag is changed to ```true``` and redeployed, will the deployment will succeed. + +## Cause +There is a bug in 2.21.2 that causes the error in the Echo pod. + diff --git a/kb/armory/General/disabling-tls-1.1-in-spinnaker-and-specifying-the-protocols-to-be-used.md b/kb/armory/General/disabling-tls-1.1-in-spinnaker-and-specifying-the-protocols-to-be-used.md new file mode 100644 index 0000000000..2039927029 --- /dev/null +++ b/kb/armory/General/disabling-tls-1.1-in-spinnaker-and-specifying-the-protocols-to-be-used.md @@ -0,0 +1,69 @@ +--- +title: Disabling TLS 1.1 in Spinnaker and Specifying the Protocols to be used +--- + +## Introduction +As TLS 1.1 has reached end of life, customers may want to disable it from the available protocols in Spinnaker, and consider which protocols and Cipher Suites to enable.   + +## Instructions +#### In Operator +The following entries would need to be added in the ```profiles``` section of the Spinnaker Manifest.  In Operator, go to ```spec.spinnakerConfig.profiles.spinnaker``` and then look to make the following modification: + +``` +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker +spec: + spinnakerConfig: + profiles: + spinnaker: + default: + cipherSuites: + - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 + - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 + - TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 + - TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 + - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 + - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 + - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 + - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 + - TLS_AES_256_GCM_SHA384 + - TLS_CHACHA20_POLY1305_SHA256 + - TLS_AES_128_GCM_SHA256 + ## For embedded systems, the below two ciphers may be necessary. Normally, no need to include these though they're a TLS1.3 cipher + # - TLS_AES_128_CCM_8_SHA256 + # - TLS_AES_128_CCM_SHA256 + tlsVersions: + - TLSv1.2 + - TLSv1.3 +[...] +``` + +Next, redeploy the manifest and it will then limit the protocol usage to TLSv1.1.  The above is an example for limiting the protocols and excluding TLSv1.1, but it can be modified to further limit TLS protocol versions. +  +#### In Halyard +In the hal config profiles directory e.g. (```~/.hal/default/profiles/```), please add the following to the ```spinnaker.yml``` file, or create a new file if it doesn't already exist.  In the file, add the following code: +``` +default: + cipherSuites: + - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 + - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 + - TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 + - TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 + - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 + - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 + - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 + - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 + - TLS_AES_256_GCM_SHA384 + - TLS_CHACHA20_POLY1305_SHA256 + - TLS_AES_128_GCM_SHA256 +## Embedded systems is where yoU MIGHT see these - would not normally include these though they're a TLS1.3 cipher +# - TLS_AES_128_CCM_8_SHA256 +# - TLS_AES_128_CCM_SHA256 + tlsVersions: + - TLSv1.2 + - TLSv1.3 +``` +Next, redeploy the manifest with ```hal deploy apply``` and it will then limit the protocol usage to TLSv1.1.  The above is an example for limiting the protocols and excluding TLSv1.1, but it can be modified to further limit TLS protocol versions. + diff --git a/kb/armory/General/docker-binding-to-matching-repositories-breaking-pipelines.md b/kb/armory/General/docker-binding-to-matching-repositories-breaking-pipelines.md new file mode 100644 index 0000000000..dcab71f310 --- /dev/null +++ b/kb/armory/General/docker-binding-to-matching-repositories-breaking-pipelines.md @@ -0,0 +1,11 @@ +--- +title: Docker binding to matching repositories breaking pipelines +--- + +## Issue +A pipeline may break due to Spinnaker ignoring any matching image artifacts available in the pipeline.  Spinnaker instead uses the tag specified in the manifest image. + +## Cause +Since the merge of [PR #4874](https://github.com/spinnaker/clouddriver/pull/4874), Clouddriver will now bind all Docker artifacts based on ***repo name***, not ***repo+image***. +This may break any non-trivial Kubernetes pipelines, especially anything with a Canary deployment. The current behavior is that Spinnaker will end up binding a random assortment of Docker images, depending on an unknown precedence order. + diff --git a/kb/armory/General/during-migration-to-iam-for-service-accounts,-clouddriver-in-ha-returns-error--access-denied-service--amazon-s3;-status-code-403.md b/kb/armory/General/during-migration-to-iam-for-service-accounts,-clouddriver-in-ha-returns-error--access-denied-service--amazon-s3;-status-code-403.md new file mode 100644 index 0000000000..2f74a24341 --- /dev/null +++ b/kb/armory/General/during-migration-to-iam-for-service-accounts,-clouddriver-in-ha-returns-error--access-denied-service--amazon-s3;-status-code-403.md @@ -0,0 +1,36 @@ +--- +title: During Migration to IAM for Service Accounts, CloudDriver in HA returns error- Access Denied Service- Amazon S3; Status Code 403 +--- + +## Issue +When trying to migrate to utilize IAM for Service accounts from the working Spinnaker solution with KIAM, the ```clouddriver-rw``` pod fails it's readiness probing checks and the logs for it show the following: +``` +*************************** +APPLICATION FAILED TO START +*************************** + +Description: + +Failed to bind properties under 'sql.connectionpools.default.password' to java.lang.String: + + Reason: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: ....; S3 Extended Request ID: .....=; Proxy: null) +``` +## Action: + +Update your application's configuration +Users can confirm that by performing a ```kubectl describe pod``` on the ```clouddriver-rw``` pod, the secrets token for service accounts are mounted on the pod. This issue impacts all of the Spinnaker pods when trying to switch to utilizing IAM for Service Accounts. +The user is not able to exec into the Clouddriver pod as the pod is doing a ```CrashLoopBackoff```. ```kubectl describe pod``` logs the following events: +``` +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + + Normal Started 3m18s (x3 over 4m9s) kubelet Started container clouddriver-rw + Warning Unhealthy 3m1s (x3 over 3m51s) kubelet Readiness probe failed: + Warning BackOff 2m48s (x3 over 3m31s) kubelet Back-off restarting failed container +``` + +## Cause +This issue could be due to a number of reasons. +* It may be that the worker nodes need permissions to read S3 bucket where the secrets are located* The ```securityContext``` is missing information on ```fsGroup```* ``````Tokens are mounted to ```/var/run/secrets/eks.amazonaws.com/serviceaccount/toke```n but they are missing the special permissions and the pod could be running as a different user (by default, a ```nobody``` account) + diff --git a/kb/armory/General/dynamics-accounts-with-github-cannot-use-token.md b/kb/armory/General/dynamics-accounts-with-github-cannot-use-token.md new file mode 100644 index 0000000000..d1ab5aa1c0 --- /dev/null +++ b/kb/armory/General/dynamics-accounts-with-github-cannot-use-token.md @@ -0,0 +1,10 @@ +--- +title: Dynamics Accounts with GitHub Cannot Use Token +--- + +## Issue +Dynamic Accounts can be set to use [GitHub.com](http://github.com/) to store credentials, but this requires usage of having account information stored as clear text.  It cannot be stored as encrypted information, and cannot use GitHub's token system either. + +## Cause +Dynamic Accounts use Spring Cloud Config to manage account access.  Spring Cloud Config for any configuration does not support S3 encryption for secrets, since the secret decryption happens on a different stage. It also cannot access GitHub via Tokens.  So for GitHub configurations, credentials to access the GitHub Account can only be stored as clear text. + diff --git a/kb/armory/General/echo-sends-duplicate-triggers.md b/kb/armory/General/echo-sends-duplicate-triggers.md new file mode 100644 index 0000000000..d32e5daa59 --- /dev/null +++ b/kb/armory/General/echo-sends-duplicate-triggers.md @@ -0,0 +1,11 @@ +--- +title: Echo Sends Duplicate Triggers +--- + +## Issue +Echo may send out multiple triggers. This can lead to doubling the size of logs, or doubling alerts and can cause unnecessary slow downs as a result +Applies to:Kubernetes V2Spinnaker with HA mode + +## Cause +Echo with replicas of more than 1 will have issues with duplicate triggers. Especially for CRON. This will happen with multiple replicas of Echo that haven't been configured with HA Mode to split them. This can cause further issues if other microservices aren't set up for HA mode as well. + diff --git a/kb/armory/General/ecs--supporting-ephemeralstorage-in-fargate.md b/kb/armory/General/ecs--supporting-ephemeralstorage-in-fargate.md new file mode 100644 index 0000000000..0fd6e50e7b --- /dev/null +++ b/kb/armory/General/ecs--supporting-ephemeralstorage-in-fargate.md @@ -0,0 +1,20 @@ +--- +title: ECS- Supporting EphemeralStorage in Fargate +--- + +## Issue +In some versions of Spinnaker and Armory Enterprise, [AWS Ephemeral Storage for Fargate](https://aws.amazon.com/about-aws/whats-new/2021/04/amazon-ecs-aws-fargate-configure-size-ephemeral-storage-tasks/) is not available.  When a user attempts to define EphemeralStorage in ECS from an artifact definition, such as with the following definition: +``` + "ephemeralStorage": { + + "sizeInGiB": 100 + + }, +``` +  +The storage of 100GB does not get assigned. + +## Cause +This issue is as a result of [https://github.com/spinnaker/kork/pull/862](https://github.com/spinnaker/kork/pull/862) and the related [dependency.](https://github.com/spinnaker/spinnaker/issues/6417) +  + diff --git "a/kb/armory/General/ecs-deployment-errors-with-\"all-server-group-names-for-cluster....are-taken\".md" "b/kb/armory/General/ecs-deployment-errors-with-\"all-server-group-names-for-cluster....are-taken\".md" new file mode 100644 index 0000000000..1a4bab7168 --- /dev/null +++ "b/kb/armory/General/ecs-deployment-errors-with-\"all-server-group-names-for-cluster....are-taken\".md" @@ -0,0 +1,15 @@ +--- +title: ECS Deployment errors with "All server group names for cluster....are taken" +--- + +## Issue +A transient issue occurs with the ECS deployment stage, across multiple pipelines within the same application.  This issue can occur even across different server group names. The following error is observed:  +`All server group names for cluster .....are taken.` +`All server group names for cluster ...are taken.` +The UI shows the following: + +## Cause +The issue is as a result of hitting a limit on the number of possible server group names in Spinnaker (e.g. there may be over 1000 versions of a specific serverGroup), and the ```NameBuilder``` that is capped at ```1000```.  +[https://github.com/spinnaker/clouddriver/blob/01f415f318a15970729b4415167e64eafeca9593/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/helpers/AbstractServerGroupNameResolver.groovy#L31](https://github.com/spinnaker/clouddriver/blob/01f415f318a15970729b4415167e64eafeca9593/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/helpers/AbstractServerGroupNameResolver.groovy#L31) +It is very rare to hit this limit of 1000 ECS server groups within a cluster, and it does not impact every v2.24 environment. + diff --git a/kb/armory/General/ecs-deployment-fails-when-using-external-launch-type-task.md b/kb/armory/General/ecs-deployment-fails-when-using-external-launch-type-task.md new file mode 100644 index 0000000000..3b17c35d86 --- /dev/null +++ b/kb/armory/General/ecs-deployment-fails-when-using-external-launch-type-task.md @@ -0,0 +1,50 @@ +--- +title: ECS Deployment fails when using EXTERNAL launch type Task +--- + +## Issue +Users automating Amazon Elastic Container Services (ECS) cannot deploy and may see errors from Clouddriver. The logs may indicate caching cycle failures when a Service in the ECS account with deployment type EXTERNAL is defined with ```taskSets```, and Spinnaker retrieves an empty ```TaskDefinitionArn```, which doesn’t ignore and tries to describe it continuously and causes this failure: +``` 'com.amazonaws.services.ecs.model.InvalidParameterException: Task Definition can not be blank. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: fcf39ec3-0e98-48f3-bb40-8b6ab19a7be5; Proxy: null)' ``` +  + +## Cause +There's a reported** issue** in OSS Spinnaker where the ECS provider causes the caching agent to fail when an ACTIVE service is defined with an EXTERNAL launch type. The error occurs when calling a task definition** **because there is no task definition** **but uses the taskSet field.  +This is the sample service definition: +``` +{ + "services": [ + { + "serviceArn": "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:service/test-cluster/test", + "serviceName": "test", + "clusterArn": "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:cluster/test-cluster", + "loadBalancers": [], + "serviceRegistries": [], + "status": "ACTIVE", + "desiredCount": 1, + "runningCount": 0, + "pendingCount": 0, + "launchType": "EC2", + "deploymentConfiguration": { + "maximumPercent": 200, + "minimumHealthyPercent": 100 + }, + "taskSets": [], + "deployments": [], + "roleArn": "arn:aws:iam::xxxxxxxxxxxx:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS", + "events": [], + "createdAt": 1640098677.085, + "placementConstraints": [], + "placementStrategy": [], + "schedulingStrategy": "REPLICA", + "deploymentController": { + "type": "EXTERNAL" + }, + "createdBy": "arn:aws:iam::xxxxxxxxxxxx:role/admin", + "enableECSManagedTags": false, + "propagateTags": "NONE", + "enableExecuteCommand": false + } + ], + "failures": [] +} +``` diff --git a/kb/armory/General/ecs-deployment-failures.md b/kb/armory/General/ecs-deployment-failures.md new file mode 100644 index 0000000000..dfc213dd29 --- /dev/null +++ b/kb/armory/General/ecs-deployment-failures.md @@ -0,0 +1,16 @@ +--- +title: ECS Deployment Failures +--- + +## Issue +Users may notice failures when deploying tasks with tags to ECS deployment targets. This behavior began on 2023/05/25 and impacts all versions of CDSH. The error on the failed pipeline would look similar to the error in the image below + +**Update Jun 1, 2023:** Hotfixes and versions have been released to address this issue + +## Cause +The failures are due to API errors with the AWS API. The issue occurs for those pipelines that have an ECS deploy stage consisting tags as shown below + +The API that accepts the tags has been causing the issue. +  +  + diff --git "a/kb/armory/General/editing-application-permissions-yields-an-error-\"cannot-read-property-'label'-of-undefined\".md" "b/kb/armory/General/editing-application-permissions-yields-an-error-\"cannot-read-property-'label'-of-undefined\".md" new file mode 100644 index 0000000000..ecfe6bd8ef --- /dev/null +++ "b/kb/armory/General/editing-application-permissions-yields-an-error-\"cannot-read-property-'label'-of-undefined\".md" @@ -0,0 +1,23 @@ +--- +title: Editing application permissions yields an error "Cannot read property 'label' of undefined" +--- + +## Issue +When trying to edit the application config and provide permissions to user by going to the application -> selecting config -> edit the application attribute -> permissions -> add user. +However, the user permission cannot be added and the following error message is seen: +Cannot read property 'label' of undefined +TypeError: Cannot read property 'label' of undefined + at https://... + at Array.map () + at PermissionsConfigurer_PermissionsConfigurer.render (https://......) + at gi (https://.....) + at fi (https://.....) + at Rj (https://.....) + at Qj (https://....) + at Kj (https://....) + at yj (https://......) +The UI shows the following: "Oh dear, something has gone wrong" + +## Cause +This error is related to the following issue:[https://github.com/spinnaker/spinnaker/issues/5986#issuecomment-686650114](https://github.com/spinnaker/spinnaker/issues/5986#issuecomment-686650114)This issue is caused by a group is set up twice with different permissions.  + diff --git a/kb/armory/General/emoji-in-redis-database-error-when-migrating-from-redis-to-mysql.md b/kb/armory/General/emoji-in-redis-database-error-when-migrating-from-redis-to-mysql.md new file mode 100644 index 0000000000..760c30c965 --- /dev/null +++ b/kb/armory/General/emoji-in-redis-database-error-when-migrating-from-redis-to-mysql.md @@ -0,0 +1,14 @@ +--- +title: Emoji in REDIS Database error when Migrating from REDIS to MySQL +--- + +## Issue +Error returned when migrating from REDIS to MySQL: +org.springframework.jdbc.UncategorizedSQLException: jOOQ; uncategorized SQLException for SQL [insert into pipeline_stages (id, legacy_id, execution_id, status, updated_at, body) values (?, ?, ?, ?, ?, ?) on duplicate key update status = ?, updated_at = ?, body = ? -- agentClass: PipelineMigrationAgent]; SQL state [HY000]; error code [1366]; +Incorrect string value: '\x\x\x\x...' + +As an example, the incorrect string value will be an emoji icon, e.g. ```\xF0\x9F\x94\x90mT...```  which is a lock emoji + +## Cause +Spinnaker doesn't parse the information properly, because it wasn't declared what kind of encoding it should process the data with + diff --git a/kb/armory/General/enable-armory-agent-for-kubernetes-to-only-discover-objects-that-are-discovered-by-spinnaker-or-armory-agent.md b/kb/armory/General/enable-armory-agent-for-kubernetes-to-only-discover-objects-that-are-discovered-by-spinnaker-or-armory-agent.md new file mode 100644 index 0000000000..14c1121351 --- /dev/null +++ b/kb/armory/General/enable-armory-agent-for-kubernetes-to-only-discover-objects-that-are-discovered-by-spinnaker-or-armory-agent.md @@ -0,0 +1,25 @@ +--- +title: Enable Armory Agent for Kubernetes to only discover objects that are discovered by Spinnaker (or) Armory Agent +--- + +## Introduction +Unlike in a standard Spinnaker install without Armory Agent, customers who have installed Armory Agent for Kubernetes in their environment will find that the Agent discovers ***all the objects in the target cluster that the Kubernetes account has access to***. +This default behavior may create default applications in Spinnaker for every deployment on the cluster, ***even if these deployments were not deployed using Spinnaker***.  Spinnaker admins can restrict the scope of the discovery engine by ensuring that Armory Agent and Spinnaker only discover the objects that Spinnaker deploys.  + +## Prerequisites +Armory Enterprise Spinnaker with Armory Agent for Kubernetes enabled + +## Instructions +When a Kubernetes object is created through Spinnaker,  the Spinnaker annotations get added to them as mentioned under [https://spinnaker.io/docs/reference/providers/kubernetes-v2/#reserved-annotations](https://spinnaker.io/docs/reference/providers/kubernetes-v2/#reserved-annotations). +The below example configuration should be defined in the Agent Plugin values to restrict the Armory Agent to discover only those objects deployed through Spinnaker: +```` + kubesvc: + cluster: kubernetes + cache: + operationWaitMs: 60000 + runtime: + defaults: + onlySpinnakerManaged: true +```` +Admins looking for a similar function for individually defined Clouddriver accounts may refer to [https://docs.armory.io/armory-enterprise/armory-admin/kubernetes-account-add/#add-the-kubeconfig-and-cloud-provider-to-spinnaker](https://docs.armory.io/armory-enterprise/armory-admin/kubernetes-account-add/#add-the-kubeconfig-and-cloud-provider-to-spinnaker) and set``` onlySpinnakerManaged: true``` on the specific Kubernetes accounts in Spinnaker. + diff --git a/kb/armory/General/enable-aws-cloudformation.md b/kb/armory/General/enable-aws-cloudformation.md new file mode 100644 index 0000000000..71f3439146 --- /dev/null +++ b/kb/armory/General/enable-aws-cloudformation.md @@ -0,0 +1,115 @@ +--- +title: Enable AWS CloudFormation +--- + +## Introduction +This document will show you how to enable the CloudFormation stage. This document also assumes that you have configured an AWS account. If you need more instructions on configuring your AWS account please refer to either [Deploying to AWS from Spinnaker (using IAM instance roles)](https://docs.armory.io/spinnaker-install-admin-guides/add-aws-account-iam/) or [Deploying to AWS from Spinnaker (using IAM credentials)](https://docs.armory.io/spinnaker-install-admin-guides/add-aws-account/) + +## Prerequisites +N/A + +## Instructions +First, we need to enable CloudDriver by adding a ```clouddriver-local.yml``` file to your hal config profiles directory +e.g. ```.hal/default/profiles/clouddriver-local.yml```. +``` +aws: + features: + cloudFormation: + enabled: true +``` +### Halconfig +Make sure to enable the aws provider, (if you don’t have the provider enable please refer to the background section to find the links). +Also you need to have setted up at least one region. To add a region you can use this instruction: +``` +export AWS_ACCOUNT_NAME=aws-1 +hal config provider aws account edit ${AWS_ACCOUNT_NAME} --regions us-east-1,us-west-2 +``` + +This is an example how should looks your hal config. +``` +aws: + enabled: true + accounts: + - name: aws-1 + requiredGroupMembership: [] + providerVersion: V1 + permissions: {} + accountId: '123456789123' + regions: + - name: us-east-1 + - name: us-west-2 + assumeRole: role/SpinnakerManagedRole + primaryAccount: aws-1 + bakeryDefaults: + templateFile: aws-ebs-shared.json + baseImages: [] + awsAssociatePublicIpAddress: true + defaultVirtualizationType: hvm + defaultKeyPairTemplate: '-keypair' + defaultRegions: + - name: us-west-2 + defaults: + iamRole: BaseIAMRole +``` +You can check your configuration by running ```hal deploy apply```. + +### CloudFormation Permissions in Policy +Another thing to check is in the AWS Console the ```cloudformation:*``` permission in the ```Action``` section inside your **Policy**, like in the following example. +``` +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": "sts:AssumeRole", + "Resource": "arn:aws:iam::123456789123:role/SpinnakerManagedRole" + }, + { + "Effect": "Allow", + "Action": [ + "ec2:*", + "cloudformation:*" + ], + "Resource": "*" + } + ] +} +``` +*for more details please refer to [Creating a Managing Account IAM Policy in your primary AWS Account](https://docs.armory.io/spinnaker-install-admin-guides/add-aws-account/#iam-user-part-3-creating-a-managing-account-iam-policy-in-your-primary-aws-account)* Part 3. + +### Creating your Pipeline +After you’ve deployed your changes, you should now be able to see the new stages when configuring your pipeline. Simply select the stage, and provide the values: example below: +**Note:** *The “Stack name” is a unique value so if you run a second time you will need to change it.* +In this example we are going to deploy a simple s3 bucket with the follow template, (Copy and paste in the **Text Source** section) +``` +{ + "AWSTemplateFormatVersion": "2010-09-09", + "Outputs": { + "BucketName": { + "Description": "S3 Bucket", + "Value": { + "Ref": "S3Bucket" + } + } + }, + "Resources": { + "S3Bucket": { + "Properties": { + "BucketName": "cf-example-s3" + }, + "Type": "AWS::S3::Bucket" + } + } +} +``` +**Note:** *You can test changing the Cloud Formation JSON Template “Resources” > “S3Bucket” > “Properties” > “BucketName” property “cf-example-s3” by another unique name.* + +### Run the Pipeline +Now you can run the pipeline and this will be conclude **SUCCEEDED**  +At the meanwhile you can see the execution details if you go to your *AWS CloudFormation console*  +**Note:** *You can execute the pipeline by generating an embedded-artifact. Deploy (CloudFormation Stack) > Source: Artifact > Expected Artifact: Artifact from execution context > Account: Embedded-artifact > Contents: “Input the CloudFormation JSON/YAML Template”.* + +### Validate +In this particular example we created an S3 bucket so in order to validate you need to go to your aws console and check in the S3 bucket section. + + diff --git a/kb/armory/General/enable-basic-form-authentication-for-spinnaker-via-halyard.md b/kb/armory/General/enable-basic-form-authentication-for-spinnaker-via-halyard.md new file mode 100644 index 0000000000..0b2d868fcb --- /dev/null +++ b/kb/armory/General/enable-basic-form-authentication-for-spinnaker-via-halyard.md @@ -0,0 +1,21 @@ +--- +title: Enable Basic Form Authentication for Spinnaker via Halyard +--- + + +If you want to enable Simple Auth for Spinnaker using Halyard: Create/update the ```.hal//profiles/gate-local.yml``` file, with these contents: +``` +security.basicform.enabled: true +spring: + security: + user: + name: + password: +``` + +And then create/update the ```.hal//profiles/settings-local.js```, with these contents +```window.spinnakerSettings.authEnabled = true;``` + +Then, apply your change with ```hal deploy apply```. +To access Spinnaker, you will be prompted with a simple form to enter the username and password. + diff --git a/kb/armory/General/enable-detailed-logging---debugging-mode-in-spinnaker-environment-service.md b/kb/armory/General/enable-detailed-logging---debugging-mode-in-spinnaker-environment-service.md new file mode 100644 index 0000000000..a87c36949e --- /dev/null +++ b/kb/armory/General/enable-detailed-logging---debugging-mode-in-spinnaker-environment-service.md @@ -0,0 +1,21 @@ +--- +title: Enable Detailed Logging / Debugging Mode in Spinnaker Environment Service +--- + +## Introduction +The environment is showing errors, but want to enable more detailed logging on a Spinnaker service to be able to troubleshoot further.  Below is an explanation of how to add that function. + +## Prerequisites +N/A + + +## Instructions +Detailed logging can be enabled by simply adding the following code to the appropriate service YAML file. (```-local.yml``` in your Halyard profiles folder.)   If the file does not exist, feel free to create one.  An example of enable debug mode on ```Echo```, the below code should be added in the ```echo-local.yml``` file.   +``` +logging: + level: + com.netflix.spinnaker.echo: DEBUG +``` +For any other service, you will need to replace the ```echo``` portion of ```com.netflix.spinnaker.**echo**``` in the code with the appropriate service.**It is highly recommended that you only do this for limited amounts of time** as the amount of logs generated will be exponentially higher, and has the potential to cause performance issues or crash your systems. +In Operator, you would use the SpinnakerService CRD and make the adjustments in the [appropriate profiles section as per the documentation found on our docs site](https://docs.armory.io/docs/installation/operator-reference/operator-config/#spinnakerservice-crd) + diff --git a/kb/armory/General/enable-two-pipelines-to-run-as-mutually-exclusive.md b/kb/armory/General/enable-two-pipelines-to-run-as-mutually-exclusive.md new file mode 100644 index 0000000000..642fe32639 --- /dev/null +++ b/kb/armory/General/enable-two-pipelines-to-run-as-mutually-exclusive.md @@ -0,0 +1,41 @@ +--- +title: Enable two pipelines to run as Mutually Exclusive +--- + +## Introduction +If two pipelines need to run as mutually exclusive, there is currently no way to achieve this in Armory Spinnaker or OSS Spinnaker without creating some kind of Custom Stage. This article will cover a workaround that can be used in the event a Custom Stage could not be made. +Please note that mutually exclusive example in this article will refer to the below scenario. +Given two pipelines, Pipeline A and Pipeline B +* If pipeline A is running, pipeline B can’t start.* Similarly, if pipeline B is running, pipeline A can’t start. + +  +There are potentially better solutions for smaller pipelines as well as caveats to taking this approach that should be considered before following this article +* If the pipelines are small/simple in nature, the Conditional Expressions can be set for all stages in order to stop them from running, thereby making for easier configuration. The method described in this article was selected instead to provide a solution in the case there are larger pipeline definitions or in the event that the pipeline's deploy stages change frequently.* There are no loop stages in Spinnaker and this article covers how to artificially create one. Please keep in mind that this can potentially cause issues such as infinite loops.* We are using two check preconditions stages since there are no options to choose a path upon failure. + +## Prerequisites +N/A + +## Instructions +There are two potential solutions to this scenaril +***Solution 1:***  Merge Pipeline A and Pipeline B into one pipeline and leverage the Check Preconditions Stage in order to direct the pipeline down the appropriate path. +***Solution 2:*** Use webhooks to re-trigger pipelines until there are no pipelines running for Pipeline A or Pipeline B. +  +Solution 1 +* Create a pipelineCreate two Check Pre-Conditions Stages +* This should check whether or not to deploy to PROD or DEV. The specific condition will vary depending on what is being passed. In this example, we are using a parameter that the user inputs before executing the pipeline.* For more information on how to define these conditions, please refer to the [OSS SpEL expression documentation](https://spinnaker.io/docs/guides/user/pipeline/expressions/) +* Input the condition into the Check-Preconditions Stage and make sure the ```Fail Pipeline``` checkbox is unchecked* Next, ensure that the entire pipeline does not fail when these stages fail and instead only discontinue their respective pipeline branches. This can be achieved by setting the Execution Options for both the Check-Preconditions Stages.* Last, setup the deploy stages for PROD and DEV. +  +  +Solution 2 +* Create two pipelines (Both pipelines will have the same configuration and only differ slightly in the Webhook configurations and Deploy Stages) +* Next, create a Webhook stage to the ```getLatestExecutionsByConfigIdsUsingGET``` Spinnaker API[https://spinnaker.io/docs/reference/api/docs.html#api-Executionscontroller-getLatestExecutionsByConfigIdsUsingGET](https://spinnaker.io/docs/reference/api/docs.html#api-Executionscontroller-getLatestExecutionsByConfigIdsUsingGET)* Set the limit to ```1``` and the status to ```RUNNING``` +* Make sure to capture the Pipeline IDs of both pipelines and input them into the Webhook URL. (Pipeline A will do a webhook call to Pipeline B's Pipeline ID and vice versa) +* When all is done, the stage should look similar to below. +* Next, create two Check Preconditions Stages just like in the first solution. For the conditions, we want to check whether or not the Webhook returned any running pipelines for either Pipeline A or Pipeline B and if there are any running pipelines for either. To do this, please refer to the SpEL expressions doc in the More Info section of the article. In this example, we will be using the below conditions to check if there are running pipelines or if there are no running pipelines. +```${execution.stages[0].context.webhook.body.toString().contains("RUNNING")}``````${!execution.stages[0].context.webhook.body.toString().contains("RUNNING")}``` +* Make sure to unset the ```Fail Pipeline``` checkbox as well as configure both Check Preconditions Stages to only fail the branch upon failure. +* From here, create a Wait Stage after the Check PreConditions Stage. (Please note that this Wait Stage should be configured after the Check PreConditions Stage that checks if there IS a running pipeline) This wait stage will act as a timeout function that will wait a set amount of time before retrying to check if there are any running pipelines. +* Next, Configure another Webhook after the Wait Stage to the ```invokePipelineConfigUsingPOST1``` Spinnaker API and input the current pipeline's ID into the Webhook URL.[https://spinnaker.io/docs/reference/api/docs.html#api-Pipelinecontroller-invokePipelineConfigUsingPOST1](https://spinnaker.io/docs/reference/api/docs.html#api-Pipelinecontroller-invokePipelineConfigUsingPOST1)This will re-trigger the pipeline in the event that a running pipeline was found for either Pipeline A or Pipeline B. +* Last, configure the deploy stage and set that stage to run after the Check Preconditions Stage that has the condition to check if the Webhook did NOT find any running pipelines. +* When this is setup properly, the pipelines should look similar to the example below. + diff --git a/kb/armory/General/enabling-caching-on-git-repos-and-clearing-clouddriver-caching-to-resolve-caching-issues.md b/kb/armory/General/enabling-caching-on-git-repos-and-clearing-clouddriver-caching-to-resolve-caching-issues.md new file mode 100644 index 0000000000..733ac3b376 --- /dev/null +++ b/kb/armory/General/enabling-caching-on-git-repos-and-clearing-clouddriver-caching-to-resolve-caching-issues.md @@ -0,0 +1,35 @@ +--- +title: Enabling Caching on Git Repos and Clearing Clouddriver Caching to Resolve Caching Issues +--- + +## Introduction +In Spinnaker +From Spinnaker 1.26, support was added for "caching" git repositories.  This allows the behavior only to clone the repo the first time it is needed.  In subsequent updates, Clouddriver completes a ```git pull``` only download updates. This is expected to dramatically improve execution times and reliability when working with big repositories. +  +  +  +  + +## Prerequisites +A valid Github account connection to the Spinnaker environment + +## Instructions +This is an opt-in feature that is disabled by default.  To enable it, administrators need to add the following change to the Clouddriver profile configuration: +``` +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker +spec: + spinnakerConfig: + profiles: + clouddriver: + artifacts: + gitrepo: + clone-retention-max-bytes: 5221225472 + clone-retention-minutes: -1 +```  +Please note: +A Git error can happen when the local repository (e.g., the cloned version in Clouddriver) has an entirely different history from the remote one. +Sometimes the repo/branch deployed and then pushed in the remote has a commit that involves changing the history of the commits etc.  In such a case, an error can occur caused by an irreconcilable change.  A Clouddriver restart will fix the issue since all the local repositories (in the Clouddriver pods) will be purged from the local storage. They will undergo an initial caching routine again upon restart. + diff --git a/kb/armory/General/enabling-the-managed-pipeline-templates-ui.md b/kb/armory/General/enabling-the-managed-pipeline-templates-ui.md new file mode 100644 index 0000000000..bbf9e195f0 --- /dev/null +++ b/kb/armory/General/enabling-the-managed-pipeline-templates-ui.md @@ -0,0 +1,33 @@ +--- +title: Enabling the Managed Pipeline Templates UI +--- + +## Introduction +Armory Spinnaker 2.19+ contains the latest version of Managed Pipeline Templates v2 (MPTv2), which is the default pipeline templating solution offered in OSS Spinnaker.Armory recommends using Armory’s Pipeline as Code feature instead of MPTv2 because it offers the following benefits: + +* Integration with GitHub, GitLab and BitBucket enabling teams to store pipelines with application code* Templates and access to the templates can be stored and managed separately from pipelines* The ability to compose complex templates and pipelines from modules +Note that Armory’s Pipeline as Code and the open source Managed Pipeline Templates are not integrated and do not work together. + + +## Prerequisites +N/A + +## Instructions +By default, the Managed Pipeline Template UI is disabled in Armory Spinnaker 2.19.5. Leaving the UI disabled maintains the same experience users had with Armory Spinnaker 2.18.x (OSS 1.18.x).If users want to enable the Managed Pipeline Templates UI, add the following to the Operator config: + +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker +spec: + spinnakerConfig: + config: + features: + pipelineTemplates: true + profiles: + deck: + settings-local.js: | + window.spinnakerSettings.feature.pipelineTemplates = true; + window.spinnakerSettings.feature.managedPipelineTemplatesV2UI = true; + + diff --git a/kb/armory/General/error---credential-not-found-aws.md b/kb/armory/General/error---credential-not-found-aws.md new file mode 100644 index 0000000000..be2f7480b3 --- /dev/null +++ b/kb/armory/General/error---credential-not-found-aws.md @@ -0,0 +1,11 @@ +--- +title: Error - Credential not found (AWS) +--- + +## Issue +When deploying to an AWS account you receive the following error: +```credential not found (name: default, accounts: [accountA, accountB, accountC...])``` + +## Cause +This can occur for installations that are deploying into AWS but not using Rosco to bake images. + diff --git a/kb/armory/General/error--you-cannot-delete-this-application-because-it-has-server-groups.md b/kb/armory/General/error--you-cannot-delete-this-application-because-it-has-server-groups.md new file mode 100644 index 0000000000..930ea6a11e --- /dev/null +++ b/kb/armory/General/error--you-cannot-delete-this-application-because-it-has-server-groups.md @@ -0,0 +1,14 @@ +--- +title: Error- You cannot delete this application because it has server groups +--- + +## Issue +When attempting to delete an application using the Spinnaker Console UI, the following error is returned: +```Delete Application - You cannot delete this application because it has server groups``` +The UI shows the following: + +After a while, either the screen logs out or the following error is seen: + +## Cause +This error is seen because there are Server Groups associated with the application, and they need to be removed. + diff --git a/kb/armory/General/error-calling-module-error-rendering-imported-module-pipeline-module.json-invalid-character.md b/kb/armory/General/error-calling-module-error-rendering-imported-module-pipeline-module.json-invalid-character.md new file mode 100644 index 0000000000..fdb3e79140 --- /dev/null +++ b/kb/armory/General/error-calling-module-error-rendering-imported-module-pipeline-module.json-invalid-character.md @@ -0,0 +1,39 @@ +--- +title: Error calling module- error rendering imported module 'pipeline/....module.json'- invalid character +--- + +## Issue +The Dinghy webhook can return the following syntax on Pull requests: +(template: dinghy-render:29:7: executing “dinghy-render” at >module “pipeline/…module.json”>: error calling module: error rendering imported module ‘pipeline/…module.json’:invalid character after object key:value pair) + +  +The logs will show the following: + +``` +time="2021-09-01T17:36:13Z" level=info msg="Found no custom configuration for repo: test-container, proceeding with master"... +time="2021-09-01T17:36:13Z" level=info msg="Skipping Spinnaker pipeline update because this branch (testing-error) is not master. Proceeding as validation." +time="2021-09-01T17:36:13Z" level=info msg="Processing request for branch: testing-error" +time="2021-09-01T17:36:13Z" level=info msg="Processing Push" +time="2021-09-01T17:36:13Z" level=info msg="Processing Push" +time="2021-09-01T17:36:13Z" level=info msg="Dinghyfile found in commit for repo test-container" +time="2021-09-01T17:36:14Z" level=error msg="Failed to parse module:.. +time="2021-09-01T17:36:14Z" level=error msg="error rendering imported module... +time="2021-09-01T17:36:14Z" level=error msg="Failed to execute buffer:\n {\n \"application\": \"test-container\",\n \"spec\": {\n \"notifications\... +time="2021-09-01T17:36:14Z" level=error msg="Failed to parse dinghyfile dinghyfile: template: dinghy-render:.....executing \"dinghy-render\" at : error calling module: error rendering imported module 'pipeline/.......module.json': invalid character 'a' after object key:value pair" +However Dinghyfile and module syntax is validated with the [ARM-CLI tool](https://docs.armory.io/docs/spinnaker-user-guides/dinghy-arm-cli/) to be valid.  For example: +>arm dinghy render dinghyfile –modules ../dinghy-templates >> /dev/null +INFO [2021-09-02 08:40:17] [Checking dinghyfile] +INFO [2021-09-02 08:40:17] [Parsing dinghyfile] +INFO [2021-09-02 08:40:17] [Parsed dinghyfile] +INFO [2021-09-02 08:40:17] [Parsing final dinghyfile to struct for validation] +INFO [2021-09-02 08:40:17] [Output: +] +INFO [2021-09-02 08:40:17] [Final dinghyfile is a valid JSON Object.] +``` + +## Cause +This error is as a result of the ```strict validator``` that has been introduced in v.2.26.2+: +[https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-26-2/#strict-json-validation](https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-26-2/#strict-json-validation) +  + + diff --git a/kb/armory/General/error-displayed-on-the-ui--exception-resolve-target-manifest.md b/kb/armory/General/error-displayed-on-the-ui--exception-resolve-target-manifest.md new file mode 100644 index 0000000000..0f466a9df3 --- /dev/null +++ b/kb/armory/General/error-displayed-on-the-ui--exception-resolve-target-manifest.md @@ -0,0 +1,16 @@ +--- +title: Error displayed on the UI- Exception (Resolve Target Manifest) +--- + +## Issue +In deployments and replicasets utilizing the delete and scale manifest stages, when the oldest or the newest version is attempted to be deleted, the following error is observed: +Exception (Resolve Target Manifest) +No manifests matching (account: spinnaker, location: default, kind: deployment, app testartifactrepo, cluster: deployment frontend, criteria: oldest) found +The UI shows the following: + + + +## Cause +The error observed is due to bug affecting all dynamic actions with Kubernetes objects. This is related to strategy selected during deployment, and the value set for ```max versions to keep```. If the chosen strategy disables old replicas when the new one is ready, then it is not required that the replicas be manually disabled or deleted. +`````` + diff --git a/kb/armory/General/error-when-deploying-manifest-kind-csidriver-with-armory-agent.md b/kb/armory/General/error-when-deploying-manifest-kind-csidriver-with-armory-agent.md new file mode 100644 index 0000000000..b09492d3b2 --- /dev/null +++ b/kb/armory/General/error-when-deploying-manifest-kind-csidriver-with-armory-agent.md @@ -0,0 +1,23 @@ +--- +title: Error when deploying manifest kind CSIDriver with Armory Agent +--- + +## Issue +When Armory Agent is utilized in ***Service Mode*** on (Armory version: 2.26.0, Armory agent plugin version: 0.9.16 and Armory agent version: 0.6.0), it is observed that with a pipeline that deploys a manifest of kind ```CSIDriver```, the stage never finishes. +The following errors are seen in Clouddriver: + +```12021-09-13 18:42:50.444 WARN 1 --- [.0-7002-exec-11] c.n.s.c.k.c.ManifestController : Failed to read manifest io.armory.kubesvc.services.ops.executor.KubesvcRuntimeException: Error occurred when Get CSIDriver/ebs.... default: (404: unknown kind CSIDriver in group for account ........eks)``` + +  +During the agent startup, CSIDriver is can be found under two agent groups, for example: ([storage.k8s.io/v1](http://storage.k8s.io/v1) and [storage.k8s.io/v1beta1)](http://storage.k8s.io/v1beta1) +time="2021-09-13T17:03:23Z" level=info msg="Kind: CSIDriver, Group: storage.k8s.io/v1, Resource=csidrivers" .....eks instanceId=..... +time="2021-09-13T17:03:23Z" level=info msg="Kind: CSIDriver, Group: storage.k8s.io/v1beta1, Resource=csidrivers" account=........-eks instanceId=...... +  +However, the error seen during the deployment does not have any group tagged to it +time="2021-09-13T18:43:57Z" level=error msg="Error for operation ........with type Get and status 404: unknown kind CSIDriver in group for account ......-eks" error="" instanceId=..... + +In the situation, the manifest for CSIDriver that is being deployed has the correct group and the pipeline does not fail, but rather remains stuck until it reaches timeout. + +## Cause +The error is due to Clouddriver not sending the ```group id``` in the response.  + diff --git "a/kb/armory/General/errors-don't-propagate-from-child-to-parent-pipelines-if-they-occur-in-stages-configured-to-\"ignore-the-failure\".md" "b/kb/armory/General/errors-don't-propagate-from-child-to-parent-pipelines-if-they-occur-in-stages-configured-to-\"ignore-the-failure\".md" new file mode 100644 index 0000000000..7a1b2dab9c --- /dev/null +++ "b/kb/armory/General/errors-don't-propagate-from-child-to-parent-pipelines-if-they-occur-in-stages-configured-to-\"ignore-the-failure\".md" @@ -0,0 +1,11 @@ +--- +title: Errors don't propagate from child to parent pipelines if they occur in stages configured to "ignore the failure" +--- + +## Issue +Error messages don't propagate from child to parent pipelines if they occur in stages configured to ```ignore the failure```.  In these instances, the parent pipeline will not have enough information to appropriately handle the child pipeline's error.   + +## Cause +This happens when a stage in the child pipeline has been set to ```ignore the failure``` if the stage fails (under ```Execution Options```).  If an error occurs in this stage, Spinnaker still considers the pipeline successful because the failure was ignored due to the setting, and even though the stage execution will display an error, the pipeline execution is not considered as "having failed" and will not signal as such.  +*note* Certain types of error will cause the entire pipeline to fail, even if the stage is set to ```ignore the failure```.  In the case of these "grand failures," Parent pipelines may fail. + diff --git a/kb/armory/General/errors-in-deploying-dinghy-while-using-aws-elastic-cache-with-encryption-in-transit.md b/kb/armory/General/errors-in-deploying-dinghy-while-using-aws-elastic-cache-with-encryption-in-transit.md new file mode 100644 index 0000000000..45217f12f7 --- /dev/null +++ b/kb/armory/General/errors-in-deploying-dinghy-while-using-aws-elastic-cache-with-encryption-in-transit.md @@ -0,0 +1,11 @@ +--- +title: Errors in deploying Dinghy while using AWS elastic cache with encryption in transit +--- + +## Issue +When attempting to deploy Dinghy while using AWS elastic cache with ***encryption in transit***, the following error is seen +```level=fatal msg="Redis Server at rediss:// could not be contacted: dial tcp: address rediss://: too many colons in address``` + +## Cause +Currently, encryption over transit is not supported.  Specifically, SSL secure connection (rediss://) is not supported, and as such the redis:// connection may be utilized. + diff --git a/kb/armory/General/errors-initializing-and-starting-up-policy-engine-configuration-hints,-orca-clouddriver-front50.md b/kb/armory/General/errors-initializing-and-starting-up-policy-engine-configuration-hints,-orca-clouddriver-front50.md new file mode 100644 index 0000000000..103757820a --- /dev/null +++ b/kb/armory/General/errors-initializing-and-starting-up-policy-engine-configuration-hints,-orca-clouddriver-front50.md @@ -0,0 +1,16 @@ +--- +title: Errors Initializing and Starting Up Policy Engine (Configuration Hints, Orca/CloudDriver/Front50) +--- + +## Issue +Policy Engine has been implemented with settings in Operator/Halyard, but it is not starting up and is causing errors in the initialization.  Example may look like something below: +``` +front50 Aug 28, 2020, 5:36:40 PM 2020-08-29 00:36:40.909 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed +front50 Aug 28, 2020, 5:36:40 PM 2020-08-29 00:36:40.210 INFO 1 --- [ main] o.s.c.a.ConfigurationClassPostProcessor : Cannot enhance @Configuration bean definition 'com.netflix.spinnaker.kork.PlatformComponents' since its singleton instance has been created too early. The typical cause is a non-static @Bean method with a BeanDefinitionRegistryPostProcessor return type: Consider declaring such methods as 'static'. +front50 Aug 28, 2020, 5:36:40 PM 2020-08-29 00:36:40.209 INFO 1 --- [ main] o.s.c.a.ConfigurationClassPostProcessor : Cannot enhance @Configuration bean definition 'pluginsAutoConfiguration' since its singleton instance has been created too early. The typical cause is a non-static @Bean method with a BeanDefinitionRegistryPostProcessor return type: Consider declaring such methods as 'static'. +front50 Aug 28, 2020, 5:36:40 PM 2020-08-29 00:36:40.206 INFO 1 --- [ main] i.a.s.f.p.PolicyEnginePluginFront50 : PolicyEngineFront50.start() +``` + +## Cause +Configurations on Policy Engine require a complete list of declarations.  It can be easy to miss a declaration. + diff --git a/kb/armory/General/errors-when-moving-account-configuration-to-external-s3-bucket.md b/kb/armory/General/errors-when-moving-account-configuration-to-external-s3-bucket.md new file mode 100644 index 0000000000..644430557e --- /dev/null +++ b/kb/armory/General/errors-when-moving-account-configuration-to-external-s3-bucket.md @@ -0,0 +1,21 @@ +--- +title: Errors when moving account configuration to external S3 bucket +--- + +## Issue +When attempting to move Kubernetes accounts from Halyard config file to an external file in AWS S3 bucket (ref: [https://spinnaker.io/docs/setup/other_config/configuration/#external-account-configuration](https://spinnaker.io/docs/setup/other_config/configuration/#external-account-configuration%29)) the accounts defined in the external ```.yml``` file are not loaded successfully. +The following example error is seen in the UI: + "credentials not found (name: ....)") when deploying app into the namespace of "...". + +  +Clouddriver logs show the following: +```2021-08-01 22:10:16.334 INFO 1 --- [ecutionAction-5] c.n.s.c.k.s.KubernetesCredentials : Kind apiService will not be cached in account 'spinnaker-test' because it cannot be listed.``` + +## Cause +Utilizing dynamic account configuration with Spring Cloud Config Server is not the recommended workflow. +  +The following document notes that: *Armory does not support using dynamic account configuration with Spring Cloud Config Server:*[https://docs.armory.io/docs/feature-status/armory-enterprise-matrix/armory-enterprise-matrix-2-26/#dynamic-accounts](https://docs.armory.io/docs/feature-status/armory-enterprise-matrix/armory-enterprise-matrix-2-26/#dynamic-accounts) + + + + diff --git a/kb/armory/General/escalating-a-case.md b/kb/armory/General/escalating-a-case.md new file mode 100644 index 0000000000..0a21d30f70 --- /dev/null +++ b/kb/armory/General/escalating-a-case.md @@ -0,0 +1,18 @@ +--- +title: Escalating a Case +--- + + +Case escalations can be a powerful tool to advise our Support Team of a change to the issue that was raised. The purpose of this ability is to allow customers to advise Support about certain changes +To ensure that a consistent experience is provided to all customers, we ask that case escalations be used sporadically and only when they are **absolutely necessary** because of a one-off situation where there is a specific need/business use-case.   +We ask that escalations be limited to the following reasons, and we would like to advise that the support team does reserve the right to decline any requests that may be raised, although we will give our best effort to meet most requests +* **Executive Visibility*** **Lack of Progress*** **Customer Imposed Deadline** +  +Note that under normal circumstances, the SLA procedure follows the response matrix as defined in the [Armory Support Information (Hours of Operation/SLAs/Procedures) page](https://kb.armory.io/s/article/Support). +## How to Escalate a Case (P1-P3) +* Log in to the ServiceNow Customer portal* In the Navigation Menu, select **Cases -> All Cases**. Locate the case that requires an escalation* At the bottom of the case, next to the **Close Case** button, there will be a **Request Escalation** button available.  Click on this button to begin the processOnce this button is clicked, two mandatory fields will appear above, asking to provide +* **Escalation Reason**: Can chose from **Executive Visibility**, **Lack of Progress**, and **Customer Imposed Deadline*** **Escalation Description**: Description about the issue to be provided to the Support Engineers, so that it is clear why the escalation should be approved.  +* Click on the **Request Escalation** button again to submit* At this point, the escalation will be reviewed by a Senior Support Engineer for its viability before **approval** or **rejection**.  In the case that a request for escalation is not approved, a reason will be provided within the case that explains why the escalation did not proceed. +## How to Escalate a Case (P0) +Customers who are looking to get ahold of Armory Support in a P0 situation where a case was raised, but now requires P0 attention due to the growing severity and radius of the issue, can look to escalate the case by phone.  The following service is only available to Armory Customers with a service agreement: [https://support.armory.io/support?id=kb_article&sysparm_article=KB0010776#phone](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010776#phone) + diff --git a/kb/armory/General/exception-starting-plugin-in-orca.md b/kb/armory/General/exception-starting-plugin-in-orca.md new file mode 100644 index 0000000000..a95ca7b321 --- /dev/null +++ b/kb/armory/General/exception-starting-plugin-in-orca.md @@ -0,0 +1,14 @@ +--- +title: Exception Starting Plugin in Orca +--- + +## Issue +The following exception occurs when Orca starts: +```com.netflix.spinnaker.kork.exceptions.IntegrationException: 'com.jpmc.cto.spinnaker.tasks.CheckEntitlementsTask' extension has unsupported constructor argument type 'com.jpmc.cto.spinnaker.PreflightPluginConfig'. Expected argument classes should be annotated with @ExpectedConfiguration or implement PluginSdks.``` + +This occurs when trying to inject a class (```PreflightPluginConfig```) that is annotated as ```@PluginConfiguration``` into the constructor of ```CheckEntitlementsTask``` (an ```@Extension```), the same way ```RandomWaitConfig``` is being injected into the extension here: [https://github.com/spinnaker-plugin-examples/pf4jStagePlugin/blob/190259a3510c8ced30ce26c385e3881e7d07a17b/random-wait-orca/src/main/kotlin/io/armory/plugin/stage/wait/random/RandomWaitPlugin.kt#L49](https://github.com/spinnaker-plugin-examples/pf4jStagePlugin/blob/190259a3510c8ced30ce26c385e3881e7d07a17b/random-wait-orca/src/main/kotlin/io/armory/plugin/stage/wait/random/RandomWaitPlugin.kt#L49)  + +## Cause +This is for users running Spinnaker 1.20.3 with the following setup in Gradle: +* spinnakerGradleVersion=8.4.0* pf4jVersion=3.2.0* korkVersion=7.75.1* orcaVersion=8.8.0 + diff --git a/kb/armory/General/execution-parameters-shared-when-run-job-stages-are-running-in-parallel.md b/kb/armory/General/execution-parameters-shared-when-run-job-stages-are-running-in-parallel.md new file mode 100644 index 0000000000..7be6769bfe --- /dev/null +++ b/kb/armory/General/execution-parameters-shared-when-run-job-stages-are-running-in-parallel.md @@ -0,0 +1,13 @@ +--- +title: Execution parameters shared when Run Job stages are running in parallel +--- + +## Issue +When two separate Run Job stages are executing simultaneously with identical names in the ```metadata.name``` field, job execution parameters are shared between both pipelines. +This may result in pipeline failure, however they may not fail if the Job configuration is still valid. In this case, the issue can still cause a number of incidents, including: +* Deploying to incorrect location* Notifications sent to the wrong contact* Incorrect parameters being passed onto an external tool + +## Cause +Armory 2.22 introduces a breaking change where Spinnaker no longer automatically appends a unique suffix to the name of jobs created by the Kubernetes Run Job stage. More info: [Release Notes for Armory Enterprise v2.22.2](https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-22-2/#suffix-no-longer-added-to-jobs-created-by-kubernetes-run-job-stage) +In practice, this means that the Kubernetes provider assumes both configurations are referring to the same Job, since they are both running under the same ```metadata.name```. When the parallel execution is triggered, the Kubernetes provider tries to modify the existing Job instead of creating a new one. + diff --git a/kb/armory/General/executionid-&-stageid-are-not-unique-in-the-webhooks-sent-by-spinnaker-customwebhook-stages.md b/kb/armory/General/executionid-&-stageid-are-not-unique-in-the-webhooks-sent-by-spinnaker-customwebhook-stages.md new file mode 100644 index 0000000000..5a29022d99 --- /dev/null +++ b/kb/armory/General/executionid-&-stageid-are-not-unique-in-the-webhooks-sent-by-spinnaker-customwebhook-stages.md @@ -0,0 +1,98 @@ +--- +title: ExecutionID & StageID are not unique in the webhooks sent by Spinnaker CustomWebhook stages +--- + +## Issue +Pipelines on separate applications are invoked by custom webhooks with the same stage and Execution IDs. + +This is observed in the custom webhook where both the pipelines the same custom webhook stage invokes: +```X-Execution-Id: ""``` +and +```X-Stage-Id: ""``` +Spinnaker is expecting that the above values will be ***unique per stage invocation***. +The following shows an example where it is not the case: +First Pipeline: +``` +{ + "type": "PIPELINE", + "id": "0123", + "application": "", + "name": "", + "buildTime": 1622432694880, + "canceled": false, + "limitConcurrent": true, + "keepWaitingPipelines": false, + "stages": [ + { + "id": "0123", + "refId": "1", +..... +...... +........ + + "failPipeline": false, + "customHeaders": { + "X-Stage-Id": [ + "012345678" + ], + "X-Execution-Id": [ + "016789" + ], + "X-Execution-User": [ + "sample@sample.com" + ], + "Content-Type": [ + "application/json" + ] + }, +``` +Second Pipeline: +``` +{ + "type": "PIPELINE", + "id": "0456", + "application": "", + "name": "", + "buildTime": 1622255063387, + "canceled": false, + "limitConcurrent": true, + "keepWaitingPipelines": false, + "stages": [ + { + "id": "0456", + "refId": "1", +.... +.... +..... + +"failPipeline": false, + "customHeaders": { + "X-Stage-Id": [ + "012345678" + ], + "X-Execution-Id": [ + "016789" + ], + "X-Execution-User": [ + "sample@sample.com" + ], + "Content-Type": [ + "application/json" + ] + }, +``` +In the example, both pipeline stages have a different Stage ID but the webhook that is invoked is identical in both pipelines. +``` + "X-Stage-Id": [ + "012345678" + ], + "X-Execution-Id": [ + "016789" +``` +In an ideal scenario they should refer to the unique Stage and Execution IDs. + +## Cause +This issue exists within Orca and the evaluation of SPeL expressions on the CustomWebhook stages and is traced back to the following Spinnaker OSS issue [https://github.com/spinnaker/spinnaker/issues/5910](https://github.com/spinnaker/spinnaker/issues/5910). +Running into this OSS issue causes the baked CRD to be deployed with evaluated the SpEL expressions of the CustomWebhook stages.  + + diff --git a/kb/armory/General/executions-after-migrating-to-terraformer-are-delayed-or-take-a-lot-more-time-to-execute.md b/kb/armory/General/executions-after-migrating-to-terraformer-are-delayed-or-take-a-lot-more-time-to-execute.md new file mode 100644 index 0000000000..ea72f354e0 --- /dev/null +++ b/kb/armory/General/executions-after-migrating-to-terraformer-are-delayed-or-take-a-lot-more-time-to-execute.md @@ -0,0 +1,76 @@ +--- +title: Executions after Migrating to Terraformer are delayed or take a lot more time to execute +--- + +## Issue +When [enabling and migrating to Terraformer](https://docs.armory.io/armory-enterprise/armory-admin/terraform-enable-integration/) (Armory Enterprise's Terraform execution service), it is observed that there are delays in pipeline executions. Users may notice increases in time when utilizing pipelines in Spinnaker with Terraform integration.  Pipelines being run locally can run significantly quicker. +For the following example, an execution using Terraformer took approx ~1:40 to complete: +Workspace ".....-deploy-destroy-events-api" already exists +Switched to workspace "....deploy-destroy-events-api". +kubernetes_secret.cp_events_api_secrets_auth: Refreshing state... [id=default/cp-events-api-secrets-auth] +kubernetes_config_map.cp_events_api_indexes: Refreshing state... [id=default/cp-events-api-indexes] +kubernetes_service_account.cp_events_api: Refreshing state... [id=default/cp-events-api] +kubernetes_config_map.cp_events_api_env_vars: Refreshing state... [id=default/cp-events-api-env-vars] +kubernetes_service.cp_events_api: Refreshing state... [id=default/cp-events-api] +kubernetes_ingress.cp_events_api_internal: Refreshing state... [id=default/cp-events-api-internal] +kubernetes_deployment.cp_events_api: Refreshing state... [id=default/cp-events-api] + +Apply complete! Resources: 0 added, 0 changed, 0 destroyed. +``` + + + +The same execution being attempted with local Terraform took approx ~35s to be completed +» time ./tf-apply.sh + +Initializing the backend... + +Initializing provider plugins... +- Reusing previous version of hashicorp/aws from the dependency lock file +- Reusing previous version of hashicorp/kubernetes from the dependency lock file +- Using previously-installed hashicorp/aws v3.55.0 +- Using previously-installed hashicorp/kubernetes v2.4.1 + +Terraform has been successfully initialized! + +You may now begin working with Terraform. Try running "terraform plan" to see +any changes that are required for your infrastructure. All Terraform commands +should now work. + +If you ever set or change modules or backend configuration for Terraform, +rerun this command to reinitialize your working directory. If you forget, other +commands will detect it and remind you to do so if necessary. + +real 0m9.729s +user 0m2.635s +sys 0m0.381s +Switched to workspace "....-deploy-destroy-events-api". + +real 0m4.379s +user 0m1.304s +sys 0m0.159s +Acquiring state lock. This may take a few moments... +kubernetes_service.cp_events_api: Refreshing state... [id=default/cp-events-api] +kubernetes_config_map.cp_events_api_env_vars: Refreshing state... [id=default/cp-events-api-env-vars] +kubernetes_service_account.cp_events_api: Refreshing state... [id=default/cp-events-api] +kubernetes_config_map.cp_events_api_indexes: Refreshing state... [id=default/cp-events-api-indexes] +kubernetes_secret.cp_events_api_secrets_auth: Refreshing state... [id=default/cp-events-api-secrets-auth] +kubernetes_ingress.cp_events_api_internal: Refreshing state... [id=default/cp-events-api-internal] +kubernetes_deployment.cp_events_api: Refreshing state... [id=default/cp-events-api] + +Apply complete! Resources: 0 added, 0 changed, 0 destroyed. +Releasing state lock. This may take a few moments... + +real 0m21.346s +user 0m7.172s +sys 0m1.329s +./tf-apply.sh 11.11s user 1.87s system 36% cpu 35.465 total +In both cases, no other actions are performed. + +## Cause +The difference in time can be due to several reasons +* Execution within Spinnaker is reliant on network saturation and network availability, and this can significantly add time to the process* Terraformer executions also can be run in parallel (multiple people can be using the Terraformer service at one time)* Depending on where the source is located, the size of the repository hosting the Terraform files may have grown, and execution of the pipeline may not be optimized +It should be noted that the ```Monitor Run Terraform``` task fetches the Terraform files from the configured repository.  The execution of this task indicates if pulling from the repository (e.g. Github) is within expected execution time or not.  If the repository pull contains additional files not related to the Terraform code (e.g. image files, large archival files) this can lead to a delay in the execution. +After that, the ```initialize terraform``` would download the providers' configuration and plugin files (e.g. AWS libraries) based on the ```.tf``` files, before they are applied (or planned/destroyed). This execution occurs whenever a stage is executed and will add time to the execution, and is also dependant on how quickly the plugins/resources are downloaded.  This is because a pod execution is expected to have ephemeral resources, and the pod will need to ensure the resources are available each time.  +This differs from a local Terraform executions, because in local executions the init stage is not running each time, and it is expected that the plugins and modules may be pre-existing.   + diff --git a/kb/armory/General/exposing-spinnaker.md b/kb/armory/General/exposing-spinnaker.md new file mode 100644 index 0000000000..3c4c4e279b --- /dev/null +++ b/kb/armory/General/exposing-spinnaker.md @@ -0,0 +1,7 @@ +--- +title: Exposing Spinnaker +--- + + +This page has been recategorized and moved to our Documents Library:[https://docs.armory.io/spinnaker/exposing_spinnaker/](https://docs.armory.io/spinnaker/exposing_spinnaker/) + diff --git a/kb/armory/General/facing-400-error-while-saving-canary-configuration.md b/kb/armory/General/facing-400-error-while-saving-canary-configuration.md new file mode 100644 index 0000000000..5e0f3788b7 --- /dev/null +++ b/kb/armory/General/facing-400-error-while-saving-canary-configuration.md @@ -0,0 +1,13 @@ +--- +title: Facing 400 Error while Saving Canary Configuration +--- + +## Issue +After enabling the **Kayenta** service in Armory, users face an issue where if adding a Canary config, it cannot be saved, as the following error message appears: +```The was an error saving your config: 400``` +As seen in the following screenshot: + + +## Cause +Currently, Armory Spinnaker supports Canary service integration including **AWS, Google, New Relic, Prometheus, Datadog, and SignalFX**. Before using the Canary analysis service in Armory, at least *one metrics service must be configured, along with at least one storage service*. The most common setup is to have one metrics service configured and one storage service (e.g. S3, GCS, etc) configured.  + diff --git a/kb/armory/General/facing-throttling-issues-when-deploying-cloud-services.md b/kb/armory/General/facing-throttling-issues-when-deploying-cloud-services.md new file mode 100644 index 0000000000..2b6e1d4685 --- /dev/null +++ b/kb/armory/General/facing-throttling-issues-when-deploying-cloud-services.md @@ -0,0 +1,11 @@ +--- +title: Facing Throttling Issues when Deploying Cloud Services +--- + +## Issue +Customers can be facing throttling issues when deploying services to AWS Cloud.  Throttling exception errors can be found in Spinnaker logs.Error example: +```ERROR: Exception ( Monitor Deploy ) Rate exceeded (Service: AmazonElasticLoadBalancing; Status Code: 400; Error Code: Throttling; Request ID: 723d8890-d134-4744-b2ea-36dd5537e428)``` + +## Cause +The polling can cause cloud providers, such as AWS, to throttle the requests on the account.If the environment has a large number of Auto-Scaling Groups and Elastic Load Balancers in the account or other services commonly querying the same APIs then it is expected to see throttling exceptions in the Spinnaker logs. + diff --git "a/kb/armory/General/failed-to-read-manifest--\"exception-wait-for-manifest-to-stabilize.-unexpected-task-failure\".md" "b/kb/armory/General/failed-to-read-manifest--\"exception-wait-for-manifest-to-stabilize.-unexpected-task-failure\".md" new file mode 100644 index 0000000000..fa45bdde41 --- /dev/null +++ "b/kb/armory/General/failed-to-read-manifest--\"exception-wait-for-manifest-to-stabilize.-unexpected-task-failure\".md" @@ -0,0 +1,22 @@ +--- +title: Failed to read manifest- "Exception (Wait for Manifest to Stabilize. Unexpected Task Failure)" +--- + +## Issue +When trying to deployer clusters to Kubernetes in versions older than 1.16, the following errors can appear in the Clouddriver logs: +2021-05-04 21:17:16.032 WARN 1 --- [0.0-7002-exec-9] c.n.s.c.k.c.ManifestController : Failed to read manifest + +com.netflix.spinnaker.clouddriver.kubernetes.op.handler.UnsupportedVersionException: No replicaSet is supported at api version extensions/v1beta1 +at com.netflix.spinnaker.clouddriver.kubernetes.op.handler.KubernetesReplicaSetHandler.status(KubernetesReplicaSetHandler.java:98) ~[clouddriver-kubernetes.jar:na] + + +2021-05-05 14:29:09.653 WARN 1 --- [utionAction-538] c.n.s.c.k.c.a.KubernetesCachingAgent : kubernetes/KubernetesCoreCachingAgent[1/1]: Failure adding relationships for service + +com.netflix.spinnaker.clouddriver.kubernetes.op.handler.UnsupportedVersionException: No replicaSet is supported at api version extensions/v1beta1 +at com.netflix.spinnaker.clouddriver.kubernetes.op.handler.KubernetesReplicaSetHandler.getPodTemplateLabels(KubernetesReplicaSetHandler.java:167) +The UI shows the following: "Exception (Wait for Manifest to Stabilize. Unexpected Task Failure)" + + +## Cause +Kubernetes deployment targets prior to version 1.16 are not supported in v2.26.x, as per the following:[https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-26-2/](https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-26-2/) + diff --git a/kb/armory/General/fiat-ignores-x509-cert-and-check-role-provider-for-permissions.md b/kb/armory/General/fiat-ignores-x509-cert-and-check-role-provider-for-permissions.md new file mode 100644 index 0000000000..a44cececfd --- /dev/null +++ b/kb/armory/General/fiat-ignores-x509-cert-and-check-role-provider-for-permissions.md @@ -0,0 +1,10 @@ +--- +title: Fiat Ignores x509 cert and Check Role Provider for Permissions +--- + +## Issue +When a x509 cert is created, Fiat seems to ignore the roles in the cert and instead will search the Role Provider for the roles. + +## Cause +The user will need to make sure they have ```x509.role-oid``` set or Fiat will switch to loading from the Role Provider and won't merge the roles with the cert. This can cause Fiat to look at the External Role Providers instead of the cert, as there is no roles merged into the cert. Therefore to merge the role, the user will have to specify the role-oid. + diff --git a/kb/armory/General/fiat-unable-to-start-in-a-cloudfoundry-environment.md b/kb/armory/General/fiat-unable-to-start-in-a-cloudfoundry-environment.md new file mode 100644 index 0000000000..c66767c4e6 --- /dev/null +++ b/kb/armory/General/fiat-unable-to-start-in-a-cloudfoundry-environment.md @@ -0,0 +1,14 @@ +--- +title: FIAT unable to start in a CloudFoundry Environment +--- + +## Issue +FIAT is showing unable to start even without any change to the Spinnaker Manifest.  However, a rolling restart was issued to the environment, and once all pods are up and running, FIAT continues to frequently restart. +Upon inspecting the FIAT logs, they will discover error messages about FIAT's inability to contact CloudDriver, such as the following: +```s.f.c.RetrofitConfig$RetryingInterceptor : [] Request for http://spin-clouddriver:7002/credentials failed. Backing off for 2000ms``` + +## Cause +The issue has to do with an issue where the FIAT service is failing to start if there are any invalid Cloud Foundry accounts. +If an invalid CF account is onboarded to spinnaker, the api ([https://spin-clouddriver:7002/credentials)](https://spin-clouddriver:7002/credentials%29) is seen to return 400 response code. +The issue did not appear until changes in 2.26.3 + diff --git a/kb/armory/General/flushall-on-spinnaker's-redis-cache.md b/kb/armory/General/flushall-on-spinnaker's-redis-cache.md new file mode 100644 index 0000000000..ea45009a64 --- /dev/null +++ b/kb/armory/General/flushall-on-spinnaker's-redis-cache.md @@ -0,0 +1,18 @@ +--- +title: FLUSHALL on Spinnaker's Redis Cache +--- + + +Question: +Should I do a Redis ```flushall``` on my Redis database? +Answer: +Before we go into the details, a good rule of thumb is to never use ```flushall``` with Spinnaker. Now let me explain why, and what issues we have seen. +First it’s important to know how Redis plays a role in Spinnaker. Redis has three primary functions: +* As a cache for [Igor](https://github.com/spinnaker/igor) +* As a cache for [Clouddriver](https://github.com/spinnaker/clouddriver) +* As a queue for [Orca](https://github.com/spinnaker/orca) +More Details: +* Igor provides a single point of integration with Jenkins, Travis and Git repositories ( BitBucket, Stash, and Github ) within Spinnaker. Igor keeps track of the credentials for multiple Jenkins and/or Travis hosts and sends events to [echo](https://github.com/spinnaker/echo) whenever build information has changed. +* Orca is the orchestration engine for Spinnaker. It is responsible for taking a pipeline or task definition and managing the stages and tasks, coordinating the other Spinnaker services. Orca pipelines are composed of stages which in turn are composed of tasks. The tasks of a stage share a common context and can publish to a global context shared across the entire pipeline allowing multiple stages to co-ordinate. For example a bake stage publishes details of the image it creates which is then used by a deploy stage. Orca persists a running execution to Redis. +* Many of Spinnakers micros services poll the provider system quite frequently, in order to be less taxing, Clouddriver and Igor will poll the provider for necessary information roughly every 30 seconds and store that information on Redis. This helps reduce the impact Spinnaker has on the provider system. + diff --git a/kb/armory/General/flushing-gate-sessions.md b/kb/armory/General/flushing-gate-sessions.md new file mode 100644 index 0000000000..fe1437a3a7 --- /dev/null +++ b/kb/armory/General/flushing-gate-sessions.md @@ -0,0 +1,39 @@ +--- +title: Flushing Gate Sessions +--- + +## Introduction +Gate depends on Sprint Boot and Spring Session to manage its HTTP sessions. When Gate authenticates a user, this session is serialized and stored in Redis. +This serialization is dependent on the versions of Spring Boot and Spring Session. +When there is a new version of those dependencies, users encounter the following error when they try to log in: +```HTTP Status 500 - Internal Server Error``` +Additionally, Deck redirects the browser to the Gate URL.To resolve the issue, perform one of the following actions: +* Remove the browser cookie for all users.* Flush all the stored Gate sessions and force users to log in again +While option 1 is simple, Armory recommends option 2 for deployments where there are a large number of users. + +## Prerequisites +You must have network access to Redis and have the ```redis-cli``` installed. If you do not have the ```redis-cli``` already installed, below is the information about installing it. +Install and configure the debugging tool +Armory recommends using the [```armory/docker-debugging-tools```](https://github.com/armory/docker-debugging-tools) and deploying it into your Spinnaker namespace. This image contains ```redis-cli```. Run the following commands: +kubectl --context $MY_CONTEXT -n $MY_NAMESPACE apply -f https://raw.githubusercontent.com/armory/troubleshooting-toolbox/master/docker-debugging-tools/deployment.yml + +kubectl --context $MY_CONTEXT -n $MY_NAMESPACE exec -it deploy/debugging-tools -- bash + +## Instructions +Step 1. Find your Redis URL: +Run the following command to exec into the Gate pod: +```k --context $MY_CONTEXT -n $MY_NAMESPACE exec -it deploy/spin-gate -c gate -- cat /opt/spinnaker/config/spinnaker.yml | grep -A2 redis``` +Capture the Redis url from the ```baseUrl``` field in the resulting output, it will look something like the following: +``` + redis: + baseUrl: redis://my-redis-url.example.com:6379 + enabled: true + host: 0.0.0.0 +``` +In this example, you would use ```my-redis-url.example.com```, without the protocol and port data.Then, run the following command in the pod you started in your cluster to flush the sessions: +```redis-cli -h my-redis-url.example.com keys 'spring:session:*' | xargs redis-cli -h my-redis-url.example.com del``` + +Step 2. Remove the debugging tools +When you are finished, you can remove the debugging tools by running the following command: +```kubectl --context $MY_CONTEXT -n $MY_NAMESPACE delete deployment debugging-tools``` + diff --git a/kb/armory/General/front50-pods-failing-to-start-after-upgrading-spinnaker-version-from-2.20.1-to-2.23.5.md b/kb/armory/General/front50-pods-failing-to-start-after-upgrading-spinnaker-version-from-2.20.1-to-2.23.5.md new file mode 100644 index 0000000000..fe7b1a420d --- /dev/null +++ b/kb/armory/General/front50-pods-failing-to-start-after-upgrading-spinnaker-version-from-2.20.1-to-2.23.5.md @@ -0,0 +1,82 @@ +--- +title: Front50 pods failing to start after upgrading Spinnaker version from 2.20.1 to 2.23.5 +--- + +## Issue +Spinnaker Halyard installations that have Front50 configured with MySQL as the backend fail to start after upgrading Spinnaker from version 2.20.x to 2.23.5 (or later). Exceptions similar to below are seen on the Front50 logs, if the environment was migrated from a Redis Front50 setup to a MySQL set up +``` +*************************** +APPLICATION FAILED TO START +*************************** + +Description: + +Parameter 0 of constructor in com.netflix.spinnaker.front50.model.application.ApplicationService required a single bean, but 2 were found: + - redisApplicationDAO: defined by method 'redisApplicationDAO' in class path resource [com/netflix/spinnaker/front50/redis/RedisConfig.class] + - applicationDAO: defined by method 'applicationDAO' in class path resource [com/netflix/spinnaker/front50/config/CommonStorageServiceDAOConfig.class] + + +Action: + +Consider marking one of the beans as @Primary, updating the consumer to accept multiple beans, or using @Qualifier to identify the bean that should be consumed +```` + +It is also possible that Front50 was set up using the PersistentStorage settings for the environment.  In this example, it was set up with S3 and then migrated to MySQL RDS +```` +*************************** +APPLICATION FAILED TO START +*************************** + +Description: + +Parameter 0 of method applicationDAO in com.netflix.spinnaker.front50.config.CommonStorageServiceDAOConfig required a single bean, but 2 were found: + - s3StorageService: defined by method 's3StorageService' in class path resource [com/netflix/spinnaker/front50/config/S3Config.class] + - sqlStorageService: defined by method 'sqlStorageService' in class path resource [com/netflix/spinnaker/config/SqlConfiguration.class] + + +Action: + +Consider marking one of the beans as @Primary, updating the consumer to accept multiple beans, or using @Qualifier to identify the bean that should be consumed +``` + +## Cause +From the logs it is evident that there were 2 beans being passed to Front50 causing the application to fail during startup. It has been noticed that ```persistantStorageType:``` in ```halyard config``` under ```/home/spinnaker/.hal/config ```does not accept MySQL as a configuration. Hence ```persistantStorageType``` is either configured with Redis, S3, GCS, etc,. However, MySQL can be enabled in ```front50-local.yml``` under ```/home/spinnaker/.hal/default/profiles/front50-local.yml```. to override the ```persistantStorageType``` from the main halyard config. +This has been the expected behaviour until 2.20 and halyard 1.9.4. However for the versions post that, any configuration mentioned in the halyard main configuration should be explicitly enabled/disabled under (```*-local.yml```), else the service would consider both the configurations. This is the reason behind Front50 considering both MySQL and Redis configurations as separate beans during startup. +Halyard Config: +```` +persistentStorage: + persistentStoreType: redis + azs: {} + gcs: + rootFolder: front50 + redis: {} + s3: + bucket: xxxxx + rootFolder: xxxxx + region: us-east-1 + pathStyleAccess: false + accessKeyId: xxxxx + secretAccessKey: xxxxx + oracle: {} +```` +```front50-local.yml``` + +```` +sql: + enabled: true + connectionPools: + default: + default: true + jdbcUrl: jdbc:mysql://your.database:3306/front50 + user: front50_service + password: ** + migration: + user: front50_migrate + jdbcUrl: jdbc:mysql://your.database:3306/front50 +s3: + enabled: false +gcs: + enabled: false +```` + + diff --git a/kb/armory/General/gate-metrics-not-available-to-the-monitoring-daemon.md b/kb/armory/General/gate-metrics-not-available-to-the-monitoring-daemon.md new file mode 100644 index 0000000000..d134418119 --- /dev/null +++ b/kb/armory/General/gate-metrics-not-available-to-the-monitoring-daemon.md @@ -0,0 +1,12 @@ +--- +title: Gate Metrics not Available to the Monitoring-Daemon +--- + +## Issue +There are two issues related to Gate metrics not being available to the monitoring daemon. +* The first issue is that Gate requires auth for ```/spectator/metrics``` if auth is enabled, including from ```localhost```.  +* Secondly, Gate metrics are not available when Gate is on the same host as Deck but on a different path due to the generated metrics_url being wrong (i.e. Deck's URL is [https://myspinnaker.com](https://myspinnaker.com/) and Gate's URL is [https://myspinnaker.com/api/v1](https://myspinnaker.com/api/v1)) + +## Cause +The first issue is that Gate requires auth for ```/spectator/metrics``` if auth is enabled, including from ```localhost```. The second issue is because the generated metrics_url for Gate does not take into context the value of the setting ```server.servlet.context-path``` in the file ```profiles/gate-local.yml```.  + diff --git a/kb/armory/General/gcp--exception--wait-for-server-group-disabled----error-while-querying-target-pool-instance-health.md b/kb/armory/General/gcp--exception--wait-for-server-group-disabled----error-while-querying-target-pool-instance-health.md new file mode 100644 index 0000000000..04cf524f50 --- /dev/null +++ b/kb/armory/General/gcp--exception--wait-for-server-group-disabled----error-while-querying-target-pool-instance-health.md @@ -0,0 +1,25 @@ +--- +title: GCP- Exception ( Wait For Server Group Disabled ) - Error while querying target pool instance health +--- + +## Issue +When creating a cluster in GKE within a pipeline , users may look to to deploy an image and then Destroy it to Disable it. +When trying to run the stage ```Destroy Server Group``` with a GCP cluster, a time out is hit after ```30 min``` in the ```Wait For Server Group Disabled``` task. +The following exception can be seen within the Spinnaker UI: +Exception ( Wait For Server Group Disabled ) +WaitForDisabledServerGroupTask of stage disableServerGroup timed out after 30 minutes 5 seconds. pausedDuration: 0 seconds, elapsedTime: 30 minutes 5 seconds, timeoutValue: 30 minutes + + + +  +  +The ```Task Status``` shows that the ```Wait for Server Group Disabled``` takes the most time when taking a look at the individual task progression: + +  +Admins will also see the following error in Clouddriver logs: +```{"@timestamp":"2021-065-23T19:46:24.349+00:00","@version":1,"message":"Error while querying target pool instance health: The resource 'projects/.....' was not found",...":"com.netflix.spinnaker.clouddriver.google.provider.agent.GoogleNetworkLoadBalancerCachingAgent","thread_name":......","level":"ERROR","level_value":40000}``` +  + +## Cause +Destroying server groups within GCP is not a supported workflow for Clouddriver. + diff --git a/kb/armory/General/gcp-error--code-=-permissiondenied-desc-=-forbidden--http-status-code-403;-transport-.md b/kb/armory/General/gcp-error--code-=-permissiondenied-desc-=-forbidden--http-status-code-403;-transport-.md new file mode 100644 index 0000000000..caac7d9f94 --- /dev/null +++ b/kb/armory/General/gcp-error--code-=-permissiondenied-desc-=-forbidden--http-status-code-403;-transport-.md @@ -0,0 +1,24 @@ +--- +title: GCP error- code = PermissionDenied desc = Forbidden- HTTP status code 403; transport- +--- + +## Issue +In this example, two clusters are operating, and one cluster has been working for a while.  The second cluster is newly created and runs on a separate Service Account.  When creating a new Google service account and associating it with Armory Agent, the new agent is not able to register with the ```gRPC``` ```server```, however the original cluster and Agent pair works without issue +A Terraform module was used to automate building a second service account and bind it to the Kubernetes service account for the agent, but a similar situation can happen in a manual process.   +When utilizing the newly-created service account to launch an agent service in a second cluster, the agent throws the following error: +``` +time="2021-08-17T18:26:30Z" level=info msg="registration starting" func="armory/kubesvc/pkg/register.(*AgentRegistry).Run.func3" file="/workspace/pkg/register/registration.go:126" instanceId=....... +time="2021-08-17T18:26:30Z" level=info msg="registering with 1 kubernetes servers" func="armory/kubesvc/pkg/register.(*AgentRegistry).register" file="/workspace/pkg/register/registration.go:249" instanceId=..... +time="2021-08-17T18:26:30Z" level=info msg="stopping all tasks" func="armory/kubesvc/pkg/throttle.(*Throttler).StopAll" file="/workspace/pkg/throttle/throttler.go:234" instanceId=...... +time="2021-08-17T18:26:30Z" level=warning msg="registration failed, restarting registrationb......" func="armory/kubesvc/pkg/register.(*AgentRegistry).logRegistrationErr" file="/workspace/pkg/register/registration.go:277" error="client error receiving ops: rpc error: code = PermissionDenied desc = Forbidden: HTTP status code 403; transport: received the unexpected content-type \"text/html; charset=UTF-8\"" instanceId=..... +``` +In the example, the agent in the first cluster running under the original Google service account continues to work properly.  If the agent in the new cluster is deployed using the original service account, both agents work properly.   +If both agents services are adjusted to use the new service account, both fail. The Golang code that fetches the OIDC token appears to be working with both the new and old service account.  If run manually from a shell on the agent pod, it returns a valid token. +  + +## Cause +The issue can be traced back to the GCP IAP settings and how they are used to authenticate the Agents to the Agent plugin endpoint. The error is due the way that the users have configured the new service accounts. +  +  +  + diff --git a/kb/armory/General/getting-the-ip-of-an-ec2-instance-with-pipeline-expressions.md b/kb/armory/General/getting-the-ip-of-an-ec2-instance-with-pipeline-expressions.md new file mode 100644 index 0000000000..4829d9831a --- /dev/null +++ b/kb/armory/General/getting-the-ip-of-an-ec2-instance-with-pipeline-expressions.md @@ -0,0 +1,12 @@ +--- +title: Getting the IP of an EC2 Instance With Pipeline Expressions +--- + + +Question +How do I get the IP of an instance in an ASG through a Pipeline Expression in Spinnaker? +Answer +You can’t get the IP address of instance through pipeline expressions, however, you can get the Instance ID and pass it to something like Jenkins and script a job to get the IP Address. Generally speaking you should not be accessing an instance by IP address, because you never know when an IP address might change. +Example Of Getting Instance ID With A Pipeline Expression +```${#stage('stage name')['id']}``` + diff --git "a/kb/armory/General/github-changed-default-base-branch-from-\"master\"-to-\"main\".md" "b/kb/armory/General/github-changed-default-base-branch-from-\"master\"-to-\"main\".md" new file mode 100644 index 0000000000..69e5f23275 --- /dev/null +++ "b/kb/armory/General/github-changed-default-base-branch-from-\"master\"-to-\"main\".md" @@ -0,0 +1,12 @@ +--- +title: GitHub Changed (default) Base Branch from "master" to "main" +--- + +## Issue +[As of Oct 1st, 2020, Github has changed their (default) base branches from "master" to "main"](https://github.com/github/renaming) .  Any new repos created by users will now use "main" as the (default) base branch.**Existing repos are unaffected.** +Artifacts and dinghyfiles coming from GitHub will need to be adjusted by adding additional configuration, until updates to Spinnaker are released to accommodate these changes.  + +## Cause +GitHub should be set as a provider, for example: +[Connecting Spinnaker to GitHub as an Artifact Source](https://docs.armory.io/docs/armory-admin/artifacts-github-connect/)[Using GitHub Artifacts](https://docs.armory.io/docs/spinnaker-user-guides/artifacts-github-use/)[Steps to Configure Github as Pipeline as Code](https://docs.armory.io/docs/armory-admin/dinghy-enable/#steps-to-follow-to-configure-pipelines-as-code) + diff --git a/kb/armory/General/github-dinghyfiles-referring-to-mixed-case-applications-fail.md b/kb/armory/General/github-dinghyfiles-referring-to-mixed-case-applications-fail.md new file mode 100644 index 0000000000..29e4d935af --- /dev/null +++ b/kb/armory/General/github-dinghyfiles-referring-to-mixed-case-applications-fail.md @@ -0,0 +1,11 @@ +--- +title: GitHub DinghyFiles referring to Mixed Case Applications Fail +--- + +## Issue +If an application is manually created in Spinnaker with a case-sensitive name (e.g. thisTestPipeline), users will encounter a ```404 error``` with regards to executing the pipeline when referring to the case-sensitive application name. +The pipeline refers to the case sensitive name properly (e.g. ```thisTestPipeline```) in the JSONThe dinghyfile is stored in a GitHub repository. + +## Cause +GitHub references to the application end up being converted into lower case.  As an example, if you attempt to execute a ```dinghyfile``` and attempt to create the application from the pipeline execution, users will see that ```thisTestPipeline``` will be converted to ```thistestpipeline``` when looking at the application within Spinnaker + diff --git a/kb/armory/General/github-issues-with-certificates-triggered-failure-to-be-able-to-execute-pipelines,-due-to-plugins-missing-unavailable.md b/kb/armory/General/github-issues-with-certificates-triggered-failure-to-be-able-to-execute-pipelines,-due-to-plugins-missing-unavailable.md new file mode 100644 index 0000000000..05a385c139 --- /dev/null +++ b/kb/armory/General/github-issues-with-certificates-triggered-failure-to-be-able-to-execute-pipelines,-due-to-plugins-missing-unavailable.md @@ -0,0 +1,21 @@ +--- +title: GitHub issues with Certificates triggered Failure to be able to execute pipelines, due to Plugins (missing/unavailable) +--- + +## Issue +On Friday, May 24th, Github experienced an incident related to an expired certificate.  As a result, customers may see services with issues validating certificates from Github. +The following error may be present at the time of the Certificate error earlier today: +```PKIX Path Validation Failed: java.security.cert.CertPathValidatorException: Validity Check failed``` +However, even after the issue was resolved from Github, Spinnaker users may find errors when executing Pipeline, and administrators see errors in logs related to plugins. +For example, for the Evaluate Artifacts Plugin, customers may see the following: +```No StageDefinitionBuilder implementation for evaluateArtifacts``` +Consequently, Spinnaker telemetry will appear healthy, but there can be an increase in failed pipelines that rely on plugins. +Please note that the error message will vary from plugin to plugin. In addition, many services may not be directly impacted, but customers should consider issuing a rolling restart for preventative reasons because of the potential impact. +Armory Customers who have an Armory Managed Environment have already had a rolling restart completed on their environment by Armory. There is no additional action needed for Managed Customers. + +## Cause +GitHub experienced an incident related to a poorly configured certificate on their CDN. This lead to additional TLS problems for their user base. +[https://www.githubstatus.com/incidents/x7njwb481j9b](https://www.githubstatus.com/incidents/x7njwb481j9b) +Since Spinnaker Plugins are loaded from GitHub, there's a possibility of the Plugin Manager removing plugins when it's unable to list them. Despite Github resolving the issue, Spinnaker services would be operational but referencing the unloaded plugins. This can cause various errors depending on the plugins used in an environment. It's also possible services are recovered without any impact. Since it's hard to verify the impact on a running environment and to each specific plugin as a safety precaution we recommend rolling restarts of all Spinnaker services. +The errors cannot be specified because they would be unique to the nature of each plugin. + diff --git a/kb/armory/General/github-notifications-do-not-update-github-commit-status.md b/kb/armory/General/github-notifications-do-not-update-github-commit-status.md new file mode 100644 index 0000000000..28f8146501 --- /dev/null +++ b/kb/armory/General/github-notifications-do-not-update-github-commit-status.md @@ -0,0 +1,23 @@ +--- +title: Github Notifications Do Not Update Github Commit Status +--- + +## Issue +Although notifications appear to be set up correctly, Spinnaker is unable to update the Github Commit Status to Github properly.    +Echo shows an error related to an unexpected URL and is unable to make API requests to Github, like the following below example: +2020-11-03 17:58:24.224 INFO 1 --- [xIoScheduler-93] c.n.s.e.n.GithubNotificationAgent : Sending Github status check for application: test +2020-11-03 17:58:24.225 INFO 1 --- [xIoScheduler-93] c.n.spinnaker.echo.github.GithubService : ---> HTTP GET 'https://api.github.com'/repos/testGit/commits/9da3da385417cd14f9a8a69518845063bc95146b +2020-11-03 17:58:24.225 INFO 1 --- [xIoScheduler-93] c.n.spinnaker.echo.github.GithubService : Authorization: token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx +2020-11-03 17:58:24.225 INFO 1 --- [xIoScheduler-93] c.n.spinnaker.echo.github.GithubService : ---> END HTTP (no body) +2020-11-03 17:58:24.225 INFO 1 --- [xIoScheduler-93] c.n.spinnaker.echo.github.GithubService : ---- ERROR 'https://api.github.com'/repos/testGit/commits/9da3da385417cd14f9a8a69518845063bc95146b +2020-11-03 17:58:24.226 INFO 1 --- [xIoScheduler-93] c.n.spinnaker.echo.github.GithubService : java.lang.IllegalArgumentException: unexpected url: 'https://api.github.com'/repos/testGit/commits/9da3da385417cd14f9a8a69518845063bc95146b + at com.squareup.okhttp.Request$Builder.url(Request.java:163) + at retrofit.client.OkClient.createRequest(OkClient.java:58) + at retrofit.client.OkClient.execute(OkClient.java:53) + at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:326) +[...] + +## Cause +The pipeline status notifications are not building Github status URL correctly.  Spinnaker is using ```'[https://api.github.com'](https://api.github.com%27/);``` to begin the URL reference, instead of just ```[https://api.github.com](https://api.github.com/)```In the above example: +```2020-11-03 17:58:24.225 INFO 1 --- [xIoScheduler-93] c.n.spinnaker.echo.github.GithubService : ---> HTTP GET 'https://api.github.com'/repos/testGit/commits/9da3da385417cd14f9a8a69518845063bc95146b``` + diff --git a/kb/armory/General/github-trigger-matches-multiple-artifacts-on-trigger-.tf-files.md b/kb/armory/General/github-trigger-matches-multiple-artifacts-on-trigger-.tf-files.md new file mode 100644 index 0000000000..68a131b1f1 --- /dev/null +++ b/kb/armory/General/github-trigger-matches-multiple-artifacts-on-trigger-.tf-files.md @@ -0,0 +1,35 @@ +--- +title: Github trigger matches multiple artifacts on trigger (.tf files) +--- + +## Issue +An organization may run into an issue where they have a pipeline setup which triggers on change in Github Enterprise. The expected artifact should look similar to this. +```` +- displayName: "changed terraform file" + + id: changed-terraform-file + + defaultArtifact: + + customKind: true + + id: default-tf-artifact + + matchArtifact: + + name: ".*\\.tf.*" + + type: github/file + + useDefaultArtifact: true +```` +This will work fine when there is only one commit with a change in one ```*.tf``` file, but it fails as soon as there are multiple commits with changes in multiple ```*.tf``` files. Here is an example error code. + +```` +Failed on startup: Expected artifact ExpectedArtifact(matchArtifact=Artifact(type=github/file, customKind=false, name=.*\.tf.*, version=null, location=null, reference=null, metadata={}, artifactAccount=null, provenance=null, uuid=null), usePriorArtifact=false, useDefaultArtifact=true, defaultArtifact=Artifact(type=null, customKind=true, name=null, version=null, location=null, reference=null, metadata={id=default-tf-artifact}, artifactAccount=null, provenance=null, uuid=null), id=changed-terraform-file, boundArtifact=null) matches multiple artifacts [Artifact(type=github/file, customKind=false, name=main.tf, version=dc461903a063ce78fbf4d6f9ce8ccda6696db6cf, location=null, reference=https://git01.pfsfhq.com/api/v3/repos/ops/terraform-demo/contents/main.tf, metadata={}, artifactAccount=null, provenance=null, uuid=null), Artifact(type=github/file, customKind=false, name=test.tf, version=dc461903a063ce78fbf4d6f9ce8ccda6696db6cf, location=null, reference=https://git01.pfsfhq.com/api/v3/repos/ops/terraform-demo/contents/test.tf, metadata={}, artifactAccount=null, provenance=null, uuid=null)] +```` + +## Cause +This was at the time an intended behaviour for Spinnaker environments using Github Enterprise. There has been a community outreach regarding this issue and as a result, changes have been implemented. +The following is a similar and related Github Issue that was used as a foundation for the work done. [https://github.com/spinnaker/spinnaker/issues/3643](https://github.com/spinnaker/spinnaker/issues/3643) + diff --git a/kb/armory/General/google-app-engine-gae-deploy-stage-not-showing-artifacts-that-have-been-added-and-can't-edit.md b/kb/armory/General/google-app-engine-gae-deploy-stage-not-showing-artifacts-that-have-been-added-and-can't-edit.md new file mode 100644 index 0000000000..f158c50daf --- /dev/null +++ b/kb/armory/General/google-app-engine-gae-deploy-stage-not-showing-artifacts-that-have-been-added-and-can't-edit.md @@ -0,0 +1,10 @@ +--- +title: Google App Engine (GAE) Deploy Stage Not Showing Artifacts That Have Been Added and Can't Edit +--- + +## Issue +When creating a pipeline for **Google App Engine (GAE)** deployment, Spinnaker does not show previously configured **Config Artifacts **under the App Engine provider in Deployment Configuration in the Deploy stage.The stage initially wouldn't correctly show the artifacts that have been added and it also wouldn't allow a user to edit them. + +## Cause +Impaired function. Code fix required. + diff --git a/kb/armory/General/google-cloud-provider---private-gke.md b/kb/armory/General/google-cloud-provider---private-gke.md new file mode 100644 index 0000000000..3769556c3f --- /dev/null +++ b/kb/armory/General/google-cloud-provider---private-gke.md @@ -0,0 +1,13 @@ +--- +title: Google Cloud Provider - Private GKE +--- + +## Introduction +Information regarding set up of private Google Kubernetes Engine cluster with different level of restricted access to cluster master endpoints, and how to configure access for Kubectl to private Google Kubernetes Engine cluster with different level of restricted access + +## Prerequisites +GCP account + +## Instructions +The following articles provided by Google have the most up-to-date information for working with Private Clusters and Private Cluster Access[https://cloud.google.com/kubernetes-engine/docs/concepts/private-cluster-concept](https://cloud.google.com/kubernetes-engine/docs/concepts/private-cluster-concept%20https://cloud.google.com/solutions/creating-kubernetes-engine-private-clusters-with-net-proxies)[https://cloud.google.com/solutions/creating-kubernetes-engine-private-clusters-with-net-proxies](https://cloud.google.com/solutions/creating-kubernetes-engine-private-clusters-with-net-proxies) + diff --git a/kb/armory/General/groovy.lang.missingmethodexception-when-referencing-github-config-artifact-for-google-app-engine.md b/kb/armory/General/groovy.lang.missingmethodexception-when-referencing-github-config-artifact-for-google-app-engine.md new file mode 100644 index 0000000000..149e2fa3f5 --- /dev/null +++ b/kb/armory/General/groovy.lang.missingmethodexception-when-referencing-github-config-artifact-for-google-app-engine.md @@ -0,0 +1,12 @@ +--- +title: groovy.lang.MissingMethodException when referencing github config artifact for Google App Engine +--- + +## Issue +```groovy.lang.MissingMethodException``` error occurs when referencing Github config artifact for Google App Engine (GAE) deploy. The config artifacts were of instance of HashMap not an Artifact object.The following is an example of the error found within Spinnaker +Exception ( Create Server Group ) +No signature of method: com.netflix.spinnaker.orca.pipeline.util.ArtifactUtils.getBoundArtifactForStage() is applicable for argument types: (com.netflix.spinnaker.orca.pipeline.model.Stage, null, HashMap) values: [Stage {id='', executionId=''}, ...] Possible solutions: getBoundArtifactForStage(com.netflix.spinnaker.orca.pipeline.model.Stage, java.lang.String, com.netflix.spinnaker.kork.artifacts.model.Artifact) + +## Cause +OSS bug:[https://github.com/spinnaker/spinnaker/issues/5836](https://github.com/spinnaker/spinnaker/issues/5836)The config artifacts were returning a HashMap format instead of an artifact format, within the Orca service + diff --git a/kb/armory/General/hashicorp-terraform-gpg-rotation-codecov-vulnerability.md b/kb/armory/General/hashicorp-terraform-gpg-rotation-codecov-vulnerability.md new file mode 100644 index 0000000000..53cde75b35 --- /dev/null +++ b/kb/armory/General/hashicorp-terraform-gpg-rotation-codecov-vulnerability.md @@ -0,0 +1,18 @@ +--- +title: Hashicorp Terraform GPG Rotation (CodeCov vulnerability) +--- + +## Issue + + +Hashicorp rotated GPG keys due to the [CodeCov vulnerability](https://discuss.hashicorp.com/t/hcsec-2021-12-codecov-security-event-and-hashicorp-gpg-key-exposure/23512).  As a result, old GPG keys were rendered invalid and Terraformer required updates to available versions to allow for the key rotations, especially on Terraform versions 0.11.x and 0.12.x.   +Customers will need to update their Terraform versions for all releases with the updated binaries & GPG keys.  Our latest release of Armory Spinnaker (2.25.0, 2.24.1, 2.23.5) did not have these latest versions as they were only recently released.  They were not available in the Terraformer stage dropdown dropdown list. + + + +*``````* + + +## Cause +HashiCorp was impacted by a security incident with a third party (Codecov) that led to potential disclosure of sensitive information. As a result, the GPG key used for release signing and verification has been rotated. Customers who verify HashiCorp release signatures may need to update their process to use the new key.To learn more, please see:[https://discuss.hashicorp.com/t/hcsec-2021-12-codecov-security-event-and-hashicorp-gpg-key-exposure/23512](https://discuss.hashicorp.com/t/hcsec-2021-12-codecov-security-event-and-hashicorp-gpg-key-exposure/23512) + diff --git a/kb/armory/General/high-number-of-github-emails-after-every-dinghyfile-update.md b/kb/armory/General/high-number-of-github-emails-after-every-dinghyfile-update.md new file mode 100644 index 0000000000..45d316afed --- /dev/null +++ b/kb/armory/General/high-number-of-github-emails-after-every-dinghyfile-update.md @@ -0,0 +1,12 @@ +--- +title: High number of GitHub emails after every Dinghyfile update +--- + +## Issue +After enabling the [Dinghy GitHub notifier](https://docs.armory.io/armory-enterprise/installation/armory-operator/op-manifest-reference/armory/#dinghy-parameters), users may find that there is additional comments in a Pull Request.  This is so that Dinghy can provide more robust feedback information to the persons initiating the Pull Request to allow them to make decisions on the PR.   +A consequence is that users may experience a high number of emails generated after every update to a module that occurs.  This is because module updates may have chain reactions to the multiple Dinghyfiles that may be using them. + +## Cause +After an update to a commonly used module, all of the associated pipelines are re-rendered, triggering a potentially high amount of API calls to Github. +This in turn generates a lot of emails from Github and may potentially manifest itself by triggering rate-limits. + diff --git a/kb/armory/General/hitting-annotation-hard-limit-when-deploying-a-configmap-secret-too-long.md b/kb/armory/General/hitting-annotation-hard-limit-when-deploying-a-configmap-secret-too-long.md new file mode 100644 index 0000000000..c302167882 --- /dev/null +++ b/kb/armory/General/hitting-annotation-hard-limit-when-deploying-a-configmap-secret-too-long.md @@ -0,0 +1,17 @@ +--- +title: Hitting annotation hard limit when deploying a Configmap/Secret (Too long) +--- + +## Issue +When users and administrators are deploying a Configmap/Secret, they may end up hitting the following error: +```  +Error occurred when Deploy secret/ in : (500: error creating resource-name in : + Secret "" is invalid: metadata.annotations: +Too long: must have at most 262144 bytes: metadata.annotations: + Too long: must have at most 262144 bytes) +``` + +## Cause +This error occurs during the ```kubectl apply``` phase for the ConfigMap/Secrets resources deployment.  Spinnaker uses ```kubectl apply``` to perform deployments.  One of the downsides of using ```kubectl apply``` is that it stores the entire ```spec``` as an annotation in the object, which is used to understand how to handle defaulted vs. deleted fields.  +What’s happening is that the data fields in the ConfigMap/Secrets are ```exceeding 262144 characters``` max limit enforced by the Kubernetes API server.  As a result, Spinnaker cannot fit in the ```last-applied-configuration``` kubectl annotation. + diff --git a/kb/armory/General/hitting-igor's-caching-thresholds.md b/kb/armory/General/hitting-igor's-caching-thresholds.md new file mode 100644 index 0000000000..0bfddc95dd --- /dev/null +++ b/kb/armory/General/hitting-igor's-caching-thresholds.md @@ -0,0 +1,16 @@ +--- +title: Hitting Igor's Caching Thresholds +--- + +## Issue +Completing a Jenkins job doesn’t trigger a pipeline execution. Pushing to a docker repository doesn’t trigger a pipeline execution +In your Igor logs, you see something like: +```Number of items (999999) to cache exceeds upper threshold (1000) in monitor=DockerMonitor partition=dockerhub``` + +Or: +```Number of items (999999) to cache exceeds upper threshold (1000) in monitor=JenkinsMonitor partition=jenkins``` + +## Cause +Potential causes could be: +* A new Jenkins master has been added* A new docker registry has been added* Igor has been down for a while* Redis has been wiped + diff --git a/kb/armory/General/http-https-redirects-with-authentication.md b/kb/armory/General/http-https-redirects-with-authentication.md new file mode 100644 index 0000000000..ad76a2e8b1 --- /dev/null +++ b/kb/armory/General/http-https-redirects-with-authentication.md @@ -0,0 +1,31 @@ +--- +title: HTTP/HTTPS Redirects with Authentication +--- + + +Troubleshooting HTTP/HTTPS Redirects +When setting up TLS encryption for Deck and Gate (the UI and API) for Spinnaker, if you have a load balancer (service, ingress, etc.) in front of your Deck/Gate that are terminating TLS and forwarding communications insecure to the Spinnaker microservices, sometimes the authentication process will redirect to the incorrect path. +For example, if you have LDAP set up and the following flow: +* Client Browser ```--HTTPS-->``` AWS ELB ```--HTTP-->``` Deck/Gate +Then you’ll likely get two invalid redirects - one to your gate address on HTTP and one on your deck address on HTTP. This is regardless of your ```override-base-url``` settings. +One option to try is to add/create ```.hal//profiles/gate-local.yml```: +``` +server: + tomcat: + protocolHeader: X-Forwarded-Proto + remoteIpHeader: X-Forwarded-For + internalProxies: .* + httpsServerPort: X-Forwarded-Port +``` +(Then run ```hal deploy apply```, and clear / update your cache). +Alternately, you can do the following: +There are a number of ongoing projects to improve this behavior (for example, when working with OAuth2.0, you can specify a ```preEstablishedRedirectUri``` via the ```--pre-established-redirect-uri``` flag). +In the interim, you can work around this issue by putting a self-signed certificate on Deck and Gate. This requires two steps: +* Create self-signed certificates for Deck (in ```pem``` format) and Gate (in ```jks``` format), and configure them to use them. You can follow the official documentation for this [here](https://www.spinnaker.io/setup/security/authentication/ssl/).Configure your load balancer to communicate with your backend with HTTPS rather than HTTP. The method of achieving this will depend on your load balancer configuration.For example, with an ALB ingress controller, you’ll want these additional annotations in your ingress configuration: +``` +metadata: + annotations: + alb.ingress.kubernetes.io/backend-protocol: "HTTPS" + alb.ingress.kubernetes.io/healthcheck-protocol: "HTTPS"​ +``` + diff --git a/kb/armory/General/iam-auth-on-pods-via-irsa.md b/kb/armory/General/iam-auth-on-pods-via-irsa.md index 9144108f4e..ce22c2631b 100644 --- a/kb/armory/General/iam-auth-on-pods-via-irsa.md +++ b/kb/armory/General/iam-auth-on-pods-via-irsa.md @@ -9,11 +9,13 @@ This capability can be integrated into Armory Enterprise, and this eliminates th ## Prerequisites The following are required: -* Creation of AWS role in accordance with the AWS documentation on [IAM roles for service accounts- Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html)* Armory v2.26+ +* Creation of AWS role in accordance with the AWS documentation on [IAM roles for service accounts- Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) * Armory v2.26+ + Note:IRSA requires a[ minimum version of the AWS SDK.](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/) If the SDK version is not currently being utilized, you may [make the app IRSA-aware (in the pod).](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/) ## Instructions To override the service accounts, the following example may be utilized: +``` apiVersion: spinnaker.armory.io/v1alpha2 kind: SpinnakerService metadata: @@ -29,13 +31,14 @@ spec: echo: kubernetes: serviceAccountName: spin-sa-echo -  +``` + The service account would then need to have annotations applied for particular roles, as in the following example: +``` apiVersion: v1 kind: ServiceAccount metadata: name: spin-sa annotations: eks.amazonaws.com/role-arn: arn:aws:iam::000000000000:role/my-spinnaker-role -  - +``` \ No newline at end of file diff --git a/kb/armory/General/igor-pod-not-running-or-starting-up-when-configuring-pipeline-triggers-500-internal-server-error.md b/kb/armory/General/igor-pod-not-running-or-starting-up-when-configuring-pipeline-triggers-500-internal-server-error.md new file mode 100644 index 0000000000..f3cde23630 --- /dev/null +++ b/kb/armory/General/igor-pod-not-running-or-starting-up-when-configuring-pipeline-triggers-500-internal-server-error.md @@ -0,0 +1,10 @@ +--- +title: Igor pod not running or starting up when configuring pipeline triggers (500 Internal Server Error) +--- + +## Issue +After configuring new repositories as part of a pipeline trigger strategy, users are noticing error messages such as ```500 - Internal Server Error``` in the UI, with the ```Igor pod``` not running. + +## Cause +Igor's start-up is dependent on having either ```CI integration```, or ```Docker Registry``` enabled. + diff --git a/kb/armory/General/impersonate-a-user-for-servicenow-troubleshooting-servicenow-admins.md b/kb/armory/General/impersonate-a-user-for-servicenow-troubleshooting-servicenow-admins.md new file mode 100644 index 0000000000..2f4a970d22 --- /dev/null +++ b/kb/armory/General/impersonate-a-user-for-servicenow-troubleshooting-servicenow-admins.md @@ -0,0 +1,20 @@ +--- +title: Impersonate a User for ServiceNow Troubleshooting (ServiceNow Admins) +--- + + +To help assist customers or Armory Users and ensure that they have roles set properly, it may be necessary to impersonate a user. This provides access to see the portal via the account's access. + +## Impersonate a User +### Via Backend Interface +* To do so, click on your name in the upper right corner and select "impersonate user"* Select the appropriate person to test with (either by search, or most recently accessed users) +* You will then be placed at the Support Portal main page with those credentials +### Via URL +* You can also make the change by going to the URL [https://armory.service-now.com/impersonate_dialog.do](https://armory.service-now.com/impersonate_dialog.do)* Selecting the appropriate user +* Click OK + +## Revert the credentials back +After the adjustments are made, please set your credentials back to the default.  This can be accomplished by +* Going to the URL [https://armory.service-now.com/impersonate_dialog.do](https://armory.service-now.com/impersonate_dialog.do) +* Selecting the appropriate user (yourself)* Click OK + diff --git a/kb/armory/General/incorrect-oauth2.0-redirects.md b/kb/armory/General/incorrect-oauth2.0-redirects.md new file mode 100644 index 0000000000..9a1f593d4d --- /dev/null +++ b/kb/armory/General/incorrect-oauth2.0-redirects.md @@ -0,0 +1,10 @@ +--- +title: Incorrect OAuth2.0 Redirects +--- + +## Issue +OAuth2.0 authentication for the Spinnaker cluster has been setting up, but users are seeing redirects to the wrong page (for example, to http instead of https) + +## Cause +Configuration will need to be adjusted to ensure the correct redirects + diff --git a/kb/armory/General/increase-the-agent-operation-wait-timeouts-on-agent-clouddriver-plugin.md b/kb/armory/General/increase-the-agent-operation-wait-timeouts-on-agent-clouddriver-plugin.md new file mode 100644 index 0000000000..05a465d1c0 --- /dev/null +++ b/kb/armory/General/increase-the-agent-operation-wait-timeouts-on-agent-clouddriver-plugin.md @@ -0,0 +1,61 @@ +--- +title: Increase the Agent operation wait timeouts on Agent Clouddriver Plugin +--- + +## Issue +Spinnaker installations that use Armory Agent for Kubernetes to perform the deployments, may experience ```timeout errors``` on Clouddriver logs if the operation on the target cluster takes more than 30 seconds. The log trace on Clouddriver shall look similar to the one shown below +``` +message="com.netflix.spinnaker.clouddriver.exceptions.OperationTimedOutException: Timeout exceeded for operation 01FHVJ66QXEHX51CSEY9PXXTPY0, type: 6, account: test-nonprod1-us-east-1_test1-dev, time: 30022 ms, timeout: 30000 ms + at io.armory.kubesvc.services.ops.KubesvcOperations.performOperation(KubesvcOperations.java:96) + at io.armory.kubesvc.services.ops.cluster.ClusteredKubesvcOperations.performOperation(ClusteredKubesvcOperations.java:70) + at io.armory.kubesvc.util.OperationUtils.perform(OperationUtils.java:76) + at io.armory.kubesvc.services.ops.executor.KubesvcExecutor.deploy(KubesvcExecutor.java:301) + at jdk.internal.reflect.GeneratedMethodAccessor703.invoke(Unknown Source) + at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + at java.base/java.lang.reflect.Method.invoke(Method.java:566) + at io.armory.kubesvc.services.registration.clouddriver.KubesvcCredentialsParser.lambda$makeKubesvcJobExecutor$0(KubesvcCredentialsParser.java:72) + at com.netflix.spinnaker.clouddriver.kubernetes.op.job.KubectlJobExecutor$$EnhancerByCGLIB$$4f088312.deploy() + at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.lambda$deploy$14(KubernetesCredentials.java:508) + at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.runAndRecordMetrics(KubernetesCredentials.java:618) + at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.runAndRecordMetrics(KubernetesCredentials.java:603) + at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.deploy(KubernetesCredentials.java:504) + at com.netflix.spinnaker.clouddriver.kubernetes.op.handler.CanDeploy.deploy(CanDeploy.java:58) + at com.netflix.spinnaker.clouddriver.kubernetes.op.manifest.KubernetesDeployManifestOperation.operate(KubernetesDeployManifestOperation.java:209) + at com.netflix.spinnaker.clouddriver.kubernetes.op.manifest.KubernetesDeployManifestOperation.operate(KubernetesDeployManifestOperation.java:46) + at com.netflix.spinnaker.clouddriver.orchestration.AtomicOperation$operate.call(Unknown Source) + at com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1$_closure2.doCall(DefaultOrchestrationProcessor.groovy:124) + at com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1$_closure2.doCall(DefaultOrchestrationProcessor.groovy) + at jdk.internal.reflect.GeneratedMethodAccessor543.invoke(Unknown Source) + at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + at java.base/java.lang.reflect.Method.invoke(Method.java:566) + at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101) + at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323) + at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263) + at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041) + at groovy.lang.Closure.call(Closure.java:405) + at groovy.lang.Closure.call(Closure.java:399) + at com.netflix.spinnaker.clouddriver.metrics.TimedCallable$CallableWrapper.call(TimedCallable.java:81) + at com.netflix.spinnaker.clouddriver.metrics.TimedCallable.call(TimedCallable.java:45) + at java_util_concurrent_Callable$call.call(Unknown Source) + at com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1.doCall(DefaultOrchestrationProcessor.groovy:123) + at com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1.doCall(DefaultOrchestrationProcessor.groovy) + at jdk.internal.reflect.GeneratedMethodAccessor537.invoke(Unknown Source) + at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + at java.base/java.lang.reflect.Method.invoke(Method.java:566) + at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101) + at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323) + at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263) + at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041) + at groovy.lang.Closure.call(Closure.java:405) + at groovy.lang.Closure.call(Closure.java:399) + at com.netflix.spinnaker.security.AuthenticatedRequest.lambda$wrapCallableForPrincipal$0(AuthenticatedRequest.java:272) + at com.netflix.spinnaker.clouddriver.metrics.TimedCallable.call(TimedCallable.java:45) + at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) + at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) + at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) + at java.base/java.lang.Thread.run(Thread.java:829)" +``` + +## Cause +The error on Clouddriver means that Clouddriver sent a deploy operation to the Agent (kubesvc) but did not receive the result back from the Agent in time. The Agent plugin on Clouddriver has a default operation timeout of 30 seconds (```30000 ms```) unless it is explicitly specified.  + diff --git a/kb/armory/General/increase-timeouts-on-agent-plugin-and-agent-to-avoid-network-latency-issues.md b/kb/armory/General/increase-timeouts-on-agent-plugin-and-agent-to-avoid-network-latency-issues.md new file mode 100644 index 0000000000..45cca20d3f --- /dev/null +++ b/kb/armory/General/increase-timeouts-on-agent-plugin-and-agent-to-avoid-network-latency-issues.md @@ -0,0 +1,57 @@ +--- +title: Increase timeouts on Agent Plugin and Agent to avoid network latency issues +--- + +## Introduction +Depending on the location of the Agents relative to Clouddriver, there might be a need to increase the timeout defined to ensure that both Agent and Clouddriver can complete their push operations. The necessity of the change may arise if the network latency between the Agent and the Clouddriver instances are high. + +## Prerequisites +Armory Enterprise Spinnaker with Armory Agent for Kubernetes enabled + +## Instructions +#### **Increase timeouts on Agent Plugin** +To increase the timeouts on Agent Plugin, add the below configuration to the Agent Plugin configuration in the following situations +Specify how long to wait for a response after a ```keepalive``` before closing the gRPC connection with the Armory Agent  +``` + kubesvc: + grpc.server.security.KeepAliveTimeOutSeconds: 180 +``` + +Specify how often should ```keepalive``` send gRPC pings to the client via the Agent plugin. +``` + kubesvc + grpc.server.security.keepAliveHeartbeatSeconds: 60​ +``` +#### **Increase timeouts on Armory Agent** +To increase the timeouts on Agents, add the below configuration to the Agent configuration in the following situations +Specify how often the gPRC keepalive message is sent from Agent to Clouddriver. The default value is ```60 sec```. Note that setting the value to 0 will set it so that keepalive messages will not be sent to Armory Enterprise Clouddriver +``` +clouddriver: + grpc: spin-clouddriver-grpc:9091 + insecure: true + keepAliveHeartbeatSeconds: 180 +Timeout before closing the gRPC connection. The Agent would wait for the time specified before closing the gRPC connection with Armory Enterprise. +clouddriver: + grpc: spin-clouddriver-grpc:9091 + insecure: true + keepAliveTimeOutSeconds:​ 180​ +``` +If customers wish to set the Agent to wait for a certain period before reconnecting to Armory Enterprise, they can set the ```reconnectTimeoutMs```.  Customers should seek to balance this setting as setting the number too low may lead to aggressive recaching of the data, leading to performance issues +``` +kubernetes: + reconnectTimeoutMs: 60 + accounts: + - kubeconfigFile: encryptedFile:secrets-manager!r:us-east-2!s:kubeconfig-secret + name: Account-1 + metrics: false + kinds: [] + omitKinds: [] + permissions: + READ: + - role-1 + WRITE: + - role-1​ +``` +The complete list of configurations for Agent and Agent Plugin can be found in the below links +Agent Plugin: [https://docs.armory.io/armory-enterprise/armory-agent/advanced-config/agent-plugin-options/](https://docs.armory.io/armory-enterprise/armory-agent/advanced-config/agent-plugin-options/)Agent:  [https://docs.armory.io/armory-enterprise/armory-agent/advanced-config/agent-options/#configuration-options](https://docs.armory.io/armory-enterprise/armory-agent/advanced-config/agent-options/#configuration-options) + diff --git a/kb/armory/General/infinite-authentication-loop---user-initially-authenticate-without-issue,-but-loops-upon-timeout.md b/kb/armory/General/infinite-authentication-loop---user-initially-authenticate-without-issue,-but-loops-upon-timeout.md new file mode 100644 index 0000000000..f642cc7454 --- /dev/null +++ b/kb/armory/General/infinite-authentication-loop---user-initially-authenticate-without-issue,-but-loops-upon-timeout.md @@ -0,0 +1,11 @@ +--- +title: Infinite Authentication Loop - User Initially Authenticate WIthout Issue, but Loops upon Timeout +--- + +## Issue +End users may find that they they are able to initially able to sign in to an environment without issue.  However, upon a timeout, the loop occurs between SAML authentication (as an example, OKTA)  and the Spinnaker environment. +Users cannot access the environment again unless they log out, and the log back in to the authenticator.  At this point, users can access the environment once again   + +## Cause +The temporary token between the authenticator and Spinnaker is still valid, but the Spinnaker Timeout Session has signed the user out.  This causes a loop between Spinnaker and the Authenticator + diff --git "a/kb/armory/General/infinite-ui-loop-\"save-changes\"-stuck-attempting-to-save-a-json-canary-config-in-the-spinnaker-ui.md" "b/kb/armory/General/infinite-ui-loop-\"save-changes\"-stuck-attempting-to-save-a-json-canary-config-in-the-spinnaker-ui.md" new file mode 100644 index 0000000000..929c98d4b5 --- /dev/null +++ "b/kb/armory/General/infinite-ui-loop-\"save-changes\"-stuck-attempting-to-save-a-json-canary-config-in-the-spinnaker-ui.md" @@ -0,0 +1,11 @@ +--- +title: Infinite UI Loop "Save Changes" stuck attempting to Save a JSON Canary Config in the Spinnaker UI +--- + +## Issue +When trying to save a Canary Config JSON via the Spinnaker UI, the ```Save Changes``` button gets stuck in a loop after it is clicked. + + +## Cause +The error occurs because the UI interface expects the ```isNew: true``` setting in the JSON file, Spinnaker cannot find it. + diff --git a/kb/armory/General/infrastructure-tab-not-showing-correct-deployment-details.md b/kb/armory/General/infrastructure-tab-not-showing-correct-deployment-details.md new file mode 100644 index 0000000000..d1896455c2 --- /dev/null +++ b/kb/armory/General/infrastructure-tab-not-showing-correct-deployment-details.md @@ -0,0 +1,10 @@ +--- +title: Infrastructure Tab not Showing Correct Deployment Details +--- + +## Issue +In a Spinnaker Application, multiple pipelines have been defined and successfully deployed. In this example, there are two deployments, (e.g, **svc-deployment-one** and **svc-deployment-two**)** **in the EKS cluster via Spinnaker.But under the **"Infrastructure" deployment tab** in the same Application, only **svc-deployment-one** can be seen.  The deployment **svc-deployment-two** does not show up in the tab. **svc-deployment-two** deployment was created in a separate Application from the original deployment.  + +## Cause +Resource relationships (for example between applications and clusters) are managed using **Kubernetes Annotations**, and Spinnaker manages these using its [**Moniker library**](https://github.com/spinnaker/moniker)**;** a Frigga replacement aimed at making naming strategies within Spinnaker more flexible. Under the Moniker settings, it specifies the resource associated with a particular application or cluster, as explained in the Spinnaker doc: [https://spinnaker.io/reference/providers/kubernetes-v2/#moniker](https://spinnaker.io/reference/providers/kubernetes-v2/#moniker) + diff --git a/kb/armory/General/ingesting-aws-cloudformation-templates-larger-than-51200-bytes-as-an-artifact.md b/kb/armory/General/ingesting-aws-cloudformation-templates-larger-than-51200-bytes-as-an-artifact.md new file mode 100644 index 0000000000..64ed246b26 --- /dev/null +++ b/kb/armory/General/ingesting-aws-cloudformation-templates-larger-than-51200-bytes-as-an-artifact.md @@ -0,0 +1,11 @@ +--- +title: Ingesting AWS CloudFormation Templates Larger than 51200 Bytes as an Artifact +--- + +## Issue +Executing an AWS CloudFormation Template which is larger than 51200 bytes as an artifact leads to the following error in Spinnaker when using the CloudFormation Stage +```'templateBody' failed to satisfy constraint: Member must have length less than or equal to 51200``` + +## Cause +The process of ingesting an Artifact, even from S3, is to take the artifact into Spinnaker so it can be manipulated and used across stages.  At that point, the file is no longer being ingested direction from S3, and is therefore, constrained by the AWS file size limitations outlined here:[https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cloudformation-limits.html](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cloudformation-limits.html) + diff --git a/kb/armory/General/ingesting-metrics-from-observability-plugin-to-datadog.md b/kb/armory/General/ingesting-metrics-from-observability-plugin-to-datadog.md new file mode 100644 index 0000000000..c40a1d7f88 --- /dev/null +++ b/kb/armory/General/ingesting-metrics-from-observability-plugin-to-datadog.md @@ -0,0 +1,17 @@ +--- +title: Ingesting Metrics from Observability Plugin to Datadog +--- + +## Introduction +With Spinnaker's changes in 2.20.x and above, the support for the existing monitoring sidecar for Spinnaker has decreased.  Instead, Armory recommends installing the Observability Plugin in the environment to pull metrics directly from Spinnaker microservices. +By negating the need for a sidecar, this change increases performance, reduces system load, and provides additional features such as metric filtering.  +While previous to v1.4.1, the Observability Plugin only natively supported NewRelic, and Prometheus metric formats, this has changed in v1.4.1. +As of v1.4.1, customers can now connect Datadog directly to the Observability Plugin without needing to route data through Prometheus for ingestion. + +## Prerequisites +Admins should install the Observability Plugin >= v1.4.1 along with Datadog.[https://docs.armory.io/armory-enterprise/armory-admin/observe/observability-configure/](https://docs.armory.io/armory-enterprise/armory-admin/observe/observability-configure/) + +## Instructions +Administrators can now ingest metric information directly into Datadog.  They can set up the observability plugin per the example shown in the Observability PlugIn ```read.me``` file. +[https://github.com/armory-plugins/armory-observability-plugin#condensed-datadog-example](https://github.com/armory-plugins/armory-observability-plugin#condensed-datadog-example) + diff --git a/kb/armory/General/initiated-login-does-not-work-with-saml-2.0.md b/kb/armory/General/initiated-login-does-not-work-with-saml-2.0.md new file mode 100644 index 0000000000..c36ec629ba --- /dev/null +++ b/kb/armory/General/initiated-login-does-not-work-with-saml-2.0.md @@ -0,0 +1,11 @@ +--- +title: Initiated Login Does Not Work With SAML 2.0 +--- + + +If you’ve set up SAML 2.0 authentication for your Spinnaker cluster and are able to login when your Identity Provider (iDP, ADFS/okta/etc.), but aren’t able to login when the Service Provider (SP, Spinnaker) initiates the login, try the following:y +```keytool -export -keystore saml.jks -alias saml -file spinnaker-saml.cer``` +Then import/configure the exported ```spinnaker-saml.cer``` in your iDP for the SAML application you created. +Essentially, Gate is signing the requests with the Java Keystore and the iDP doesn’t know how to understand the signed requests until it is aware of the signing certificate. +This is somewhat documented [here](https://www.spinnaker.io/setup/security/authentication/saml/#identity-provider-setup). + diff --git a/kb/armory/General/instance-is-not-healthy-but-the-deployment-shows-as-succeeded-.md b/kb/armory/General/instance-is-not-healthy-but-the-deployment-shows-as-succeeded-.md new file mode 100644 index 0000000000..6289125b28 --- /dev/null +++ b/kb/armory/General/instance-is-not-healthy-but-the-deployment-shows-as-succeeded-.md @@ -0,0 +1,23 @@ +--- +title: Instance is not healthy but the deployment shows as succeeded +--- + +## Issue +In an ECS deployment, the UI shows "successfully deployed" but fails health-check fail 5 seconds later. Health checks are fail but the application still shows as being deployed.The following stage for example is being utilized: +```` + "completeOtherBranchesThenFail": true, + "dependsOn": [], + "executionOptions": { + "successful": true + }, + "failPipeline": false, + "name": "Deploy app Stack [dev, ........", + "stageTimeoutMs": 900000, + "type": "deploy" +} +```` +The pipeline completes as succeeded but in the ```server group logs``` there are messages for ```failures``` (service unhealthy). + +## Cause +Ideally, the deployment should only succeed if the underlying service/task is healthy. This issue is currently being investigated further by Armory Engineering. + diff --git a/kb/armory/General/instance-registration-teardown.md b/kb/armory/General/instance-registration-teardown.md index bec28331c0..b57aa11521 100644 --- a/kb/armory/General/instance-registration-teardown.md +++ b/kb/armory/General/instance-registration-teardown.md @@ -13,7 +13,7 @@ When the instance registration is enabled, Deck makes some API requests to Gate Validate that the change has been successfully implemented in Gate:      View the Gate logs and verify that the following log DOES NOT exist: ```i.a.c.cloud.config.CloudConfiguration : Cloud service is enabled```     In case the the armory.cloud config was never enabled, and the customer doesn't have a clientId/clientSecret. The following block can be safely ignored: - +``` armory.cloud: enabled: false iam: @@ -22,7 +22,7 @@ Validate that the change has been successfully implemented in Gate: clientSecret: api: baseUrl: https://api.cloud.armory.io -  +``` It has many parameters involved for which customers are applicable or not. For example: If using Armory-operator > 1.4.1 and Armory version >=2.27.1 -> APPLICABLEIf using an Armory-operator =2.27.1 -> APPLICABLEIf using an Armory-operator NOT APPLICABLE @@ -31,6 +31,7 @@ To avoid service disruption please apply the following configuration in your Arm ## Instructions Here is the manifest: +```` apiVersion: spinnaker.armory.io/v1alpha2 kind: SpinnakerService metadata: @@ -62,4 +63,4 @@ spec: Armory.ArmoryHeader: enabled: true version: 0.2.2 - +```` diff --git a/kb/armory/General/integration-with-nexus.md b/kb/armory/General/integration-with-nexus.md new file mode 100644 index 0000000000..f372a393f6 --- /dev/null +++ b/kb/armory/General/integration-with-nexus.md @@ -0,0 +1,36 @@ +--- +title: Integration with Nexus +--- + +## Introduction +The following instructions provide a sample configuration to integrate Spinnaker with Nexus as an artifact repository. + +## Prerequisites +1. Nexus as an artifact repository2. Kubernetes Operator with Kustomize templates in a deployment. + +## Instructions +In Operator, the following configuration change can be made under: ```spec.spinnakerConfig.config.repository``` to enable connection to the Nexus repository needed.   +Please ensure that the following values below are adjusted to match the repository's values: +* baseUrl* repo* nodeId* username* password +```` +spec: + spinnakerConfig: + config: + repository: + nexus: + enabled: true + searches: + - name: Armorytest # Example. Please name as necessary + permissions: + READ: + - '[]' + WRITE: + - '[]' + baseUrl: https://armory.io # Example. Please use the correct URL + repo: testrepo + nodeId: testNodeID + username: testuser + password: testpassword +```` +Please note that this is from preliminary testing and only to provide a lead on what the configuration may look like. Any issues related to Nexus may be filed through Spinnaker Github Issue Tracker as a New Issue/Feature with the OSS community[https://github.com/spinnaker/spinnaker/issues](https://github.com/spinnaker/spinnaker/issues) + diff --git a/kb/armory/General/internal-error,-failed-calling-webhook--when-deploying-armory-continuous-deployment-after-upgrading-the-armory-operator.md b/kb/armory/General/internal-error,-failed-calling-webhook--when-deploying-armory-continuous-deployment-after-upgrading-the-armory-operator.md new file mode 100644 index 0000000000..8b8c05c0a1 --- /dev/null +++ b/kb/armory/General/internal-error,-failed-calling-webhook--when-deploying-armory-continuous-deployment-after-upgrading-the-armory-operator.md @@ -0,0 +1,27 @@ +--- +title: Internal error, failed calling webhook when deploying Armory Continuous Deployment after upgrading the Armory Operator +--- + +## Issue +Users cannot deploy Spinnaker (Armory Enterprise) after installing a new Armory Operator. +After running the following command: ```kubectl -n armory apply -k overlays/dev/```, admins see the following exception: +```Error from server (InternalError): error when creating "overlays/dev/": Internal error occurred: failed calling webhook "webhook-spinnakerservices-v1alpha2.spinnaker.armory.io": Post "https://spinnaker-operator.armory.svc:443/validate-spinnaker-armory-io-v1alpha2-spinnakerservice?timeout=10s": context deadline exceeded``` +  +Or an exception like this: +```Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook-spinnakerservices-v1alpha2.spinnaker.armory.io": Post "https://spinnaker-operator.armory-operator.svc:443/validate-spinnaker-armory-io-v1alpha2-spinnakerservice?timeout=10s": no endpoints available for service "spinnaker-operator"``` +`````` + +## Cause +The reason for this issue is that when deploying a new Armory Operator, the old ```spinnakervalidatingwebhook``` was not removed. +During installation the Operator by following instructions: [https://docs.armory.io/armory-enterprise/installation/armory-operator/op-quickstart/#single-manifest-file-option](https://docs.armory.io/armory-enterprise/installation/armory-operator/op-quickstart/#single-manifest-file-option), +the step +```kubectl apply -f deploy/crds/``` will create two ```customresourcedefinition.apiextensions.k8s.io```: +* ```armoryaccounts.spinnaker.armory.io``` +* ```spinnakerservices.spinnaker.armory.io``` +The ```spinnakervalidatingwebhook``` ```ValidatingWebhookConfiguration``` creates two validating admission webhooks: + +* ```webhook-spinnakeraccounts-v1alpha2.spinnaker.armory.io``` +* ```webhook-spinnakerservices-v1alpha2.spinnaker.armory.io``` + +The mismatched ```caBundle``` of ```webhook-spinnakerservices-v1alpha2.spinnaker.armory.io``` will generate an Internal error as shown. + diff --git a/kb/armory/General/invalid-token-with-named-profiles-assume-role.md b/kb/armory/General/invalid-token-with-named-profiles-assume-role.md new file mode 100644 index 0000000000..d77b53f37d --- /dev/null +++ b/kb/armory/General/invalid-token-with-named-profiles-assume-role.md @@ -0,0 +1,11 @@ +--- +title: Invalid Token with Named Profiles Assume Role +--- + +## Issue +When using Terraformer Named Profiles with AWS and assuming a role, an error is returned when running a pipeline.An example of the error is: +```Error: error using credentials to get account ID: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid. status code: 403, request id:``` + +## Cause +When trying to set up AWS credentials to assume a role in a named profile, a session key is required.  If the session key is not added to the configuration, it will cause the pipeline to fail. + diff --git a/kb/armory/General/is-it-possible-to-restrict-user-access-to-certain-envrionments-with-rbac.md b/kb/armory/General/is-it-possible-to-restrict-user-access-to-certain-envrionments-with-rbac.md new file mode 100644 index 0000000000..f8aa1b612f --- /dev/null +++ b/kb/armory/General/is-it-possible-to-restrict-user-access-to-certain-envrionments-with-rbac.md @@ -0,0 +1,14 @@ +--- +title: Is it Possible to Restrict User Access to Certain Envrionments with RBAC +--- + +## Introduction +Customers may want to restrict certain users to deploying within certain environments.  This may because of least privilege rules, or team distribution.   + +## Prerequisites +Customers will need to have purchased a tier of CDaaS that includes RBAC.  This would include setting up a Trial version with our Sales team or purchase a full Enterprise license. + +## Instructions +Currently, it is not possible to make this restriction, but it is a part of our delivery of our RBAC implementation.  Please look forward to more information in mid/late Summer 2022.   +For more information, please subscribe to this knowledge article, as it will be updated once RBAC functionality has launched.   + diff --git a/kb/armory/General/jason-mcintosh's-handy-information-about-setting-up-monitoring.md b/kb/armory/General/jason-mcintosh's-handy-information-about-setting-up-monitoring.md new file mode 100644 index 0000000000..4c12e900fd --- /dev/null +++ b/kb/armory/General/jason-mcintosh's-handy-information-about-setting-up-monitoring.md @@ -0,0 +1,19 @@ +--- +title: Jason McIntosh's handy information about Setting Up Monitoring +--- + +## Introduction +Below is a reference to Jason's recommendations to JPMC about monitoring their environment.  + +A lot of the information below has been recreated in the KB article  +[https://support.armory.io/support?id=kb_article&sysparm_article=KB0010370](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010370) +The information below is Jason's raw notes about monitoring [https://cloud-armory.slack.com/archives/CC0TJ4K24/p1626192030226800?thread_ts=1626186324.221500&cid=CC0TJ4K24](https://cloud-armory.slack.com/archives/CC0TJ4K24/p1626192030226800?thread_ts=1626186324.221500&cid=CC0TJ4K24).  Please note that these are general best practices for just about any software deployment in the cloud.  They should not be shared directly with the customers as direct recommendations.  The KB article attempts to provide some clues, but shies away from providing a full end to end solution, as customers Dev Ops and DBAs should ultimately be responsible for figuring out what external monitoring is necessary.   + +## Prerequisites +N/A + +## Instructions +* Watch CloudWatch metrics on your instances, particularly burst budget limits on EBS volumes or CPU usage.  INFRASTRUCTURE monitoring of key metrics is KEY - and these are platform agnostic that “Ops” teams should be aware of and alert on.* Watch Redis & MySQL metrics - storage, IO wait times, etc.  In particular Transactions per second limits.  DBA’s usually know these intimately well (at the very least, watch some of the common ones, IO Wait time, Transaction burst capacity and transactions per second rates, as well as CPU utilization… AND STORAGE USAGE!!)Watch your APPS - particularly JVM metric data, queue times, cache latency.  Specifically +* [https://blog.spinnaker.io/monitoring-spinnaker-part-1-4847f42a3abd](https://blog.spinnaker.io/monitoring-spinnaker-part-1-4847f42a3abd)* [https://blog.spinnaker.io/monitoring-spinnaker-sla-metrics-a408754f6b7b](https://blog.spinnaker.io/monitoring-spinnaker-sla-metrics-a408754f6b7b) +* And our SLI/SLO doc ```[internal use only]```: [https://paper.dropbox.com/doc/Armory-SLISLO-Documentation--BPVZrXCL8ulTK2w9Nv8p5tkjAg-UiwecmNjt0DGV7vE4NQ6a](https://paper.dropbox.com/doc/Armory-SLISLO-Documentation--BPVZrXCL8ulTK2w9Nv8p5tkjAg-UiwecmNjt0DGV7vE4NQ6a)* ANYTHING in java where GC collection is over 5-10% is a sign of memory exhaustion and needs to be increased.  If continuous increasing there’s something leaking and/or causing a leak (can be external or internal, e.g. a thread blocked by an external resource can eventually fill up, and is tricky to track, ETC)* I HIGHLY suggest an APM -  LIKE datadog/NewRelic/etc. that can monitor the JVM itself at a deeper level than what is reported via metrics.  This is an extra layer that has a 2-5% overhead, but gives you way more visibility easier.  You can get SOME of this data from the metrics, but… not always * Distributed tracing - often get with an APM, but not always.  Zipkin/sleuth - these are USEFUL tools for tracking flow through a system to know if a paramter broke things.* JMX - if you can, this can provide some INSANE diagnostics, but requires a connection to the JVM (similar to APM agents).  NICE thing is a lot of things work well with this - e.g. Tomcat used to provide GREAT JMX controls to do things like JDBC pool configurations & thread manipulation during RUNTIME.  SUPER useful for viewing things like that.  Less common to see in todays world but still POTENTIALLY useful if configured/enabled for remote RMI calls.* Monitor your OTHER systems, like the EKS environment, API rate limits on artifact stores, throttling of anything, etc. that are often signs of systems struggling.  These are not always “super critical” but can be signs that spinnaker is overwhelming your transtiive infrastructure.  Key ones: GitHub APIs for example, or if you have burst rate limits on AWS or Google APIs.* WATCH any of your artifact stores, CI pipelines, etc.  E.g. Jenkins jobs & polling Jenkins can HAMMER it, but ALSO can fail if too many changes and not enough capacity to watch for those changes. + diff --git a/kb/armory/General/jenkins-ci-driver-can-start-but-spinnaker-fails-to-monitor-the-jenkins-job-tomcat-server.md b/kb/armory/General/jenkins-ci-driver-can-start-but-spinnaker-fails-to-monitor-the-jenkins-job-tomcat-server.md new file mode 100644 index 0000000000..6743f0ea38 --- /dev/null +++ b/kb/armory/General/jenkins-ci-driver-can-start-but-spinnaker-fails-to-monitor-the-jenkins-job-tomcat-server.md @@ -0,0 +1,13 @@ +--- +title: Jenkins CI driver can start but Spinnaker fails to monitor the Jenkins job (Tomcat Server) +--- + +## Issue +When a Jenkins CI provider is set up, it is possible to add a master with a Jenkins service account/API token. The pipeline is kicked off from a Jenkins job and is able to execute.However, it fails almost immediately with an ```HTTP 400 error bad request``` upon trying to monitor the status of the job it ran.The error starts around the line: +```Error running MonitorJenkinsJobTask for pipeline[**]``` +The Jenkins environment would be hosted in a Tomcat server. In this test and example, the environment is: +* Java 8u144* Tomcat 8.5.23* Jenkins 2.176.4 + +## Cause +Tomcat ends up blocking the characters within a Jenkins' URL.  For example, when Jenkins' API URL includes  the pipe single character ```|``` or in the "exclude" portion of a query string ```{}```.When either the URL or query string includes those characters, Tomcat would throw a 500 response; not including it gave the aforementioned log message with a 400 response.  + diff --git a/kb/armory/General/jenkins-timeout-issues.md b/kb/armory/General/jenkins-timeout-issues.md new file mode 100644 index 0000000000..efcfb6ce9f --- /dev/null +++ b/kb/armory/General/jenkins-timeout-issues.md @@ -0,0 +1,32 @@ +--- +title: Jenkins Timeout Issues +--- + +## Issue +Admins see timeout issues when connecting Jenkins with Spinnaker such as below: +2019-05-14 19:31:28.366 WARN 1 --- [0.0-8088-exec-1] .m.m.a.ExceptionHandlerExceptionResolver : Resolved [com.netflix.hystrix.exception.HystrixRuntimeException: jenkins-build-qa-getJobs failed and no fallback available.] +2019-05-14 19:31:28.380 WARN 1 --- [0.0-8088-exec-3] .m.m.a.ExceptionHandlerExceptionResolver : Resolved [com.netflix.hystrix.exception.HystrixRuntimeException: jenkins-build-qa-getJobs failed and no fallback available.] +2019-05-14 19:31:29.842 ERROR 1 --- [RxIoScheduler-3] c.n.s.igor.jenkins.JenkinsBuildMonitor : Failed to update monitor items for monitor=JenkinsBuildMonitor:partition=build-dev + +com.netflix.hystrix.exception.HystrixRuntimeException: jenkins-build-dev-getProjects failed and no fallback available. +... +Caused by: retrofit.RetrofitError: connect timed out + at retrofit.RetrofitError.networkError(RetrofitError.java:27) ~[retrofit-1.9.0.jar:na] + at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:395) ~[retrofit-1.9.0.jar:na] + at retrofit.RestAdapter$RestHandler.invoke(RestAdapter.java:240) ~[retrofit-1.9.0.jar:na] + at com.sun.proxy.$Proxy93.getProjects(Unknown Source) ~[na:na] + at sun.reflect.GeneratedMethodAccessor154.invoke(Unknown Source) ~[na:na] + at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_201] + at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_201] + at com.netflix.spinnaker.kork.telemetry.InstrumentedProxy.invoke(InstrumentedProxy.java:87) ~[kork-telemetry-3.8.1-5814b41-stable145.jar:na] + at com.sun.proxy.$Proxy93.getProjects(Unknown Source) ~[na:na] + at com.netflix.spinnaker.igor.jenkins.client.JenkinsClient$getProjects.call(Unknown Source) ~[na:na] + at com.netflix.spinnaker.igor.jenkins.service.JenkinsService$_getProjects_closure1.doCall(JenkinsService.groovy:78) ~[igor-web-1.1.1-63d06a5-stable160.jar:na] +... +Caused by: java.net.SocketTimeoutException: connect timed out + at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_201] + at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_201] + +## Cause +Jenkins can run into communication Timeout Issues with Spinnaker.  The timeout can be adjusted in Igor to compensate for this issue. + diff --git a/kb/armory/General/jobs-run-by-a-service-account-in-a-different-namespace-generates-generic-errors-index--1-out-of-bounds-for-length-0.md b/kb/armory/General/jobs-run-by-a-service-account-in-a-different-namespace-generates-generic-errors-index--1-out-of-bounds-for-length-0.md new file mode 100644 index 0000000000..858a00b2a4 --- /dev/null +++ b/kb/armory/General/jobs-run-by-a-service-account-in-a-different-namespace-generates-generic-errors-index--1-out-of-bounds-for-length-0.md @@ -0,0 +1,26 @@ +--- +title: Jobs run by a service account in a different namespace generates generic errors (Index -1 out of bounds for length 0) +--- + +## Issue +Armory has found that when a job is configured to run as a specific service account, and it can't locate that service account within the same namespace context of the job, it will throw an exception in the Run Job Config tab: + +``` +Exception +Index -1 out of bounds for length 0 +waitOnJobCompletion: +Internal Server Error +Index -1 out of bounds for length 0 +``` + +The error is generic and doesn't provide much context.  +  +However, the error message in the Deploy Status tab is more indicative of the root cause:   +```Error creating: pods "test-job-name" is forbidden: error looking up service account namespace1/fake-sa: serviceaccount "fake-sa" not found``` +  +Please note that the error message in the Deploy Status tab is based on ephemeral data, and administrators may get the following message if it's been longer than Kubernetes has been configured to store events: +```No recent events found - Kubernetes does not store events for long.``` + +## Cause +Armory cannot find the service account specified in the job manifest because it doesn't exist, is misspelled, or is in a different namespace from the job itself.   + diff --git a/kb/armory/General/jsonpath-update-allows-constraints-on-additional-webhooks-payload-values-directory-specific.md b/kb/armory/General/jsonpath-update-allows-constraints-on-additional-webhooks-payload-values-directory-specific.md new file mode 100644 index 0000000000..48336ad433 --- /dev/null +++ b/kb/armory/General/jsonpath-update-allows-constraints-on-additional-webhooks-payload-values-directory-specific.md @@ -0,0 +1,67 @@ +--- +title: JSONPath Update Allows Constraints on Additional Webhooks Payload Values (Directory Specific) +--- + +## Introduction +Due to Armory's contribution to the Spinnaker OSS project, users can now define constraints based on nested JSON values in the webhook.  +[https://github.com/spinnaker/echo/pull/1078](https://github.com/spinnaker/echo/pull/1078) +The purpose of this example is to show how the trigger of a pipeline can be constrained to only occur when changes occur in a subdirectory of a Repo.   Folder changes in a Github webhook are one of the many values that do not exist on the parent/root of the webhook payload.  The above change now allows users to more finely tune when a trigger is executed. + +## Prerequisites +Github webhook trigger should be set up and tested beforehand +Environment should be running 1.25.x or 2.25.x or higher. + +## Instructions +First, please look at the following snippet of the Webhook Payload from Github, truncated and sanitized of information   +``` +{ + "ref": "refs/heads/main", + "before": "9999999999x999x9999x9xxxx99999x99999xxxx9", + "after": "x9999999x99999x99999x99999xxxx99xxx9999x", + "repository": { + "id": 999999999, + "node_id": "XXXxXXXXxX9xxXXxxxxxXXxxXXX9Xxx=", + "name": "test-armoryTest", + "full_name": "armory/test-armoryTest", + "private": true, +[...] + "head_commit": [ + { + "id": "x9999999x99999x99999x99999xxxx99xxx9999x", +{...] + "added": [ + + ], + "removed": [ + + ], + "modified": [ + "TerraformFiles/File1.txt" + ] + } + ], +[...] +} +``` + +As can be seen in the example, the folder information is located within the commits information, customers previously would not be able to run any constraints based on the data located in the ```added```, ```removed```, or ```modified``` sections.  Thanks to the update, JSONPath can be used to access these values. + +### Setting up a Payload Constraint based on a Directory +Please read through the [documentation on setting up a webhook trigger](https://spinnaker.io/setup/triggers/github/) and that it is set up according to the documentation. The logic behind this set up is explained further in this [article in the OSS documentation](https://spinnaker.io/guides/user/pipeline/triggers/webhooks/). +In our example, the payload will be sent to the ```https:///webhooks/webhook/gitrepo``` location.  The ```gitrepo``` portion of the payload destination can be defined as a different value, but it should be noted down for Step 2b below.   +It is recommended that if users want to test their constraints, they can do so with a ***simple wait stage***.  Apply the trigger to the wait stage and test it out, before adding the constraint to the final pipeline.   +* Log in to the Spinnaker instance.  Proceed to the pipeline that will be configured with the trigger.To first test the set up, it is recommended that users test a trigger without any constraints.  In the configuration stage of the pipeline, go to the ```Automated Triggers``` Section +* Set the *Type* to *Webhook* Set the *Source* to the destination value when the webhook was set in Github.  (e.g. in our example, ```gitrepo```) +* Save the Pipeline +* Complete a PR request in the GitRepo and ensure the pipeline is triggered.  Please note that it can take a little time for the pipeline to complete the modifications so that the webhook will trigger the pipeline +* Once the test is completed successfully, it is possible to now add constraints.  Log into the pipelineIn the Automated Triggers, we will now define the Payload Constraints to trigger *based on any files modified in the directory* +* In the Key section, set the key using ```JsonPath expression``` ([https://github.com/json-path/JsonPath](https://github.com/json-path/JsonPath)). In our example, ```$.head_commit.modified``` will restrict based on values in the ```head_commit``` root member object, and the ```modified``` sub member object of the payload +* In the Value section, set it to the directory (e.g. in our example, ```TerraformFiles/*``` allowing us to trigger the pipeline based on any changes to the ```TerraformFiles``` folder in the repo)* If an ***AND*** condition needs included, add an additional payload constraint to this trigger +To provide an ***OR ***condition, since we would like to add a trigger *based on any files added into the directory* +* In the Key section, set the key using ```JSONPath expression``` again, but this time, in our example, ```$.head_commit.added``` will restrict based on values in the ```head_commit``` root member object, and the ```added``` sub member object of the payload* In the Value section, set it to the directory (e.g. in our example, ```TerraformFiles/*``` allowing us to trigger the pipeline based on any additional files added to the ```TerraformFiles``` folder in the repo) +* If an ***AND*** condition needs included, add an additional payload constraint to this trigger +* Save the pipeline* What should now happen is that if any files are modified or added to the directory specified, the webhook should trigger the pipeline.  Any files removed will not trigger the pipeline, nor will any changes anywhere else in the repo trigger the pipeline.  It is recommended that this be tested to ensure the pipeline trigger is functioning as expected +The [following information about JSONPath](https://github.com/json-path/JsonPath) can further assist users with defining constraints beyond the example provided.   Potentially, users can constrain webhooks based on any parts of the payload, including on changes to particular files, or changes made by particular users.   + + + diff --git a/kb/armory/General/k8s-v1.21-causing-valut-intergation-outages-with-kuberenetes-auth-method.md b/kb/armory/General/k8s-v1.21-causing-valut-intergation-outages-with-kuberenetes-auth-method.md new file mode 100644 index 0000000000..e51a7f86a3 --- /dev/null +++ b/kb/armory/General/k8s-v1.21-causing-valut-intergation-outages-with-kuberenetes-auth-method.md @@ -0,0 +1,53 @@ +--- +title: K8s v1.21 causing Valut intergation outages (with kuberenetes auth method) +--- + +## Issue +On ```Kubernetes 1.21```, the ServiceAccount issuer Discovery [feature](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-issuer-discovery) is on stable release and is enabled by default. + +This means that the ***JWT format of the service accounts is changing to have a more secure format***. + + +Previous format: +``` +{ + "iss": "kubernetes/serviceaccount", + "kubernetes.io/serviceaccount/namespace": "spinnaker", + "kubernetes.io/serviceaccount/secret.name": "test-token-5v2cp", + "kubernetes.io/serviceaccount/service-account.name": "test", + "kubernetes.io/serviceaccount/service-account.uid": "0ecb5560-7d43-4883-ae85-d07cf635d2d2", + "sub": "system:serviceaccount:spinnaker:test" +} +``` + +New format: +``` +{ + "aud": [ + "https://kubernetes.default.svc" + ], + "exp": 1661509326, + "iat": 1629973326, + "iss": "https://oidc.server.something", + "kubernetes.io": { + "namespace": "spinnaker", + "pod": { + "name": "debugging-tools-6464df994b-46wsq", + "uid": "90451169-29cb-4e2d-8ee8-4c1e2c293a3c" + }, + "serviceaccount": { + "name": "test", + "uid": "affc78ef-fa4b-4ba8-bb00-f9cc51d65408" + }, + "warnafter": 1629976933 + }, + "nbf": 1629973326, + "sub": "system:serviceaccount:spinnaker:test" +} +``` + +## Cause +This breaks the vault ***kubernetes auth method*** with vault throwing the message:```***ISS claim invalid***``` + +This is causing Spinnaker and Spinnaker-operator not to be able to retrieve secrets from Vault***``````*** + diff --git a/kb/armory/General/kayenta-404s-in-armory-spinnaker.md b/kb/armory/General/kayenta-404s-in-armory-spinnaker.md new file mode 100644 index 0000000000..f6edcbb7e6 --- /dev/null +++ b/kb/armory/General/kayenta-404s-in-armory-spinnaker.md @@ -0,0 +1,16 @@ +--- +title: Kayenta 404s in Armory Spinnaker +--- + + +If you’ve enabled Kayenta in Armory Spinnaker 2.0, and are running into any of these issues: +* Canary configs not saving properly* Infrastructure/Clusters page details pane not loading* 404s in the browser console attempting to hit ```/v2/canaryConfig``` +Then make the following change: +Create/update the ```.hal//profiles/gate-local.yml``` file, with these contents: +``` +services: + kayenta: + canaryConfigStore: true +``` +Then, apply your change with ```hal deploy apply```. + diff --git a/kb/armory/General/kubernetes-and-docker-accounts-are-not-visible-in-spinnaker-console-ui.md b/kb/armory/General/kubernetes-and-docker-accounts-are-not-visible-in-spinnaker-console-ui.md new file mode 100644 index 0000000000..d1c8212126 --- /dev/null +++ b/kb/armory/General/kubernetes-and-docker-accounts-are-not-visible-in-spinnaker-console-ui.md @@ -0,0 +1,45 @@ +--- +title: Kubernetes and Docker Accounts are not visible in Spinnaker Console UI +--- + +## Issue +Customers may find that when adding new Kubernetes accounts or new Docker Registry accounts and deploying the manifest, the accounts do not show up in the UI. +For Docker Registry accounts, they will show that the ```Registry Name``` is ```blank``` for possible options whenever referring to the account values: +For Kubernetes accounts, they will show that the ```Accounts``` field is ```blank``` for possible options whenever referring to the account values: +  +Attempting to access the Console UI with a new, flushed session (e.g in an Incognito Browser) and/or restarting services does not resolve the issue. The Clouddriver logs show that the account is not being picked up when restarting. For example the Kubernetes account ```test-aws-1``` or ```test-common``` for the Docker account are defined in the ```SpinnakerService``` custom resource, but are not visible in the logs. +```` +2021-12-16 15:13:33.240 INFO 1 --- [ main] .....hikari.HikariDataSource : tasks - Starting... + +2021-12-16 15:13:33.267 INFO 1 --- [ main] c....hikari.HikariDataSource : tasks - Start completed. + +2021-12-16 15:13:34.914 INFO 1 --- [ main] c.n.s.c.security.ProviderUtils : Adding accounts [docker-registry, ....registry, apm] of type DockerRegistryNamedAccountCredentials... + +2021-12-16 15:13:35.619 INFO 1 --- [ main] c.n.spinnaker.cats.sql.cache.SqlCache : Configured for kubernetes + +2021-12-16 15:13:35.619 INFO 1 --- [ main] c.n.spinnaker.cats.sql.cache.SqlCache : Configured for com.netflix.spinnaker.clouddriver.sql.SqlProvider +```` + +  +Similarly, the Clouddriver logs show that the Kubernetes accounts do not show ```test-aws-1```, but all other previously set accounts can be seen: +```` +2021-12-16 15:13:35.619 INFO 1 --- [ main] ....spinnaker.cats.sql.cache.SqlCache : Configured for kubernetes + +2021-12-16 15:13:35.619 INFO 1 --- [ main] ....spinnaker.cats.sql.cache.SqlCache : Configured for com.netflix.spinnaker.clouddriver.sql.SqlProvider + +2021-12-16 15:13:35.619 INFO 1 --- [ main] c.n.spinnaker.cats.sql.cache.SqlCache : Configured for com.netflix.spinnaker.clouddriver.sql.SqlProvider + +2021-12-16 15:13:35.620 INFO 1 --- [ main] c.n.spinnaker.cats.sql.cache.SqlCache : Configured for com.netflix.spinnaker.clouddriver.docker.registry.provider.DockerRegistryProvider + +2021-12-16 15:13:35.620 INFO 1 --- [ main] c.n.spinnaker.cats.sql.cache.SqlCache : Configured for com.netflix.spinnaker.clouddriver.core.provider.CoreProvider + +..... +2021-12-16 15:13:38.714 INFO 1 --- [ main] .s.KubernetesCredentialsLifecycleHandler : Adding 2 agents for new account a-sandbox + +2021-12-16 15:13:39.584 INFO 1 --- [ main] .s.KubernetesCredentialsLifecycleHandler : Adding 2 agents for new account b-stage +```` + + +## Cause +This issue can stem from a number of reasons such as incorrect account configuration, insufficient permissions, or limits set at the namespace level. + diff --git a/kb/armory/General/kubernetes-namespaces-still-appearing-after-deletion-in-spinnaker-ui.md b/kb/armory/General/kubernetes-namespaces-still-appearing-after-deletion-in-spinnaker-ui.md new file mode 100644 index 0000000000..b7b619b0e8 --- /dev/null +++ b/kb/armory/General/kubernetes-namespaces-still-appearing-after-deletion-in-spinnaker-ui.md @@ -0,0 +1,10 @@ +--- +title: Kubernetes Namespaces still appearing after deletion in Spinnaker UI +--- + +## Issue +Kubernetes accounts that are removed and deleted from Spinnaker configuration are still showing up in the UI with old resources. When attempting to interact with these resources there is an error and users cannot use them. + +## Cause +This can happen if the environment runs Clouddriver with an SQL backend.There is an issue with the SQL backend where it doesn't respect the resource TTLs ([https://github.com/spinnaker/spinnaker/issues/4803](https://github.com/spinnaker/spinnaker/issues/4803)). + diff --git a/kb/armory/General/kubernetes-service-name-change-creates-a-new-service-old-service-not-deleted.md b/kb/armory/General/kubernetes-service-name-change-creates-a-new-service-old-service-not-deleted.md new file mode 100644 index 0000000000..9c1db9216c --- /dev/null +++ b/kb/armory/General/kubernetes-service-name-change-creates-a-new-service-old-service-not-deleted.md @@ -0,0 +1,10 @@ +--- +title: Kubernetes Service Name Change Creates A New Service (old service not deleted) +--- + +## Issue +When modifying the k8s manifest to change the service name, a new service is getting generated. The old service is not getting deleted automatically. Manual service delete needs to be done. + +## Cause +This is working as expected.  Kubernetes does not delete the old service as it is a function that can break other functions elsewhere.  It will rely on a manual judgement from the user to complete the "cleanup" + diff --git a/kb/armory/General/kubernetes-v1-provider-removed.md b/kb/armory/General/kubernetes-v1-provider-removed.md new file mode 100644 index 0000000000..f67a9fc9ca --- /dev/null +++ b/kb/armory/General/kubernetes-v1-provider-removed.md @@ -0,0 +1,18 @@ +--- +title: Kubernetes V1 Provider Removed +--- + +## Issue +When you upgrade to Armory Spinnaker 2.21 and still use the Kubernetes V1 provider, you will encounter the following error: +``` +2020-07-31 22:11:55.146 WARN 1 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'configurationRefreshListener' defined in URL [jar:file:/opt/clouddriver/lib/clouddriver-web-6.10.0-20200625140019.jar!/com/netflix/spinnaker/clouddriver/listeners/ConfigurationRefreshListener.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'kubernetesV2ProviderSynchronizable': Invocation of init method failed; nested exception is java.lang.IllegalArgumentException: The legacy Kubernetes provider (V1) is no longer supported. Please migrate all Kubernetes accounts to the standard provider (V2). +``` + +This error is followed by the following error message that gets repeated: +``` +q2020-07-31 22:12:31.005 ERROR 1 --- [gentScheduler-0] c.n.s.c.r.c.ClusteredAgentScheduler : Unable to run agents redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool at redis.clients.jedis.util.Pool.getResource(Pool.java:59) ~[jedis-3.1.0.jar:na] at redis.clients.jedis.JedisPool.getResource(JedisPool.java:234) ~[jedis-3.1.0.jar:na] at com.netflix.spinnaker.kork.jedis.telemetry.InstrumentedJedisPool.getResource(InstrumentedJedisPool.java:60) ~[kork-jedis-7.45.9.jar:7.45.9] at com.netflix.spinnaker.kork.jedis.telemetry.InstrumentedJedisPool.getResource(InstrumentedJedisPool.java:26) ~[kork-jedis-7.45.9.jar:7.45.9] at com.netflix.spinnaker.kork.jedis.JedisClientDelegate.withCommandsClient(JedisClientDelegate.java:54) ~[kork-jedis-7.45.9.jar:7.45.9] at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.acquireRunKey(ClusteredAgentScheduler.java:183) ~[cats-redis-6.10.0-20200625140019.jar:6.10.0-20200625140019] at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.acquire(ClusteredAgentScheduler.java:136) ~[cats-redis-6.10.0-20200625140019.jar:6.10.0-20200625140019] at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.runAgents(ClusteredAgentScheduler.java:163) ~[cats-redis-6.10.0-20200625140019.jar:6.10.0-20200625140019] at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.run(ClusteredAgentScheduler.java:156) ~[cats-redis-6.10.0-20200625140019.jar:6.10.0-20200625140019] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na] at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[na:na] at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na] Caused by: java.lang.IllegalStateException: Pool not open at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:759) ~[commons-pool2-2.7.0.jar:2.7.0] at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:402) ~[commons-pool2-2.7.0.jar:2.7.0] at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349) ~[commons-pool2-2.7.0.jar:2.7.0] at redis.clients.jedis.util.Pool.getResource(Pool.java:50) ~[jedis-3.1.0.jar:na] ... 14 common frames omitted +``` + +## Cause +The Kubernetes V1 provider is removed in Armory Spinnaker 2.21. + diff --git a/kb/armory/General/lambda-creation-for-aws-exception--no-message-available.md b/kb/armory/General/lambda-creation-for-aws-exception--no-message-available.md new file mode 100644 index 0000000000..917d2c13ad --- /dev/null +++ b/kb/armory/General/lambda-creation-for-aws-exception--no-message-available.md @@ -0,0 +1,19 @@ +--- +title: Lambda Creation for AWS-Exception- No message available +--- + +## Issue +When creating Lambda functions in Spinnaker, users may see the following non-descriptive error: + +  +In checking Clouddriver after receiving the error, the following errors can be found in the logs: +WARN 1 --- [nio-7002-exec-9] c.n.s.k.w.e.GenericExceptionHandlers : Handled error in generic exception handler +com.netflix.spinnaker.clouddriver.orchestration.AtomicOperationConverterNotFoundException: null +... +com.netflix.spinnaker.clouddriver.orchestration.AnnotationsBasedAtomicOperationsRegistry.getAtomicOperationConverter(AnnotationsBasedAtomicOperationsRegistry.groovy:71) + + +## Cause +This error is related to the following Github issue request[https://github.com/spinnaker/spinnaker/issues/6355](https://github.com/spinnaker/spinnaker/issues/6355) +Which was subsequently looking to be resolved with the following PR:[https://github.com/spinnaker/clouddriver/pull/5272](https://github.com/spinnaker/clouddriver/pull/5272) + diff --git a/kb/armory/General/large-fiat-headers-can-cause-permission-errors-due-to-timeouts-and-400-errors..md b/kb/armory/General/large-fiat-headers-can-cause-permission-errors-due-to-timeouts-and-400-errors..md new file mode 100644 index 0000000000..7bb6345ac8 --- /dev/null +++ b/kb/armory/General/large-fiat-headers-can-cause-permission-errors-due-to-timeouts-and-400-errors..md @@ -0,0 +1,11 @@ +--- +title: Large Fiat headers can cause permission errors due to timeouts and 400 errors. +--- + +## Issue +If a header is passed to Fiat that is too big in size, it can cause errors in the logs such as below +```java.lang.IllegalArgumentException: Request header is too large``` + +## Cause +The default header size for Tomcat is only 8KB. If this limit is exceeded, Fiat will timeout and throw ```400 errors``` as it is unable to parse the entire header. + diff --git a/kb/armory/General/load-balancer-service-svc-spinnaker-demo-does-not-exist.md b/kb/armory/General/load-balancer-service-svc-spinnaker-demo-does-not-exist.md new file mode 100644 index 0000000000..57acaca9ec --- /dev/null +++ b/kb/armory/General/load-balancer-service-svc-spinnaker-demo-does-not-exist.md @@ -0,0 +1,13 @@ +--- +title: Load Balancer Service svc-spinnaker-demo Does Not Exist +--- + +## Issue +Deployments fail with the error when baking the manifest and deploy to prod.  +```Load balancer service svc-spinnaker-demo does not exist``` +or something very similar. This only happens when using a HELM chart, and if you copy the HELM chart into plain text to deploy it works. + +## Cause +While using a HELM chart as well as Red/Black or Highlander Deployment strategy, Kubernetes has a bug where it will not see your manifest properly and thus error out. This is a known issue. +[https://github.com/spinnaker/spinnaker/issues/5040#issuecomment-663736648](https://github.com/spinnaker/spinnaker/issues/5040#issuecomment-663736648) + diff --git "a/kb/armory/General/locked-pipelines-cannot-be-unlocked-except-by-admin-access-users-when-using-\"run-as\".md" "b/kb/armory/General/locked-pipelines-cannot-be-unlocked-except-by-admin-access-users-when-using-\"run-as\".md" new file mode 100644 index 0000000000..f8f1e8a1ba --- /dev/null +++ "b/kb/armory/General/locked-pipelines-cannot-be-unlocked-except-by-admin-access-users-when-using-\"run-as\".md" @@ -0,0 +1,14 @@ +--- +title: Locked Pipelines Cannot be Unlocked Except by Admin Access Users When Using "Run As" +--- + +## Issue +It is possible that Pipelines created with RunAs permissions will result in having restricted permissions to unlock a pipeline.  This can occur when creating a pipeline that has specific permissions for a user to RunAs a service account, either created by an overall admin, or by a Dinghyfile which is being used to create the pipeline.  The following is an error that can occur as a result +spin-front50 front50 2020-09-14 17:28:02.675 ERROR 1 --- [.0-8080-exec-10] c.n.s.f.c.AuthorizationSupport : User does not have access to service account +spin-front50 front50 2020-09-14 17:28:02.675 ERROR 1 --- [.0-8080-exec-10] n.s.f.s.FiatAccessDeniedExceptionHandler : Encountered exception while processing request POST: +spin-front50 front50 +spin-front50 front50 org.springframework.security.access.AccessDeniedException: Access is denied + +## Cause +Permissions are inadequate for a multitude of reasons.  The user who is performing the ```RunAs``` may not be a member of the correct group to allow access to the Service Account.  It is also possible that the Service Account that is being ```RunAs``` may not be the correct one, so it is recommended that you check with the Spinnaker Administrator and AD Admin to ensure the correct one was declared.   + diff --git a/kb/armory/General/long-force-cache-refresh-kubernetes.md b/kb/armory/General/long-force-cache-refresh-kubernetes.md new file mode 100644 index 0000000000..4baf3d50dc --- /dev/null +++ b/kb/armory/General/long-force-cache-refresh-kubernetes.md @@ -0,0 +1,10 @@ +--- +title: Long Force Cache Refresh (Kubernetes) +--- + +## Issue +A long Kubernetes manifest deploy, scale, or undo rollout stage, and more specifically a ```Force Cache Refresh``` task that lasts 12 minutes is a sign of Orca being unable to acknowledge the cache refresh that occurred in Clouddriver + +## Cause +Several stages require that the target manifest be updated in the cache and do it via a ```Force Cache Refresh``` task. The ```manifestName``` is a representation of the manifest you’re targeting that Orca will use when inspecting cached values (from Clouddriver). It is made of the Kubernetes kind and the manifest name. Earlier versions of Spinnaker were incorrectly saving the kind as is. + diff --git "a/kb/armory/General/manual-installation-of-project-aurora-for-spinnaker\342\204\242-no-helm.md" "b/kb/armory/General/manual-installation-of-project-aurora-for-spinnaker\342\204\242-no-helm.md" new file mode 100644 index 0000000000..b4de2ccc79 --- /dev/null +++ "b/kb/armory/General/manual-installation-of-project-aurora-for-spinnaker\342\204\242-no-helm.md" @@ -0,0 +1,225 @@ +--- +title: Manual Installation of Project Aurora for Spinnaker™ (No Helm) +--- + +## Introduction +**If Helm cannot be used in the environment**, it is possible to install the dependencies for Project Aurora for Spinnaker™ manually. +This document walks through installing the Remote Network Agent (RNA) and the Argo Rollouts Controller, which are both required for Project Aurora. If the target deployment cluster already has Argo Rollouts installed, that section of the installation can be skipped. + +## Prerequisites +### Before Beginning +Complete all steps in [Get Started with Project Aurora for Spinnaker™](https://docs.armory.io/docs/installation/aurora-install/) up to the section titled [Enable Aurora in target Kubernetes clusters](https://docs.armory.io/docs/installation/aurora-install/#enable-aurora-in-target-kubernetes-clusters). +### **Only proceed with this document if Helm cannot be used to complete the section Enable Aurora in target Kubernetes clusters.** + +## Instructions +## Create a namespace +In the target cluster where apps will be deployed, create a namespace for the Remote Network Agent: +```kubectl create ns aurora``` +The examples on this page assume a namespace called ```aurora``` for the Remote Network Agent installation. Replace the namespace in the examples if using a different namespace. +## ****Install Argo Rollouts +Project Aurora for Spinnaker™ requires that Argo Rollouts Controller 1.x or later is installed in each target Kubernetes cluster along with the Armory Agent. +For information about how to install Argo Rollouts, see [Controller Install](https://argoproj.github.io/argo-rollouts/installation/#controller-installation)[ation](https://argoproj.github.io/argo-rollouts/installation/#controller-installation) in the Argo documentation. +## ****Configure permissions +Create a ```ClusterRole```, ```ServiceAccount```, and ```ClusterRoleBinding``` for the Remote Network Agent by applying the following manifest to the ```aurora``` namespace: +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: spin-cluster-role +rules: +- apiGroups: + - "" + resources: + - pods + - pods/log + - ingresses/status + - endpoints + verbs: + - get + - list + - update + - patch + - delete +- apiGroups: + - "" + resources: + - services + - services/finalizers + - events + - configmaps + - secrets + - namespaces + - ingresses + - jobs + verbs: + - create + - get + - list + - update + - watch + - patch + - delete +- apiGroups: + - batch + resources: + - jobs + verbs: + - create + - get + - list + - update + - watch + - patch +- apiGroups: + - apps + - extensions + resources: + - deployments + - deployments/finalizers + - deployments/scale + - daemonsets + - replicasets + - replicasets/finalizers + - replicasets/scale + - statefulsets + - statefulsets/finalizers + - statefulsets/scale + verbs: + - create + - get + - list + - update + - watch + - patch + - delete +- apiGroups: + - monitoring.coreos.com + resources: + - servicemonitors + verbs: + - get + - create +- apiGroups: + - spinnaker.armory.io + resources: + - '*' + - spinnakerservices + verbs: + - create + - get + - list + - update + - watch + - patch +- apiGroups: + - admissionregistration.k8s.io + resources: + - validatingwebhookconfigurations + verbs: + - '*' +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + namespace: aurora + name: spin-sa +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: spin-cluster-role-binding +subjects: + - kind: ServiceAccount + name: spin-sa + namespace: aurora +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: spin-cluster-role +## ****Configure the Remote Network Agent +For information about adding accounts, see the ```kubernetes.accounts[]``` options in the [Agent Configuration documentation](https://docs.armory.io/docs/armory-agent/agent-options/#configuration-options). +apiVersion: v1 +kind: ConfigMap +metadata: + name: armory-agent-config + namespace: aurora +data: + armory-agent.yaml: | + hub: + connection: + grpc: agents.cloud.armory.io:443 + auth: + armory: + clientId: ${CLIENT_ID_FOR_AGENT_FROM_ABOVE} + secret: ${CLIENT_SECRET_FOR_AGENT_FROM_ABOVE} + tokenIssuerUrl: https://auth.cloud.armory.io/oauth/token + audience: https://api.cloud.armory.io + verify: true + kubernetes: + accounts: [] +## ****Deploy the Remote Network Agent +Apply the following Remote Network Agent deployment manifest to the namespace created on the target cluster for the Agent (```aurora``` for the examples on this page): +apiVersion: apps/v1 +kind: Deployment +metadata: + labels: + app: spin + app.kubernetes.io/name: armory-agent + app.kubernetes.io/part-of: spinnaker + cluster: spin-armory-agent + name: spin-armory-agent +spec: + replicas: 1 + selector: + matchLabels: + app: spin + cluster: spin-armory-agent + template: + metadata: + labels: + app: spin + app.kubernetes.io/name: armory-agent + app.kubernetes.io/part-of: spinnaker + cluster: spin-armory-agent + spec: + serviceAccount: spin-sa + containers: + - image: armory/agent-kubernetes:0.1.3 + imagePullPolicy: IfNotPresent + name: armory-agent + env: + - name: ARMORY_HUB + value: "true" + ports: + - name: health + containerPort: 8082 + protocol: TCP + - name: metrics + containerPort: 8008 + protocol: TCP + readinessProbe: + httpGet: + port: health + path: /health + failureThreshold: 3 + periodSeconds: 10 + successThreshold: 1 + timeoutSeconds: 1 + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + volumeMounts: + - mountPath: /opt/spinnaker/config + name: volume-armory-agent-config + # - mountPath: /kubeconfigfiles + # name: volume-armory-agent-kubeconfigs + restartPolicy: Always + volumes: + - name: volume-armory-agent-config + configMap: + name: armory-agent-config + # - name: volume-armory-agent-kubeconfigs + # secret: + # defaultMode: 420 + # secretName: kubeconfigs-secret +## Next steps +After completing the above steps, return to [Get Started with Project Aurora for Spinnaker™](https://docs.armory.io/docs/installation/aurora-install/) and continue from [Verify the Agent Deployment](https://docs.armory.io/docs/installation/aurora-install/#verify-the-agent-deployment). + diff --git a/kb/armory/General/metric-data-does-not-flow-into-splunk-despite-setting-up-hec-connection.md b/kb/armory/General/metric-data-does-not-flow-into-splunk-despite-setting-up-hec-connection.md new file mode 100644 index 0000000000..4ca36eca21 --- /dev/null +++ b/kb/armory/General/metric-data-does-not-flow-into-splunk-despite-setting-up-hec-connection.md @@ -0,0 +1,10 @@ +--- +title: Metric data does not flow into Splunk despite setting up HEC Connection +--- + +## Issue +Customers setting up an HEC connection [https://docs.splunk.com/Documentation/Splunk/8.2.4/Data/UsetheHTTPEventCollector](https://docs.splunk.com/Documentation/Splunk/8.2.4/Data/UsetheHTTPEventCollector) with Spinnaker Logging may encounter an issue where data is not being pushed to the connector + +## Cause +HEC connector with default ports uses 8088 or 8089 depending on the Splunk version by default.  This will happen if you do not configure a TCP listener, and can cause issues in Splunk with attaining data if the port is already in use + diff --git a/kb/armory/General/migrate-clouddriver's-redis-to-its-own-instance.md b/kb/armory/General/migrate-clouddriver's-redis-to-its-own-instance.md new file mode 100644 index 0000000000..1db6196a9f --- /dev/null +++ b/kb/armory/General/migrate-clouddriver's-redis-to-its-own-instance.md @@ -0,0 +1,10 @@ +--- +title: Migrate Clouddriver's Redis To Its Own Instance +--- + + +* Take a snapshot of your existing Redis cluster* Create a new cluster with the snapshot from the previous step. This will include all the keys from the previous cluster that you can clear out later.Configure Clouddriver to use the new cluster. In your ```/opt/spinnaker/config/clouddriver-local.yml``` add the following: +redis: + connection: redis://${YOUR_REDIS_HOST}:6379​ +* Redeploy your Spinnaker instance + diff --git a/kb/armory/General/migrating-to-the-policy-engine-plugin-from-the-policy-engine-extension.md b/kb/armory/General/migrating-to-the-policy-engine-plugin-from-the-policy-engine-extension.md new file mode 100644 index 0000000000..68cdd03ad0 --- /dev/null +++ b/kb/armory/General/migrating-to-the-policy-engine-plugin-from-the-policy-engine-extension.md @@ -0,0 +1,39 @@ +--- +title: Migrating to the Policy Engine Plugin from the Policy Engine Extension +--- + +## Introduction +With the [deprecation of the Policy Engine Extension](https://docs.armory.io/docs/armory-admin/policy-engine-enable/policy-engine-ext-enable/) and its replacement with the Policy Engine Plugin, customers who are still using the Policy Engine Extension can use the following information to move to the Policy Engine Plugin.  The Policy Engine Plugin allows for Armory to provide a more robust set of features to our customers, and is a more further developed solution towards enabling policies in their Spinnaker environment.Plugins require access to the external resource repository to remain updated, but if the environment is air-gapped, please take a look at the following KB article that advises about [how to deploy plugins in air-gapped situations](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010314) + +## Prerequisites +Administration access to the Spinnaker Environment + +## Instructions +To migrate to the Policy Engine Plugin from the extension, perform the following steps: +Turn off the Policy Engine extension. You cannot run both the extension and plugin at the same time. You must disable the extension project and then enable the plugin. +**Halyard**: Remove the following snippet from ```.hal/default/profiles/spinnaker-local.yml```: +armory: + opa: + enabled: true + url: :/v1​ +**Operator**: Remove the ```opa``` blocks from the Front50 and Clouddriver sections of your ```SpinnakerService``` manifest: +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker +spec: + spinnakerConfig: + profiles: + front50: #Enables Save time validation of policies + armory: + opa: + enabled: true + url: :/v1 + clouddriver: #Enables Runtime validation of policies + armory: + opa: + enabled: true + url: :/v1​ + +* If you redeploy Armory Enterprise and apply these changes before you enable the Policy Engine Plugin, no policies get enforced. * Enable the [Policy Engine Plugin](https://docs.armory.io/docs/armory-admin/policy-engine-enable/policy-engine-plug-enable/). If you have an existing OPA server with policies that you want to use, you can provide that OPA server in the plugin configuration. You do not need to create a new OPA server or migrate your policies. + diff --git a/kb/armory/General/missing-file-in-rosco-image-got-exception-create-bake.md b/kb/armory/General/missing-file-in-rosco-image-got-exception-create-bake.md new file mode 100644 index 0000000000..26af3bd359 --- /dev/null +++ b/kb/armory/General/missing-file-in-rosco-image-got-exception-create-bake.md @@ -0,0 +1,49 @@ +--- +title: Missing file in rosco image got Exception (Create Bake) +--- + +## Issue +Spinnaker is open-source and can be customized by anyone. +One scenario is a customer customized Spinnaker and upgraded from version 2.27.x to 2.28.0. When running Bake (MANIFEST) stage, an exception shows: +Exception (Create Bake) +Bake failed: Execution failed (Exit value: -559038737. Caused by java.io.IOException: Cannot run program "kustomize" (in directory "."): error=13, Permission denied) + +  + +## Cause +This exception shows a permission denied, but the root reason is that the ```kustomize``` tool is not in the rosco container. +  +**How to check:** +  +1. Acccess into spin-rosco container, for example: +```kubectl exec -it spin-rosco-7c6d6444c8-t6w5h -- sh``` +In the spin-rosco container, you will be directed to the directory ```/packer```. +```/packer $``` +Under ```/packer``` directory, there are ```packer``` tool and directory ```kustimize``` +  +2. Check what inside ```/packer``` directory +/packer $``` ls -lart``` + +total 124000 +-rwxr-xr-x 1 spinnake nogroup 126976000 Jul 11 18:17 packer +drwxr-xr-x 1 spinnake nogroup 23 Jul 11 18:17 kustomize +drwxr-xr-x 1 spinnake nogroup 37 Jul 11 18:17 . +drwxr-xr-x 1 root root 51 Nov 4 19:23 .. +/packer $ + + +  +3. Access into directory ```kustimize```, you suppose be able to see the ```kustimize``` tool. + +/packer $ cd kustomize/ +/packer/kustomize $ ls -lart +total 33076 +-rwxr-xr-x 1 spinnake nogroup 33869824 Oct 8 2020 kustomize +drwxr-xr-x 1 spinnake nogroup 37 Jul 11 18:17 .. +drwxr-xr-x 1 spinnake nogroup 23 Jul 11 18:17 . +/packer/kustomize $ + + +  +In the above failure case's spin-rosco container, there were no directory ```kustimize``` and  ```kustimize``` tool showed. + diff --git a/kb/armory/General/missing-run-as-user-field-for-triggers-in-spinnaker-ui.md b/kb/armory/General/missing-run-as-user-field-for-triggers-in-spinnaker-ui.md new file mode 100644 index 0000000000..e961a73f9f --- /dev/null +++ b/kb/armory/General/missing-run-as-user-field-for-triggers-in-spinnaker-ui.md @@ -0,0 +1,12 @@ +--- +title: Missing Run As User field for triggers in Spinnaker UI +--- + +## Issue +Customers may find that the** **```Run As User``` field under Automated Triggers is missing in UI and has the Permissions field instead. + +  + +## Cause +When the pipeline permissions is enabled, it is expected that ```Run As User``` field in Spinnaker UI is not visible. + diff --git a/kb/armory/General/monitored-deploy-rollout.md b/kb/armory/General/monitored-deploy-rollout.md new file mode 100644 index 0000000000..a7608dc65f --- /dev/null +++ b/kb/armory/General/monitored-deploy-rollout.md @@ -0,0 +1,15 @@ +--- +title: Monitored Deploy Rollout +--- + +## Introduction +Monitored deploy strategy operated like **RollingRedBlack** with the addition that it allows for 3rd party service (**DeploymentMonitor**) to provide input into the progress of the deploy. +After every step, Spinnaker will call into the DeploymentMonitor specified in the stage (and configured in ```orca.yml```).The DeploymentMonitor can then perform analysis on the newly deployed instances and provide input on whether the deployment should continue or not. + +## Prerequisites +N/A + +## Instructions +In order to use this feature, admins must have the config flag enabled as well as manually craft json for the deploy stage. +Here is a link from OSS 1.17 when the strategy was introduced that references more of the code. [https://github.com/spinnaker/orca/commit/a1fb5283840caaee01a1bec8824053fcb70e6205](https://github.com/spinnaker/orca/commit/a1fb5283840caaee01a1bec8824053fcb70e6205) + diff --git a/kb/armory/General/mysql-communication-failure-when-using-tls1.1.md b/kb/armory/General/mysql-communication-failure-when-using-tls1.1.md new file mode 100644 index 0000000000..df4f12437a --- /dev/null +++ b/kb/armory/General/mysql-communication-failure-when-using-tls1.1.md @@ -0,0 +1,23 @@ +--- +title: MySQL communication failure when using TLS1.1 +--- + +## Issue +An organization may run into failures when components attempt communication with a MySQL database service via TLS1.1 on newer versions of Spinnaker.This connectivity issue may manifest in different ways, since a TLS-related error message may not always explicitly be shown. As a result, Spinnaker services of varying degree may end up failing as they are upgraded to a newer Java version. +This is an issue that [originates due to changes in JVM](https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-reference-using-ssl.html) and deprecations that have been announced, and it is not strictly speaking, an Armory/Spinnaker Issue. +Sample error: +spin-orca-56cbc4f4d-zg6tk orca Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate) +spin-orca-56cbc4f4d-zg6tk orca at java.base/sun.security.ssl.HandshakeContext.(HandshakeContext.java:170) +spin-orca-56cbc4f4d-zg6tk orca at java.base/sun.security.ssl.ClientHandshakeContext.(ClientHandshakeContext.java:103) +spin-orca-56cbc4f4d-zg6tk orca at java.base/sun.security.ssl.TransportContext.kickstart(TransportContext.java:222) +spin-orca-56cbc4f4d-zg6tk orca at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:449) +spin-orca-56cbc4f4d-zg6tk orca at com.mysql.cj.protocol.ExportControlled.performTlsHandshake(ExportControlled.java:336) +spin-orca-56cbc4f4d-zg6tk orca at com.mysql.cj.protocol.StandardSocketFactory.performTlsHandshake(StandardSocketFactory.java:188) +spin-orca-56cbc4f4d-zg6tk orca at com.mysql.cj.protocol.a.NativeSocketConnection.performTlsHandshake(NativeSocketConnection.java:99) +spin-orca-56cbc4f4d-zg6tk orca at com.mysql.cj.protocol.a.NativeProtocol.negotiateSSLConnection(NativeProtocol.java:329) +spin-orca-56cbc4f4d-zg6tk orca ... 168 common frames omitted + +## Cause +With TLS 1.1 approaching an end-of-life deprecation, it was discovered that Java 11.0.11 removes certain cyphers that would enable TLS1.1 to work. +This entails that any component that communicates using TLS1.1 will fail (by default). It was also observed that the ***communication did not auto-negotiate to TLS1.2***, even though it is supposedly supported by the version of the MySQL drivers that were tested (8.0.19+). + diff --git a/kb/armory/General/mysql-table-name-change-error-when-rolling-back-spinnaker-undo-renamed-values.md b/kb/armory/General/mysql-table-name-change-error-when-rolling-back-spinnaker-undo-renamed-values.md new file mode 100644 index 0000000000..80926b5588 --- /dev/null +++ b/kb/armory/General/mysql-table-name-change-error-when-rolling-back-spinnaker-undo-renamed-values.md @@ -0,0 +1,18 @@ +--- +title: MySQL Table Name Change Error When Rolling Back Spinnaker (Undo Renamed Values) +--- + +## Issue +Rolling back Spinnaker versions causes pods to continually have errors in the logs, including the following first example for Front50 (example is rolling back from 2.19.x to 2.18.x +```` +2020-06-22 21:52:27.001 INFO 1 --- [ main] .s.f.m.p.DefaultPluginArtifactRepository : Warming Cache +2020-06-22 21:52:27.723 ERROR 1 --- [ main] .s.f.m.p.DefaultPluginArtifactRepository : Unable to warm cache: {} + +org.springframework.jdbc.BadSqlGrammarException: jOOQ; bad SQL grammar [select max(last_modified_at) as `last_modified_at` from plugin_artifacts]; nested exception is java.sql.SQLSyntaxErrorException: Table 'front50_kinnon.plugin_artifacts' doesn't exist + at org.jooq_3.12.3.MYSQL.debug(Unknown Source) ~[na:na] + at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:93) ~[spring-jdbc-5.1.14.RELEASE.jar:5.1.14.RELEASE] +```` + +## Cause +When upgrading, the values change, and when reverting, the values do not change back to the old values, thus errors will occur accessing the database.Example from 2.18.x to 2.19.x, ```plugin_artifacts``` table does get renamed to ```plugin_info``` and does not revert back when downgrading.  + diff --git a/kb/armory/General/network-latency-between-clouddriver-and-agent-causing-timeout-issues.md b/kb/armory/General/network-latency-between-clouddriver-and-agent-causing-timeout-issues.md new file mode 100644 index 0000000000..7d7b068b59 --- /dev/null +++ b/kb/armory/General/network-latency-between-clouddriver-and-agent-causing-timeout-issues.md @@ -0,0 +1,60 @@ +--- +title: Network latency between Clouddriver and Agent causing Timeout issues +--- + +## Issue +Users may run into failures because of network latency between Clouddriver and the Agent instances.  Users may observe the following error within the Clouddriver logs when executing a pipeline on a Spinnaker instance that uses Armory Agent for Kubernetes: +message="com.netflix.spinnaker.clouddriver.exceptions.OperationTimedOutException: Timeout exceeded for operation 01FHVJ66QXEHX51DMY9PDSTPY0, type: 6, account: xxx, time: 30022 ms, timeout: 30000 ms + at io.armory.kubesvc.services.ops.KubesvcOperations.performOperation(KubesvcOperations.java:96) + at io.armory.kubesvc.services.ops.cluster.ClusteredKubesvcOperations.performOperation(ClusteredKubesvcOperations.java:70) + at io.armory.kubesvc.util.OperationUtils.perform(OperationUtils.java:76) + at io.armory.kubesvc.services.ops.executor.KubesvcExecutor.deploy(KubesvcExecutor.java:301) + at jdk.internal.reflect.GeneratedMethodAccessor703.invoke(Unknown Source) + at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + at java.base/java.lang.reflect.Method.invoke(Method.java:566) + at io.armory.kubesvc.services.registration.clouddriver.KubesvcCredentialsParser.lambda$makeKubesvcJobExecutor$0(KubesvcCredentialsParser.java:72) + at com.netflix.spinnaker.clouddriver.kubernetes.op.job.KubectlJobExecutor$$EnhancerByCGLIB$$4f088312.deploy() + at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.lambda$deploy$14(KubernetesCredentials.java:508) + at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.runAndRecordMetrics(KubernetesCredentials.java:618) + at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.runAndRecordMetrics(KubernetesCredentials.java:603) + at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.deploy(KubernetesCredentials.java:504) + at com.netflix.spinnaker.clouddriver.kubernetes.op.handler.CanDeploy.deploy(CanDeploy.java:58) + at com.netflix.spinnaker.clouddriver.kubernetes.op.manifest.KubernetesDeployManifestOperation.operate(KubernetesDeployManifestOperation.java:209) + at com.netflix.spinnaker.clouddriver.kubernetes.op.manifest.KubernetesDeployManifestOperation.operate(KubernetesDeployManifestOperation.java:46) + at com.netflix.spinnaker.clouddriver.orchestration.AtomicOperation$operate.call(Unknown Source) + at com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1$_closure2.doCall(DefaultOrchestrationProcessor.groovy:124) + at com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1$_closure2.doCall(DefaultOrchestrationProcessor.groovy) + at jdk.internal.reflect.GeneratedMethodAccessor543.invoke(Unknown Source) + at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + at java.base/java.lang.reflect.Method.invoke(Method.java:566) + at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101) + at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323) + at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263) + at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041) + at groovy.lang.Closure.call(Closure.java:405) + at groovy.lang.Closure.call(Closure.java:399) + at com.netflix.spinnaker.clouddriver.metrics.TimedCallable$CallableWrapper.call(TimedCallable.java:81) + at com.netflix.spinnaker.clouddriver.metrics.TimedCallable.call(TimedCallable.java:45) + at java_util_concurrent_Callable$call.call(Unknown Source) + at com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1.doCall(DefaultOrchestrationProcessor.groovy:123) + at com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1.doCall(DefaultOrchestrationProcessor.groovy) + at jdk.internal.reflect.GeneratedMethodAccessor537.invoke(Unknown Source) + at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + at java.base/java.lang.reflect.Method.invoke(Method.java:566) + at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101) + at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323) + at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263) + at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041) + at groovy.lang.Closure.call(Closure.java:405) + at groovy.lang.Closure.call(Closure.java:399) + at com.netflix.spinnaker.security.AuthenticatedRequest.lambda$wrapCallableForPrincipal$0(AuthenticatedRequest.java:272) + at com.netflix.spinnaker.clouddriver.metrics.TimedCallable.call(TimedCallable.java:45) + at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) + at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) + at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) + at java.base/java.lang.Thread.run(Thread.java:829)" + + +## Cause +After forwarding the ```deploy operation``` to the Agent, Clouddriver waits for the Agent to respond within 30 seconds.  However, if the response from the Agent takes more than 30 sec, then users may find the error above.  Administrators can override the default value of 30 seconds. + diff --git a/kb/armory/General/nexus--could-not-parse-error-response-unexpected-character-'<'-code-60--expected-a-valid-value-.md b/kb/armory/General/nexus--could-not-parse-error-response-unexpected-character-'<'-code-60--expected-a-valid-value-.md new file mode 100644 index 0000000000..0d044f7940 --- /dev/null +++ b/kb/armory/General/nexus--could-not-parse-error-response-unexpected-character-'<'-code-60--expected-a-valid-value-.md @@ -0,0 +1,18 @@ +--- +title: Nexus- Could not parse error response Unexpected character ('<' (code 60))- expected a valid value +--- + +## Issue +Customers may experience the following when trying to use Spinnaker with Nexus.  They may find that there is a 401 Unauthorized response. +Clouddriver shows that the caching agent logs contain ```operative-docker``` registries with no errors: +2021-10-19 05:30:36.441 INFO 1 --- [ecutionAction-8] .d.r.p.a.DockerRegistryImageCachingAgent : Describing items in operative-docker/DockerRegistryImageCachingAgent[1/2] +2021-10-19 05:30:36.450 INFO 1 --- [ecutionAction-8] .d.r.p.a.DockerRegistryImageCachingAgent : Caching 2474 tagged images in operative-docker/DockerRegistryImageCachingAgent[1/2] +2021-10-19 05:30:36.450 INFO 1 --- [ecutionAction-8] .d.r.p.a.DockerRegistryImageCachingAgent : Caching 2474 image ids in operative-docker/DockerRegistryImageCachingAgent[1/2] +  +Below are messages that were provided by ```operative.com``` from Nexus.  No other errors can be found via Spinnaker.  +2021-09-12 06:17:42,136+0000 WARN ....] .....org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl - Could not parse error response Unexpected character (' +  + +## Cause +This could be as a result of a disabled account on Nexus.  Customers should look to review and audit their Nexus accounts. + diff --git a/kb/armory/General/no-expected-artifacts-field.md b/kb/armory/General/no-expected-artifacts-field.md new file mode 100644 index 0000000000..82ec8a8d46 --- /dev/null +++ b/kb/armory/General/no-expected-artifacts-field.md @@ -0,0 +1,10 @@ +--- +title: No "Expected Artifacts" Field +--- + +## Issue +Spinnaker is installed, and users are trying to configure artifacts, but don’t see the ```Expected Artifacts``` field in the configuration tab of the Spinnaker pipelines + +## Cause +Functions need to be enabled in order for them to appear available in Deck. + diff --git "a/kb/armory/General/no-ingress-is-supported-when-using-api-version-\"networking.k8s.io-v1\".md" "b/kb/armory/General/no-ingress-is-supported-when-using-api-version-\"networking.k8s.io-v1\".md" new file mode 100644 index 0000000000..4670d99726 --- /dev/null +++ "b/kb/armory/General/no-ingress-is-supported-when-using-api-version-\"networking.k8s.io-v1\".md" @@ -0,0 +1,29 @@ +--- +title: No ingress is supported when using API version "networking.k8s.io/v1" +--- + +## Issue +Customers may encounter issues with their Spinnaker environment when making adjustments to the API version for incoming (Ingress) network traffic to their Spinnaker environment. +The following error, for example, may be found in CloudDriver logs: +```` +com.netflix.spinnaker.clouddriver.kubernetes.op.handler.UnsupportedVersionException: No ingress is supported at api version networking.k8s.io/v1 +at com.netflix.spinnaker.clouddriver.kubernetes.op.handler.KubernetesIngressHandler.attachedServices(KubernetesIngressHandler.java:122) ~[clouddriver-kubernetes.jar:na] +at com.netflix.spinnaker.clouddriver.kubernetes.op.handler.KubernetesIngressHandler.addRelationships(KubernetesIngressHandler.java:108) ~[clouddriver-kubernetes.jar:na] +at com.netflix.spinnaker.clouddriver.kubernetes.caching.agent.KubernetesCachingAgent.lambda$loadSecondaryResourceRelationships$5(KubernetesCachingAgent.java:255) ~[clouddriver-kubernetes.jar:na] +at java.base/java.util.HashMap$KeySet.forEach(HashMap.java:928) ~[na:na] +at com.netflix.spinnaker.clouddriver.kubernetes.caching.agent.KubernetesCachingAgent.loadSecondaryResourceRelationships(KubernetesCachingAgent.java:248) ~[clouddriver-kubernetes.jar:na] +at com.netflix.spinnaker.clouddriver.kubernetes.caching.agent.KubernetesCachingAgent.buildCacheResult(KubernetesCachingAgent.java:209) ~[clouddriver-kubernetes.jar:na] +at com.netflix.spinnaker.clouddriver.kubernetes.caching.agent.KubernetesCachingAgent.loadData(KubernetesCachingAgent.java:192) ~[clouddriver-kubernetes.jar:na] +at com.netflix.spinnaker.cats.agent.CachingAgent$CacheExecution.executeAgentWithoutStore(CachingAgent.java:87) ~[clouddriver-api.jar:na] +at com.netflix.spinnaker.cats.agent.CachingAgent$CacheExecution.executeAgent(CachingAgent.java:77) ~[clouddriver-api.jar:na] +at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler$AgentExecutionAction.execute(ClusteredAgentScheduler.java:338) ~[cats-redis.jar:na] +at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler$AgentJob.run(ClusteredAgentScheduler.java:308) ~[cats-redis.jar:na] +at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na] +at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na] +at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na] +at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na] +at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na] +```` +## Cause +Based on the OSS source code, CloudDriver older than release 1.26.x/2.26.x does not support incoming traffic from ```networking.k8s.io/v1``` + diff --git a/kb/armory/General/no-plugins-error-on-service-startup.md b/kb/armory/General/no-plugins-error-on-service-startup.md new file mode 100644 index 0000000000..0fa6f2398f --- /dev/null +++ b/kb/armory/General/no-plugins-error-on-service-startup.md @@ -0,0 +1,14 @@ +--- +title: No plugins error on service startup +--- + +## Issue +When starting a service with plugins, the following error appears: +``` +org.pf4j.util.FileUtils : Expanded plugin zip 'armory-observability-plugin-v1.2.0.zip' in 'armory-observability-plugin-v1.2.0' +org.pf4j.AbstractPluginManager : No plugins +``` + +## Cause +Service starts with a different User ID vs Spinnaker User ID and is unable to run certain commands (eg. unzipping the plugin folder). + diff --git a/kb/armory/General/nodeselector-toleration-settings-do-not-get-propagated-while-using-operator.md b/kb/armory/General/nodeselector-toleration-settings-do-not-get-propagated-while-using-operator.md new file mode 100644 index 0000000000..715daeea4d --- /dev/null +++ b/kb/armory/General/nodeselector-toleration-settings-do-not-get-propagated-while-using-operator.md @@ -0,0 +1,13 @@ +--- +title: nodeSelector/toleration settings do not get propagated while using Operator +--- + +## Issue +Customers may find that that in a single-manifest deployment of Armory while utilizing Operator, when attempting to move the Armory pods to specific nodes with nodeSelectors and tolerations, the settings do not get propagated to the Deployment or Pods. +The configuration is added for nodeSelector/toleration under ```spec.spinnakerConfig.service-settings```, however the change is not reflected in the Deployment or Pods. + +## Cause +This may be due to a prior bug in the [Spinnaker documentation](https://spinnaker.io/docs/reference/halyard/custom/#nodeselectors) (which specifies nodeSelector**s**, not nodeSelector). +The Spinnaker docs had a typo due to which ```nodeSelector``` did not propagate. ```nodeSelector:``` is the  field that was supported by Kubernetes. +  + diff --git a/kb/armory/General/nouniquebeandefinitionexception-error-with-armory-cloud-enabled.md b/kb/armory/General/nouniquebeandefinitionexception-error-with-armory-cloud-enabled.md new file mode 100644 index 0000000000..0add65a0ae --- /dev/null +++ b/kb/armory/General/nouniquebeandefinitionexception-error-with-armory-cloud-enabled.md @@ -0,0 +1,21 @@ +--- +title: NoUniqueBeanDefinitionException error with Armory Cloud enabled +--- + +## Issue +When upgrading from ***2.25.x*** to ***2.26.x***, Clouddriver, Echo, Front50, Igor fail to startup.Clouddriver logs show the following: +2021-10-18 16:10:04.637 WARN 1 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'catsSqlAdminController' defined in URL [jar:file:/opt/clouddriver.... + + +2021.09.17.18.53.46.release-1.26.x.jar!/com/netflix/spinnaker/cats/sql/controllers/CatsSqlAdminController.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'fiatPermissionEvaluator' defined in URL [jar:file:/opt......fiat-api-1.27.0.jar!/com/netflix/spinnaker/fiat/shared/FiatPermissionEvaluator.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'fiatService' defined in class path resource [com/netflix/spinnaker/fiat/shared/FiatAuthenticationConfig.class]: Unsatisfied dependency expressed through method 'fiatService' parameter 1; nested exception is org.springframework.beans.factory.NoUniqueBeanDefinitionException: No qualifying bean of type 'com.netflix.spinnaker.okhttp.SpinnakerRequestInterceptor' available: expected single matching bean but found 2: armoryCloudRequestInterceptor,spinnakerRequestInterceptor +Echo logs show the following: +```2021-10-18 16:29:55.282 WARN 1 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'pipelineTriggerJob' defined in URL [jar:file:/opt/echo/lib/echo-scheduler-2021.04.29.16.51.35.release-1.26.x.jar!/com/netflix/spinnaker/echo/scheduler/actions/pipeline/PipelineTriggerJob.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'pipelineInitiator' defined in URL [jar:file:/opt/echo/lib/echo-pipelinetriggers-2``` +Igor logs show the following: +```2021-10-18 16:24:21.990 WARN 1 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'fiatPermissionEvaluator' defined in URL [jar:file:/opt/igor/lib/fiat-api-1.27.0.jar!/com/netflix/spinnaker/fiat/shared/FiatPermissionEvaluator.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'fiatService' defined in class path resource [com/netflix/spinnaker/fiat/shared/FiatAuthenticationConfig.class]: Unsatisfied dependency expressed through method 'fiatService' parameter 1; nested exception is org.springframework.beans.factory.NoUniqueBeanDefinitionException: No qualifying bean of type 'com.netflix.spinnaker.okhttp.SpinnakerRequestInterceptor' available: expected single matching bean but found 2: ``` + + +## Cause +This is due to a known issue, and pertains to the existence of beans of the same type as one already present in OSS.   + +The fix for the issue is in ```armory-commons``` and the following PR provides more information:[https://github.com/armory-io/armory-commons/pull/199](https://github.com/armory-io/armory-commons/pull/199) + diff --git a/kb/armory/General/oauth-setup-for-github.md b/kb/armory/General/oauth-setup-for-github.md new file mode 100644 index 0000000000..e3815be819 --- /dev/null +++ b/kb/armory/General/oauth-setup-for-github.md @@ -0,0 +1,7 @@ +--- +title: Oauth Setup for Github +--- + + +This page has been recategorized and moved to our Documents Library:[https://docs.armory.io/spinnaker-install-admin-guides/authn-github/](https://docs.armory.io/spinnaker-install-admin-guides/authn-github/) + diff --git a/kb/armory/General/objects-disappearing-under-the-cluster-tab-and-a-slow--deck-ui-cluster-page.md b/kb/armory/General/objects-disappearing-under-the-cluster-tab-and-a-slow--deck-ui-cluster-page.md new file mode 100644 index 0000000000..f2309db4fe --- /dev/null +++ b/kb/armory/General/objects-disappearing-under-the-cluster-tab-and-a-slow--deck-ui-cluster-page.md @@ -0,0 +1,49 @@ +--- +title: Objects disappearing under the Cluster tab and a slow Deck UI Cluster page +--- + +## Issue +The objects under Clusters tab disappear after a certain time (3-6 hours). +In the duration that the cluster data shown in the UI is absent, the Clouddriver pods do not restart. Another symptom is that the Cluster page is slow at updating the pod being updated. +The following errors are seen running the caching agents: +``` WARN 1 --- [utionAction-174] c.n.s.c.cache.LoggingInstrumentation : kubernetes:rr-workflow-.....-3/KubernetesCoreCachingAgent[1/1] completed with one or more failures in 0.378s``` + + +## Cause +Caching agents being unable to complete will prevent the cache from being up to date as new deployments are not cached. +The output of making a request to ```http://:7002/cache/introspection``` would be for example like: +``` +[ +2 { +3 "details": {}, +4 "id": "...../CustomKubernetes(DestinationRule)[1/1]", +5 "lastExecutionDurationMs": 413, +6 "lastExecutionStartDate": "2021-08-17 21:44:26.928", +7 "lastExecutionStartMs": 1629150266928, +8 "provider": "kubernetes", +9 "totalAdditions": 0, +10 "totalEvictions": 0 +11 }, +12 { +13 "details": {}, +14 "id": "...../KubernetesCoreCachingAgent[1/1]", +15 "lastExecutionDurationMs": 1934, +16 "lastExecutionStartDate": "2021-08-17 21:44:21.928", +17 "lastExecutionStartMs": 1629150271928, +18 "provider": "kubernetes", +19 "totalAdditions": 121, +20 "totalEvictions": 0 +21 }, +22 { +23 "details": {}, +24 "id": "......KubernetesUnregisteredCustomResourceCachingAgent[1/1]", +25 "lastExecutionDurationMs": 1, +26 "lastExecutionStartDate": "2021-08-17 21:44:19.927", +27 "lastExecutionStartMs": 1629150249927, +28 "provider": "kubernetes", +29 "totalAdditions": 0, +30 "totalEvictions": 0 +31 } +``` +This endpoint contains statistics about how much time it took for each caching agent to complete.  All of them should run under 10 minutes (successfully) or data will disappear from UI. + diff --git a/kb/armory/General/obscuring-last-names-in-public-articles-and-cases---internal-users.md b/kb/armory/General/obscuring-last-names-in-public-articles-and-cases---internal-users.md new file mode 100644 index 0000000000..4c799cc594 --- /dev/null +++ b/kb/armory/General/obscuring-last-names-in-public-articles-and-cases---internal-users.md @@ -0,0 +1,24 @@ +--- +title: Obscuring Last Names in Public Articles and Cases - Internal Users +--- + + +To help with providing privacy options for Armory Employees, Support has added an option to obscure last names and how they are displayed.  This is to give options to employees so that their full names are not available publicly to be scraped from the site, in public articles. +This is a global change that will affect cases and articles all at once.  It cannot be changed just for KB articles and not for cases and vice versa.   +#### What was done +A new field on the ```sys_user``` table was created to show a ```Hide Full Name``` checkbox.   Once implemented, code will be implemented to change the displayed name so that only the final initial will be shown.   +For example, ```Jane Smith``` will be changed to ```Jane S.``` + + +Once this checkbox is checked, the below code will run to calculate the value of the user's display name +``` +if(current.u_hide_full_name == true) { + var ephemeral = current.last_name.toString(); + current.first_name + ' ' + ephemeral[0] + "."; +} +``` +This will concatenate the user's first name and last name and hide the last name anywhere the user is mentioned. This includes but is not limited to +* Table Queries on a sys_user reference field* Knowledge Article Authors* Case Assigned Users + +By default, the ```Hide Full Name``` checkbox is unchecked. All internal users are free to check this for themselves if they wish to hide their full names in the ServiceNow platform. + diff --git a/kb/armory/General/opa-policies-not-working-after-spinnaker-upgrade-.md b/kb/armory/General/opa-policies-not-working-after-spinnaker-upgrade-.md new file mode 100644 index 0000000000..317c3c52ec --- /dev/null +++ b/kb/armory/General/opa-policies-not-working-after-spinnaker-upgrade-.md @@ -0,0 +1,16 @@ +--- +title: OPA Policies not working after Spinnaker Upgrade +--- + +## Issue +An organization may have issues after upgrading Spinnaker where Pipelines with OPA policies attached will fail to deploy. +Below is an example of the sequence of events: +* An organization creates a policy and attaches it to a pipeline.Example - ```"Every pipeline must have a XXXXXX component UUID".```* The organization tries to modify this pipeline, but an error arises even if the change is valid on a previous version of Spinnaker.Example - Successful tests in 2.26.x may not succeed in 2.27.x* After hitting the Save Changes button, Spinnaker shows a banner with:```"There was an error saving your pipeline: policy Every pipeline must have a XXXXXX component UUID.". Please refer to the attached file XXXXXX.rego to see the policy definition.``` +An organization may also see Gate logs containing an error message with: +```com.netflix.spinnaker.gate.controllers.PipelineController$PipelineException.``` + +## Cause +This error is caused by changes to the RegoScript data structure within OPA. +For example, within Spinnaker 2.26.x, the ```input.pipeline.expectedArtifacts``` hierarchy was updated to exist under: ```input.pipeline.any.expectedArtifacts``` +As a result, existing RegoScripts would need to be updated. Depending on the policies defined, customers may need to account for these changes with version upgrades.   + diff --git a/kb/armory/General/opa-policy-deployment-error--failed-to-shutdown-server-gracefully.md b/kb/armory/General/opa-policy-deployment-error--failed-to-shutdown-server-gracefully.md new file mode 100644 index 0000000000..a44cdf66de --- /dev/null +++ b/kb/armory/General/opa-policy-deployment-error--failed-to-shutdown-server-gracefully.md @@ -0,0 +1,17 @@ +--- +title: Opa Policy Deployment Error- Failed to shutdown server gracefully +--- + +## Issue +Opa Policy Deployment suddenly causes the Spinnaker UI to fail even though there is no change in the deployment configuration. In addition the Spinnaker UI cannot be accessed.  +OPA logs show the following: +```` +"POST","req_path":"/v1/data/spinnaker/.....t","resp_body":"{\"decision_id\":\"2878445b-1487-49ff-af64-7d000aedcffe\"}","resp_bytes":54,"resp_duration":29624.175444,"resp_status":200,"time":"2021-08-26T23:05:44Z"} +{"err":"error while shutting down: (0) context deadline exceeded. ","level":"error","msg":"Failed to shutdown server gracefully.","time":"2021-07-26T23:05:44Z"} +{"level":"info","msg":"Stopping decision logger.","plugin":"decision_logs","time":"2021-08-26T23:05:44Z"} +{"level":"debug","msg":"maxprocs: No GOMAXPROCS change to reset","time":"2021-08-26T23:05:44Z"} +```` +## Cause +This error stems from the source code: +[https://github.com/open-policy-agent/opa/blob/320f9a6b06101687979abb3a180d521c743e45cc/runtime/runtime.go#L682](https://github.com/open-policy-agent/opa/blob/320f9a6b06101687979abb3a180d521c743e45cc/runtime/runtime.go#L682) + diff --git a/kb/armory/General/openshift--how-to-supply-the-complete-certificate-chain.md b/kb/armory/General/openshift--how-to-supply-the-complete-certificate-chain.md new file mode 100644 index 0000000000..40c555def3 --- /dev/null +++ b/kb/armory/General/openshift--how-to-supply-the-complete-certificate-chain.md @@ -0,0 +1,11 @@ +--- +title: OpenShift- How to supply the complete certificate chain +--- + +## Issue +Validator error when attempting to apply changes: +```certificate signed by unknown authority``` + +## Cause +OpenShift is not supplying the full certificate chain for verifying the certificate from the server.The issue occurs when a user's certificate originates from Kubernetes in the ```kubeconfig``` file (base64 encoded PEM format certs).The **OpenShift cluster** includes the **leaf cert**, which is used by the API server. A number of supporting certs in the ```kubeconfig``` file are also provided, but not the certificates needed to verify the **leaf cert**. + diff --git a/kb/armory/General/operator-cannot-connect-to-halyard-due-to-a-tcp-timeout.md b/kb/armory/General/operator-cannot-connect-to-halyard-due-to-a-tcp-timeout.md new file mode 100644 index 0000000000..4e60d73a33 --- /dev/null +++ b/kb/armory/General/operator-cannot-connect-to-halyard-due-to-a-tcp-timeout.md @@ -0,0 +1,12 @@ +--- +title: Operator cannot connect to Halyard due to a TCP timeout +--- + +## Issue +While attempting to deploy Spinnaker using Operator, the process may fail with the following TCP timeout error message displayed in Operator's pod logs: +```{"level":"error","ts":1636061223.53345,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"spinnakerservice-controller","request":"spinnaker-service/spinnaker","error":"Post http://localhost:/v1/config/deployments/manifests: dial tcp :: i/o timeout"``` + +## Cause +The Spinnaker Operator container is based on an ***Alpine Linux*** distribution. Alpine does not, by default, include ```/etc/nsswitch.conf```. Golang, in the absence of ```/etc/nsswitch.conf```, defaults to a DNS-first lookup and will exhaust DNS lookups on any hostname, including ```localhost```, before it falls back to using ```/etc/hosts``` for hostname lookup. +If an entry for ```localhost``` is configured/mapped within the DNS server for the cluster in question, when Operator starts, it performs a DNS look-up for ```localhost```. This catches the DNS entry for ```localhost.```, which resolves to an IP address displayed in the timeout error message. This IP does not exist, and since it is not running Halyard, it results in a TCP timeout. + diff --git a/kb/armory/General/operator-error-when-performing-a-spinnaker-upgrade-in-an-air-gapped-environment.md b/kb/armory/General/operator-error-when-performing-a-spinnaker-upgrade-in-an-air-gapped-environment.md new file mode 100644 index 0000000000..bbf942fb2d --- /dev/null +++ b/kb/armory/General/operator-error-when-performing-a-spinnaker-upgrade-in-an-air-gapped-environment.md @@ -0,0 +1,13 @@ +--- +title: Operator error when performing a Spinnaker upgrade in an air-gapped environment +--- + +## Issue +When upgrading Spinnaker's version in an air-gapped environment using a pull-through repository, a user may come across the following error in the Operator logs: +``` +"{"level":"error","ts":1641324429.9751217,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"spinnakerservice-controller","request":"spinnaker-service/spinnaker","error":"Unable to retrieve profile \"settings.js\": Error reading contents of S3 using AWS SDK. Region: , Bucket: , Object: profiles/deck//settings.js. Check connectivity and permissions to that bucket. Error: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request " +``` + +## Cause +Since the error hints at missing files, the pull-through repository might not be up to date with the latest artifacts from Armory's official repo. + diff --git a/kb/armory/General/operator-is-throwing-error-message-about-modified-object.md b/kb/armory/General/operator-is-throwing-error-message-about-modified-object.md new file mode 100644 index 0000000000..6b254c8796 --- /dev/null +++ b/kb/armory/General/operator-is-throwing-error-message-about-modified-object.md @@ -0,0 +1,11 @@ +--- +title: Operator is throwing error message about modified object +--- + +## Issue +Administrators may be seeing an error similar to the one below in their Operator pods: +```Reconciler error","controller":"spinnakerservice-controller","request":"spinnaker/spinnaker","error":"Operation cannot be fulfilled on spinnakerservices.spinnaker.armory.io \"spinnaker\": the object has been modified; please apply your changes to the latest version and try again``` + +## Cause +This is a transient error that frequently occurs when an object is modified mid-reconcile loop. Users are only allowed to modify and commit one change before returning it back to Kubernetes and try again another time.  + diff --git a/kb/armory/General/orca-mishandling-spel-expressions.md b/kb/armory/General/orca-mishandling-spel-expressions.md new file mode 100644 index 0000000000..58fca6bde0 --- /dev/null +++ b/kb/armory/General/orca-mishandling-spel-expressions.md @@ -0,0 +1,19 @@ +--- +title: Orca mishandling SpEL Expressions +--- + +## Issue +An organization may experience Orca mishandling SpEL Expressions. An example payload is below.  +``` +payload: '{"subexperienceApprover":"${execution[''stages''].?[type == ''manualJudgment''][0][''context''][''lastModifiedBy'']}","clusterName": "${parameters[''Cluster Name'']}","subExperience": "${parameters[''Sub-experience Name'']}","GCPProjectID": "${parameters[''GCP Project ID'']}","namespace": "${parameters[''Namespace'']}","writeGroups": ["${parameters[''AD Write Group(s)'']}"],"readGroups": ["${parameters[''AD Read Group(s)'']}"],"location": "${parameters[''Cluster Location'']}","endpoint":"${parameters[''Public Endpoint'']}"}' +``` + +If this example payload is run through spinnaker, Orca will evaluate each of the parameters individually instead of having that payload be there for future runs to evaluate each of the needed parameters on the actual user run. The run through of Spinnaker will be set as (```executionId```, ```NameSpace```, etc..) This issue can also arise from Orca evaluating SpEL Expressions when none exist, which can cause a failed pipeline. +Related to the below Github Issue. +[https://github.com/spinnaker/spinnaker/issues/5910](https://github.com/spinnaker/spinnaker/issues/5910) + +## Cause +The SpEL expression toggle in Deck is not working as expected. This is a bug that affects Armory and Spinnaker Version 2.24 and below. There are legitimate cases where there is a mix of some SpEL expressions that should be evaluated and cases where they shouldn't be evaluated. This is exacerbated by SpEL Expressions starting with ```$``` in Manifest Files which can cause Orca some confusion. + + + diff --git a/kb/armory/General/orca-stuck-in-crashloopbackoff-db--mysql.md b/kb/armory/General/orca-stuck-in-crashloopbackoff-db--mysql.md new file mode 100644 index 0000000000..b5962a087b --- /dev/null +++ b/kb/armory/General/orca-stuck-in-crashloopbackoff-db--mysql.md @@ -0,0 +1,23 @@ +--- +title: Orca Stuck in CrashLoopBackoff (DB- mySQL) +--- + +## Issue +The orca pod is stuck in ```crashloopbackoff``` with an error connecting to mySQL. +Example error: +```liquibase.exception.DatabaseException: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure``` +Restarting the pod is not resolving this issue. + +## Cause +Orca pods were most likely misconfigured. To dig deeper, we can look into orca pod logs.Since this is related to mySQL, we can check ```orca-mysql``` pod logs. From the pod logs, we can see it is a connectivity issue due to the ```communication link failure``` error listed above. +We can now look into the spinnaker services to see if all pods have connectivity.Run the command: +```kubectl get svc -n spinnaker``` +*example output:* + +We saw that the ```orca-mysql``` service was using ```NodePort```. To further examine, we described the orca-mysql service. To do this, you can run the following command: ```kubectl describe svc orca-mysql -n spinnaker```*example output:* +** + +We found the following error under the **events** section: +```*v1.Service spinnaker/orca-mysql failed syncing: Service "orca-mysql" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'``` + + diff --git a/kb/armory/General/orca-with-sql-high-memory-usage---tables-information-dive.md b/kb/armory/General/orca-with-sql-high-memory-usage---tables-information-dive.md new file mode 100644 index 0000000000..e7ae69054b --- /dev/null +++ b/kb/armory/General/orca-with-sql-high-memory-usage---tables-information-dive.md @@ -0,0 +1,28 @@ +--- +title: Orca with SQL high memory usage / Tables information dive +--- + + +What:Spinnaker executions fails to show.Spinnaker ui is very slow to load pipeline view with executions.Orca consumes a lot of memory.Orca dies after SSL hand shake error.Why:This error can be because the data in SQL of Orca executions is too big and Orca can't load the data.Can happen because 2 factors, but basically too much data.1) One of Orca rows has reached over the max_allowed_packet size you have defined or over the max (1Gb)2) A pipeline execution in sum has reached over this maximum.How to fix this? +Clean the data.Clear the tables pipelines, pipeline_stages, orchestrations and orchestration_stages.Note: Data can be plenty and deleting it may take a while. Hence this script creates a tmp table to store id's and then deletes from here. No risk in you query dying in the middle.Note1: Always perform a DB snapshot before deleting data in clients Prod Env. +Howto Cleanup Old Executions in Orca MySQL Database +=================================================== +Important Tables:- orchestration_stages: Task Stage Executions (ref back to orchestrations) +- orchestrations: Tasks (in Tasks tab)(e.g. Save Pipeline Task) +- pipeline_stages: Stage Executions (ref back to pipelines) +- pipelines: Pipeline ExecutionsTo Remove Pipeline Executions:- Create temporary tables with the IDs of rows to remove: + CREATE TEMPORARY TABLE old_pipelines SELECT id FROM pipelines WHERE (updated_at / 1000) +Why Why: +For each pipeline execution Orca saves data.Each time a user loads an application pipeline view (and hence it's executions) Orca loads said executionsIf a user selects to show even more executions Orca fetches that data.Spinnaker can run smooth and fine for all your applications but one, that can be due that said application pipelines are run more often or have bigger artifacts or are run on a loop or are in the form of parent-child with multiple children or grandchildren.  +Why why why: +Orca stores the entire execution in a single column "body". +If a pipeline calls another pipeline ALL the execution of parent pipeline is stored in the child pipeline data instead of an id pointing to said data. + +What to avoid: + +Recurring pipelines: +```A pipeline calling itself grows significantly big rather quick. For a small footprint pipeline with 10kb artifacts your pipeline you can reach 30MB rows in SQL in 10 iterations.``` +Each bake stage saves the baked artifact 4 times on SQL +Each Deploy Stage saves ALL artifacts 2 times on SQL (ALL of them, not just the one used for the deploy) + + diff --git a/kb/armory/General/packet-for-query-is-too-large---mysql-max_allowed_packet.md b/kb/armory/General/packet-for-query-is-too-large---mysql-max_allowed_packet.md new file mode 100644 index 0000000000..eb0e83716c --- /dev/null +++ b/kb/armory/General/packet-for-query-is-too-large---mysql-max_allowed_packet.md @@ -0,0 +1,11 @@ +--- +title: Packet for query is too large - MySQL (max_allowed_packet) +--- + +## Issue +Upon changing to MySQL, some pipelines fail, or fail to trigger.  Upon investigation, error messages for the issue can be found in a stack trace.  As an example, you will find errors in CloudWatch about the Packet for query is too large such as the one below: +```org.springframework.dao.TransientDataAccessResourceException: jOOQ; SQL [insert into pipelines (id, legacy_id, `partition`, status, application, build_time, start_time, canceled, updated_at, body, config_id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) on duplicate key update status = ?, body = ?, start_time = ?, canceled = ?, updated_at = ?, config_id = ? -- user: john.smith@abc.com origin: api]; Packet for query is too large (size1 > size2). You can change this value on the server by setting the 'max_allowed_packet' variable.; nested exception is com.mysql.cj.jdbc.exceptions.PacketTooBigException: Packet for query is too large (size 1 > size 2). You can change this value on the server by setting the 'max_allowed_packet' variable.``` + +## Cause +```max_allowed_packet``` definition is too small to handle the size of the data incoming to the database.  There is a limit that is defined on the database.Here is a further explanation of the setting in MySQL 8 as an example:[https://dev.mysql.com/doc/refman/8.0/en/packet-too-large.html](https://dev.mysql.com/doc/refman/8.0/en/packet-too-large.html) + diff --git a/kb/armory/General/pagerduty---setting-up-for-new-users.md b/kb/armory/General/pagerduty---setting-up-for-new-users.md new file mode 100644 index 0000000000..9786694884 --- /dev/null +++ b/kb/armory/General/pagerduty---setting-up-for-new-users.md @@ -0,0 +1,63 @@ +--- +title: PagerDuty - Setting up for New Users +--- + + + +### Administration to Setup User in the Pager Duty Queues + +BlackMountain IT team will set up new hiring PagerDuty user account.  +  +Technical Support Team (currently everyone in Support team has manager level privilege) needs to enroll new hire’s account into Technical Support Team.  +  +The target on-call schedule enrolls after 6 weeks after the new hires starting date. +[https://armory.pagerduty.com/teams/PSUCQEP/users](https://armory.pagerduty.com/teams/PSUCQEP/users) +  + +### Link your Account with ServiceNow + +As a part of the process, once your account is created, it will need to be linked in ServiceNow.  The following are  +steps that need to be completed +* Each user needs to be mapped using their PagerDuty ID.  In each ```User``` object in Service Now, there is a ```PagerDuty ID```* To set this up, log in to **ServiceNow** as an **Admin*** Search for the **Users** sectionSearch for the Support Engineer and click on their Name.  Add the ```PagerDuty ID``` +* The PagerDuty ID needs to match with the User’s ID in PagerDuty.  Once the User’s PagerDuty Account is created, the User’s ID (```P------```) needs to be added* Log in to PagerDuty* Find the User; an easy way to do this is to go to **People** → **Users** → and Search for the user* Click on the User, and then you’ll find their PagerDuty ID in the URL as per the screenshot below + + +* Add the ```PagerDuty ID``` in ServiceNow* Click **Update**, and the information will be now added, and the account will be linked. + +### Set up PagerDuty Profile in PagerDuty portal (through OKTA) + + +#### Profile settings + +In the PagerDuty Website, Click on your portrait in the upper right hand side → Click on My Profile.  +* Make sure you fill up all related information (on-call phone number, Email, etc) +* Go to your Contact Information Tab and ensure that the Mobile App Notifications are enabled (test as well)* Go to Notification Rules, make sure the high-urgency notification rule contains all trigger method (Phone call, push notify on phone, etc) +### Setting Up Slack +Please make sure that you have set up notifications to your preference for any alerts surrounding the user group ***@support-team-p0alerts***.  The general expectation is that when a message comes in to this group, it will be for a P0 case, and that the team should be all hands on that case as a high priority once the alert comes in, during regular Support work hours. +If that requires special exceptions on alerts for that group, please make the appropriate changes (such as notifying via sound to get your attention) + +### If you are using an Android device: + +To prevent any accidents where a P-0 may come in, but you are not alerted, please ensure you have gone through your PagerDuty Settings in the application and adjusted your +* System Notification Settings* High Urgency Notification Settings* High Urgency Override System Volume * High Urgency Override Do Not Disturb settings +  +**Note:** If you are an Android user, there is a large possibility that you cannot override your system volume on newer Android OS versions.  If you are in this camp, you may need to set up a macro in an application like Tasker. +  +For Tasker, an example of the logic to set up would be to increase the volume to MAX if an incoming call comes in from your PagerDuty contacts.  This way, when a call comes in via PagerDuty to alert you, it will notify you.  PagerDuty allows for the creation of a contact card in your Android which has all the possible incoming numbers from their service, and then a macro rule that you would set would be to basically maximize your volume if a call comes in from one of the numbers in the contact card +  +  +  +  +  +  +  +  + +### If you are using an IOS device: + +* You need to go to Settings → Notifications → PageDuty→ turn on Allow Notifications* In PagerDuty settings, you need to turn on `Push Notifications` and install the PagerDuty Contact Card.  + +* Please note, for iOS users, installing the contact card allows you to enable an override for Do Not Disturb. and will ensure silencing unknown numbers doesn't affect calls from PagerDuty. + +* Make tests through PagerDuty portal, and check if you can receive phone calls, notifications properly on your phone. + diff --git a/kb/armory/General/pagerduty-notifications-integration-with-spinnaker.md b/kb/armory/General/pagerduty-notifications-integration-with-spinnaker.md new file mode 100644 index 0000000000..625fd9909c --- /dev/null +++ b/kb/armory/General/pagerduty-notifications-integration-with-spinnaker.md @@ -0,0 +1,64 @@ +--- +title: PagerDuty Notifications Integration with Spinnaker +--- + +## Introduction +PagerDuty is an incident response application that IT departments commonly use to alert, triage, and manage incidents based on different use cases.  PagerDuty empowers organizations with insights to proactively manage events that may impact customers across their IT environment.  It supports multiple integrations from Slack to Atlassian products such as Jira. + +In this document, we will look into integrating PagerDuty with Spinnaker Notifications.  Spinnaker Notifications allow simple notification messages to be passed upon pipeline execution completion.  These messages will be passed on to PagerDuty to a named target.  The functionality of the PagerDuty Notifications in Spinnaker operates slightly differently from how the Slack or Email notification systems work and requires additional configuration. +For more flexibility with PagerDuty's API systems and the type of data sent to PagerDuty, please explore Spinnaker's Webhook stages. + +## Prerequisites +* Permission to the Spinnaker application* PagerDuty token generated by an administrator + +## Instructions +Below is a breakdown of how PagerDuty Notifications can be utilized.   + +## Table of Contents +*  * [Set-Up](#mcetoc_1hack6h8sk)* [Using the Integration](#mcetoc_1hack6h8sl) + + +### Set-Up +Administrators will need to set up Spinnaker's UI to allow the feature to be configured.  The following needs to be added to the ```SpinnakerService.yml``` file: +``` +spec: + spinnakerConfig: + profiles: + deck: + settings-local.js: + window.spinnakerSettings.feature.pagerDuty = true; + Administrators will also need to make the following changes to the Gate and Echo services in the ```SpinnakerService.yml``` +spec: + spinnakerConfig: + profiles: + echo: + pager-duty: + enabled: true + token: *** + gate: + pagerDuty: + base-url: https://api.pagerduty.com + enabled: true + token: *** +``` +### Using the Integration +* After applying the config changes, users can log in to Spinnaker and configure their applications.  Users can select the PagerDuty Service when editing the application.* If no services are shown within the drop-down, Gate has an issue loading them, and the token/configurations should be checked. +* Once a service is selected, Spinnaker will pull the FIRST integration key from the services integration list in PagerDuty.  It then saves the integration key as a new field on the application in Spinnaker ```pdApiKey```.  Orca has a stage that reads this key as part of a task and uses this to call the callback notification service.[https://github.com/spinnaker/orca/blob/b34b257f925bde8bc1d25b7c38696329438725cf/orca-echo/src/main/groovy/com/netflix/spinnaker/orca/echo/tasks/PageApplicationOwnerTask.groovy#L36](https://github.com/spinnaker/orca/blob/b34b257f925bde8bc1d25b7c38696329438725cf/orca-echo/src/main/groovy/com/netflix/spinnaker/orca/echo/tasks/PageApplicationOwnerTask.groovy#L36)Users can add a stage of the ```pageApplicationOwner``` type and edit the JSON, and set it like the below example: +``` +{ + "applications": [ + "demo" + ], + "details": { + "from": "Overnight Monitoring Team" + }, + "message": "[DEMO] Application ABC has completed", + "name": "Trigger pager duty", + "type": "pageApplicationOwner" +} +``` +Note that if the ```from``` field is NOT set, it defaults to the triggering via the user's email.  Other PagerDuty settings can be configured in Deck as shown here [https://github.com/spinnaker/deck/blob/c768e88fbc893d0bd5dc86959320a7b7d67443e5/packages/core/src/config/settings.ts#L132-L137](https://github.com/spinnaker/deck/blob/c768e88fbc893d0bd5dc86959320a7b7d67443e5/packages/core/src/config/settings.ts#L132-L137). + +* Once the Pipeline has executed the stage, the notification will be sent to PagerDuty via the Events API.  If other PagerDuty APIs need to be used, please explore the Webhook stage as an option. +  + diff --git a/kb/armory/General/parameters-with-kubernetes-manifest-&-using-spel.md b/kb/armory/General/parameters-with-kubernetes-manifest-&-using-spel.md new file mode 100644 index 0000000000..187262e9fd --- /dev/null +++ b/kb/armory/General/parameters-with-kubernetes-manifest-&-using-spel.md @@ -0,0 +1,15 @@ +--- +title: Parameters with Kubernetes Manifest & Using SpEL +--- + + +Question: +How do I dynamically promote a build with parameters in a manifest file in Spinnaker using Kubernetes? +Answer: +* The first step is to understand how to use parameters in your kubernetes manifest file. Here is a link to some good documentation: [Parameterize Kubernetes Manifests](https://www.spinnaker.io/guides/user/kubernetes-v2/parameterize-manifests/#parameterize-kubernetes-manifests)* The second step is to create a pipeline expression to dynamically decide where to deploy the code based on some action. For our example we are using a manual judgement stage to determine if the code should go to Prod or Staging. +Example: +```Promote:${#judgement('Promote') == '' ? parameters['namespace'] : #judgement('Promote')}``` + +More Resources: +If you would like to learn more about SpEL and Pipeline expressions with Spinnaker you can find it here: [Pipeline Expressions Guide](https://www.spinnaker.io/guides/user/pipeline-expressions/#pipeline-expressions-guide) + diff --git a/kb/armory/General/password-resets.md b/kb/armory/General/password-resets.md new file mode 100644 index 0000000000..2d70a20f63 --- /dev/null +++ b/kb/armory/General/password-resets.md @@ -0,0 +1,9 @@ +--- +title: Password Resets +--- + + +Password Resets can be accomplished by self-resetting their password +To do so, navigate to the support portal at [https://support.armory.io](https://support.armory.io) +* In the Upper Right corner, click on Login* Click on **Forgot Password?** and a new screen will load* On the Identify screen, enter the email address of the user to be reset* On the Verify screen, enter the username (which should also be the same email address) and click next* An email will be sent to the email address shortly.  Follow the instructions in the email to reset the password + diff --git a/kb/armory/General/pausing-in-flight-kubernetes-deployments-in-spinnaker.md b/kb/armory/General/pausing-in-flight-kubernetes-deployments-in-spinnaker.md new file mode 100644 index 0000000000..a5a31ff6e4 --- /dev/null +++ b/kb/armory/General/pausing-in-flight-kubernetes-deployments-in-spinnaker.md @@ -0,0 +1,18 @@ +--- +title: Pausing in-flight Kubernetes deployments in Spinnaker +--- + +## Introduction +A user would like to pause a Kubernetes deployment via the Spinnaker UI directly.  + +## Prerequisites +* An actively-running Kubernetes deployment that shows up in the Spinnaker UI.* Output of ```{your_gate_endpoint}/manifests/{your_account_name}/{your_namespace}/deployment%20{your_deployment_name}``` has ```status.paused.state: false``` and ```status.stable.state: false```* ```settings-local.js``` for Deck has ```window.spinnakerSettings.kubernetesAdHocInfraWritesEnabled = true;``` as per the *Kubernetes infrastructure in the UI* documentation: [https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-26-0/#kubernetes-infrastructure-in-the-ui](https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-26-0/#kubernetes-infrastructure-in-the-ui) + + +## Instructions +To achieve this via the Spinnaker UI +* Click on the actively running deployment you would like to pause.* A menu displaying detailed information on that deployment will show up on the right-hand side.* Inside it, from the ```Deployment Actions``` drop-down - select ```Pause```.  +It is important to note that the button **only** appears while the deployment is running, and before it’s stable. +Depending on the size, this may entail that it only shows up for a brief period of time during each pipeline execution. Because of this, a suggested workaround to pause it is to manually issue a ```kubectl rollout pause deployment.v1.apps/``` command, as per the official Kubernetes documentation: +[https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#pausing-and-resuming-a-deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#pausing-and-resuming-a-deployment) + diff --git "a/kb/armory/General/pipeline-execution-error-\"service-selector-must-have-no-label-keys-in-common-with-target-workload\".md" "b/kb/armory/General/pipeline-execution-error-\"service-selector-must-have-no-label-keys-in-common-with-target-workload\".md" new file mode 100644 index 0000000000..e62857d4e9 --- /dev/null +++ "b/kb/armory/General/pipeline-execution-error-\"service-selector-must-have-no-label-keys-in-common-with-target-workload\".md" @@ -0,0 +1,12 @@ +--- +title: Pipeline Execution Error "Service selector must have no label keys in common with target workload" +--- + +## Issue +After migrating Spinnaker to ```2.24.x+``` from a previous version, and upon utilizing the migrated pipelines from the previous version, the following error is observed at the Deploy stage: +```"Service selector must have no label keys in common with target workload"``` +The UI shows the following: + +## Cause +This is resulting from the following caveat ([https://spinnaker.io/docs/guides/user/kubernetes-v2/traffic-management/#caveats](https://spinnaker.io/docs/guides/user/kubernetes-v2/traffic-management/#caveats)) and an issue impacting prior versions of Spinnaker ([https://github.com/spinnaker/clouddriver/pull/4504](https://github.com/spinnaker/clouddriver/pull/4504)). + diff --git a/kb/armory/General/pipeline-randomly-cancelled.md b/kb/armory/General/pipeline-randomly-cancelled.md new file mode 100644 index 0000000000..e8cbf3ab4f --- /dev/null +++ b/kb/armory/General/pipeline-randomly-cancelled.md @@ -0,0 +1,10 @@ +--- +title: Pipeline Randomly Cancelled +--- + +## Issue +Pipeline randomly stops running and shows the status as Terminal. + +## Cause +Stage failure options are not set properly. + diff --git "a/kb/armory/General/pipeline-stuck-on-\"wait-for-manifest-to-stabilize\"-task.md" "b/kb/armory/General/pipeline-stuck-on-\"wait-for-manifest-to-stabilize\"-task.md" new file mode 100644 index 0000000000..277e87f483 --- /dev/null +++ "b/kb/armory/General/pipeline-stuck-on-\"wait-for-manifest-to-stabilize\"-task.md" @@ -0,0 +1,15 @@ +--- +title: Pipeline stuck on "Wait for Manifest to Stabilize" Task +--- + +## Issue +When executing a pipeline the ```Wait For Manifest To Stabilize``` stage gets stuck until the timeout is reached. + +## Cause +This can be caused by a few different reasons.  In order to narrow down the potential root cause, please check the following: +* If the execution was initiated by *non-admin* user and when an *admin* user starts the the pipeline, it gets stuck * If manually triggering the pipeline shows the same behavior +If there is a difference in behavior when checking the above, then the issue may be due to the moniker naming convention. What happens is that the request to fetch a manifest from Clouddriver does two permission checks: +* Does the user have permission to the account in question?Does the user have permission to the app specified on the manifest's app +* In cases where a ```moniker.app``` is explicitly specified on the manifest, then this is the app that is used.* In cases where there is no ```moniker.app``` specified, then the app is determined by parsing using ```frigga``` as per [this article](https://github.com/Netflix/frigga/blob/d24ab6f96529827d7d99e2e3b23e3417f8435cf6/src/main/java/com/netflix/frigga/Names.java#L95). + + diff --git a/kb/armory/General/pipelines-are-triggered-by-anonymous-without-sufficient-permission.md b/kb/armory/General/pipelines-are-triggered-by-anonymous-without-sufficient-permission.md new file mode 100644 index 0000000000..bd8aecd75c --- /dev/null +++ b/kb/armory/General/pipelines-are-triggered-by-anonymous-without-sufficient-permission.md @@ -0,0 +1,10 @@ +--- +title: Pipelines are triggered by anonymous without sufficient permission +--- + +## Issue +Pipelines are able to be triggered even though the user does not have sufficient permissions + +## Cause +The ```runAsUser``` option is set in the pipeline JSON, overriding the users permissions, as these credentials take precedence.  + diff --git a/kb/armory/General/pipelines-as-code-dinghy-starts-with-multiplebranchesenabled--true-even-though-it-has-been-explicitly-set-to-false.md b/kb/armory/General/pipelines-as-code-dinghy-starts-with-multiplebranchesenabled--true-even-though-it-has-been-explicitly-set-to-false.md new file mode 100644 index 0000000000..9f3a56b4a6 --- /dev/null +++ b/kb/armory/General/pipelines-as-code-dinghy-starts-with-multiplebranchesenabled--true-even-though-it-has-been-explicitly-set-to-false.md @@ -0,0 +1,20 @@ +--- +title: Pipelines as Code (Dinghy) starts with multipleBranchesEnabled- true even though it has been explicitly set to false +--- + +## Issue +When disabling multiple branches for Pipelines as Code as described under [https://docs.armory.io/plugins/pipelines-as-code/install/configure/#multiple-branches](https://docs.armory.io/plugins/pipelines-as-code/install/configure/#multiple-branches), It may be noticed that the configuration ```multipleBranchesEnabled``` is never set to ```false``` even though it is set to false under the Dinghy profiles settings. This may cause ```dinghyfile``` changes on the master/main branch to be pushed to Spinnaker unexpectedly.  + +## Cause +As per the current structure, the configuration ```multipleBranchesEnabled``` is set to ```true``` by default unless it is explicitly set to false. However, there has been an issue that has been identified where disabling ```multipleBranchesEnabled``` does not have any effect and is always set to true when Dinghy starts up. This can be seen in the Dinghy startup logs, which are similar to the logs shown below +``` +time="2023-12-11T15:47:19Z" level=info msg="Configured with settings from file: /opt/spinnaker/config/spinnaker.yml" +time="2023-12-11T15:47:19Z" level=info msg="Configured with settings from file: /opt/spinnaker/config/dinghy.yml" +time="2023-12-11T15:47:19Z" level=info msg="Configured with settings from file: /opt/spinnaker/config/spinnaker-local.yml" +time="2023-12-11T15:47:19Z" level=info msg="Configured with settings from file: /opt/spinnaker/config/dinghy-local.yml" +time="2023-12-11T15:47:19Z" level=info msg="The following settings have been loaded: {\"instanceId\":\"dinghy\",\"templateOrg\":\"anonymous\",\"templateRepo\":\"spinnaker-kustomize-patches\",\"dinghyFilename\":\"dinghyfile\",\"autoLockPipelines\":\"false\",\"githubCredsPath\":\"/home/spinnaker/.armory/cache/github-creds.txt\",\"githubToken\":\"**REDACTED**\",\"githubEndpoint\":\"https://api.github.com\",\"stashCredsPath\":\"/home/spinnaker/.armory/cache/stash-creds.txt\",\"stashEndpoint\":\"http://localhost:7990/rest/api/1.0\",\"fiatUser\":\"dinghyserviceaccount\",\"logging\":{\"remote\":{\"enabled\":false,\"endpoint\":\"https://debug.armory.io/v1/logs\",\"customerId\":\"99fb5f10-ff80-4f54-be7b-ce098c0fdaa3\",\"version\":\"2.30.3\"}},\"secrets\":{\"vault\":{\"enabled\":true,\"url\":\"http://10.100.146.32:8200\",\"authMethod\":\"KUBERNETES\",\"role\":\"spinnaker\",\"path\":\"kubernetes\",\"username\":\"\",\"password\":\"\",\"userAuthPath\":\"\",\"namespace\":\"\",\"Token\":\"\"}},\"parserFormat\":\"json\",\"jsonValidationDisabled\":true,\"repoConfig\":[{\"provider\":\"github\",\"repo\":\"dinghy_implementation\",\"branch\":\"template-testing\"},{\"provider\":\"github\",\"repo\":\"dinghy_implementation\",\"branch\":\"template-testing-1\"}],\"orca\":{\"enabled\":\"true\",\"baseUrl\":\"http://spin-orca.anon-operator-spin:8083\"},\"front50\":{\"enabled\":\"true\",\"baseUrl\":\"http://spin-front50.anon-operator-spin:8080\"},\"deck\":{},\"echo\":{\"baseUrl\":\"http://spin-echo.anon-operator-spin:8089\"},\"gate\":{},\"fiat\":{\"authUser\":\"dinghyserviceaccount\",\"enabled\":\"true\",\"baseUrl\":\"http://spin-fiat.anon-operator-spin:7003\"},\"redis\":{\"baseUrl\":\"redis://spin-redis.anon-operator-spin:6379\"},\"server\":{\"Host\":\"\",\"Port\":8081,\"Ssl\":{\"Enabled\":false,\"CertFile\":\"\",\"KeyFile\":\"\",\"KeyPassword\":\"\",\"CAcertFile\":\"\",\"ClientAuth\":\"\"}},\"http\":{\"CacertFile\":\"\",\"ClientCertFile\":\"\",\"ClientKeyFile\":\"\",\"ClientKeyPassword\":\"\"},\"LogEventTTLMinutes\":1440,\"sql\":{\"baseUrl\":\"anon-spin-mysql-db-instance-1.cddfke67dweh.us-east-2.rds.amazonaws.com:3306\",\"user\":\"admin\",\"password\":\"**REDACTED**\",\"databaseName\":\"dinghy\",\"eventlogsOnly\":false},\"dinghyIgnoreRegexp2Enabled\":true,\"userWritePermissionsCheckEnabled\":false,\"ignoreUsersWritePermissions\":null,\"multipleBranchesEnabled\":true,\"upsertPipelineUsingOrcaTaskEnabled\":false}" +``` +It may be noticed that ```dinghy-local.yml``` mounted onto the Dinghy pod still has the correct configs. This issue is currently being investigated further by Armory Engineering and is tied to the below versions +2.30.32.30.42.30.5 +2.31.0 is not affected by this issue + diff --git a/kb/armory/General/pipelines-being-triggered-with-stale-artifacts.md b/kb/armory/General/pipelines-being-triggered-with-stale-artifacts.md new file mode 100644 index 0000000000..c827e4347e --- /dev/null +++ b/kb/armory/General/pipelines-being-triggered-with-stale-artifacts.md @@ -0,0 +1,10 @@ +--- +title: Pipelines being triggered with Stale Artifacts +--- + +## Issue +After a Spinnaker upgrade, one or more pipelines is triggered with old/stale artifacts. This can be caught by the build number moving back on the affected pipelines from a bigger to a smaller value (eg. from build # 100 to 20). This behaviour has been observed on pipelines with an ECR and Jenkins trigger.  + +## Cause +While the cause has not yet been confirmed, the culprit is suspected to lie with leftover stale data in Orca's queue, left in there prior to the upgrade. + diff --git a/kb/armory/General/pipelines-with-gae-githttpsusername-githttpspassword-fail-with-'not-authorized'-exception.md b/kb/armory/General/pipelines-with-gae-githttpsusername-githttpspassword-fail-with-'not-authorized'-exception.md new file mode 100644 index 0000000000..cc3ed6f2e4 --- /dev/null +++ b/kb/armory/General/pipelines-with-gae-githttpsusername-githttpspassword-fail-with-'not-authorized'-exception.md @@ -0,0 +1,12 @@ +--- +title: Pipelines with GAE gitHttpsUsername/gitHttpsPassword fail with 'Not authorized' Exception +--- + +## Issue +Pipelines with GAE ```gitHttpsUsername```/```gitHttpsPassword``` are seen to fail with a ```Not authorized``` exception.  + +## Cause +This is owing to GitHub starting to enforce 2FA and that doesn't work for automated environments. The problem with the library failing on using a password is a known issue, related to 2FA: +[https://github.com/eclipse/dirigible/issues/194](https://github.com/eclipse/dirigible/issues/194) + + diff --git a/kb/armory/General/pod-details-do-not-show-for-some-pods.md b/kb/armory/General/pod-details-do-not-show-for-some-pods.md new file mode 100644 index 0000000000..d1855e91c5 --- /dev/null +++ b/kb/armory/General/pod-details-do-not-show-for-some-pods.md @@ -0,0 +1,14 @@ +--- +title: Pod details do not show for some pods +--- + +## Issue +In the Spinnaker Console UI, Pod details are not showing up on the right panel for some pods. +This issue is observed indefinitely on some pods, and can be seen with other deployments on the same cluster.  The element issue can happen suddenly. Previously the pod details could be viewed for the same deployments. The UI shows the following: +  +The issue can also manifest itself in applications when attempting to list the namespace in the Kubernetes account.  The other namespaces are not seen on the Infrastructure tab. +The UI shows the following: + +## Cause + This issue is tied to reaching limits in ```tp caching```, ```performance```, or a combination of both.  It may also be tied back to changes to the permissions applied on the Kubernetes Service Account + diff --git a/kb/armory/General/pods-going-to-unhealthy-state-due-to-zombie-processes.md b/kb/armory/General/pods-going-to-unhealthy-state-due-to-zombie-processes.md new file mode 100644 index 0000000000..76c686d14d --- /dev/null +++ b/kb/armory/General/pods-going-to-unhealthy-state-due-to-zombie-processes.md @@ -0,0 +1,10 @@ +--- +title: Pods Going to Unhealthy State due to Zombie Processes +--- + +## Issue +Every 2-3 days, some or all of your pods may start to go unhealthy. Checking processes reveals that there are zombie SSL processes that are defunct but not shutting down, eventually slowing and crashing both pods as well as kubectl and Docker. This problem can happen on any OS or Cloud, if you have TLS enabled and Monitoring set upApplies to:EKSAWSKubernetesUbuntuDocker + +## Cause +There is a bug or issue with healthchecks/livenessProbe and SSL. Every 10 seconds, (or however long between health checks) the service will create a new process but fail to remove it. This zombie process will take up space and resources, until so many are spawned that it will crash the pod it’s supposed to be checking on.  + diff --git a/kb/armory/General/pods-running-jobs-or-scripts-remain-after-completed-state.md b/kb/armory/General/pods-running-jobs-or-scripts-remain-after-completed-state.md new file mode 100644 index 0000000000..ad8a89df91 --- /dev/null +++ b/kb/armory/General/pods-running-jobs-or-scripts-remain-after-completed-state.md @@ -0,0 +1,11 @@ +--- +title: Pods Running Jobs or Scripts Remain After Completed State +--- + +## Issue +Pods launched and completed for running jobs or scripts remain even after completion state.  They do not automatically clean themselves after completion + +## Cause +Kubernetes doesn't have an automated way to clean up jobs without enabling a feature flag on the master (which isn't possible on *most* managed Kubernetes services). +Spinnaker project also opted to *not* clean up jobs on completion because we found that our primary users didn't have any logging infrastructure in place to view logs after the fact which means that debugging jobs would be impossible for some users. + diff --git a/kb/armory/General/policy-engine---no-policy-decision-detected,-missing-spinnaker.persistence.pipelines.md b/kb/armory/General/policy-engine---no-policy-decision-detected,-missing-spinnaker.persistence.pipelines.md new file mode 100644 index 0000000000..81ee101413 --- /dev/null +++ b/kb/armory/General/policy-engine---no-policy-decision-detected,-missing-spinnaker.persistence.pipelines.md @@ -0,0 +1,11 @@ +--- +title: Policy Engine - No Policy Decision Detected, missing spinnaker.persistence.pipelines +--- + +## Issue +Even though it may appear that your declarations are correct, Policy Engine can possibly return the below error when initializing.  +```There was an error saving your pipeline: Policy Error: No policy decision detected. This is likely because a missing spinnaker.persistence.pipelines.before package.. [dismiss]``` + +## Cause +Setting Policy Engine to ```failOpen: false``` can cause the issue if no policies are declared.When Spinnaker sends a specific payload via Policy Engine to OPA to be validated by a policy and the policy doesn't exist, it will return the error message.  By default, with this setting Front50 will ```fail close```. + diff --git a/kb/armory/General/policy-engine---opa-for-styra-das-deployments.md b/kb/armory/General/policy-engine---opa-for-styra-das-deployments.md new file mode 100644 index 0000000000..d12f4982ab --- /dev/null +++ b/kb/armory/General/policy-engine---opa-for-styra-das-deployments.md @@ -0,0 +1,14 @@ +--- +title: Policy Engine - OPA for STYRA DAS Deployments +--- + +## Issue +Policy Engine will return errors when using the OPA configuration using the following format for the OPA URL when also using Styra DAS (Declarative Authorization Service): +```.../v1/data``` +For example, the following OPA URL: +```http://opa.tempcompany.com:8181/v1/data``` + + +## Cause +Styra DAS deployments require changes to the OPA URL to read policies. Policy Engine expects that policies are in separate file/package namespaces for OPA to consume.  Styra DAS deployments place all policies in a single ```rules.rego``` package instead. The OPA URL needs to be adjusted as the policies will be in a different location for consumption. + diff --git a/kb/armory/General/policy-engine--pluginruntimeexception--failed-to-write-file-'plugins-armory.policyengine-policy-engine-vx.x.x'-to-plugins-folder.md b/kb/armory/General/policy-engine--pluginruntimeexception--failed-to-write-file-'plugins-armory.policyengine-policy-engine-vx.x.x'-to-plugins-folder.md new file mode 100644 index 0000000000..feae7a8a3e --- /dev/null +++ b/kb/armory/General/policy-engine--pluginruntimeexception--failed-to-write-file-'plugins-armory.policyengine-policy-engine-vx.x.x'-to-plugins-folder.md @@ -0,0 +1,34 @@ +--- +title: Policy Engine- PluginRuntimeException- Failed to write file 'plugins/Armory.PolicyEngine-policy-engine-vX.X.X' to plugins folder +--- + +## Issue +When attempting to enable Policy Engine, the following error, or similar can be found in gate logs when attempting to use Armory Policy Engine with OSS and Halyard +``` +org.pf4j.PluginRuntimeException: Failed to write file 'plugins/Armory.PolicyEngine-policy-engine-vX.X.X.zip' to plugins folder + at com.netflix.spinnaker.kork.plugins.update.SpinnakerUpdateManager.write(SpinnakerUpdateManager.kt:174) + at com.netflix.spinnaker.kork.plugins.update.SpinnakerUpdateManager.download(SpinnakerUpdateManager.kt:102) + at com.netflix.spinnaker.kork.plugins.update.SpinnakerUpdateManager.downloadPluginReleases$kork_plugins(SpinnakerUpdateManager.kt:66) + at com.netflix.spinnaker.kork.plugins.v2.SpinnakerPluginService$initialize$1.invoke(SpinnakerPluginService.kt:81) + at com.netflix.spinnaker.kork.plugins.v2.SpinnakerPluginService$initialize$1.invoke(SpinnakerPluginService.kt:49) + at com.netflix.spinnaker.kork.plugins.v2.SpinnakerPluginService.withTiming(SpinnakerPluginService.kt:170) + at com.netflix.spinnaker.kork.plugins.v2.SpinnakerPluginService.initialize(SpinnakerPluginService.kt:71) + at com.netflix.spinnaker.kork.plugins.v2.PluginFrameworkInitializer.postProcessBeanDefinitionRegistry(PluginFrameworkInitializer.kt:37) + at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:280) + at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:109) + at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:707) + at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:533) + at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141) +[...] +Caused by: java.nio.file.NoSuchFileException: /tmp/pf4j-update-downloader00000000000000000000/policy-engine-vX.X.X.zip -> plugins/Armory.PolicyEngine-policy-engine-vX.X.X.zip + at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) + at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) + at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:478) + at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:267) + at java.base/java.nio.file.Files.move(Files.java:1422) + at com.netflix.spinnaker.kork.plugins.update.SpinnakerUpdateManager.write(SpinnakerUpdateManager.kt:172) + ... 47 common frames omitted +``` +## Cause +Additional definitions may be required to use Plugin with OSS, using Halyard. The plugin's original design and suggested use case is to be used with Armory Enterprise Operator and Halyard and is maintained with those installations in mind. + diff --git a/kb/armory/General/policy-engine-error-typeerror--e.takeuntil-is-not-a-function-when-accessing-project-tab.md b/kb/armory/General/policy-engine-error-typeerror--e.takeuntil-is-not-a-function-when-accessing-project-tab.md new file mode 100644 index 0000000000..09dd987cb7 --- /dev/null +++ b/kb/armory/General/policy-engine-error-typeerror--e.takeuntil-is-not-a-function-when-accessing-project-tab.md @@ -0,0 +1,22 @@ +--- +title: Policy Engine Error (TypeError- e.takeUntil is not a function) when Accessing Project Tab +--- + +## Issue +Users may encounter the following error when accessing the ```Projects Tab``` in the Spinnaker Console UI, while the Policy Engine plugin is enabled: +``` +TypeError: e.takeUntil is not a function + at Hr.componentDidMount (https://testapi.armory.io/plugins/deck/Armory.PolicyEngine/0.2.0-rc/index.js:18:39048) + at Ji (https://test.armory.io/3.js:370968:132) + at Sj (https://test.armory.io/3.js:371011:229) + at push.exports.unstable_runWithPriority (https://test.armory.io/3.js:371086:467) + at cg (https://test.armory.io/3.js:370878:325) + at Jj (https://test.armory.io/3.js:371004:370) + at yj (https://test.armory.io/3.js:370995:376) + at https://test.armory.io/3.js:370879:115 + at push.exports.unstable_runWithPriority (https://test.armory.io3.js:371086:467) + at cg (https://test.armory.io/3.js:370878:325) +``` +## Cause +Code changes need to be made to Policy Engine to make it compatible changes in Spinnaker + diff --git a/kb/armory/General/policy-engine-regoscript-examples.md b/kb/armory/General/policy-engine-regoscript-examples.md new file mode 100644 index 0000000000..6f0fcc50dc --- /dev/null +++ b/kb/armory/General/policy-engine-regoscript-examples.md @@ -0,0 +1,14 @@ +--- +title: Policy Engine RegoScript Examples +--- + +## Introduction +Customer looking for some guidance on Policy Engine scripts and creating policies should first look to the [OPA/Rego Script guidance](https://www.openpolicyagent.org/docs/latest/policy-language/) page.  However, customers may want some examples to get started and the guidance below can be a good reference to start with. + +## Prerequisites +Customers should [first install the Policy Engine Plugin](https://docs.armory.io/armory-enterprise/armory-admin/policy-engine/policy-engine-enable/policy-engine-plug-enable/) before proceeding + +## Instructions +Below are some standard examples.  If customers would like additional assistance beyond the below information, we recommend discussing options for a Professional Services engagement with the customer's account manager. +[https://docs.armory.io/armory-enterprise/armory-admin/policy-engine/policy-engine-use/example-policies/](https://docs.armory.io/armory-enterprise/armory-admin/policy-engine/policy-engine-use/example-policies/) + diff --git a/kb/armory/General/potential-outages-when-vault-integration-is-on-w--k8s-auth-method-and-updating-to-k8s-1.21.md b/kb/armory/General/potential-outages-when-vault-integration-is-on-w--k8s-auth-method-and-updating-to-k8s-1.21.md new file mode 100644 index 0000000000..ada1bab62a --- /dev/null +++ b/kb/armory/General/potential-outages-when-vault-integration-is-on-w--k8s-auth-method-and-updating-to-k8s-1.21.md @@ -0,0 +1,52 @@ +--- +title: Potential outages when Vault integration is on (w/ K8s auth method) and updating to K8s 1.21 +--- + +## Issue +On Kubernetes 1.21 the [ServiceAccount Issuer Discovery feature](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-issuer-discovery) is on stable release and is enabled by default. This means that the JSON Web Token (JWT) format of the service accounts is changing to have a more secure formatPrevious format: + +``` +{ + "iss": "kubernetes/serviceaccount", + "kubernetes.io/serviceaccount/namespace": "spinnaker", + "kubernetes.io/serviceaccount/secret.name": "test-token-5v2cp", + "kubernetes.io/serviceaccount/service-account.name": "test", + "kubernetes.io/serviceaccount/service-account.uid": "0ecb5560-7d43-4883-ae85-d07cf635d2d2", + "sub": "system:serviceaccount:spinnaker:test" +} +``` +  +New format: +``` +{ + "aud": [ + "https://kubernetes.default.svc" + ], + "exp": 1661509326, + "iat": 1629973326, + "iss": "https://oidc.server.something", + "kubernetes.io": { + "namespace": "spinnaker", + "pod": { + "name": "debugging-tools-6464df994b-46wsq", + "uid": "90451169-29cb-4e2d-8ee8-4c1e2c293a3c" + }, + "serviceaccount": { + "name": "test", + "uid": "affc78ef-fa4b-4ba8-bb00-f9cc51d65408" + }, + "warnafter": 1629976933 + }, + "nbf": 1629973326, + "sub": "system:serviceaccount:spinnaker:test" +} +``` + +## Cause +The change in formatting will potentially break the Vault Kubernetes authentication method depending on the Vault Kubernetes backend configuration, throwing the message: + +```ISS claim invalid``` + +  +Therefore, this will cause Spinnaker and Spinnaker-operator not to be able to retrieve secrets from Vault. + diff --git a/kb/armory/General/primary-and-secondary-contacts-and-customer-service-portal-administrators.md b/kb/armory/General/primary-and-secondary-contacts-and-customer-service-portal-administrators.md new file mode 100644 index 0000000000..14e814a5e5 --- /dev/null +++ b/kb/armory/General/primary-and-secondary-contacts-and-customer-service-portal-administrators.md @@ -0,0 +1,34 @@ +--- +title: Primary and Secondary Contacts and Customer Service Portal Administrators +--- + + +Primary and Secondary Contacts are the users that are listed within a [customer account](https://armory.service-now.com/customer_account_list.do?sysparm_userpref_module=25f7d04cc322310015519f2974d3ae43&sysparm_view=case&sysparm_query=customer%3Dtrue%5EEQ&sysparm_clear_stack=true) (ServiceNow Admin access required).  These are the contacts outlined by the company as their main administrators, and points of contact for Armory.  We will use these contacts for any communications such as for NPS surveys and any other issues that we must contact them about (e.g. security items/alerts) +Customer accounts can have more than 2 administrators as long a user is a part of the ```[Armory - Customer Administration](https://armory.service-now.com/sys_user_group.do?sys_id=f552d87edb86281079f53ec8f4961979&sysparm_record_target=sys_user_group&sysparm_record_row=4&sysparm_record_rows=24&sysparm_record_list=ORDERBYname) ```group.   +### What are Customer Service Portal Administrators? +Administrators will manage the user accounts and will be the point of user security/contact for an account.  The idea is that the Administrators will be responsible for maintaining the health of their users on their account.  We are providing this self management so that companies can add and remove access for their users as necessary, so that we don't have to be administrative bottlenecks.  This also keeps the security and access update responsibilities with our accounts instead of making it an Armory responsibility.   +### What are the Administrators Additional Abilities? +On the surface, they have the ability to add user accounts, and also to disable/re-enable accounts.  However, because users are able to request changes for their environments to Managed/Armory Cloud, and ask for additional information about their accounts, there is a potential for malicious attack via user access.  Administrators can therefore, also audit their user lists as well.  +Administrators can also assign additional administrators to the account. +### What will our responsibilities be? +Largely, we will be administering the Administrators.  This will lessen the burden for CS in General. +### Why have we moved from Self Registration to Self Service? +From a security standpoint, for SOC compliance, the general burden of auditing whether a particular user is supposed to have a level of access, or still have access is now up to the client.  It's no longer going to be necessary for Armory to send them an audit notice to check the people that they have assigned to have access.   For example, if customers could self register, we would normally provide the company a list of users, and ask them to check it, once every year or so, depending on the cadence.  +Now, with the way the access has been set, it is within the customer's power to remove access for an employee who has left their organization.  This fits better with their own security needs +#### What security issues does this help prevent?  +For managed services especially, people who have access to the ticketing system have access to request a change in their environment.  If the access was left open for anyone in the company to get access, this can potentially mean a bad actor could gain access and request a detrimental change to their environment (Speaking as a worst case scenario) +Similarly, we could end up providing information as support, to unauthorized personnel.  (Speaking as a worst case scenario again) +Keep in mind that if the customer has a massive list of changes, that we can still assist with creating those accounts.  We would just require a list including first and last names and email addresses.   +## Adding/Editing Primary & Secondary Administrators  + +If a primary and secondary contact needed to be added to a customer account, they will need to be added via the customer account.  **Any changes should be recorded in Slack or a Ticket.  If a customer wants to edit the administrators, please confirm if the people being removed should still be administrators or not.** If they need to be removed, follow the instructions further below +* Log in to the Service Now portal, and proceed to the Service Management Administration console* Search for ```Accounts``` in the left navigation menu* Under ```Customer Service```, there is a section called ```Accounts```* Find the Customer's Account by name.  Click on it* There are two fields.  The ```Primary Contact``` and the ```Secondary Contact```.  For either one, click on the Magnifying Glass* Click on the name of the user that will be the contact.  * Click on the ```Update``` Button* A message will appear on the screen confirming the update is processed and that the roles were added to the user's account.* Advise the user and have them test and see if they have access to **Manage Accounts** from the main portal page.  You can also use the "Impersonate User" function to test, if you have Service Now Administrative Priviledges +## Adding Additional Administrators +If a primary and secondary contact are already added to a customer account, and they wish to have more administrators, then we will need to add them manually to the Group.  **Any changes should be recorded in a ticket or in Slack for the purpose of documentation** +* Log in to the Service Now portal, and proceed to the Service Management Administration console* Search for **Groups** in the left navigation menu* Under **User Administration**, there is a section called **Groups*** Find the ```Armory - Customer Administration``` Group.  Click on it* Go to the **Group Members** Tab* Click **Edit**.  Find the user and add them to the **Group Members List** (Right pointing arrow). Click on **Save*** Click on the **Update** Button +## Removing Administrators +Removing a Primary and Secondary contact will not remove their administrative access.  Disabling their login will stop their access.  If a customer needs to change their contacts, **please also confirm if they wish for those contacts to remain administrators or not**.  **This interaction should be recorded in a ticket for documentation purposes** +* Log in to the Service Now portal, and proceed to the Service Management Administration console* Search for **Groups** in the left navigation menu* Under **User Administration**, there is a section called **Groups*** Find the ```Armory - Customer Administration``` Group.  Click on it* Go to the **Group Members** Tab* Click **Edit**.  Find the user and remove them to the **Group Members List** (Left pointing arrow). Click on **Save*** Click on the **Update** Button +  + + diff --git a/kb/armory/General/producing-artifacts-in-the-webhook-stage.md b/kb/armory/General/producing-artifacts-in-the-webhook-stage.md new file mode 100644 index 0000000000..9a353f701d --- /dev/null +++ b/kb/armory/General/producing-artifacts-in-the-webhook-stage.md @@ -0,0 +1,28 @@ +--- +title: Producing Artifacts in the Webhook Stage +--- + +## Introduction +The Webhook Stage is capable of producing artifacts which can be used in downstream stages. This is done by returning a valid array of artifacts in the response of the webhook that you're calling. For example, your webhook may be responsible for generating an artifact and pushing it into an object store such as S3.  + +## Prerequisites +N/A + +## Instructions +To correctly inform Spinnaker of this artifact, your webhook should respond with the following JSON response. +``` +{ + "artifacts": [ + { + "type": "s3/object", + "reference": "s3://webhook-example-bucket/webhook-example-object.yaml" + } + ] +} +``` +In order to use these artifacts within the pipeline, you must first add them under the pipeline **Expected Artifacts section**. + +Once you've added a Webhook stage, be sure to add your artifact under the **Produces Artifacts** section in the stage configuration. + +For more information on Webhooks, please take a look at the following documentation in our Docs library: [https://docs.armory.io/docs/spinnaker-user-guides/webhooks/](https://docs.armory.io/docs/spinnaker-user-guides/webhooks/) + diff --git a/kb/armory/General/provide-logs-from-pods-to-armory-support-for-troubleshooting-purposes.md b/kb/armory/General/provide-logs-from-pods-to-armory-support-for-troubleshooting-purposes.md new file mode 100644 index 0000000000..d1269dd27b --- /dev/null +++ b/kb/armory/General/provide-logs-from-pods-to-armory-support-for-troubleshooting-purposes.md @@ -0,0 +1,16 @@ +--- +title: Provide Logs from Pods to Armory Support for Troubleshooting Purposes +--- + +## Introduction +In the event that the customer's environment is Airgapped, or unable to [send logs to be monitored by Armory](https://kb.armory.io/s/article/Confirming-Environment-Logs-and-Environment-Debugging-ID-with-Armory-Support), users will need to extract the logs from the pods and provide them as attachments when opening tickets + +## Prerequisites +Access to the pods via kubectl + +## Instructions +It is possible to provide a snippet of the logs by accessing the pod and then copying and pasting the relevant sections  +```kubectl logs -n %namespace %podname -f``` +If instead, a set of complete logs is required from the pod, the logs can also be extracted by using the following command +```kubectl logs -n %namespace %podname > %servicename.log``` + diff --git a/kb/armory/General/pull-request-validation-error---file-not-found-module.md b/kb/armory/General/pull-request-validation-error---file-not-found-module.md new file mode 100644 index 0000000000..f380167950 --- /dev/null +++ b/kb/armory/General/pull-request-validation-error---file-not-found-module.md @@ -0,0 +1,10 @@ +--- +title: Pull Request Validation Error - File not Found (Module) +--- + +## Issue +When executing a PR Validation in GitHub on a new change ([https://docs.armory.io/docs/armory-admin/dinghy-enable/#pull-request-validations](https://docs.armory.io/docs/armory-admin/dinghy-enable/#pull-request-validations)), users are unable to validate the execution. If users run ARM-CLI ([https://github.com/armory-io/arm](https://github.com/armory-io/arm)) locally to validate the dinghyfile and modules, it returns no errors.When executing the PR Validation, there is an error about ```File Not Found``` on the modules + +## Cause +Executing a PR that includes a change to the **dinghyfile** along with a change to **modules** (addition of them** **or removal of them) causes the failure.  This is because the PR validation is validating against the repo structure of the module files in the master branch at the time of execution.  Because the modules are being added at the same time as the dinghyfile, they are "missing" because they do not already exist in the structure + diff --git a/kb/armory/General/quiet-the-echo-service.md b/kb/armory/General/quiet-the-echo-service.md new file mode 100644 index 0000000000..cbf8e9ac96 --- /dev/null +++ b/kb/armory/General/quiet-the-echo-service.md @@ -0,0 +1,75 @@ +--- +title: Quiet the Echo Service +--- + +## Introduction +When planning and upgrade or maintenance in Spinnaker, Echo by default will keep on triggering notification which may cause unwanted errors or issues with stuck pipelines. Quieting Echo during the maintenance time will prevent unwanted echo triggers during that time. With triggers suppressed this effectively stops pipelines from being run if they are flagged to respect the quiet period.  +There are two options to quieting the Echo service. The first option is to disable communication to Orca. This will prevent ANY trigger from firing and cannot be scheduled. The second option is enable ```quietPeriod``` in Echo. This allows a quiet period to be scheduled and allows the operator to configure which triggers should be suppressed. + + +## Prerequisites +Suppressing all triggers by disabling communication to Orca +In the Echo profile file ```echo-local.yml``` add the following configuration ```orca.enabled: false``` and redeploy Echo. This will suppress all triggers until this setting is changed back to ```true``` and the Echo service is redeployed. + +## Instructions +Enable quietPeriod +**NOTE:** Pipelines **MUST** be configured to respect the quiet period for this to work!!! +The following sample config will suppress the list of triggers in ```suppressedTriggerTypes``` from ```startIso``` to ```endIso``` if ```quietPeriod.enabled: true```. The triggers will automatically start working again after the ```endIso``` time. It can also be manually disabled by changing ```quietPeriod.enabled``` back to ```false```. +In Echo’s profile +In the Echo profile file ```echo-local.yml``` in Halyard's hal config profiles directory e.g. (```~/.hal/default/profiles/```) or in Operator under the ```spec.spinnakerConfig.profiles.echo, ```add the following configuration and replace everything in between ```<>``` in the ```startISO``` and ```endISO``` parameters and change the list of ```suppressedTriggerTypes``` as needed. + +quietPeriod: + enabled: true + startIso: + endIso: + # see https://github.com/spinnaker/echo/blob/64cb72de7648a82d392db459e98026d1a9c16959/echo-model/src/main/java/com/netflix/spinnaker/echo/model/Trigger.java#L77 for list of triggers + # The following will suppress all triggers including a Manual trigger + suppressedTriggerTypes: + - cron + - webhook + - git + - concourse + - jenkins + - docker + - pubsub + - dryrun + - pipeline + - plugin + - helm + - manual + +In Deck’s settings-local.js +In Deck’s ```settings-local.js``` file in Halyard's hal config profiles directory e.g. (```~/.hal/default/profiles/```) add the following:  +```window.spinnakerSettings.feature.quietPeriod = true;``` +  +In Operator under the ```spec.spinnakerConfig.profiles.deck.settings-local.js:``` +spec: + spinnakerConfig: + profiles: + deck: + settings-local.js: + window.spinnakerSettings.feature.quietPeriod = true; +`````` +Update Pipeline configs to respect quietPeriod +For a pipeline to respect Echo’s ```quietPeriod``` setting it must have the following set in its pipeline definition at the same indent level as ```stages``` and ```triggers```: +```"respectQuietPeriod": true``` +If this is not set it is assumed that the pipeline does NOT respect Echo’s quietPeriod setting. +**Example pipeline config snippet** + +{ + "appConfig": {}, + "expectedArtifacts": [], + "keepWaitingPipelines": false, + "lastModifiedBy": "admin", + "limitConcurrent": true, + "respectQuietPeriod": true, + "spelEvaluator": "v4", + "stages": [], + "triggers": [], + "updateTs": "1625697650000" +} + + +**Update config via the Deck UI** +**** + diff --git a/kb/armory/General/redis-to-sql-migration--can't-see-old-history.md b/kb/armory/General/redis-to-sql-migration--can't-see-old-history.md new file mode 100644 index 0000000000..908292219b --- /dev/null +++ b/kb/armory/General/redis-to-sql-migration--can't-see-old-history.md @@ -0,0 +1,12 @@ +--- +title: Redis to SQL Migration- Can't see old history +--- + +## Issue +After Migration from Redis to AWS Elasticache SQ, users cannot see the old execution history.    +When going to the Spinnaker UI, the pipelines that have the old history. Get the ```pipeline ID``` by clicking on configure and copying the last part of the URL after ```/configure/```. This will be referred to as ```${PIPELINE-ID}``` in the rest of the article. +The Redis in production needs to have execution keys, and users will need this execution key to proceed. + +## Cause +None + diff --git a/kb/armory/General/reduce-kubeconfig-size.md b/kb/armory/General/reduce-kubeconfig-size.md new file mode 100644 index 0000000000..8677a51605 --- /dev/null +++ b/kb/armory/General/reduce-kubeconfig-size.md @@ -0,0 +1,10 @@ +--- +title: Reduce kubeconfig Size +--- + +## Issue +If you’re seeing issues with halyard taking forever or complaining that your kubeconfig is too large, it may be worth reducing your kubeconfig or removing unnecessary certificate authorities from your kubeconfig. + +## Cause +Kubeconfig file sizes can cause a long time to load because of the way kubeconfig yaml files are parsed + diff --git a/kb/armory/General/renaming-a-configmap-creates-a-new-copy-and-does-not-delete-the-original.md b/kb/armory/General/renaming-a-configmap-creates-a-new-copy-and-does-not-delete-the-original.md new file mode 100644 index 0000000000..cb96f52309 --- /dev/null +++ b/kb/armory/General/renaming-a-configmap-creates-a-new-copy-and-does-not-delete-the-original.md @@ -0,0 +1,11 @@ +--- +title: Renaming a ConfigMap creates a new copy and does not delete the original +--- + +## Issue +When a ConfigMap is modified or renamed, a new copy is created. The ConfigMap with the original name is not impacted or deleted. + +## Cause +Retaining previous versions of resources like ConfigMaps is the intended behavior. +According to Spinnaker's default Resource Management Policy - if a resource is versioned, it is **always** deployed with a new sequence number, in the format of ```vNNN```, unless no change has been made to it. This is the default behavior for resources like ConfigMaps and ReplicaSets, which don't have their own built-in update policies. + diff --git a/kb/armory/General/renaming-an-application-in-spinnaker.md b/kb/armory/General/renaming-an-application-in-spinnaker.md new file mode 100644 index 0000000000..d99d18e648 --- /dev/null +++ b/kb/armory/General/renaming-an-application-in-spinnaker.md @@ -0,0 +1,11 @@ +--- +title: Renaming An Application in Spinnaker +--- + + +Question: +Can I Rename An Application in Spinnaker? +Answer: +No, and yes. Can you, yes. Should you, no. The preferred approach is to create a new application and copy and paste your pipelines over from the other application. +A primary reason of why not to rename the application is because Spinnaker names resources based on your application name. For example if you have a deployment stage, it will deploy the resources based on the application name. If you change the application name, there is a chance you could orphan those resources. + diff --git a/kb/armory/General/render-and-validate-dinghy-pipelines-with-armory-cli.md b/kb/armory/General/render-and-validate-dinghy-pipelines-with-armory-cli.md new file mode 100644 index 0000000000..c4ba87e087 --- /dev/null +++ b/kb/armory/General/render-and-validate-dinghy-pipelines-with-armory-cli.md @@ -0,0 +1,15 @@ +--- +title: Render and Validate Dinghy Pipelines with Armory CLI +--- + +## Introduction +ARM CLI is a standalone tool that can also let you render Dinghyfiles and modules offline +This walkthrough and explanation from Armory Engineer, Jossue Gamez, provides an example for how to use Armory CLI with Dinghy pipelines with several examples: +* Basic* RawData* Conditionals* Globals* MakeSlice* MakeSlice with Conditionals and Raw Data + +## Prerequisites +N/A + +## Instructions +How to Render and Validate Dinghy Pipelines: + diff --git a/kb/armory/General/replicaset-versioning-is-erratic,-sometimes-will-work,-sometimes-will-not.md b/kb/armory/General/replicaset-versioning-is-erratic,-sometimes-will-work,-sometimes-will-not.md new file mode 100644 index 0000000000..0c71c1b716 --- /dev/null +++ b/kb/armory/General/replicaset-versioning-is-erratic,-sometimes-will-work,-sometimes-will-not.md @@ -0,0 +1,20 @@ +--- +title: ReplicaSet Versioning is erratic, sometimes will work, sometimes will not +--- + +## Issue +When trying to use ReplicaSet Versioning in order to maintain a solid backup system, an organization may run into a known issue where versioning will perform erratically and either not update the version at all or even revert the versioning number. Examplev000 > v001 > v002 > v003 > v002 > v003 +Additionally, previous versions can be deleted after some time, which removes the value of having a backup. + +## Cause +For some of these issues, this is a cache invalidation issue which can arise in any Spinnaker or Armory version prior to 2.23. + +The difference between older versions of Spinnnaker is that live manifest calls only searches for the live manifest of what’s it is currently deploying (i.e. ```replicaset $APPLICATION```), while ```2.23.x``` does query all the related artifacts before evaluating runtime labels (e.g. the previous versions currently deployed to get newer version). + +This can be observed in the ```orca``` logs when deploying applications, but as of version ```2.23.x, ```live manifests calls happen in ```clouddriver```. + +This issue can be exacerbated if an organization uses a stage called ```Delete ReplicaSets``` which is done via a ```custom runJobManifest``` instead of the Spinnaker-managed ```Delete (Manifest)``` stage, + +These are most likely the reasons the Spinnaker cache becomes invalid and makes versioning run incorrectly on the next pipeline call. This can be erratic, especially if wait stages are used.  If the caching agent happens to get a chance to refresh the cache before the next stage, these problems will not occur.  + + diff --git a/kb/armory/General/resolve-cluster-wide-dockerhub-rate-limit-warning.md b/kb/armory/General/resolve-cluster-wide-dockerhub-rate-limit-warning.md new file mode 100644 index 0000000000..796b396bf8 --- /dev/null +++ b/kb/armory/General/resolve-cluster-wide-dockerhub-rate-limit-warning.md @@ -0,0 +1,35 @@ +--- +title: Resolve cluster-wide DockerHub Rate Limit Warning +--- + +## Issue + +During Spinnaker deployment, some or all the component pods are in an ```ImagePullBackOff``` status. This can be confirmed by checking the full warning message by describing one of the affected pods: +  + +```kubectl -n describe pod ``` +  + +In the Events block of the affected pod the following message similar to the following should be found: +  + + Warning Failed 14m (x4 over 15m) kubelet \ + Failed to pull image "docker.io/armory/echo:2.26.10": \ + rpc error: code = Unknown desc = Error response from daemon: \ + toomanyrequests: You have reached your pull rate limit. You may \ + increase the limit by authenticating and upgrading: \ + https://www.docker.com/increase-rate-limit +  +This is likely to happen where multiple users are working with a single Kubernetes Cluster, or if a cluster grows extremely large and has many changes/redeployments.  + + + + + +## Cause + +This is because DockerHub rate limits anonymous container pulls from their service to a maximum of 100 every 6 hours while free authenticated DockerHub users are limited to 200 container image pulls every 6 hours . +  +In order to get around this limit, users will need to create a DockerHub account and configure registry credentials within the namespace that Spinnaker is being deployed. Therefore, pulling container images as an authenticated user gives 100 more image pull requests before hitting the limit. + + diff --git "a/kb/armory/General/resource-requests-on-custom-stages-does-not-work--error--got-\"map\",-expected-\"string\".md" "b/kb/armory/General/resource-requests-on-custom-stages-does-not-work--error--got-\"map\",-expected-\"string\".md" new file mode 100644 index 0000000000..0c1ee24d81 --- /dev/null +++ "b/kb/armory/General/resource-requests-on-custom-stages-does-not-work--error--got-\"map\",-expected-\"string\".md" @@ -0,0 +1,22 @@ +--- +title: Resource requests on custom stages does not work- Error- got "map", expected "string" +--- + +## Issue +An organization may want to set a ```resource request (CPU)``` on jobs for custom stages.  Improper settings can result in an error similar to the one below:```Create failed: error: error validating “STDIN”: error validating data: [ValidationError(Job.spec.template.spec.containers[0].resources.requests.cpu): invalid type for io.k8s.apimachinery.pkg.api.resource.Quantity: got “map”, expected “string”,``` +An example manifest snippet is provided below. + spec: + restartPolicy: Never + containers: + - name: ###### + image: ######:latest + imagePullPolicy: Always + resources: ##### I ADDED THIS TO REQUEST RESOURCES + requests: ##### I ADDED THIS TO REQUEST RESOURCES + cpu: 100m ##### I ADDED THIS TO REQUEST RESOURCES +The above manifest will not work and results in the error: ```resources.requests.cpu > invalid type > got “map”, expected “string”```*,* +These settings can cause organizations to encounter situations where resources are not being used effectively or efficiently. + +## Cause +This error is due to a known bug in Spinnaker Operator.  The manifest field defined in the ```KubernetesPreconfiguredJobProperties``` class has ```io.kubernetes.client.openapi.models.V1``` Job type.  This type is defined in the Kubernetes Open API models package.  This package contains special JSON serializers e.g ```io.kubernetes.client.custom.Quantity.QuantityAdapter```.  The serializers must be used to get the correct JSON format. + diff --git a/kb/armory/General/resources-are-duplicated-in-the-ui-when-migrating-to-armory-agent.md b/kb/armory/General/resources-are-duplicated-in-the-ui-when-migrating-to-armory-agent.md new file mode 100644 index 0000000000..7f590717da --- /dev/null +++ b/kb/armory/General/resources-are-duplicated-in-the-ui-when-migrating-to-armory-agent.md @@ -0,0 +1,13 @@ +--- +title: Resources are duplicated in the UI when Migrating to Armory Agent +--- + +## Issue +Admins and users may notice that after deploying Armory Agent, as a part of the process, customers will need to then remove the previous accounts set up in Clouddriver.  After removing the accounts from Clouddriver, resources appear duplicated on Deck. The UI would show something similar to the following example: + +In this case, nothing was changed on the Kubernetes clusters, and the resources present are the same. +  + +## Cause +Spinnaker would normally clean up any caching issues of unknown objects periodically as noted on these [OSS docs](https://spinnaker.io/docs/setup/productionize/persistence/clouddriver-sql/#configure-clouddriver-to-use-mysql)However, this issue is a known symptom where Spinnaker not performing a refresh of its cache.  As a result, the previous settings that were in Clouddriver and the cached values will remain even after migrating the Kubernetes accounts from Clouddriver to Agent.  The objects are persisted in the both old and the new cache of Clouddriver.  + diff --git a/kb/armory/General/restarting-spinnaker-unexpectedly-triggers-many-jenkins-jobs-when-using-redis.md b/kb/armory/General/restarting-spinnaker-unexpectedly-triggers-many-jenkins-jobs-when-using-redis.md new file mode 100644 index 0000000000..5321abd58f --- /dev/null +++ b/kb/armory/General/restarting-spinnaker-unexpectedly-triggers-many-jenkins-jobs-when-using-redis.md @@ -0,0 +1,12 @@ +--- +title: Restarting Spinnaker Unexpectedly Triggers Many Jenkins Jobs when using Redis +--- + +## Issue +When restarting Spinnaker, many unexpected Jenkins jobs are triggered.  This process can cause a lot of unintended consequences for the environment.   +This issue occurs when the default Spinnaker Redis service and the Redis database and service are not persistent.  (e.g., The environment is not using an external Redis or RDS MySQL) + +## Cause +If we use the default ```spin-redis``` service in SpinnakerService Custom Resource (CR), and the Redis service is not persisted.  The Redis service will restart when Spinnaker restarts.   +As a consequence, the Jenkins jobs will be triggered, and this is by design.  Restarting a setup with Redis configured in this way will also cause the loss of the execution data. + diff --git a/kb/armory/General/restrict-application-creation.md b/kb/armory/General/restrict-application-creation.md new file mode 100644 index 0000000000..f155c2be97 --- /dev/null +++ b/kb/armory/General/restrict-application-creation.md @@ -0,0 +1,81 @@ +--- +title: Restrict Application Creation +--- + +## Introduction +Before version 2.17, there was no way to prevent application creation in Spinnaker. In Armory Spinnaker 2.17 and later, Fiat can now control application creation through the use of a new permission option: ```CREATE```. +**Note**: When configuring permissions, you must explicitly configure permissions for each user role. The default for a user role is no permissions, which means it cannot perform any actions. +This document assumes that you have enabled and configured Fiat. +**This was tested on version 2.17 and may change in later versions.** + +## Prerequisites +Fiat Needs to be Enabled + +## Instructions +Fiat is the authorization (authz) microservice of Spinnaker, which looks for the permissions from different sources. In 2.17, a new sources were added, providing more flexibility for applying permissions. This page focuses on the ```prefix``` source to control permissions for any applications whose name starts with a given prefix. To use this functionality, you need to enable Fiat to use the new sources and set prefixes as one of the sources. ```auth.permissions.source.application.prefix```.  +Perform the following steps: +* In ```fiat-local.yml```, set the value for ```auth.permissions.provider.application``` parameter to ```aggregate```.Add prefixes as a source: + +```````````` + auth.permissions.source.application.prefix: + enabled: true +```````````` + +Define the permissions for a prefix: + - prefix: + permissions: + READ: + - "" + - "" + - "" + WRITE: + - "" + EXECUTE: + - "user role n>"​ +The below example does the following: + +* Enables Fiat to read from the new sources.* Sets ```prefix``` as one of these new sources.* Defines the prefix ```apptest-x```.Defines permissions for all applications that match the prefix ```apptest-*``` based on roles: +* Add READ, WRITE and EXECUTE permissions to ```role-one```.* Add READ permissions to ```role-two```. + + + +In the Halyard Profiles folder, in the``` fiat-local.yml``` file add to or create the file if it doesn't exist: + auth.permissions.provider.application: aggregate + auth.permissions.source.application.prefix: + enabled: true + prefixes: + - prefix: "apptest-*" + permissions: + READ: + - "role-one" + - "role-two" + WRITE: + - "role-one" + EXECUTE: + - "role-one" +  `````` + + +As a result, any application that matches the prefix ```apptest-*``` with ```role-two``` are read-only. +To restrict application creation specifically, add ```fiat.restrictApplicationCreation``` and set it to ```true```: +**Note: Currently, the prefix source is the only source that support the CREATE permission.**In the Halyard Profiles folder, in the``` fiat-local.yml``` file: + fiat.restrictApplicationCreation: true + auth.permissions.provider.application: aggregate + auth.permissions.source.application.prefix: + enabled: true + prefixes: + - prefix: "*" + permissions: + CREATE: + - "role-one" + READ: + - "role-one" + - "role-two" + WRITE: + - "role-one" + EXECUTE: + - "role-one" + +The above example assigns CREATE permission to users with the ```role-one``` role. Users without the ```role-one``` role cannot create any applications in Spinnaker because only ```role-one``` has CREATE permission. +* Finally, apply your configuration changes to Spinnaker by running the following command: ```hal deploy apply```. + diff --git a/kb/armory/General/restricting-the-user-into-specific-namespace-in-eks-cluster.md b/kb/armory/General/restricting-the-user-into-specific-namespace-in-eks-cluster.md new file mode 100644 index 0000000000..4740b19b52 --- /dev/null +++ b/kb/armory/General/restricting-the-user-into-specific-namespace-in-eks-cluster.md @@ -0,0 +1,10 @@ +--- +title: Restricting the user into specific namespace in EKS cluster +--- + +## Issue +Admins may want to be able to restrict certain users into specific namespaces in their EKS clustersk8s community has asked for a similar feature: [https://github.com/kubernetes/kubernetes/issues/56582](https://github.com/kubernetes/kubernetes/issues/56582) + +## Cause +N/A - Feature request + diff --git "a/kb/armory/General/rosco's-\"bakescompleted_seconds_count\"-metric-unavailable-in-prometheus.md" "b/kb/armory/General/rosco's-\"bakescompleted_seconds_count\"-metric-unavailable-in-prometheus.md" new file mode 100644 index 0000000000..9d9d31901d --- /dev/null +++ "b/kb/armory/General/rosco's-\"bakescompleted_seconds_count\"-metric-unavailable-in-prometheus.md" @@ -0,0 +1,15 @@ +--- +title: Rosco's bakesCompleted_seconds_count metric unavailable in Prometheus +--- + +## Issue +After installing the Observability Plugin, the ```bakesCompleted``` metric set (count, max & sum) from Rosco is not displayed in Prometheus even though the Prometheus Targets screen shows that all pods being up and being scraped regularly. +  +Port forwarding the Rosco pod and hitting the ```/aop-prometheus``` endpoint shows metrics are registered but not the ```bakesCompleted``` metrics set. The metrics are also not visible in Grafana. + + +## Cause +For ```Observability Plugin version v1.2.1``` - A successful bake is required after plugin installation for the metric to be registered. + +If ```Observability plugin v1.2.0``` is being used there is a [bug (resolved in v1.2.1)](https://github.com/armory-plugins/armory-observability-plugin/issues/26) that will prevent some metrics being displayed in Prometheus as well as the above. + diff --git a/kb/armory/General/rosco-will-crash-causing-a-failed-pipeline-erratically.md b/kb/armory/General/rosco-will-crash-causing-a-failed-pipeline-erratically.md new file mode 100644 index 0000000000..9d292c6a78 --- /dev/null +++ b/kb/armory/General/rosco-will-crash-causing-a-failed-pipeline-erratically.md @@ -0,0 +1,11 @@ +--- +title: Rosco will crash causing a failed pipeline erratically +--- + +## Issue +An organization may run into what appears to be an intermittent and erratic failure of ```Rosco``` causing a failed pipeline. This will typically happen after several weeks or months of usage. This will happen intermittently and without any seeming major cause.  + +## Cause +Rosco will give error logs stating failed metrics and metric batches. This error will not provide specific details, and the errors will be intermittent with no clear cause or trail.  +The above can be a red herring.  This issue can result from a lack of resources - typically running out of memory for Bake AMIs. Depending on the size of the application it may or may not trigger this error. + diff --git a/kb/armory/General/run-a-generic-shell-script-with-spinnaker.md b/kb/armory/General/run-a-generic-shell-script-with-spinnaker.md new file mode 100644 index 0000000000..1354d0e729 --- /dev/null +++ b/kb/armory/General/run-a-generic-shell-script-with-spinnaker.md @@ -0,0 +1,92 @@ +--- +title: Run a Generic Shell Script with Spinnaker +--- + +## Introduction +Spinnaker is a tool specialized in completing Deployments *natively*, without having to write shell scripts. That said, sometimes when end users may need to run custom logic related to deployments. Spinnaker has a stage called ```Script```to help deal with this logic.  However, this execution relies on a Jenkins job under the hood and has been deprecated in favor of the ```Jenkins``` stage. +If end users want to run a script, then Kubernetes jobs can be used to execute, but then the user will need to develop a Docker container that contains the script. A solution is to use a generic docker container and mount the script in a Kubernetes ```ConfigMap```. +In this way, end users can define the script contents as part of the deployment ```yaml``` for the ```ConfigMap```. + +## Prerequisites +N/A + +## Instructions + +## Table of Contents +*  * [Basic Method](#basic-approach)* [Taking the Shell Script from Version Control](#taking-the-shell-script-from-version-control) + + +Basic Method +Stage: ```Deploy (Manifest)``` +``` +apiVersion: v1 +data: + script.sh: |- + echo "Hello world!" + kubectl get pods +kind: ConfigMap +metadata: + name: script-configmap + namespace: german +--- +apiVersion: batch/v1 +kind: Job +metadata: + labels: + app: script-job + name: script-job + namespace: german +spec: + backoffLimit: 2 + template: + spec: + containers: + - command: + - sh + - /opt/script/script.sh + image: 'bitnami/kubectl:1.12' + name: script + volumeMounts: + - mountPath: /opt/script + name: script-configmap + readOnly: false + restartPolicy: Never + serviceAccountName: spinnaker-service-account + volumes: + - configMap: + name: script-configmap + name: script-configmap +``` +Here we define a generic ```Deploy (Manifest)``` stage, supplying the above text contents. +End users define the ConfigMap as well as the Job. The script is part of the ConfigMap, and the Job runs a generic Docker container from the Docker hub with the tool ```kubectl```. The job will execute a command, which is the script mounted in the ConfigMap. +In the situation there is a need to edit the script, this can be done without generating another Docker image.  Another consideration is that jobs can be run only once in Kubernetes and are immutable, so the end user will also need to add a ```Delete (Manifest)``` stage for cleaning up/deleting old jobs before another ```Deploy (manifest)``` stage is executed. +  +Retrieving the Shell Script from Version Control +The basic approach is fine for small scripts, but writing something more complex inside a ConfigMap definition is no fun.Fortunately, a trick in Spinnaker allows users to read file contents from an artifact, which can be a ```GitHub``` repo file, ```Bitbucket``` repo file, ```AWS S3``` file, and any supported Spinnaker artifact that refers to text files.See the complete list of supported artifact sources: [https://spinnaker.io/docs/setup/other_config/artifacts/](https://spinnaker.io/docs/setup/other_config/artifacts/) +In the following example, a GitHub repository is used as an artifact source for the script file. +End users will a ```Webhook``` stage to make a request to Spinnaker’s ```Clouddriver``` service.  The service already has all the credentials needed for downloading artifacts.  However, administrators [may need to enable and supply those artifact credentials through Spinnaker configuration](https://docs.armory.io/continuous-deployment/armory-admin/artifacts-github-connect/#configure-github-as-an-artifact-source), if they don't already exist. Clouddriver will use the same logic it already uses in other parts of Spinnaker to download the file and return it to the ```Webhook``` stage. The output of this stage (and hence the file) is available in the pipeline execution context, so user can then refer to it using [Pipeline Expressions](https://www.spinnaker.io/guides/user/pipeline/expressions/) in the ConfigMap definition: + +#### Example of the Webhook stage configuration +```` +Webhook URL: http://spin-clouddriver:7002/artifacts/fetch +Method: PUT +Payload: { + "artifactAccount": "", + "reference": "https://api.github.com/repos///contents/", + "type": "github/file", + "version": "master" + } +```` +#### Example of the ```Deploy (Manifest) - ConfigMap``` +``` +apiVersion: v1 +data: + script.sh: '${#stage("Webhook")["context"]["webhook"]["body"]}' +kind: ConfigMap +metadata: + name: script-configmap + namespace: german +```  +Then the ```Run job``` stage can contain only the yaml description of the kubernetes job. +  + diff --git a/kb/armory/General/run-job---producing-artifacts.md b/kb/armory/General/run-job---producing-artifacts.md new file mode 100644 index 0000000000..c39e42290a --- /dev/null +++ b/kb/armory/General/run-job---producing-artifacts.md @@ -0,0 +1,71 @@ +--- +title: Run Job - Producing Artifacts +--- + +## Introduction +If you're using the *Run Job (Manifest)* stage in Spinnaker, it is sometimes helpful to be able to produce arbitrary artifacts. For example, if you have a run job that produces a Kubernetes manifest, you can have the run job output the manifest in base64 as an *embedded/base64* artifact. + +## Prerequisites +N/A + +## Instructions +### How to Achieve It +Have your Run Job output the artifacts into ```STDOUT``` so they get added to the stage JSON: +```SPINNAKER_CONFIG_JSON={"artifacts": [{YOUR_ARTIFACT_JSON}]}``` +* Configure the Run Job to Capture Output from ```Logs ```(not ```Artifact```)* Configure the stage ```Produces Artifacts``` section to match your artifact +### Example +We have a run job that’s going to produce a Kubernetes manifest that looks like this: + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: hello-today +spec: + replicas: 1 + selector: + matchLabels: + app: hello-monday + template: + metadata: + labels: + app: hello-monday + spec: + containers: + - image: 'justinrlee/nginx:monday' + name: 'hello-today' +`````` + +The base64-formatted version of this is roughly like this (depending on spacing of the above): +```YXBpVmVyc2lvbjogYXBwcy92MQpraW5kOiBEZXBsb3ltZW50Cm1ldGFkYXRhOgogIG5hbWU6IGhlbGxvLXRvZGF5CnNwZWM6CiAgcmVwbGljYXM6IDEKICBzZWxlY3RvcjoKICAgIG1hdGNoTGFiZWxzOgogICAgICBhcHA6IGhlbGxvLW1vbmRheQogIHRlbXBsYXRlOgogICAgbWV0YWRhdGE6CiAgICAgIGxhYmVsczoKICAgICAgICBhcHA6IGhlbGxvLW1vbmRheQogICAgc3BlYzoKICAgICAgY29udGFpbmVyczoKICAgICAgLSBpbWFnZTogJ2p1c3RpbnJsZWUvbmdpbng6bW9uZGF5JwogICAgICAgIG5hbWU6ICdoZWxsby10b2RheScK``` + +### Create a Spinnaker stage of type “Run Job (Manifest)”  +For the purposes of the example, we’re just echo-ing the output, but you could generate the base64 in a number of ways, such as piping helm to base64: ```helm template | base64```. +Put this in the Manifest Configuration: + +apiVersion: batch/v1 +kind: Job +metadata: + name: echo + namespace: spinnaker +spec: + template: + spec: + containers: + - + command: + - echo + - "SPINNAKER_CONFIG_JSON={\"artifacts\": [{\"type\":\"embedded/base64\",\"name\": \"hello-manifest\", \"reference\": \"YXBpVmVyc2lvbjogYXBwcy92MQpraW5kOiBEZXBsb3ltZW50Cm1ldGFkYXRhOgogIG5hbWU6IGhlbGxvLXRvZGF5CnNwZWM6CiAgcmVwbGljYXM6IDEKICBzZWxlY3RvcjoKICAgIG1hdGNoTGFiZWxzOgogICAgICBhcHA6IGhlbGxvLW1vbmRheQogIHRlbXBsYXRlOgogICAgbWV0YWRhdGE6CiAgICAgIGxhYmVsczoKICAgICAgICBhcHA6IGhlbGxvLW1vbmRheQogICAgc3BlYzoKICAgICAgY29udGFpbmVyczoKICAgICAgLSBpbWFnZTogJ2p1c3RpbnJsZWUvbmdpbng6bW9uZGF5JwogICAgICAgIG5hbWU6ICdoZWxsby10b2RheScK\"}]}" + image: alpine + name: manifest-generator + restartPolicy: Never + + +### Configure the “Output” section as follows: +Capture Output From: ```Logs```Container Name: ```manifest-generator``` +Scroll to the “Produces Artifacts” section, and click “Define Artifact”. Specify these options: +Account: ```embedded-artifact```Name: ```hello-manifest``` +This will generate an artifact ```Display name``` which can be used in later stages to reference the generated artifact. For example, this might be ```alpine-fox-34``` +To use the artifact, create a **Deploy (Manifest) Stage** that depends on the Run Job stage. Specify the following: +Account: Choose your Kubernetes cluster or accountManifest Source: ```Artifact```Manifest Artifact: Select the display name from the previous stage (such as ```alpine-fox-34```) +Save and run your pipeline + diff --git a/kb/armory/General/run-job-manifest-configuration-has-a-non-zero-backoff-does-not-show-results-of-multi-failure.md b/kb/armory/General/run-job-manifest-configuration-has-a-non-zero-backoff-does-not-show-results-of-multi-failure.md new file mode 100644 index 0000000000..6db4feb0ef --- /dev/null +++ b/kb/armory/General/run-job-manifest-configuration-has-a-non-zero-backoff-does-not-show-results-of-multi-failure.md @@ -0,0 +1,29 @@ +--- +title: Run Job (Manifest) Configuration has a non-zero backoff does not show results of multi-failure +--- + +## Issue +An organization may run into a known issue where in a Run Manifest stage, the Run Job (Manifest) Configuration has a non-zero backoff limit e.g. +apiVersion: batch/v1 + +kind: Job + +metadata: + + name: reschedule-iop-tickets + + namespace: spinnaker-jobs + +spec: + + backoffLimit: 2 + + template: + +The console output shows the results of the first attempt only. There is no way to see whether the pods are re-attempting it or failing somewhere else. + +## Cause +It is a known issue when the pod crashes, logs are only created in the first instance where the crash occurs. The logs from the latest pod created are pulled. +Sometimes there are multiple executions of a ```runjob```, but there is no place in the UI to let know the users what happened to each pod. +This issue is known to Armory Engineering and the OSS community. + diff --git a/kb/armory/General/run-pipeline-stage-picks-up-and-uses-pipeline-artifacts-from-previous-stages.md b/kb/armory/General/run-pipeline-stage-picks-up-and-uses-pipeline-artifacts-from-previous-stages.md new file mode 100644 index 0000000000..48c61a6cb2 --- /dev/null +++ b/kb/armory/General/run-pipeline-stage-picks-up-and-uses-pipeline-artifacts-from-previous-stages.md @@ -0,0 +1,14 @@ +--- +title: Run Pipeline Stage picks up and uses Pipeline Artifacts from previous stages +--- + +## Issue +When Deploying a Pipeline that consists of multiple ```run pipeline``` stages, the artifacts from the initial stage are saved in a cache and can be used in subsequent stages. +However, this may not be the ideal behavior for some pipelines, especially for Canary processes.   +As an example and test of how this can be an issue, +* Run a canary analysis pipeline with Canary Enabled first. * If the execution is successful, it may be necessary to run the same Canary Analysis pipeline with the Canary disabled around 5-10 times to deploy it to 5-10 different K8s clusters.* The runs after the initial execution will use the Canary Artifacts produced in the initial stage into the subsequent stages, possibly leading to a conflict. + +## Cause +The default behavior is to pass artifacts downstream to other pipelines running from the parent pipeline.  +There is no explicit method to turn this setting off via the Spinnaker Console UI.  However, users can define a value to adjust this behavior and prevent the artifacts from passing into subsequent pipelines. + diff --git a/kb/armory/General/runjob-does-not-execute-due-to-write-permissions-in-fiat.md b/kb/armory/General/runjob-does-not-execute-due-to-write-permissions-in-fiat.md new file mode 100644 index 0000000000..2b8b11f809 --- /dev/null +++ b/kb/armory/General/runjob-does-not-execute-due-to-write-permissions-in-fiat.md @@ -0,0 +1,11 @@ +--- +title: RunJob Does not Execute Due to Write Permissions in FIAT +--- + +## Issue +Group has been provided Read access via Fiat and is unable to execute an existing pipeline.  As an example, the below error occurs when attempting to execute +```Access denied to application k8s2 - required authorization: WRITE``` + +## Cause +```runJobManifest``` does not work without write permission due to permissions and access set within OSS Spinnaker.  This was previously done by design.  + diff --git a/kb/armory/General/s3-buckets-for-front50-revision-history.md b/kb/armory/General/s3-buckets-for-front50-revision-history.md new file mode 100644 index 0000000000..006cec0d8b --- /dev/null +++ b/kb/armory/General/s3-buckets-for-front50-revision-history.md @@ -0,0 +1,12 @@ +--- +title: S3 buckets for Front50, revision history +--- + + +## Question: +If we are using S3 buckets for front50, can we have revision history? + +## Answer: +Yes you can have pipeline version history if you use S3. Just make sure to enable versioning when you create the S3 bucket. +**Relevant Links: **[https://docs.aws.amazon.com/AmazonS3/latest/user-guide/enable-versioning.html](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/enable-versioning.html) + diff --git a/kb/armory/General/salesforce-to-servicenow-service-portal-migration.md b/kb/armory/General/salesforce-to-servicenow-service-portal-migration.md new file mode 100644 index 0000000000..68ca2220d7 --- /dev/null +++ b/kb/armory/General/salesforce-to-servicenow-service-portal-migration.md @@ -0,0 +1,22 @@ +--- +title: Salesforce to ServiceNow Service Portal Migration +--- + + + +In the end of February, we will be migrating all of our Support Issues, Knowledge Base, and Customers from Salesforce to ServiceNow. We will be re-routing our URLs (e.g. [https://kb.armory.io](https://kb.armory.io/), [https://portal.armory.io,](https://portal.armory.io%2C/) [https://registration.armory.io](https://registration.armory.io/)) to the ServiceNow portal on our go-live date, **March 1st, 2021** +## **The following are useful articles for our Customers to help them get started** +* [Support Portal User Registration](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010200)* [Password Resets](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010247)* [Creating Cases and Case Management in ServiceNow](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010136)* [Creating Change Requests in ServiceNow](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010267)* [ServiceNow Service Portal Customer Administrators](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010240)* [Global Technical Support - Case Statuses](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010011)* [Case Escalations](https://armory.service-now.com/support?id=kb_article&sysparm_article=KB0010203) +What does this mean for our customers? +* Customers will be using ServiceNow moving forward, and will not need to worry about accessing information through Salesforce again* Access controls will be placed so that only privileged Service Portal Customer Administrator Users will be able to register users. This is to emphasize self management for our customers while providing them added security on access to their company tickets.  They can also disable users* Users have the ability to review tickets for the entire account* If a user has **ever** logged a Support Issue in Salesforce, we have migrated both the issues and users. Issue statuses will also be synced along with any comments* A better ticketing process, with added connection opportunities with our customers +What does this mean for Armory? +* The knowledge base will be more easily navigated, with an easier to see layout* Groundwork to allow for future functionality and expansion plans* A portal designed to provide customers with a definitive support experience +Customer Administrators for Customer Accounts +* We will be reaching out to customers to provide us some information with regards to who should be administering their account on ServiceNow* At least a primary and secondary admin for a customer account will be requested* For more information about Customer Administrators and their rights, please review [https://kb.armory.io/s/article/ServiceNow-Service-Portal-Customer-Administrators](https://kb.armory.io/s/article/ServiceNow-Service-Portal-Customer-Administrators) +How to Gain Access +Registration will now be handled by customer's provided administrators. Administrators should be approached to attain a ServiceNow service portal account. +Automated self-registration is no longer available, to ensure that customers have the control to manage the security and access. +Service portal accounts provide: +* Ability to open and manage tickets that the user has opened* Access to non-public articles +Please do not hesitate to reach out if you have any further questions.  We thank you for your patience and help during this transitional time. + diff --git a/kb/armory/General/searching-applications-folder-with-execution-id-times-out.md b/kb/armory/General/searching-applications-folder-with-execution-id-times-out.md new file mode 100644 index 0000000000..ce0721d50e --- /dev/null +++ b/kb/armory/General/searching-applications-folder-with-execution-id-times-out.md @@ -0,0 +1,16 @@ +--- +title: Searching Applications folder with Execution ID times out +--- + +## Issue +When calling ```**/applications/{application_name}/executions/search with an execution_id**```,** **the calls time out for some pipelines. This time out error makes it difficult to find specific executions as needed. Example Search Query +curl --location --request GET 'http://{spinnaker-gate}/applications/{application_name}/executions/search?eventId={some_uuid}' \ +--header 'Accept: application/json' \ +--header 'Content-Type: application/json' \ +--header 'Cache-Control: no-cache' \ +--header 'Authorization: Basic {token}' \ +--header 'Cookie: SESSION={session_id}' + +## Cause +The number of executions for the queried pipeline can grow so large that it can reach the default timeout limit and return a failure. This is expected behavior for a scaled up Spinnaker that has had many pipelines put through it. + diff --git a/kb/armory/General/securing-dinghy-deployments-occurring-via-github.md b/kb/armory/General/securing-dinghy-deployments-occurring-via-github.md new file mode 100644 index 0000000000..3c7993e475 --- /dev/null +++ b/kb/armory/General/securing-dinghy-deployments-occurring-via-github.md @@ -0,0 +1,24 @@ +--- +title: Securing Dinghy Deployments Occurring via GitHub +--- + +## Introduction +Customers running an environment with Dinghy deployments via GitHub may find the need to secure their deployments.  Administrators often would like to know how Spinnaker secures a GitHub Pull Request and limits the trigger to authorized users. +From the standpoint of Spinnaker, changes to the Repo should be managed with Github permissions.  In order to secure the environment, most of the changes and security must be completed on the GitHub side of the equation.   +When Dinghy is configured, any Github user who has access to the Github Repo configured to use the ```dinghyfile``` can modify the ```dinghyfile``` so that it will create a pipeline to deploy to any environment.  This is because the credentials assigned to the user are on the Github level and are not passed as a part of the webhook payload to Spinnaker, and Spinnaker cannot use those permissions to identify access permissions of the user.  +Within the Spinnaker environment itself, Dinghy as a service is provided administration permissions so that it can execute on the changes pushed to it through the webhook. +Once item to note as well is that a user could also potentially modify permissions and triggers in the ```dinghyfile``` that would then allow them to trigger the pipeline without logging in by using the ```runAsUser``` property of the trigger.   This definition should always be reviewed before approving any changes to the ```dinghyfile``` +One other possibility that customers can look to leverage is to use Policy Engine to provide additional checks on the content of the deployment are met before executing.   + +## Prerequisites +Access to Github Repo to administer changes + +## Instructions +The restrictions in this case, must come from the Github policies and architecture/design of the repo.  As it stands of this writing, Github does not support RBAC on the folder/file level.   +There are a couple best practices surrounding Github architecture that will assist administrators to restrict the access and capabilities of end users.   +* Have ```CODEOWNERS``` defined for the ```dinghyfile```: [About code owners - GitHub Docs](https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners)* Have a ```Branch protection rule``` set up that requires a Pull Request review from the ```CODEOWNERS```: [Managing a branch protection rule - GitHub Docs](https://docs.github.com/en/github/administering-a-repository/managing-a-branch-protection-rule) +Once these two items are set, this will force a review of changes made to the Github repository and allow companies to manage the access control to their pipelines with regards to Dinghy + +In an Armory enabled environment, customers can also look to leverage Policy Engine to create a policy to validate the application and account combination at runtime.  The following blog post outlines the basics about how to accomplish policy creation: [https://www.armory.io/blog/policy-deployment-and-self-governance-with-spinnaker/](https://www.armory.io/blog/policy-deployment-and-self-governance-with-spinnaker/) +For more information about Policy Engine, please read the following documentation on [Enabling Policy Engine](https://docs.armory.io/docs/armory-admin/policy-engine-enable/) and [Using Policy Engine](https://docs.armory.io/docs/spinnaker-user-guides/policy-engine-use/)  + diff --git a/kb/armory/General/serverless-support-with-spinnaker.md b/kb/armory/General/serverless-support-with-spinnaker.md new file mode 100644 index 0000000000..c782c21cae --- /dev/null +++ b/kb/armory/General/serverless-support-with-spinnaker.md @@ -0,0 +1,12 @@ +--- +title: Serverless Support with Spinnaker +--- + + +Question: +Does Spinnaker support Serverless or other FAAS technologies? +Answer: +At the moment Spinnaker doesn’t have any native support for Serverless technologies. If these are a critical part of your deployment workflow you can always utilize Jenkins to do these as part of your application rollout. Since Spinnaker has a native integration with Jenkins, you can create jobs that roll out your Serverless function and trigger them as part of your Pipeline, making them *feel* native. This is one of the many ways to make Spinnaker more extensible.  +**Related Documentation:**[Working with Jenkins](https://docs.armory.io/user-guides/working-with-jenkins/) +Armory is currently exploring what Serverless support might look like within Spinnaker so if you have any ideas or feedback, be sure to let us know at [https://go.armory.io/ideas](https://go.armory.io/ideas). + diff --git a/kb/armory/General/servicenow---add-new-version-of-spinnaker.md b/kb/armory/General/servicenow---add-new-version-of-spinnaker.md new file mode 100644 index 0000000000..7a4f60e7e0 --- /dev/null +++ b/kb/armory/General/servicenow---add-new-version-of-spinnaker.md @@ -0,0 +1,23 @@ +--- +title: ServiceNow - Add New Version of Spinnaker +--- + + +Adding Versions to ServiceNow and Jira need to be done anytime a new major version of Spinnaker is created. +ServiceNow Changes +Only ServiceNow Admins can perform this change +Change your Application to ```Global``` +* Click on the Gear in the upper right* Click on the Developer Tab* Chose the Application ```Global``` from the drop down +* The first thing to do is to go to the [Dictionary Entry for the Armory Version in Service Now](https://support.armory.io/sys_dictionary.do?sysparm_referring_url=sn_customerservice_case.do%3Fsys_id%3D82bfa58edb18349079f53ec8f49619f4%4099%40sysparm_record_rows%3D4%4099%40sysparm_view%3Darmory_support%4099%40sysparm_record_target%3Dsn_customerservice_case%4099%40sysparm_record_list%3Dassigned_toISEMPTY%255EORassigned_to%253D2a48f6742f042010bd855d8b2799b6c1%255Estate%2521%253D3%255Estate%2521%253D5%255Estate%2521%253D-5%255Eassigned_to%253D2a48f6742f042010bd855d8b2799b6c1%255EORDERBYassignment_group%4099%40sysparm_record_row%3D1&sysparm_view=armory_support&sysparm_query=name%3Dtask%5Eelement%3Du_armory_version&sysparm_query_encoded=name%3Dtask%5Eelement%3Du_armory_version)* Scroll to the bottom and click on ```Choices``` tab. Note the biggest Sequence numberClick New +* Table: Task [task]* Sequence: Make it ```3```* Element: u_armory_version* Language: en* Label: 2.xx.x (e.g. 2.26.x)* Value: Same as in 4e* Click on Submit +* The new value should be in the listChange your Application to ```Customer Service```Click on the Gear in the upper right +* Click on the Developer Tab* Chose the Application ```Customer Service``` from the drop down +* Back in the [Dictionary Entry for the Armory Version in Service Now](https://support.armory.io/sys_dictionary.do?sysparm_referring_url=sn_customerservice_case.do%3Fsys_id%3D82bfa58edb18349079f53ec8f49619f4%4099%40sysparm_record_rows%3D4%4099%40sysparm_view%3Darmory_support%4099%40sysparm_record_target%3Dsn_customerservice_case%4099%40sysparm_record_list%3Dassigned_toISEMPTY%255EORassigned_to%253D2a48f6742f042010bd855d8b2799b6c1%255Estate%2521%253D3%255Estate%2521%253D5%255Estate%2521%253D-5%255Eassigned_to%253D2a48f6742f042010bd855d8b2799b6c1%255EORDERBYassignment_group%4099%40sysparm_record_row%3D1&sysparm_view=armory_support&sysparm_query=name%3Dtask%5Eelement%3Du_armory_version&sysparm_query_encoded=name%3Dtask%5Eelement%3Du_armory_version)In the Choices Tab, click on New +* Table: Case [sn_customerservice_case]* Sequence: Use the same number established in step 4b* Element: u_armory_version* Language: en* Label: Use the same number established in step 4e* Value: Use the same number established in step 4f* Click on Update +Reorganize the sequence of the values.  +* The previous versions should move down in sequence (both the value in the ```Case table``` and in the ```Tasks Table```).* Versions that are now out of should be moved into the ```1xxx``` sequence order.  Please follow the pattern (e.g. When ```2.28.x``` is added, ```2.25.x``` should move to ```1910)``` + +  +Jira Changes +* Go to Projects → Armory Support* Click on Project Settings (left menu)* Click on Fields (left menu) → Click on Actions → Edit Fields* Click on Custom Fields (left nav bar menu) → Find ```Armory Version``` → Click on three dots and Click on ```Contexts and Default Values```* Click on Edit Options → Add the new Value (e.g. 2.26.x) Click on Add Value* Move Value to the top + diff --git a/kb/armory/General/servicenow---all-about-change-requests,-navigation-and-anatomy.md b/kb/armory/General/servicenow---all-about-change-requests,-navigation-and-anatomy.md new file mode 100644 index 0000000000..a7949d24dc --- /dev/null +++ b/kb/armory/General/servicenow---all-about-change-requests,-navigation-and-anatomy.md @@ -0,0 +1,11 @@ +--- +title: ServiceNow - All About Change Requests, Navigation and Anatomy +--- + + +Change requests have a numbering scheme that starts with ```ACHG``` +They are located in their own section called ```Armory Change Requests``` that can be searched using the Sidebar Navigator + +If the view does not match as above, please change the view by clicking on the **hamburger** in the upper right corner of the Change record, and go to **View** → **Armory Change Request** +Change request interactions occur via Jira, and so, there is no interaction from with ServiceNow other than to pass information back to customers. + diff --git a/kb/armory/General/servicenow---all-about-support-cases,-navigation-and-anatomy.md b/kb/armory/General/servicenow---all-about-support-cases,-navigation-and-anatomy.md new file mode 100644 index 0000000000..0855514dde --- /dev/null +++ b/kb/armory/General/servicenow---all-about-support-cases,-navigation-and-anatomy.md @@ -0,0 +1,26 @@ +--- +title: ServiceNow - All About Support Cases, Navigation and Anatomy +--- + + +Support cases begin with ```CS``` and are located in the Cases section that can be searched using the Sidebar Navigator + +If the view does not match as above, please change the view by clicking on the **hamburger** in the upper right corner of the Change record, and go to **View** → **Armory Support** +Watch List +Located closer to the top of the case, Armory users can add additional personnel to become up to date about a case. This can include Customers, Armory users, and even third parties. Anyone with an email address will be valid +Responses and Interactions +There is a tab called **Comments and Work Notes,** at the bottom portion of the case. This is a record of the interactions, internal or otherwise. It is also where we can respond back to the customer +There are two fields here. One called Work Notes, which are internal only, and will have a Yellow border to the interaction. Another called Additional Comments, which are customer visible, and have a Grey border to the interaction. +Escalations +All customers can now escalate, and all users can escalate based on their ticket +There is an escalation tab for each case. If a case becomes escalated, it will be recorded in this tab. There is also a background process that happens as a result of an escalation, that is documented here: [Support Engineers - SNOW Escalation Response](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010246) +The following KB explains how the process works from a customer's perspective: [Escalating a Case](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010203) +Resolution Notes +This area shows the resolution information on a case. It will have any additional information about the case that may not be covered in the Responses and interactions +Attached Knowledge/Created Knowledge +If any KB articles were either attached as a result of resolving the case, or created, they will be located in these tabs at the very bottom of the case. +Navigating within Cases to find additional information + +For any field within a Case where there's an associated record, users will see an info icon (i) next to the field. +If you click on the (i) icon, you can see a preview of the record, and a button named ```open record``` can be seen. That button will respect right click navigation to open to a new window/tab and also Command+click to open in a new tab + diff --git a/kb/armory/General/servicenow---allowing-customers-to-see-tickets-across-organization-divisions.md b/kb/armory/General/servicenow---allowing-customers-to-see-tickets-across-organization-divisions.md new file mode 100644 index 0000000000..0ce4911c94 --- /dev/null +++ b/kb/armory/General/servicenow---allowing-customers-to-see-tickets-across-organization-divisions.md @@ -0,0 +1,14 @@ +--- +title: ServiceNow - Allowing Customers to See Tickets Across Organization Divisions +--- + + +Customers (e.g. Apple) may want their team to be able to see all tickets that are opened across the entire Apple Organization when they are logged in and looking at their (ServiceNow) Company's tickets.  +In Apple's case, they have quite a few divisions: +Apple - ACSApple - CloudTechApple - JMETIn this example, user Ravi who is a part of CloudTech needs to be able to see all tickets across Apple.  Likewise, all users in JMET will be able to see all tickets across all Apple divisions when using the user view to manage their organization's tickets in ServiceNow +Austin created a business rule that allows this to happen.  To set this for an organization, they need will need +* Multiple Accounts Created ([https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010283](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010283)) (In this example, Apple - ACS, Apple - CloudTech, Apple - JMET were all created)* A Master Organization Account needs to be created, but ***it should not*** have the ```Customer checkbox``` checked ([https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010283](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010283)) (In this example, Apple was created) +Once this is done, perform the following for each account +* Log in to the ServiceNow Backend* Find the ```Account``` for each division (e.g. Apple - ACS)* In the ```Parent``` field, add the Master Organization account (e.g. Apple)* Save and Update* Do the same for each other Division (e.g. Apple - CloudTech, Apple - JMET) + + diff --git a/kb/armory/General/servicenow---attaching-kb-articles.md b/kb/armory/General/servicenow---attaching-kb-articles.md new file mode 100644 index 0000000000..291267dc61 --- /dev/null +++ b/kb/armory/General/servicenow---attaching-kb-articles.md @@ -0,0 +1,13 @@ +--- +title: ServiceNow - Attaching KB Articles +--- + + +Attaching KB Articles to a Case (newly created or old) +Attaching KB articles advises about how useful our KBs are currently. This can be done at any time during the case, but definitely needs to be reviewed and completed at the end of a case +Attaching KB articles that have already been created +* Log in to the ```ServiceNow Admin Portal```* Locate the ```Case``` in question* At the very bottom, where the **SLAs **Tab is, there is a tab for **Attached Knowledge*** Click on the **Edit...** button* Find all KBs and their versions in the **left column** and click on the **right arrow** in the middle to add it to the attached Knowledge list.* Click on Save. The tab should now have the attached articles +Attaching KB Articles that are Newly Created +There are several ways to create a KB article as a part of a ticket, but if you are not going to click on the **Create Article Button** to generate the article and association with the case, then [you can do so separately](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010045) and then attach it to the case following the steps below +* At the bottom of the case, where the **SLAs** tab is, there is a tab for **Created Knowledge*** Click on the **New** button* Next to the **Article Field**, there's a **magnifying glass** to the right of it. **Click on it** to locate the article in question that has already be created* Click on the **article display number field** for the relevant article in the window that has popped up* Click on **Submit*** The article should now be listed + diff --git a/kb/armory/General/servicenow---create-edit-knowledge-base-articles.md b/kb/armory/General/servicenow---create-edit-knowledge-base-articles.md new file mode 100644 index 0000000000..186b306daa --- /dev/null +++ b/kb/armory/General/servicenow---create-edit-knowledge-base-articles.md @@ -0,0 +1,82 @@ +--- +title: ServiceNow - Create/Edit Knowledge Base Articles +--- + + + +### Overview + +Creating new KB articles in ServiceNow is relatively easier than it’s ever been.  There are multiple entry points to creating KB articles to make it easier and remove configuration concerns (i.e. Security).  There are a few terminologies to keep in mind when we talk about KBs and their articles, and so here are a few things to take a look at. + +For writing tips on KB formatting, designing and laying out articles, please consult with [https://armory.slab.com/posts/knowledge-article-writing-tips-b6fgrxv5](https://armory.slab.com/posts/knowledge-article-writing-tips-b6fgrxv5) + +This article is specifically about Creation of KB Articles.  For more on different subjects information, please look at +Creating a Knowledge Base +Creating a Template + +### Terms + +**Knowledge Base -** A way to sort and organize articles.  Think of these as Libraries, or File Cabinets.  Each KB has their own security. (e.g. a KB can be created so that only Support users can see certain documents, and another KB can be created so that customers can see these other documents). Create the correct KB in the correct knowledge base to expose information only to those people. +**Searches - **Using the ServiceNow Search Engine will search across all articles that the users have access to.  Each KB is denoted by their iconography, so that users will know where the KB article came from +**Articles -** Individual information documents contained within a Knowledge Base.  Security can be applied to individual articles that differs from the KB, but is in general, not recommended as then, security management will be more difficult.   +**Templates -** A way to organize Article Information so that they have a standard look.  Templates can be used across different Knowledge Bases, so uniform information (e.g. How To articles) can be served with different security, depending on which KB they are created under +**Article Workflow** - Each Article created can follow an approval workflow processes, or can be instantly published.  If the article has to go through an approval process, the article will then need to be created, saved, and then will be sent to the group to approve or not.  This happens whenever an article is published, a new version is added, or can also happen when an article is retired. + + + +### Article Creation + +#### Deciding where to put your Article +There are four Knowledge Bases to consider when creating a KB article +**KB Name****What it's used for****Who can see it**Armory Knowledge BaseInformation to work on issues and how to do things which are general knowledge. Non proprietary informationPublicArmory Knowledge Base: Customer-Only ArticlesInformation of a Professional Services/Solutions Architecture in nature. Information which customers are paying for, such as providing design choice and informationArmory + Registered Customers of the PortalArmory Internal: Support and Snow ProcessesSupport information for the management of Service Now (SNOW)Armory OnlyArmory Internal: Confidential Technical AritclesTechnical information that is Armory only, and should not be advised to the customerArmory Only + + + + +#### Starting Points + +There are several entry points to creating articles.  However, do NOT format your articles in Word or other processors before pasting them in the editor.  This will add a lot of unnecessary HTML.  It is highly recommended to use the Service Now WYSWIG editor. + +Via Service Now Admin +* Log in to ```Service Now Admin Console```* In the Knowledge Section, select ```Create New```* Choose the correct Knowledge Base for the article.  This will determine the security settings and who can view the Article* Select the ```Template``` for the article.* Click ```Next``` +Via the Public KB site +* Go to the Public Facing Site ([https://armory.service-now.com/support](https://armory.service-now.com/support))* Click on the ```Knowledge``` link in the main page* Log in to ServiceNow (upper right corner)* From the Main Knowledge page, click on the ```ellipses (…)``` next to the **Explore our Knowledge Bases** and select ```Create Article```.  This will bring the user to a selection screen like in 1c. Select the Knowledge Base (to determine the security), the Template, and then click ```Next``` +Via a Case View +* At any time, users with correct access can create an article from a Case view.  However, if the article is being created after a case is resolved, and the resolution is already entered, the case resolution will be copied over.  In all other cases, clicking on the **Create Article** button creates a blank article* This will open up a selection drop down pop up to select the **Article Template Type.  **Note that **all article types** are available, so it is important to select an article type available for the KB (e.g. For Public Knowledge KBs, **General Information, How To,** and** Issues** types are available)* The resolution information will be added to the declared **SEO Field** for the **Article Template** (e.g. ```Information``` for ```How To```, or ```Issue``` for ```Troubleshooting```)* Select the **Knowledge Base** where the article will reside.  This is a required selection. + + + +#### Creating the Article + +* Once the Knowledge Base and template are selected, users will be presented a screen with mandatory fields and information, and a pop up asking if they want to ```populate the fields with default values```. Select OK, as this will set the group for approving the changes, and the type of article (HTML)Fill in the Information for the Article as needed +* Select the Knowledge ```Category``` for the article (required)* Short Description: ``````* Fill in the rest of the information required.  Any field that has a ***** next to it must be filled out +* If the article needs to be saved and resumed later, right click on the top banner and select save.  To find the article again, search for the ```unpublished``` section of the Knowledge navigation* When completed, either select the ```Submit``` button to save the article (for the first time) and then open the article and ```Publish```, or ```Publish``` if the article was saved once already* Approvers will look over the article and either provide feedback or approve the publishing of the article. + + +### Handy Dandy Article Pointers + +* **When Marking Code** - Code blocks AND code lines can be used in ServiceNow’s WYSIWYG editor.  If users highlight a selection and then press the **code sample** button, it will mark that section with the **```code tag```**.  If nothing is selected and then press the code sample button, it will open up the ```**Code Block insertion window**```**Adding Images to an Article** - Adding images to an article should be done as an ```attachment``` instead of as an ```image library```. When adding an image, there is a dropdown to select whether it will be added as an attachment or as a part of the image library.  There are several reasons to select image library. Image Library options are removed. +* Usually, only a single article needs access to a particular image, so it makes more sense to add it as an attachment.  * In addition, if an image is added to the image library, the image cannot be overwritten and must be manually deleted before being able to be re-added if there are changes (This option is no longer available.  Image Library option removed)* **Users can cut and paste images into the article**. However, sometimes, the image may not be editable/adjustable.  If this happens, it may need to be opened in an image program like GIMP and then copied and pasted into the ServiceNow KB article.* The following is a detailed article by ServiceNow to understand how Images work within KBs [https://hi.service-now.com/kb_view.do?sysparm_article=KB0696767](https://hi.service-now.com/kb_view.do?sysparm_article=KB0696767) +* **Numbered Lists Types** - When using numbered lists, users can also create sublists with a different numbering schema (e.g., like in this article).  Once the sublist is created, highlight it, and click on the **dropdown arrow** next to the numbered list in the WYSIWYG toolbar and select the number schema* Break the article into Sections with Headers whenever possible.  e.g. For an **Article How to Send Metrics to Datadog**, break it up into **Install Prometheus, Set Up Datadog, Connect Datadog**If the article can be broken up into several smaller articles, it is better to modularize and link to other articles because: +* It will be possible to refer back to that starting article.  e.g. **How to Install Prometheus **may be relevant to multiple articles, so users may want to break that out into its own article* If the process changes for **How to Install Prometheus**, authors only have to update that one article and because it’s linked to it from the others, they don’t need to seek out and update all the other articles when there is a change +* Use numbered steps whenever possible* Try to include pictures wherever possible, to remove ambiguity* **Please note providing any solution that is not Armory designed**, or not provided by a trusted Armory partner (e.g. GCP, AWS, etc…) should have a disclaimer before providing a link.  This is to ensure that Armory is not liable for the solution, or any security issues from the software. + +Example: [https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010161](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010161) +```Please note that Armory did not design this solution, and we advise that any customers looking to implement this solution perform their due diligence on this solution, including testing on a pre-production environment before implementing​``` + + + + +### Editing a Knowledge Article + +There are multiple ways to accomplish edits on an article +Via the Service Now Admin Console +* In the Navigation, go to the Knowledge area → Articles → Published* Search for the article by the name* Click on the article number* In the upper right, click on **Checkout*** Make the edits again and **publish** or **submit** again. +Via the Public KB site +* Access the Public Site.  Sign in * Find the article to make changes to* Open it.  In the Upper right corner of the article, there’s the ability to click on the **Ellipses (…)** and **Edit*** In the upper right, click on **Checkout*** Make the edits again and **publish** or **submit** again + +### Attaching KB Articles to your Case +As a part of our efforts to see how effective KB articles are, please ensure that you are attaching KB articles to your cases and providing the appropriate solution code when necessary +[https://support.armory.io/support?id=kb_article&sysparm_article=KB0010382](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010382) + + diff --git a/kb/armory/General/servicenow---creating-cases-and-change-requests-on-behalf-of-customers.md b/kb/armory/General/servicenow---creating-cases-and-change-requests-on-behalf-of-customers.md new file mode 100644 index 0000000000..b6a41cc3f7 --- /dev/null +++ b/kb/armory/General/servicenow---creating-cases-and-change-requests-on-behalf-of-customers.md @@ -0,0 +1,30 @@ +--- +title: ServiceNow - Creating Cases and Change Requests on Behalf of Customers +--- + + +**Note:** For information on distinguishing Change Requests and Cases, please review the following KB article on the subject.  Customers are recommended to use this for reference during their orientation as well: [https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010266](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010266) +### General Recommendations +Overall, Armory would prefer it if customers open their own support cases and especially change requests.  While the following article will provide some additional information about how we can do so if necessary, the general guidance is for customers to do so on their own.  Here are some of the reasons why it is the preference: +* Customers will be advised to do this from the beginning, and we should follow this policy* It is not scalable for us to do this style of "white glove" treatment, and is enforcing this style of request* This can cause delays in the process while the case is being created and can cause customers to wait for the representative to open their cases for them* It introduces a level of interpretation by our own employees +### Special Note about Change Requests +Change requests as well have additional red flags associated with them, especially with regards to SOC2 compliance.  As a result, it is not recommended, and we do not have general instructions for how to do so +A change request is a qualified customer request to make changes to their Armory environment.  This request can impact their +* Service capacity* Performance* Costs* Security +* As a result, liability now enters the picture, and a unbroken chain of authorization from the customer needs to be established* Armory employees, if they were to make a mistake in their understanding of the changes, could potential authorize a change on behalf of the customer which the customer did not agree toWhile this appears to be providing a service to the customer, it will actually introduce delays +* Normal Process: Customer opens ticket -> Managed has the verified information for the change -> Managed implements changes* Opening CR on behalf of customer: Armory opens ticket -> Authorization and confirmation needs to be obtained with the customer and ***recorded in the change request***** -> **Managed has the verified information for the change -> Managed implements changes +* For further discussions on this matter, please consult with the following thread in slack discussing the implications: [https://cloud-armory.slack.com/archives/CN5TLLFH6/p1633700093124300](https://cloud-armory.slack.com/archives/CN5TLLFH6/p1633700093124300) +### Creating a Change Request +It is technically possible from the admin console with the proper permissions to do so, but as advised above and in the Slack thread, the process introduces a lot of arduous checks.  This process and should only be used an exception to the rule.   +Therefore, creating a change request on behalf of a customer should only be done by a ServiceNow Admin.  If it is necessary, please reach out to an admin, and they can assist. +### Creating a Case  +A case can be created on behalf of a customer in a few situations.  One would be if a customer accidentally creates a Change Request, but meant to create a Case to be investigated by Armory Support, then we can go ahead and do so.  If a particular person would like to do an exception and create a case for a customer to help them out, that would also be a rare occasion.  However, as stated above, this is not a recommended process. +#### From the Customer Portal +* Go to the [ServiceNow home page](https://support.armory.io) and log in* In the left navigation bar, select Cases -> Create a CasePlease fill in the information in the case as follows: +**Account: Click on the search icon and look for the customer's company****Contact:** Click on the search icon and add a **customer point of contact** on the case**Armory Version**: Add the version of the environment**Impact:** Select the customer impact**Workaround Rollback Available: **Select, based on situation**Feature:** Click on the search icon and add the Feature the issue falls under according to the [Armory Enterprise Compatibility Matrix (ECM)](https://docs.armory.io/docs/feature-status/armory-enterprise-matrix/)**Affected Environment: **Change to the environment type that has the issue**Service:** Click on the search icon and add the Service the issue falls under according to the [Armory Enterprise Compatibility Matrix (ECM)](https://docs.armory.io/docs/feature-status/armory-enterprise-matrix/)**Customer Availability: **Chose a time from the dropdown. Do not leave default value**Subject: ** Title of the case**Description:** Provide notes according to the KB article we created that describes the [investigation about the issue, relative documents, logs, and information about the case](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010136).  **** +* Click on the **Submit Button** +#### From the ServiceNow Admin Portal +* Go to the [ServiceNow home page](https://support.armory.io) and log in* Click on **Home** to access the backend* Look up **Cases** under the **Armory Section of the Navigation bar**. You can also just [bookmark the following link to get to the cases](https://support.armory.io/nav_to.do?uri=%2Fsn_customerservice_case_list.do%3Fsysparm_userpref_module%3D647356c81b4be810ec88b8c2cc4bcbc0%26sysparm_clear_stack%3Dtrue).* Click on the **New Button** to create a new casePlease fill in the information in the case as follows: +**Assignment Group:** Click on the search icon or type **Armory - Technical Support****Account:** Click on the search icon and look for the **customer's company****Assigned to:** Please leave blank (unless you are taking the case, or have an agreement with someone to take the case)**Contact:** Click on the search icon and add a **customer point of contact** on the case**Feature:** Click on the search icon and add the Feature the issue falls under according to the [Armory Enterprise Compatibility Matrix (ECM)](https://docs.armory.io/docs/feature-status/armory-enterprise-matrix/)**Customer Availability: **Chose a time from the dropdown. Do not leave default value**Service:** Click on the search icon and add the Service the issue falls under according to the [Armory Enterprise Compatibility Matrix (ECM)](https://docs.armory.io/docs/feature-status/armory-enterprise-matrix/)**Priority: **[Set according to our SLAs](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010007)**Armory Version**: Add the version of the environment**Workaround Rollback Available**: Select, based on situation**Environment: **Change to the environment type that has the issue**State: **New**Short Description: ** Title of the case**Description:** Provide notes according to the KB article we created that describes the [investigation about the issue, relative documents, logs, and information about the case](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010136).   +* Click on the **Submit button** + diff --git a/kb/armory/General/servicenow---creating-users.md b/kb/armory/General/servicenow---creating-users.md new file mode 100644 index 0000000000..98d8c60ef3 --- /dev/null +++ b/kb/armory/General/servicenow---creating-users.md @@ -0,0 +1,21 @@ +--- +title: ServiceNow - Creating Users +--- + + +**Note: **Do not click on the New button at the ```Users``` table to create a new user.  This will create a user under the ```Users``` table, but the user needs to be created under the ```Contact``` table instead. +#### Check if the User Exists +* Navigate to the Admin site and check out the ```Users``` table to see if the person was already created, but just can't log in.[https://support.armory.io/nav_to.do?uri=%2Fsys_user_list.do%3Fsysparm_userpref_module%3Dc5aa0fff0a0a0aa7009a39da035ea396%26sysparm_clear_stack%3Dtrue](https://support.armory.io/nav_to.do?uri=%2Fsys_user_list.do%3Fsysparm_userpref_module%3Dc5aa0fff0a0a0aa7009a39da035ea396%26sysparm_clear_stack%3Dtrue)* If the user exists already, check to see if the account is ```active```, and ```not locked out```.  If those value are set correctly they will just need to [reset their password](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010247)* Otherwise, we'll need to add the user +Create/Add the User (Customer) +* To start navigate to [https://support.armory.io](https://support.armory.io) and Log in as Support.* Go to **Manage Accounts > Users List. ** If you are in the Administration View, look for **Create a User** under the **Armory** heading in the Navigator Bar***** This should redirect you to a list view of all current users in your account. As an example, for Support, this is will show all the people in our account, which is our team. (If doing this as an internal user, this should list all users for you).* Click on the **Create User** button which should redirect you to a form.* Simply fill out the mandatory fields denoted by the * icon and the user should be added once submitted.* Again, **internal Armory users can select which account to create the new user in** while external users can only add users in their current account. Internal Armory users can create users for customers when needed.  *Please Copy and Paste the information provided by the customer to ensure there are fewer errors*.* **Please Note: **The account you select for a user, determines their security permissions, and which company the user is tied to. Please review to make sure that the correct account is selected, or adjustments may need to be made later on.  * Send the user instructions about how to reset their password to gain access.  [https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010247](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010247) +The customer admin field will always be left as uneditable and should be pre-populated when navigating to the Create New User form. This is so that we can track of who created what users. +For external users to have the ability to create users, they must have the ```customer_admin``` role. By default, we've provided this role to two users in each account. For more information about [Primary and Secondary accounts, please visit the KB](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010201) +Create/Add the User (Armory Internal Users) +* To start navigate to [https://support.armory.io](https://support.armory.io) and Log in as Support.* Go to **Manage Accounts > Users List. ** If you are in the Administration View, look for **Create a User** under the **Armory** heading in the Navigator Bar***** This should redirect you to a list view of all current users in your account. As an example, for Support, this is will show all the people in our account, which is our team. (If doing this as an internal user, this should list all users for you).* Click on the **Create User** button which should redirect you to a form.* Simply fill out the mandatory fields denoted by the * icon and the user should be added once submitted.* Select the Account the Armory user should be associated with.  In this case, it will be the correct department.  If you are uncertain which department a person belongs to, please consult Rippling for the latest information. [https://app.rippling.com/employee-list](https://app.rippling.com/employee-list)* Once the user is created, we will need to correlate the user with the Jira user.  In the ServiceNow Administration backend, please search ```Users``` in the Navigation Bar, or go to [the following link](https://support.armory.io/nav_to.do?uri=%2Fsys_user_list.do%3Fsysparm_userpref_module%3D56e8b9ce3718200044e0bfc8bcbe5d00%26sysparm_clear_stack%3Dtrue)* Find the user(s) that you have created.  Click on the user name.Check that the user has the correct ```Group``` assigned in the additional tabs at the bottom of the profile.  They should be in the same group as the company designated for the user which was selected upon creation +* If the incorrect group is assigned, click on ```Edit...``` * Remove the group assigned and save* Click on ```Edit...``` again* Add the correct group and save. +In the ```Jira ID``` field, fill in the user's # from Jira. +* Log in to Jira in a separate window* Search for the person in [the People Search](https://armory.atlassian.net/jira/people/search) Click on the name* In the URL, after the ```https://armory.atlassian.net/jira/people/``` portion of the URL, is the user's Jira ID.  Copy this and paste it into the person's ServiceNow Jira ID +If the person is a part of Support, they will also need their PagerDuty ID added  +* Log in to PagerDuty in a separate window* Search for the person in [the People Search](https://armory.pagerduty.com/users-new). Click on the name* In the URL, after the ```https://armory.pagerduty.com/users/``` portion of the URL, is the user's Jira ID.  Copy this and paste it into the person's ServiceNow PagerDuty ID Field +* Click on the ```Update``` Button in the user's profile in ServiceNow + diff --git a/kb/armory/General/servicenow---customer-account-company-creation.md b/kb/armory/General/servicenow---customer-account-company-creation.md new file mode 100644 index 0000000000..45714ec165 --- /dev/null +++ b/kb/armory/General/servicenow---customer-account-company-creation.md @@ -0,0 +1,20 @@ +--- +title: ServiceNow - Customer Account/Company Creation +--- + + +To create an account, you will need to add the organization account both in ServiceNow and in Jira.  This is so that case, when created, will refer to the correct organization when synchronizing across both systems. +  +  +* Go to ServiceNow Admin Portal* Head to the Accounts page under Customer ServiceClick on the "New" button +* **Name:** Official name for the company,* **Install Type:** if they will be managed/self-installed. Most will be Self-Installed with the future transition to Managed.* **Customer:** Checked* **Premium**: If customer has Premium Enterprise support, please check (SLAs and responses)* **CDaaS:** If the customer has a CDaaS license, please check* **Account Owner:** AE who is involved (check Salesforce)* **Technical Account Manager**: TAM who is involved (check Salesforce)* **JIRA ID**: Required for the Sync (see below about filling it out)* **Saleforce ID**: Required for Sync of TAM/AE (see below about filling it out)* **Slack Customer Channel/War Room Channel**: Required to post messages in the customer War Room for new cases (see below about filling it out)* **Parent** (Optional): This is required only for those companies where they wish all users to be able to see all tickets across the company.  (e.g. Apple) +Log into Jira +* Click on the top menu, Projects → Support* In the left menu, click on Customers → Click on Add Organizations* In the Dropdown field, past the exact same name used in step 3a. This should be Case Sensitive.* Click on the Create New option underneath, and click on the Add button* Click on the Organization's name in the list. In the URL, you'll see a number after the organization portion in the path. This is the** JIRA ID*** Add the Account to BOB Issues as well* Click on the Gear icon (Next to your portrait) -> Issues -> Custom Fields* Search for ```Customer``` -> Click on ```...``` -> Select ```Context and Default Value``` -> Click ```Edit Options```* Add the new value to the list (Scroll to the bottom, fill in the field, and click on Add) +* Add this number to SNOW (see Step 3e).* Update the Customer Account record/Save itOpen the account again and add Salesforce Information (Allows for AE/TAM sync via Zapier) +* Ask a Salesforce Admin (e.g. Kinnon) to go into Salesforce and find the Customer account* Copy the SysID from Service now (Click on the ```Hamburger``` in the Upper Left corner -> Select ```Copy SysId```)* Add this value to the equivalent account in the ```ServiceNow Sys ID``` field.  Click ```Save``` in the Salesforce Account* Copy the Salesforce ID from the URL* Add this value to the account's ```Salesforce ID``` field* Update the Customer Account record/Save it +Add the Slack Channels +* Find the Customer's Slack Channel.  Each Customer must have a War Room, but they may not have a Custome Facing channel* Right click on the ```Channel``` -> Click on ```Copy``` -> Click on ```Copy Link```* Isolate the Channel ID.  For example the ```CMEHUF72A``` portion of ```https://cloud-armory.slack.com/archives/CMEHUF72A``` in the Support Channel Link* Add it to the account's ```Slack Customer WR``` or ```Slack Customer CH``` field* Update the Customer Account record/Save it + +Note, it is also possible that depending on the organization, it may be necessary to set up a series of accounts that allow users to see tickets of the same organization: [https://support.armory.io/support?id=kb_article&sysparm_article=KB0010307](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010307) +  + diff --git a/kb/armory/General/servicenow---escalation-response-for-support-engineers.md b/kb/armory/General/servicenow---escalation-response-for-support-engineers.md new file mode 100644 index 0000000000..0e44df1393 --- /dev/null +++ b/kb/armory/General/servicenow---escalation-response-for-support-engineers.md @@ -0,0 +1,24 @@ +--- +title: ServiceNow - Escalation Response for Support Engineers +--- + + +The following outlines what happens after an escalation is sent from a customer.  +The customer side of the process can be found in the following article:[https://support.armory.io/support?id=kb_article&sysparm_article=KB0010203](https://support.armory.io/support?id=kb_article&sysparm_article=KB0010203) +Once an escalation is submitted, it opens an "escalation", which can be thought of as a special record type in ServiceNow.  The parts are +* A case that now has an escalated flag (can be found in the Navigation bar under the Customer Service Section, Cases, Escalated)* A linked escalation (can be found in the Navigation bar under the Customer Service Section, Escalations, All) +## If Viewing from the Case Record + +When you click on a Case Record and open it up, there is an escalation tab that can be seen near the middle/bottom of the case.  This tab contains information about the escalation on hand, including the reason (Executive Visibility, Lack of Progress, and Customer Imposed Deadline) and justification provided.  The Active Escalation contains the Escalation record number.  If you wish to jump to this record, click on the ```i``` icon to preview the record and then an option to ```Open Record``` will be available to go to the actual escalation record. +Note that all communication with the customer needs to happen from the ```**Case Record**```. + +## If Viewing from the Escalation Record + + +When you click on an Escalation Record and open it up, there will be information about the escalation contained within, including ```Source Record```, which contains a reference to the Case Number.  If you wish to jump to this case record, click on the ```i``` icon to preview the record and then an option to ```Open Record``` will be available to go to the actual case record. +Note that the only area that should be changed is at the bottom tab, called **Approvers**. Any communication or work should be done normally from the original case.  The **Approvers** tab will allow users to either approver or reject the escalation.   +* Select the approver name (e.g. your name) * Scroll to the bottom where it says Actions on Selected Rows* Click on the drop down and select from **Approve**, or **Reject.** * Communicate via the **case** back to the customer about the approval, or advise about the reason for the rejection in full detail. + +## Process for Response +* Escalation is submitted by the end user. * Support team is notified about the Escalation* Review the escalation reason, including the reason and description provided by the submitter* Approve or Reject the escalation provided by the submitter.  This can be accomplished either by going to the escalation, as advised above, or going to the **Self-Service, My Approvals** section via the Navigation Bar* Respond back to the submitter via the case about the reasons, and outcome of their escalation* Note that customers can re-escalate a case if it is rejected.  Please make sure that it is explained clearly why the rejection was not accepted, and **if possible, provide an avenue for a dialog with the customer** so that they do not have to keep re-escalating in hopes of getting assistance.   + diff --git a/kb/armory/General/servicenow---finding-a-resource-case,-change-request,-etc....md b/kb/armory/General/servicenow---finding-a-resource-case,-change-request,-etc....md new file mode 100644 index 0000000000..515ea8965a --- /dev/null +++ b/kb/armory/General/servicenow---finding-a-resource-case,-change-request,-etc....md @@ -0,0 +1,13 @@ +--- +title: ServiceNow - Finding a Resource (Case, Change Request, etc...) +--- + + + +If you are looking to find Cases or Change requests, there are several ways to go about doing so. +* Navigate to the ServiceNow Backend* In the upper right corner of the interface, click on the magnifying glass and put the case number in there, and hit enter +You can also navigate to the All Cases table or All Change Requests table and run a search on the table +* Go to the ServiceNow BackendIn the Navigation bar +* For Cases: Search for ```Case```, and either look for the ```Cases``` option under the Armory heading, or the ```All``` option under the ```Customer Service, Cases``` heading.* For Change requests: Search for ```Change``` in the navigation bar and either look for the ```Change Requests``` option under the Armory heading, or the ```All``` option under the Change heading. +* Using the [quick search](https://armory.slab.com/posts/service-now-managed-and-tam-team-information-p1idav2r#searches) bar, you can enter the parameters of the case number and use it to filter the table. Wildcards will also be accepted. + diff --git a/kb/armory/General/servicenow---navigation-of-the-backend-site.md b/kb/armory/General/servicenow---navigation-of-the-backend-site.md new file mode 100644 index 0000000000..58abe8df2f --- /dev/null +++ b/kb/armory/General/servicenow---navigation-of-the-backend-site.md @@ -0,0 +1,32 @@ +--- +title: ServiceNow - Navigation of the Backend Site +--- + + +Sites in ServiceNow +ServiceNow Navigation works with a front end and backend. For all Armory users, all that they will need to do is log in to the website at [https://support.armory.io](https://support.armory.io). To reach the backend, click on ```Home``` on the navigation bar. Non Armory users, will only be brought back to the homepage, when clicking ```Home``` +Most of the time will be spent in the Backend of the portal + + + + +Customer Portal + + + + +Backend Admin Portal +Sidebar Navigator +The sidebar navigator is the area that most Armory users will be using to navigate around ServiceNow. This navigator will take you from section to section, allow you to go to Favorites and Historical sections that you have been to. There are a few tips that will help with the Navigation +One thing to note is that the Sidebar Navigator can at anytime, be collapsed, by using the Left Arrow, in the bottom left of the screen. +Filter Navigator +The Filter Navigator field acts as a search filter for the various different sections that users will want to navigate to. If you want to look for Cases, just type that there, and it will bring up options related to cases + +Favorites +When you find sections of importance that you would like to return to, click on the **Star** to fill in the star and favorite it. You can favorite both an entire section, or individual parts of a subsection. +Once the items are favorited, they will appear in the Star tab. +You can re-order your Favorites by clicking on the pencil icon at the bottom of the Sidebar Navigator +History +The history tab will display historical searches and pages that have been navigated to by your account. It is a great way to trace back steps if you accidentally lose the section that you have been working in + + diff --git a/kb/armory/General/servicenow---notifications-on-case-comments.md b/kb/armory/General/servicenow---notifications-on-case-comments.md new file mode 100644 index 0000000000..804e3fa1b5 --- /dev/null +++ b/kb/armory/General/servicenow---notifications-on-case-comments.md @@ -0,0 +1,21 @@ +--- +title: ServiceNow - Notifications on Case Comments +--- + + +Armory Technical Support Engineers receive notifications on case updates based on a workflow designed to send DMs to the Support Engineers.  The following explains how to manage the notifications and what the icons mean  +If the TSE receives a Slack DM from the Slackbot with icon  , it will be a full notification with comment information because it came from the customer + +If the TSE receives a Slack DM from the Slackbot icon and +* Full notification with comment information, it means it was from someone at Armory (not the case owner), posting on the case, visible to the customer.* Minimal notification it was from the case owner (you), so information is stripped down to a link to the case if the TSE needs to jump to the case. +If the TSE receives a Slack DM from the Slackbot icon +* Full notification with comment information, it will be from someone at Armory (not the case owner), posting an internal comment on the case.* Minimal notification it was from the case owner (you), so information is stripped down to a link to the case if you need to jump to the case. +#### Why do it this way? +The purpose of notifications are two-fold.  The first is to inform the TSE that there is new action on the case that was assigned to them.   For their own posts, it is important that TSEs recognize whether the post they made was processed by ServiceNow or not, and also to ensure that they have a notification to let them know if their own posts were internal or external comments in case a mistake was made +The other purpose is to inform the TSE of updates that they didn't make, that happened to the case.  In these instances, TSEs are informed about comments left by others on the case, including the customer and other Armory users +#### Muting Notifications +Every User has a check box for Case Comment Slack Notification, but it only applies to Support for now.  The workflow checks against this selection to determine if a notification should be sent out or not. + +#### Flow Designer WorkFlow +(Do not edit without proper training, and always test on a copy before modifying actual).  We have added a filter to clean up the HTML parts of the comments so that they will appear as close to plain text as possible.  This is a business rule that runs on the comments in the table (sys_journal_fields)[https://support.armory.io/$flow-designer.do?sysparm_nostack=true#/flow-designer/de6dd0cbdbff7410ac71c26c2996193c](https://support.armory.io/$flow-designer.do?sysparm_nostack=true#/flow-designer/de6dd0cbdbff7410ac71c26c2996193c) + diff --git a/kb/armory/General/servicenow---sales-engineers---opening-and-managing-support-cases.md b/kb/armory/General/servicenow---sales-engineers---opening-and-managing-support-cases.md new file mode 100644 index 0000000000..09d9672503 --- /dev/null +++ b/kb/armory/General/servicenow---sales-engineers---opening-and-managing-support-cases.md @@ -0,0 +1,24 @@ +--- +title: ServiceNow - Sales Engineers - Opening and Managing Support Cases +--- + + +For further information about Sales Engineers opening Support Cases, [please check out the slab doc](https://armory.slab.com/posts/sales-engineering-opening-service-now-cases-gqo9yxu5) +### Opening Cases on Behalf of Customers +Sales Engs can easily open cases on behalf of customers and have them assigned to themselves. **Support is here to help with educating and helping Sales Engs, but should not manage the interactions with the customers.** +While it **is technically possible** to add a customer's email to the watchlist of a case so that they get updated, please do not do this when requesting assistance from Support. As advised, Support is supporting the Sales Eng, and adding a customer to the watchlist will mean customers will get access to ask Support questions directly. +This is one method to open a case, and is probably the easiest for Sales Engs. +* Go to the ServiceNow Portal [https://support.armory.io](https://support.armory.io) and log in to the site* In the left navigation bar from the main portal, select Cases -> Create a Case: [https://support.armory.io/support?id=csm_get_help&sys_id=de45c412c312310015519f2974d3ae1b](https://support.armory.io/support?id=csm_get_help&sys_id=de45c412c312310015519f2974d3ae1b)Please fill in the information in the case as follows: +**Account: Armory - Sales Engineer****Contact:** Click on the search icon and add a **your name as the  contact** on the case**Armory Version**: Add the version of the environment**Impact:** Select the customer impact**Workaround Rollback Available: **Select, based on situation**Feature:** Click on the search icon and add the Feature the issue falls under according to the [Armory Enterprise Compatibility Matrix (ECM)](https://docs.armory.io/docs/feature-status/armory-enterprise-matrix/)**Affected Environment: **Change to the environment type that has the issue.  In this case, likely **non-production****Service:** Click on the search icon and add the Service the issue falls under according to the [Armory Enterprise Compatibility Matrix (ECM)](https://docs.armory.io/docs/feature-status/armory-enterprise-matrix/) **Customer Availability: **Chose a time from the dropdown. Do not leave default value  **Subject: **Title of the case **Description:** Provide notes according to the KB article we created that describes the [investigation about the issue, relative documents, logs, and information about the case](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010136).  ****  +* Click on the **Submit Button** +  +### Viewing Cases / Updating Cases +The following allows you to see all the cases that you have opened on behalf of customers, and the ones your colleagues have opened.  Please note that if you would like to view the cases of customers **after they have become Armory customers**, use the [TAM/CSM Dashboard](https://armory.slab.com/posts/service-now-getting-started-information-about-service-now-ta-ms-managed-etc-p1idav2r#tam-csm-dashboard-customer-account-cases-and-information) +* **Log in** to the portal at [https://support.armory.io](https://support.armory.io/).  * In the Navigation Bar, select **Cases** -> **All Cases*** From this screen, there are options to view open and closed cases created by yourself on behalf of customers, and access to view the overall account's open and closed cases (the cases opened by your colleagues in the Sales Engineers Team)* Click on the case that needs to be reviewedOnce in the case, there are options available to update the case +* **Contact**: The main contact can be updated here to any registered user on the same account.  Note that users that have deactivated logins can also be chosen, so users should be careful when making a selection* **Watch list**: Allows the addition of more users to view a case.  Note that external users to the account can also be set in the watch list so that they can observe the case* **Activity**: Update the information about the case.  A WYSIWYG option is available when posting.  Attachments can also be added. Once ready, click **Post **to commit the information to the activity feed* Once changes are completed, click on **Save** in the bottom right to update the case +* You can either reply back to the case within the portal, or by replying to the emails sent to you +  +### Closing Cases +* **Log in** to the portal at [https://support.armory.io](https://support.armory.io/).  * In the Navigation Bar, select **Cases** -> **All Cases*** From this screen, there are options to view open and closed cases created by the user, and if the privilege is provided, access to view the overall account's open and closed cases.  * Click on the case that needs to be closed* In the bottom left corner, there is button to **Close Case.  **Click on this button to resolve and close a case without input from support* Please note that cases, once closed, cannot be opened again and will be in a **read only** state.  To open up a case about the issue again, please open a new case and provide any update information that can assist our team with a new resolution.  +  + diff --git a/kb/armory/General/servicenow---using-lists-and-views.md b/kb/armory/General/servicenow---using-lists-and-views.md new file mode 100644 index 0000000000..010ff25a98 --- /dev/null +++ b/kb/armory/General/servicenow---using-lists-and-views.md @@ -0,0 +1,35 @@ +--- +title: ServiceNow - Using Lists and Views +--- + + +Using Lists and Views +A lot of the information in ServiceNow is presented in lists. Lists are powerful, and allow users to sort through reams of information, extract that information, and create custom **personal** views and filters that can be saved. +For the purposes of this demo, we'll look at the Cases table, but the information can be used in various places, including other tables and reports. + +A list has generally a few areas of interest and layout. We'll be covering them here so that people can access and find information in the way that they want to +Customize Columns + + + +One of the first things to make sure about your list view is to set up the columns that are most important to your account. By clicking on the Gear Icon in the upper left corner, a selection will appear allow the user to add or remove columns, and even reset back to the default columns. The columns available for selection depends on the type of record the list is displaying (e.g. the columns available to display will be different compared to a user list). + + +Filters +There are a few ways to filter information and search information. Filters are extremely powerful and will allow users to present a view of data that makes sense to the user. For example, maybe it make sense to show all the cases from JPMC. In this case, Click on the Filter icon to unfurl the options + +* From that expanded filter view, users can now chose a **field** and **operation**, and the values they would like to filter on.* Users can add additional conditions such as **AND** or **OR**, and add **a sort order**.This filter can then be **Saved** so it can be accessed quickly later. +* The user will be prompted to name the filter* They can then advise if it should be visible to either yourself, Everyone, or a Group* Click **Save** and the filter will now be available to be used +* Or, press **Run**, to execute on the filter* If you need to make further refinements, or edits to the filter, simply click on the filter icon again, and make additions and run again. +A Saved filter can then be re-accessed, or managed by clicking on the "hamburger" on the upper left corner and then selecting the saved filter from the list. You can also manage your filters by clicking on **edit personal filters**. +**Filter out**, can also be used to add to your current filter. For any value in any column, right click on it, and select to "filter out" and it will automatically be added into the filter query. +Searches +Searches can be performed either from the very top of the list view, or in individual columns. The searches will return values based on the established filters + +You can also use wildcards as a part of your search. (e.g., if you have a case number, and only want to look for the final 5 digits, use the search ```*#####```*, *like ```*23456```) Any search will then be added as a part of the "filter" at the top. +If you do not see the search line under the column headers, click on the **Magnifying Glass** next to the Gear Icon in the list view +Miscellaneous Items +* Any list that is created, can be exported as a variety of different sheets, including Excel, CSV, JSON, and PDF. Just Right Click close to the header, and select Export → (File Format)* You can sort by any value by clicking on the "hamburger" in any column and choosing to sort ascending (a to z) or descending (z to a)* Clicking on the ```i``` icon next to any record opens a preview of the record that will show the most relevant information pertaining to the record.* Select any number of the records, and at the bottom, there's a selection that advises Actions on the Selected Rows. At times, it may be needed to perform a function on multiple records, and this would be the method to do it. This includes deleting.You can also Group your list results, providing collapsable lists groups. As an example, you can Group Account, and then have collapsable lists for each account +* In the List view, click on the hamburger icon for the column or right click, and an option to **Group by ** will be available.* You can also click on the main hamburger in the upper left corner of the list and select **Group By** → Select the category of to group by* To undo and go back to a full list view, just click on any column and select **Ungroup** +* Any List and view that you find is very important to have available on hand, can be "favorited" so that it is a part of your list in your Navigation menu. To favorite your current list, click on the main hamburger icon in the upper left corner of the list and select **Create Favorite**. You'll then be prompted to select a name for the favorite, icon to distinguish it, and colors to make it more easily identifiable + diff --git a/kb/armory/General/servicenow-service-portal-customer-administrators.md b/kb/armory/General/servicenow-service-portal-customer-administrators.md new file mode 100644 index 0000000000..8359dea8e2 --- /dev/null +++ b/kb/armory/General/servicenow-service-portal-customer-administrators.md @@ -0,0 +1,19 @@ +--- +title: ServiceNow Service Portal Customer Administrators +--- + + +Customer Administrators are privileged customer users who have been granted additional access and management rights to the ServiceNow Portal. Customers should inform Support or the designated TAM about any changes that need to happen with regards to their assigned administrator. Because the administrator has additional privileges, please be aware of the following information when assigning administrators.  +Add a User +* Go to the ServiceNow service portal ([https://support.armory.io](https://support.armory.io/))* Log in with your credentials* Click on Manage Accounts on the side menu → Click on Users List* Click on the "Create User" button. Fill in all information about the new user* Click on Submit user. The user will now be created, and the person can now log in or reset their password to access the support portal +If for some reason a user cannot be created, please reach out to Armory Technical Support in Slack at @support-team.  If a mass list needs to be added, feel free to engage Armory Technical Support or the Technical Account Manager/Customer Success Manager assigned to your company.  +Toggle Between Enable/Disable a User +Disabled users cannot log in or reset their password.  Administrators can also enable disabled users, and it is recommended that they do so once an employee has left their company.  Armory is also available to assist in this process if needed.  Please reach out to Armory Technical Support in Slack at @support-team +* Go to the ServiceNow service portal ([https://support.armory.io](https://support.armory.io/))* Log in with your credentials* Click on Manage Accounts on the side menu → Click on Users List* Click on the user that you wish to disable* Select to Disable Login to disable a user, or Enable Login to enable a user* Click on "Save"   +  +Changing Administrators +To add additional administrators, please follow the following instructions. Armory is also available to assist in this process if needed.  Please reach out to Armory Technical Support in Slack at @support-team +* Go to the ServiceNow service portal ([https://support.armory.io/](https://support.armory.io/))* Log in with your credentials* Click on Manage Accounts on the side menu → Click on Grant Administrator Role * Select from the drop down from the users currently registered * Click on "submit" +  +If you wish to remove any administrators, please reach out to Armory Technical Support in Slack at @support-team + diff --git a/kb/armory/General/set-execution-history-lookback.md b/kb/armory/General/set-execution-history-lookback.md new file mode 100644 index 0000000000..791e04367f --- /dev/null +++ b/kb/armory/General/set-execution-history-lookback.md @@ -0,0 +1,13 @@ +--- +title: Set Execution History Lookback +--- + + +If you have a pipeline that hasn’t run in a long time, you might only ever see one pipeline execution in its history – and as soon as you run the pipeline again, it disappears, leaving only the latest. +This is because Spinnaker, by default, limits lookback to only the last 14 days of activity; if no execution history is found in that time, it only returns the last known execution, regardless how many you ask it to display. +This can’t currently be turned off, but you can adjust it with configuration. +You’ll need to edit (or create) your ```orca-local.yml``` file, and then add this bit, adjusting the number of days to look back as you’d like: +tasks: + daysOfExecutionHistory: 365 +Hopefully you’re deploying more than once a year! At least this will let you look back over the past year for previous deployments, and not just the last two weeks. + diff --git a/kb/armory/General/setting-a-github-status-check-on-pr-from-a-pipeline-status.md b/kb/armory/General/setting-a-github-status-check-on-pr-from-a-pipeline-status.md new file mode 100644 index 0000000000..51257c299f --- /dev/null +++ b/kb/armory/General/setting-a-github-status-check-on-pr-from-a-pipeline-status.md @@ -0,0 +1,12 @@ +--- +title: Setting a Github Status Check on PR from a Pipeline Status +--- + + +A common situation Administrators can hit is creating a pull request on Github with some Terraform changes that are run through pipelines. Administrators want to verify the Terraform changes cleanly and “plan” before applying them. +An example is when someone has a typo, and the plan fails. Spinnaker has a solution for this. + +  +The following blog shows how to add the status check to help notify the person making changes and help self-diagnose and prevent issues. +[https://spinnaker.armory.io/blog/spinnaker-tips-setting-a-github-status-check-on-a-pr-from-a-pipeline/](https://spinnaker.armory.io/blog/spinnaker-tips-setting-a-github-status-check-on-a-pr-from-a-pipeline/) + diff --git a/kb/armory/General/setting-an-sql-cleanup-job-for-clouddriver.md b/kb/armory/General/setting-an-sql-cleanup-job-for-clouddriver.md new file mode 100644 index 0000000000..a9acacbce0 --- /dev/null +++ b/kb/armory/General/setting-an-sql-cleanup-job-for-clouddriver.md @@ -0,0 +1,25 @@ +--- +title: Setting an SQL Cleanup Job for Clouddriver +--- + +## Introduction +Spinnaker allows for the easy configuration of a cleanup job for the Clouddriver MySQL backend if it is enabled.  This can be helpful to remove orphaned entries and maintain a reasonable size of the SQL backend. +Enabling this setting causes clouddriver-caching's clean up agent to periodically purge old clusters and accounts. OSS recommends setting to true when using the Kubernetes provider. + +## Prerequisites +N/A + +## Instructions +#### In Halyard: +This would be added to the ```clouddriver-local.yaml``` file in the hal config profiles directory e.g. (~/.hal/default/profiles/) +``````sql.unknown-agent-cleanup-agent.enabled: true`````` +Alternatively, Users can manually go into the MYSQL Database and remove the offending entries.  +#### In Operator: +This can be set at **spec.spinnakerConfig.profiles.clouddriver** +spec: + spinnakerConfig: + profiles: + clouddriver: + sql.unknown-agent-cleanup-agent.enabled: true +Alternatively, Users can manually go into the MYSQL Database and remove the offending entries.  + diff --git a/kb/armory/General/setting-up-logging-without-tokens.md b/kb/armory/General/setting-up-logging-without-tokens.md new file mode 100644 index 0000000000..6b9c27626f --- /dev/null +++ b/kb/armory/General/setting-up-logging-without-tokens.md @@ -0,0 +1,26 @@ +--- +title: Setting up Logging without Tokens +--- + +## Introduction +In some environments, customers may find that they want to set up logging without Tokens or Security.  If Tokens are not available, Logging can still be set up, although it is not recommended in generally due to possible security risks. + +## Prerequisites +Customers should look to set up the logging endpoint with the following additional instructions +[https://docs.armory.io/armory-enterprise/armory-admin/observe/integrations-logging/](https://docs.armory.io/armory-enterprise/armory-admin/observe/integrations-logging/) + +## Instructions +Customers looking to set up logging can remove the need for a token in the definition as per the following example.  Once again, it is not recommended due to general security best practices. +```` +spec: + spinnakerConfig: + profiles: + echo: + rest: + enabled: true + endpoints: + - wrap: true + url: ":/services/collector/event?" + template: '{"event":{{event}} }' + insecure: true +```` \ No newline at end of file diff --git a/kb/armory/General/shared-configuration-repository.md b/kb/armory/General/shared-configuration-repository.md new file mode 100644 index 0000000000..46da20d128 --- /dev/null +++ b/kb/armory/General/shared-configuration-repository.md @@ -0,0 +1,16 @@ +--- +title: Shared Configuration Repository +--- + + +Question: +How do you use the same or multiple Spinnaker installations to deploy to multiple AWS accounts while sharing a single configuration repository? +Answer: +Having multiple Spinnaker installations, with multiple AWS accounts, share the same configuration repository is possible. First let’s cover a few things that need to be baselined, after that we can jump over to our docs for more details. +Spinnaker Environment: +This is where Spinnaker (the service) lives. Usually correlated with an appropriate DNS entry. ex: [https://spinnaker.yourcompany.com](https://spinnaker.yourcompany.com/). This is useful for different levels of isolations. There’s multiple methods of isolations you can do: multiple clouds (Kubernetes, AWS), AWS accounts, Kubernetes namespaces, different VPCs, or even just different instances with new datastores. +Deployment Target: +This is where Spinnaker is configured to deploy *to*. This is useful for managing applications across different cloud providers, accounts. +Documentation +With the above understanding jump over to our documentation:[https://docs.armory.io/admin-guides/shared_configuration_repo/](https://docs.armory.io/admin-guides/shared_configuration_repo/) + diff --git a/kb/armory/General/shell-script-introduce-massive-warning-in-spinnaker-ui-failed-to-evaluate-not-found.md b/kb/armory/General/shell-script-introduce-massive-warning-in-spinnaker-ui-failed-to-evaluate-not-found.md new file mode 100644 index 0000000000..b2937541f3 --- /dev/null +++ b/kb/armory/General/shell-script-introduce-massive-warning-in-spinnaker-ui-failed-to-evaluate-not-found.md @@ -0,0 +1,14 @@ +--- +title: ${ in shell script introduce massive warning in Spinnaker UI (Failed to evaluate ... not found) +--- + +## Issue +When running a shell script in a stage, the Spinnaker UI and logs show a massive warning message, but everything works fine. +``````Failed to evaluate ... not found`````` +A screenshot is below: + + +## Cause +End users seeing this issue may discover that it is tied to the shell scripts include the ```${}``` notation. For example, ```${n}```. +The format ```${}``` is the same as a Spring Expression Language (SpEL) expression. Orca is confused because it sees ```${n}``` as an expression, and there is no ***`n`*** in the context. + diff --git a/kb/armory/General/sourceami-must-be-defined-in-two-areas-if-a-template-json-is-used-in-aws-bakes.md b/kb/armory/General/sourceami-must-be-defined-in-two-areas-if-a-template-json-is-used-in-aws-bakes.md new file mode 100644 index 0000000000..12cb70fd6b --- /dev/null +++ b/kb/armory/General/sourceami-must-be-defined-in-two-areas-if-a-template-json-is-used-in-aws-bakes.md @@ -0,0 +1,20 @@ +--- +title: sourceAMI Must be Defined in Two Areas if a template JSON is used in AWS bakes +--- + +## Issue +Rosco supports adding ```sourceAMI``` which would serve as the base image for baking a custom AMI.  This setting can be changed/completed from the UI.  However, if a template JSON contains the filter ```source_ami_filter```, the Source AMI would also be based on the filter conditions. + +## Cause +There are two steps when the Bake process kicks in. + +Rosco would validate the ```sourceAMI``` defined in the configuration (or from UI) to see if the source AMI exists.  It validates by making a request to Clouddriver, which in turn uses the AWS APIs to determine if the ```sourceAMI``` is valid. +---> HTTP GET http://spin-clouddriver.gowri-operator-spin:7002/aws/images/find?q=CentOS+Linux+7+x86_64+HVM+EBS*&account=gowri-aws®ion=us-east-1 + + +Once the source AMI is valid, it invokes the packer command with the sourceAMI seen as the variable along with others.  Below is a sample command it invokes. +```packer, build, -color=false, -var, aws_region=us-east-1, -var, aws_instance_type=t2.micro, -var, aws_source_ami=ami-d7e1d2bd, -var, aws_target_ami=nginx-all-20220427155605-vendor-centos-7, -var, aws_ssh_username=centos, -var, aws_associate_public_ip_address=true, -var, package_type=rpm, -var, packages=nginx, -var, configDir=/opt/rosco/config/packer, -var, aws_subnet_id=subnet-02fd76fd0e0db6662, -var, aws_vpc_id=vpc-0a54022472f8eeb4c, /opt/rosco/config/packer/aws-vendor-centos-7-latest.json``` + +If the ```sourceAMI``` is not defined in the Rosco configurations, the bake will fail during the validation step. +Users may also have a criteria/filter before fetching the source AMI that uses a prefix.  ``````If the filter is defined, Packer does not respect the variables (such as sourceAMI) passed to the Packer command.  Without the filter, the accurate sourceAMI cannot be located because Spinnaker does not support fields like ```owners``` that's available on the filter in the template JSON. + diff --git a/kb/armory/General/specify-a-subnet-during-bake-stage.md b/kb/armory/General/specify-a-subnet-during-bake-stage.md new file mode 100644 index 0000000000..fc8324b7b7 --- /dev/null +++ b/kb/armory/General/specify-a-subnet-during-bake-stage.md @@ -0,0 +1,8 @@ +--- +title: Specify a Subnet During Bake Stage +--- + + +There may be occasions where it’s useful to specify the subnet used in your Bake stage. Targeting a specific subnet can ensure your [Packer Builders](https://www.packer.io/docs/builders/index.html) (which Spinnaker relies on for baking images) are assigned a subnet which meets the requirements of the network (e.g. a subnet which does not auto-assign a public IP). +Specifying a subnet is handled through the ```Extended Attributes``` of the Bake stage’s Bake Configuration. You will need to know the ```id``` of your subnet and associate it with the ```aws_subnet_id``` key. + diff --git a/kb/armory/General/spel-expression-failure-interpreting--with-terraform-templates-.md b/kb/armory/General/spel-expression-failure-interpreting--with-terraform-templates-.md new file mode 100644 index 0000000000..0e21a6019a --- /dev/null +++ b/kb/armory/General/spel-expression-failure-interpreting--with-terraform-templates-.md @@ -0,0 +1,21 @@ +--- +title: SpEL expression failure interpreting dollar sign bracket with Terraform Templates +--- + +## Issue +A SpEL expression failure appears while using Terraformer to serialize data from a ```Terraform Plan``` execution.  The execution creates a ```JSON object``` instead of to a ```JSON string```.The customer pipeline does the following: + +* In the Terraformer Stage, a Terraform Plan outputs a binary object as a Spinnaker Artifact. +* Terraform shows passing through a ```plan file``` which causes Terraform to output a ```JSON plan file``` as an ```output``` of the stage.The outputted ```plan file``` is stored as an artifact that can be used as a ```tfvars object``` for the next Terraformer stage. +* The command ```toJson``` is used to ingest the JSON object and transform it into a string.  The string will be passed in via a ```tfvars``` object to the subsequent ```Terraform Apply Stage``` (with a static/defined by using a module that uploads the ```tfvars var``` to a bucket/object) +* Lastly, the pipeline calls ```kube``` to run a 3rd party API to download the object through the name passed to it. + +The issue is that if the output has a ***SpEL escape*** in it ```${``` then it fails. + +## Cause +The cause is that the ```SpEL processor``` in Spinnaker throws a ```null``` value if the SpEL expression has ```${``` in it, and is visible in Spinnaker's code [here](https://github.com/spinnaker/kork/blob/d895e2b820baadc0986dd4451e7b18f030a9a865/kork-expressions/src/main/java/com/netflix/spinnaker/kork/expressions/ExpressionsSupport.java#L198) +Also, if the users are using an older Terraform version (< 0.12), then it is also using the template provider from Terraform.The template provider exposes data sources to use templates to generate strings for other Terraform resources or outputs. + +❗️ HashiCorp has deprecated the template provider since v0.12 +* The announcement is [here](https://registry.terraform.io/providers/hashicorp/template/latest/docs)* Users should use the [template file](https://www.terraform.io/language/functions/templatefile?_ga=2.68579986.223689003.1655389310-554336950.1655389310) function instead + diff --git "a/kb/armory/General/spel-expression-issue--must-be-of-type-string--\"integer\".md" "b/kb/armory/General/spel-expression-issue--must-be-of-type-string--\"integer\".md" new file mode 100644 index 0000000000..fd01126f5a --- /dev/null +++ "b/kb/armory/General/spel-expression-issue--must-be-of-type-string--\"integer\".md" @@ -0,0 +1,10 @@ +--- +title: SpEL expression issue- Must be of type string- "integer" +--- + +## Issue +Error noticed: ```spec.endpoints.metricRelabelings.replacement in body must be of type string: "integer"```This error indicates that replacement expects a string instead of an integer.  In the CRD some values may contain Regex expressions like ```${}```.  These characters in Spinnaker are used to evaluate SPEL expressions.  There is a way to escape these characters but doesn't work for numbers at this time. + +## Cause +Spinnaker is evaluating the expression and returning an integer causing the error. + diff --git a/kb/armory/General/spinnaker-access-denied-error-with-armory-agent-deployed-and-fiat-enabled-exception-monitor-pipeline.md b/kb/armory/General/spinnaker-access-denied-error-with-armory-agent-deployed-and-fiat-enabled-exception-monitor-pipeline.md new file mode 100644 index 0000000000..2642a809d6 --- /dev/null +++ b/kb/armory/General/spinnaker-access-denied-error-with-armory-agent-deployed-and-fiat-enabled-exception-monitor-pipeline.md @@ -0,0 +1,18 @@ +--- +title: Spinnaker Access denied error with Armory Agent deployed and FIAT enabled (Exception (Monitor Pipeline)) +--- + +## Issue +Admins and end users may find that with FIAT enabled and Armory Agent Deployed, Agent shows connection, but execution of pipelines will fail.  The following error may be outputted: +Exception ( Monitor Pipeline ) +Exception in child pipeline stage (deploy-armory-agent-devtest: DeployManifest): Access denied to account test-spinnaker + +  +FIAT is deployed without issue, and is enabled in the Clouddriver Config File.  + +## Cause +When trying to perform any manifest operation (deploy/patch/delete etc) with either ```Scale Agent``` or ```Clouddriver Kubernetes v2``` provider, Clouddriver will first fetch permissions from FIAT to validate that the user is authorized to do that operation. +FIAT synchronizes authorization data of each account and its permissions every certain period of time specified on the ```fiat.writeMode.syncDelayMs``` property as stated within the [Authorization Architecture](https://spinnaker.io/docs/reference/architecture/authz_authn/authorization/#sync). The default value is ```600000 milliseconds``` (10 minutes). +Therefore, it’s possible that FIAT synchronizes auth data at minute zero, Clouddriver defines new credentials at one minute, and then a new operation is performed with the new accounts. +We will then need to wait 9 minutes for FIAT to synchronize the new accounts. Otherwise, during that period ```Access denied to account``` errors will happen for new operations of the newly defined accounts.  This is assuming Clouddriver has the default 10 mins defined at the properly, ```fiat.writeMode.syncDelayMs```. + diff --git a/kb/armory/General/spinnaker-and-helm.md b/kb/armory/General/spinnaker-and-helm.md new file mode 100644 index 0000000000..328856672d --- /dev/null +++ b/kb/armory/General/spinnaker-and-helm.md @@ -0,0 +1,7 @@ +--- +title: Spinnaker and Helm +--- + + +Which one works best?  Helm or Spinnaker?The following blog post aims to answer this question about how both work together:https://www.armory.io/blog/deploy-helm-charts-with-spinnaker/ + diff --git a/kb/armory/General/spinnaker-cloud-provider-limit.md b/kb/armory/General/spinnaker-cloud-provider-limit.md new file mode 100644 index 0000000000..73b6f20fbf --- /dev/null +++ b/kb/armory/General/spinnaker-cloud-provider-limit.md @@ -0,0 +1,10 @@ +--- +title: Spinnaker Cloud Provider Limit +--- + + +Question: +Is there a limit to the number of Kubernetes clusters that one can define in Spinnaker? +Answer: +Nope! There isn’t a hard limit to the number of Kubernetes (or other) clusters you can define in Spinnaker. Depending on the number of clusters and resources you’re running, you may need to scale out multiple Clouddriver instances to handle caching. + diff --git a/kb/armory/General/spinnaker-does-not-properly-clean-up-old-artifacts-and-resources-when-redeploying.md b/kb/armory/General/spinnaker-does-not-properly-clean-up-old-artifacts-and-resources-when-redeploying.md new file mode 100644 index 0000000000..0e8b910073 --- /dev/null +++ b/kb/armory/General/spinnaker-does-not-properly-clean-up-old-artifacts-and-resources-when-redeploying.md @@ -0,0 +1,10 @@ +--- +title: Spinnaker does not properly clean up old artifacts and resources when redeploying +--- + +## Issue +An organization may encounter an issue that Spinnaker does not delete old artifacts during a deployment, which are not part of the new deployment. This happens for Kubernetes deployments, which leads to a dirty cluster state and potential security risks e.g old istio rules being existent, and causes significant manual work needed to clean it up. + +## Cause +This issue is caused by how OSS Spinnaker communicates with Kubernetes, there is currently no automatic way of determining what is and what isn't a used artifact or resource. This issue is being tracked on OSS Spinnaker [https://github.com/spinnaker/spinnaker/issues/4236](https://github.com/spinnaker/spinnaker/issues/4236)There is also related work going on in this issue [https://github.com/spinnaker/spinnaker/issues/4550](https://github.com/spinnaker/spinnaker/issues/4550)There are two sides to it. On the one side for singular pipelines that have a set number of resources, this can be handled with a delete resources stage as mentioned in the issue above.For multiple resource pipelines, it gets trickier as there currently isn't an ideal way of tracking those resources  + diff --git a/kb/armory/General/spinnaker-fails-to-deploy-to-google-appengine-flex-due-to-name-being-too-long.md b/kb/armory/General/spinnaker-fails-to-deploy-to-google-appengine-flex-due-to-name-being-too-long.md new file mode 100644 index 0000000000..e8ac48c66c --- /dev/null +++ b/kb/armory/General/spinnaker-fails-to-deploy-to-google-appengine-flex-due-to-name-being-too-long.md @@ -0,0 +1,37 @@ +--- +title: Spinnaker Fails to Deploy to Google AppEngine Flex Due to Name Being Too Long +--- + +## Issue +Spinnaker fails to deploy to Google AppEngine due to name being too long. Error will look something like the following: +Failed to deploy to App Engine with command gcloud app deploy /var/tmp/clouddriver +/gcr.io-np-ca-analytics-xxx-pospbsb:1.0.21/./b5769c9f-d0a6-4f2c-95e3- +038f9cfcb880.yaml --version=caanalyticspossalespubsub-pospbsb-v000 --no-promote +--stop-previous-version --project=np-ca-analytics-xxx --account=spinnaker- +onboarding@pr-developer-tools.iam.gserviceaccount.com --image-url=gcr.io/np-ca- +analytics-xxx/pospbsb:1.0.21: stdout: , stderr: Services to deploy: + +WARNING: This deployment will result in an invalid SSL certificate for service +[possalespubsub]. The total length of your subdomain in the format $VERSION_ID- +dot-$SERVICE_ID-dot-$APP_ID should not exceed 63 characters. Please verify that +the certificate corresponds to the parent domain of your application when you +connect. descriptor: [/var/tmp/clouddriver/gcr.io-np-ca-analytics-xxx- +pospbsb:1.0.21/b5769c9f-d0a6-4f2c-95e3-038f9cfcb880.yaml] source: +[/var/tmp/clouddriver/gcr.io-np-ca-analytics-xxx-pospbsb:1.0.21] target project: +[np-ca-analytics-xxx] target service: [possalespubsub] target version: [caanalyticspossalespubsub-pospbsb-v000] target url: +[http://caanalyticspossalespubsub-pospbsb-v000.possalespubsub.np-ca-analytics- +xxx.uc.r.appspot.com] + +(add --promote if you also want to make this service available from +[https://possalespubsub-dot-np-ca-analytics-xxx.uc.r.appspot.com]) +Do you want to continue (Y/n)? +Beginning deployment of service [possalespubsub]... WARNING: Deployment of service +[possalespubsub] will ignore the skip_files field in the configuration file, +because the image has already been built. ERROR: (gcloud.app.deploy) +INVALID_ARGUMENT: Combined version and service (module) name is too long. The combined length must be less than 48 characters. Note that for internal reasons, +each hyphen or sequence of hyphens in the version or service name counts as one +extra character. + +## Cause +The length of Spinnaker Application Name + Spinnaker Stack + Spinnaker Detail (optional) + Spinnaker Version (4 characters) combined cannot be greater than or equal to 48. + diff --git a/kb/armory/General/spinnaker-improperly-errors-even-when-spel-expressions-are-commented-out.md b/kb/armory/General/spinnaker-improperly-errors-even-when-spel-expressions-are-commented-out.md new file mode 100644 index 0000000000..cb2f0d7066 --- /dev/null +++ b/kb/armory/General/spinnaker-improperly-errors-even-when-spel-expressions-are-commented-out.md @@ -0,0 +1,11 @@ +--- +title: Spinnaker Improperly Errors Even When SPEL Expressions Are Commented Out +--- + +## Issue +Some users may find Spinnaker attempting to evaluate ```${}```-wrapped expressions not meant for Spinnaker to evaluate as SpEL. +An example may be artifacts that has some comments in it, unfortunately some of the comments are containing valid SpEL expressions.  + +## Cause +Orca's evaluation of the entire pipeline context for SpEL expressions occurs outside the control of individual stages. Due to this, the expressions won't skip over all of the SpEL expressions even if commented out.  + diff --git "a/kb/armory/General/spinnaker-is-down-with-the-error-of-\"oom-command-not-allowed-when-used-memory->-'maxmemory\"-redis.md" "b/kb/armory/General/spinnaker-is-down-with-the-error-of-\"oom-command-not-allowed-when-used-memory->-'maxmemory\"-redis.md" new file mode 100644 index 0000000000..33361ac2b2 --- /dev/null +++ "b/kb/armory/General/spinnaker-is-down-with-the-error-of-\"oom-command-not-allowed-when-used-memory->-'maxmemory\"-redis.md" @@ -0,0 +1,11 @@ +--- +title: Spinnaker is Down with the Error of "OOM command not allowed when used memory > 'maxmemory" (Redis) +--- + +## Issue +The Spinnaker environment is down and unaccessible, and comes with the Redis error as following: +```{"timestamp":1601481892xxx,"status":500,"error":"Internal Server Error","message":"OOM command not allowed when used memory > 'maxmemory'.; nested exception is redis.clients.jedis.exceptions.JedisDataException: OOM command not allowed when used memory > 'maxmemory'."}``` + +## Cause +The **O```O```M command not allowed when used memory > ‘maxmemory’** error means that Redis was configured with a memory limit and that particular limit was reached.In other words: its memory is full, it can’t store any new data. + diff --git a/kb/armory/General/spinnaker-kubernetes-v2-faqs.md b/kb/armory/General/spinnaker-kubernetes-v2-faqs.md new file mode 100644 index 0000000000..ab23acf3c4 --- /dev/null +++ b/kb/armory/General/spinnaker-kubernetes-v2-faqs.md @@ -0,0 +1,47 @@ +--- +title: Spinnaker Kubernetes V2 FAQs +--- + + +How to Create a Pipeline in Kubernetes V2 with Spinnaker? +Generally speaking, to create a Kubernetes V2 pipeline, you would create a ‘deployment’ manifest for your server group. You can store it in either GitHub, S3, or GCS. In the pipeline you can configure it to trigger when there is a change. Details to each can be found in the OSS docs. +You can find more details [here](https://www.spinnaker.io/reference/providers/kubernetes-v2/#using-externally-stored-manifests). +How Does Spinnaker Handle Multi-Cluster and Multi-Region Deploys to Kubernetes? +In Spinnaker you can configure multiple Kubernetes accounts with different clusters located in different regions, and then each deploy stage can target a specific account. So you can have a number of deploy stages, each going to a different Kubernetes accounts. +How Does Spinnakers Kubernetes V2 Handle Istio Changes? +There is no extra support for Istio right now. Some things may work perfectly, while others may not work as expected. Armory and the OSS community are still determining how to best fit Istio into the provider. +How do you Handle Pipelines that Sidecar Containers? Are These Just Defined in the yml? +Yes. You can add any containers you need to the ‘deployment’/’replicaSet’ manifest. This can be initContainers, side cars, etc. +How do You Handle Ingress with Let’s Encrypt SSL Certificates? +It is possible to deploy an ingress resource in the same way you can deploy any manifest. An nginx-ingress controller is configured by a config-map, which can be redeployed by Spinnaker to make changes. Since an ingress controller is a daemon deployed as a pod, it can also be redeployed by Spinnaker if desired. +See [here](https://www.spinnaker.io/reference/providers/kubernetes-v2/#services-ingresses) for more information. +Can V1 kube and V2 kube Live on the Same Cluster? +Yes. However, in the Spinnaker UI you will see every Kubernetes resource twice. Once, when the V1 provider sees it, and once, when the V2 provider sees it. +Is it possible to apply secrets using the [Base (Future Simple) Helm Secrets Plugin](https://github.com/futuresimple/helm-secrets)? +It is not possible today. +How do we handle deployment in multiple clusters? For example, if our environment includes more than one Kubernetes cluster. +With the account concept in Kubernetes, an account is just a Kubernetes cluster. When you configure an account you configure it to use an specific kubeconfig file. This defines how we communicate and authenticate on our clusters. If you have multiple clusters you can configure multiple accounts within Spinnaker using different kubeconfig files that have the address for the API. When you set the deployment step you set up a different account to deploy to, and Spinnaker will use the credentials found in the kubeconfig. +There are two different types of pipelines: Strategy and Pipeline. What’s the difference? +A pipeline is generally just the way your application rolls out. That would be like saying deploying to a particular environment Manual Judgment deploying into another, we do things like kicking of Jenkins jobs. Strategy is a way of taking all of the different pieces of that pipeline and being able to select them as a way you want to deploy. +It seems that Spinnaker is Polling Data from Kubernetes. How Often is this Data Refreshed? +By default Spinnaker tries to pull every 30 seconds, but you can configure the interval. +How’s the Performance of Spinnaker When it’s Keeping Track of Hundreds of Pipelines? +Spinnaker was built to run at scale - it is safe to run hundreds of pipelines with Spinnaker. However, that comes with the caveat that you have to manage and scale Spinnaker components. We have a guide at [spinnaker.io](https://spinnaker.io/) that describes which components can scale horizontally, depending on your workload. In the case of pipelines, there is a component called Front50, and you can add more replicas to prevent issues with large numbers of pipelines. +Is the V2 Provider Still in Beta and Subject to Significant Changes? +Release 1.10 sees this provider out of Beta, but there are still many large features that will get added to the provider. For example Traffic Management for Red/Black, Rolling Red/Black, Canary, etc. Also added label manipulation and Istio. +Changes to the Managed Pipeline Templates to allow operators to define custom pipelines that the users can use instead of the users making pipelines by hand over and over again. +Artifact provenance for software supply chain management, this will allow you to ask questions like, which commitment made into production since the last release, what changes in my config map trigger this release. +Any Plans to Move from kubectl to Kubernetes APIs? +There are no plans at the moment. One of the reasons we use kubectl, especially for deployment, is that there is some merging logic that is built into kubectl that we are taking advantage of. So the primary way that people interact with the Kubernetes API today is through kubectl. +As it stands using any of the APIs besides the official one and kubectl is very difficult and odds are they are largely unsupported. +Can the Manifests be Uploaded Using a Spinnaker API? +The best practice is to store them outside the pipeline. If you want to do it inline you will not upload the manifests to Spinnaker, what you would do be to upload the pipeline which will contain those manifests. You can upload a task saying, deploy this manifests to the Spinnaker API, if you have the manifests in mind and you have it´s representation, you can send that to Spinnaker to deploy it. +With a Pull Model, can the Dev and Prod Stages Call a Service or CLI to Pull the Appropriate Manifest(s) for Their Env? +The manifest can come from different places, GitHub, GitLab, S3. +Is Force Cache Refresh Duration Dependent to the Number of Objects in Manifest of Stage? Number of Objects in Manifest of Stage Might Increase the Duration? +The Force Cashe Refresh task in the deploy stage is basically making sure that after it has submitted everything that the Spinnaker orchrestation engine has an accurate view of what´s happening on Kubernetes. So Spinnaker tries as much as possible to read from it´s cashe the reason being, with the number of clients that will be looking at Spinnaker UI for example you will have thousands and thousands of requests duplicated against the server, so everything tries to read at that Cashe as most as possible, the issue of course is updating that Cashe it´s tricky because you have a lot of different compiting mechanism making sure that things are up to date, so, when you have a lot of manifests it don´t actually add a whole lot of time to Force Cashe Refresh stage, that time is mostly dominated by how many cluster you have or how many resource are in those clusters in the first place when Spinnaker is indexing them because everytime it´s rehersing the Cashe, it has to look at all the resources in all the cluster and then take any updates that happened like during deployment and the more resources you have the longer that step takes. +For Helm Charts Usage is There a Way to Locally Test the Bake Process? +Helm bakes the same way that Spinnaker does it, we just use helm template under the hood, it´s a subcommand within the top level helm command, if you have a chart you want to see how it will be rendered with Spinnaker you can just run that with helm template, you can search more information on their page helm.sh. +Can We Use ChartMuseum with Spinnaker Instead of Putting Helm Charts to S3? +Absolutely, ChartMuseum is effectively just an http server, managing Helm charts is easier but in the end it´s just an http server. As long as you can configure ChartMuseum at an end point you can configure Spinnaker to pull that chart of of ChartMuseum, the main thing that you need to remember is that whatever is serving your charts just needs to send them back as a tarball. + diff --git a/kb/armory/General/spinnaker-not-removing-the-old-auto-scaling-group-after-a-new-deployment.md b/kb/armory/General/spinnaker-not-removing-the-old-auto-scaling-group-after-a-new-deployment.md new file mode 100644 index 0000000000..cb98e3003f --- /dev/null +++ b/kb/armory/General/spinnaker-not-removing-the-old-auto-scaling-group-after-a-new-deployment.md @@ -0,0 +1,25 @@ +--- +title: Spinnaker not removing the old Auto Scaling Group after a new deployment +--- + +## Issue +Customers may find that there is a significant increase of the EC2 instances in some accounts.   +Upon further investigation, this can be attributed to Spinnaker not removing the old Auto Scaling Group after a new deployment. This results in the retention of a lot of versions for each service, and generates an increase in costs, greater pressure on services, and hitting API rate limits. +In this example, the deployment timeout per pipeline is set for 30 minutes.  +In the UI in the Deploy Configurations for the Red/Black strategy, the option for "Rollback to previous server group if deployment fails" has been set as well: + + + + + + +## Cause +What occurs is that the new server group fails to pass the health check. +As a result, no rollback happens to a previous version, and the new version does not get deleted either. Spinnaker tries to restart the failing server(s) repeatedly. +If the deployment is repeated again, with yet another failing version, server groups continue to be created. The environment then has numerous instances in AWS that are in a failing state. The Health checks for these instances show that the deployed ones fail, and are also restarted by Spinnaker. +  +The logs show the following: +```2021-09-14 14:20:52.665 WARN 1 --- [ handlers-15] c.n.s.o.c.t.i.WaitForUpInstancesTask : [integration.spinnaker@...... Short circuiting initial target capacity determination after 10 minutes (serverGroup: aws:....._orders-main-stage-v030, executionId: .......)``` +```Stage Deploy in ....... timed out after 30 minutes 6 seconds``` +After that the server is left unhealthy in the server group and does not get removed. + diff --git a/kb/armory/General/spinnaker-only-supports-unique-account-names-across-both-aws-and-ecs.md b/kb/armory/General/spinnaker-only-supports-unique-account-names-across-both-aws-and-ecs.md new file mode 100644 index 0000000000..99dd474d69 --- /dev/null +++ b/kb/armory/General/spinnaker-only-supports-unique-account-names-across-both-aws-and-ecs.md @@ -0,0 +1,28 @@ +--- +title: Spinnaker only supports unique account names across both AWS and ECS +--- + +## Issue +When the same account name is provided in AWS and ECS, some issues can arise such as incorrect credentials being used by the Lambda plugin, as it cannot distinguish between accounts defined with the same name in both resources in the manifest. +As an example, in Operator, if ```example-account``` is defined in both AWS and ECS as below, it will cause issues when being referred to in Lambda stages +spec:  spinnakerConfig:    config:      providers: aws: + enabled: true + accounts: + - name: example-account + providerVersion: V1 + accountId: '1234567890' + regions: + - name: us-east-2 + assumeRole: role/some-assumeRole + lifecycleHooks: [] + primaryAccount: example-account + ecs: + enabled: true + accounts: + - name: example-account + awsAccount: example-account + primaryAccount: example-account + +## Cause +Conflict in reference from other services about which account and access credentials should be used. + diff --git a/kb/armory/General/spinnaker-pipelines-not-displaying-in-infrastructure-tab.md b/kb/armory/General/spinnaker-pipelines-not-displaying-in-infrastructure-tab.md new file mode 100644 index 0000000000..c46bc6aa6e --- /dev/null +++ b/kb/armory/General/spinnaker-pipelines-not-displaying-in-infrastructure-tab.md @@ -0,0 +1,10 @@ +--- +title: Spinnaker pipelines not displaying in Infrastructure tab +--- + +## Issue +An organization may notice a behavior where spinnaker pipelines created under common application names do not display in the Infrastructure tab, despite the pipeline deploying correctly. + +## Cause +Resources in Spinnaker applications are grouped based on the ```managedBy``` and ```app``` annotations in the ***deployment manifest***. Resources may not be categorized correctly in Spinnaker if they are not correctly annotated in their deployment manifest. + diff --git a/kb/armory/General/spinnaker-supports-kustomize-up-to-v3.8.5-as-the-render-engine-in-bake-manifest-configuration,-but-not-v4.x-artifact-not-found.md b/kb/armory/General/spinnaker-supports-kustomize-up-to-v3.8.5-as-the-render-engine-in-bake-manifest-configuration,-but-not-v4.x-artifact-not-found.md new file mode 100644 index 0000000000..5ac4e36fa4 --- /dev/null +++ b/kb/armory/General/spinnaker-supports-kustomize-up-to-v3.8.5-as-the-render-engine-in-bake-manifest-configuration,-but-not-v4.x-artifact-not-found.md @@ -0,0 +1,13 @@ +--- +title: Spinnaker supports KUSTOMIZE up to v3.8.5 as the Render Engine in Bake (Manifest) Configuration, but not v4.x (Artifact not found) +--- + +## Issue +When using ```Kustomize``` as the declared Render Engine, user may get exceptions as seen below: + +```Exception ( Monitor Deploy ) The following required artifacts could not be bound: '[ArtifactKey(type=docker/image, name=250312325083.dkr.ecr.us-east-1.amazonaws.com/hive-won-airings, version=0.1.5, location=null, reference=250312325083.dkr.ecr.us-east-1.amazonaws.com/hive-won-airings:0.1.5)]'. Check that the Docker image name above matches the name used in the image field of your manifest. Failing the stage as this is likely a configuration error.``` + + +## Cause +The latest versions of ```Kustomize```, for example, v4.5.5, etc., are not supported within in the cluster due to the mismatch with the API version declared.   + diff --git a/kb/armory/General/spinnaker-timeout-issues-with-scale-agent--increasing-grpc-message-size.md b/kb/armory/General/spinnaker-timeout-issues-with-scale-agent--increasing-grpc-message-size.md new file mode 100644 index 0000000000..b2550eb0bc --- /dev/null +++ b/kb/armory/General/spinnaker-timeout-issues-with-scale-agent--increasing-grpc-message-size.md @@ -0,0 +1,38 @@ +--- +title: Spinnaker Timeout Issues with Scale Agent- Increasing gRPC Message Size +--- + +## Issue +In a Spinnaker deployment, it’s common to encounter timeout issues when deploying custom resources via Armory’s Scale Agent. These issues can be accompanied by error messages such as ```Timeout exceeded for operation``` and ```gRPC message exceeds maximum size```. This article will explore the cause of these errors and provide a guide on how to increase the gRPC message size in Spinnaker to resolve the problem. +In Spinnaker, deploying a Custom Resource to a Kubernetes Cluster via Scale Agent can result in timeout exceeded for Operation error. On the UI, the deploy manifest stage has the below error as an example. + Exception ( Deploy Manifest ) +Timeout exceeded for operation. OperationId: '01HDHRPM9018Z3WHN316JBEYSV' Type: 'List' Account: 'aws-spin-support-agent' Kind: 'CustomResourceDefinition' Namespace: 'default' +  +The Agent logs have the below errors as an example. + time="2023-10-25T19:53:18Z" level=error msg="error sending operation result" account=aws-spin-support-agent agentId=spin-armory-agent-5955c56d4b-sqdq6 error="rpc error: code = Canceled desc = stream terminated by RST_STREAM with error code: CANCEL" operationId=01HDM6C0A3QQG0KM1GQ53HSHDP operationNamespace= operationType=List +time="2023-10-25T19:53:21Z" level=error msg="error sending operation result" account=aws-spin-support-agent agentId=spin-armory-agent-5955c56d4b-sqdq6 error="rpc error: code = Canceled desc = stream terminated by RST_STREAM with error code: CANCEL" operationId=01HDM6C0A3QQG0KM1GQ53HSHDP operationNamespace= operationType=List +time="2023-10-25T19:53:24Z" level=error msg="error sending operation result" account=aws-spin-support-agent agentId=spin-armory-agent-5955c56d4b-sqdq6 error="rpc error: code = Canceled desc = stream terminated by RST_STREAM with error code: CANCEL" operationId=01HDM6C0A3QQG0KM1GQ53HSHDP operationNamespace= operationType=List +time="2023-10-25T19:53:24Z" level=error msg="Max retries reached for operation result" account=aws-spin-support-agent agentId=spin-armory-agent-5955c56d4b-sqdq6 error="giving up after 3 attempts, last error: rpc error: code = Canceled desc = stream terminated by RST_STREAM with error code: CANCEL" operationId=01HDM6C0A3QQG0KM1GQ53HSHDP operationNamespace= operationType=List +time="2023-10-25T19:53:24Z" level=warning msg="High latency detected sending operation result to clouddriver: spent 6.045102696s" account=aws-spin-support-agent agentId=spin-armory-agent-5955c56d4b-sqdq6 operationId=01HDM6C0A3QQG0KM1GQ53HSHDP operationNamespace= operationType=List + +  +The Clouddriver logs the below exceptions as an example. +2023-10-25 19:52:15.284 INFO 1 --- [ault-executor-1] i.a.k.services.RegistrationService : Sending operationId 01HDM6A3M8NMF28RYCPKFNRJ8V type List for account aws-spin-support-agent to agent spin-armory-agent-5955c56d4b-sqdq6 +2023-10-25 19:52:16.336 WARN 1 --- [-worker-ELG-3-1] i.g.n.s.io.grpc.netty.NettyServerStream : Exception processing message + +io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 4194304: 4534004 + at io.grpc.Status.asRuntimeException(Status.java:526) ~[grpc-api-1.45.1.jar:1.45.1] + at io.grpc.internal.MessageDeframer.processHeader(MessageDeframer.java:391) ~[grpc-core-1.45.1.jar:1.45.1] + at io.grpc.internal.MessageDeframer.deliver(MessageDeframer.java:271) ~[grpc-core-1.45.1.jar:1.45.1] + at io.grpc.internal.MessageDeframer.deframe(MessageDeframer.java:177) ~[grpc-core-1.45.1.jar:1.45.1] + at io.grpc.internal.AbstractStream$TransportState.deframe(AbstractStream.java:210) ~[grpc-core-1.45.1.jar:1.45.1] + at io.grpc.internal.AbstractServerStream$TransportState.inboundDataReceived(AbstractServerStream.java:255) ~[grpc-core-1.45.1.jar:1.45.1] + at io.grpc.netty.shaded.io.grpc.netty.NettyServerStream$TransportState.inboundDataReceived(NettyServerStream.java:226) ~[grpc-netty-shaded-1.45.1.jar:1.45.1] + at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler.onDataRead(NettyServerHandler.java:508) ~[grpc-netty-shaded-1.45.1.jar:1.45.1] + + +  + +## Cause +The error messages suggest a problem with the size of gRPC messages sent during custom resource deployment. Specifically, the error ```RESOURCE_EXHAUSTED: gRPC message exceeds maximum size``` indicates that the gRPC message size is larger than the default limit of 4MB. To resolve this issue, we need to increase the maximum gRPC message size in Spinnaker. + diff --git a/kb/armory/General/spinnaker-timesout-after-60000-milliseconds-when-updating-to-spinnaker-2.19.x-from-2.18.x-or-lower.md b/kb/armory/General/spinnaker-timesout-after-60000-milliseconds-when-updating-to-spinnaker-2.19.x-from-2.18.x-or-lower.md new file mode 100644 index 0000000000..6c3e0dee5a --- /dev/null +++ b/kb/armory/General/spinnaker-timesout-after-60000-milliseconds-when-updating-to-spinnaker-2.19.x-from-2.18.x-or-lower.md @@ -0,0 +1,13 @@ +--- +title: Spinnaker Timesout After 60000 milliseconds when Updating to Spinnaker 2.19.x from 2.18.x or lower +--- + +## Issue +Spinnaker errors out when upgrading to a new version.  Issue occurs only when going to 2.19.x.  It does not happen when downgrading below 2.18.x nor does it happen when going from a lower version than 2.18.x to 2.18.x +Halyard log will show an error similar to: +```Interrupting task [Generate config] (**************) - RUNNING that timed out after 60000 millis.``` + +## Cause +There can be multiple reasons for this issue to occur.  How Spinnaker handles kubeconfig files is different starting in 2.19.x.  This is due to a regression bug found in Halyard 1.8.x, and it may only download one kubeconfig file because the decryption is not being done by Halyard. +With Spinnaker 2.19.x forward, the bug may cause Halyard to attempt to download all kubeconfig files, increasing the chance of Network Latency Errors/Failures. + diff --git a/kb/armory/General/splunk-ami-issues---io-wait-warning,-connection-lost-splunk-crashed-with-splunk-service.md b/kb/armory/General/splunk-ami-issues---io-wait-warning,-connection-lost-splunk-crashed-with-splunk-service.md new file mode 100644 index 0000000000..c8e3240d51 --- /dev/null +++ b/kb/armory/General/splunk-ami-issues---io-wait-warning,-connection-lost-splunk-crashed-with-splunk-service.md @@ -0,0 +1,10 @@ +--- +title: Splunk AMI Issues - IO WAIT Warning, Connection Lost/Splunk Crashed with Splunk Service +--- + +## Issue +Following is information about setting up Splunk with Logging, that is internal only.  Information was researched by Chris Jeggo.  Customers should be pointed to Splunk for Troubleshooting Splunk issues, but for our internal reference, this is some data that was found over the course of working with Splunk + +## Cause +Splunk AMI Troubleshooting, as Splunk AMI has some restrictions and issues out of the box + diff --git a/kb/armory/General/spring-expression-language-spel-samples.md b/kb/armory/General/spring-expression-language-spel-samples.md new file mode 100644 index 0000000000..6d01ccba29 --- /dev/null +++ b/kb/armory/General/spring-expression-language-spel-samples.md @@ -0,0 +1,153 @@ +--- +title: Spring Expression Language (SpEL) samples +--- + +## Introduction +Spring Expression Language (SpEL) is a powerful tool that can help customers with a variety of tasks around pipelines.  This can include passing information to subsequent stages and transforming information that was provided from previous stages. +Customers wanting to see some larger examples of how SpEL can be leveraged, can take a look at the following KB articles that provide some real-life situations where SpEL helps solve some common issues from customers: +* Passing Rosco-Packer information to subsequent stages: [https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010305](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010305)* Using SpEL to add Target Groups in a Deployment Pipeline for AWS CloudFormation: [https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010280](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010280) +  +Below are some common usage examples of using in Evaluate Variables stage and Check Preconditions stages + +## Prerequisites +Spinnaker is configured. + +## Instructions +### Sample 1: Evaluate a Job stage's status +Pipeline sample: a job stage followed by an evaluate stage  +Evaluate Variables Configuration, provides the ```status``` value of the stage as the output + + +``````${#stage('Job 4390')['status']}​`````` + +### Sample 2: Capture Job Stage Output Value +Evaluate Variables Configuration, provides the particular output from the previous stage. Outputs the sample named ```a``` from the stage. + + + +``````${#stage("Job 4390").outputs.a}`````` + + +  + +### Sample 3: Call parameter value +Evaluate Variables Configuration. Pulls the parameters called ```a1``` from the stage + + +``````${parameters.a1}`````` + +###   +### Sample 4: Call parameter all values +Evaluate Variables Configuration. Pulls the parameter all values from the stage + + +``````${parameters.values()}`````` + +###   +### Sample 5: Use java class  +Evaluate Variables Configuration.  Implementing ```Java class``` for evaluations.  + + +``````${new java.lang.String(parameters.a1).contains("a")}`````` + + + +``````${new java.lang.String(a).indexOf(':')}`````` + + + +``````${new java.lang.String(a).substring(1,8)}`````` + +###   +### Sample 6: Use Helper functions + + +Evaluate Variables Configuration. Simple values output transformations. +```alphanumerical``` command returns only A-z and 0-9 values from the parameter ```a1```. + + +``````${#alphanumerical('a1')}`````` + + +  +```Map01``` example shows how to make equivalent value maps.  ```readJson``` inputs a JSON string into a Map so that it can be processed further.  In this example, it is reading it from the ```map01``` values. +  + +  +``` + +{"a":1,"b":2} + +${#readJson(map01)}​ + +  +```deployedServerGroups``` command takes the name of the deploy stage and then returns the Server Group created by the stage +```deployedServerGroups[0].serverGroup``` command returns the first value in the list. + + +${#deployedServerGroups()} +${deployedServerGroup[0].serverGroup} + + ``` + +### Sample 7: Capture Webhook Stage Output Values +Pipeline sample: a webhook stage followed by an evaluate stage  +Evaluate Variables Configuration captures the webhook stage values specified.  In the example, it is the ```WebhookBefore``` stage and the output values. + + +``````${#stage("WebhookBefore").context.webhook.body}​`````` + + +###   +### Sample 8: Capture the Size of Map of JSON + + + +Evaluate Variables Configuration.  Size evaluates the size of the list and returns the count on the size. + + +``````${alreadyDeployedBeforeRun.size()}​`````` + + +###   +### Sample 9: Use toString().contains() + + +Evaluate Variables Configuration. Checks to see if the string is contained within the value.  + + +``````${alreadyDeployedBeforeRun.toString().contains(deployedServerGroupName)}`````` + +  + +### Sample 10: Perform Calculations +Evaluate Variables Configuration.  + +``` + +${afterDeployServerGroup.size()} +${sizeOfAfter-sizeOfAlready} +${delta > 0} + + +###   +### Sample 11: Use expression in Check Preconditions stage +Pipeline sample: an evaluate stage followed by a Check Preconditions stage  +Define Check Preconditions + +Edit Precondition to use Expression: + + +Expression: +``````${!errorCreated}`````` + +``` +  +### Reference: +[Use the Spring Expression Language (SpEL) in Spinnaker Pipelines](https://docs.armory.io/armory-enterprise/spinnaker-user-guides/expression-language/) +[Pipeline Expressions Guide](https://spinnaker.io/docs/guides/user/pipeline/expressions/) +[Pipeline Expression Reference](https://spinnaker.io/docs/reference/pipeline/expressions/) + + + + diff --git a/kb/armory/General/spring-expression-language-tricks.md b/kb/armory/General/spring-expression-language-tricks.md new file mode 100644 index 0000000000..68b2f9fa22 --- /dev/null +++ b/kb/armory/General/spring-expression-language-tricks.md @@ -0,0 +1,38 @@ +--- +title: Spring Expression Language Tricks +--- + +## Introduction +Spinnaker uses the Spring Expression Language (SpEL) for pipeline expressions, so you can do a lot of interesting things with Spinnaker expressions. +Here are a couple of the more interesting examples that we’ve come across (this page will grow) + +## Prerequisites +N/A + +## Instructions +Basic Math +Take the number of instances for a deployment and multiply by a non-integer value:Note: “Get Deployment” is the name of the “Find Artifacts from Resource (Manifest)” stage that looks at a Kubernetes deployment object. +```${ (0.8 * #stage("Get Deployment")["outputs"]["manifest"]["spec"]["replicas"]).intValue() }``` +Helper Properties +Create a ```parameter environment``` with 3 options +Access the value of the variable via a helper property, in this case, ```parameter```. For example: +```The environment is: ${ parameters.environment }``` +Some Helper Properties are already defined, for example: +```The execution id is automatically set: ${execution['id']}``` +Helper Functions +Read and print a JSON, for example: +```${#readJson('{"development": "dev", "staging": "stage", "production":"prod"}').toString()}``` +You can also access a value in the JSON with the ```environment``` parameter from the [Helper Properties](https://kb.armory.io/pipelines/spel-tricks/#helper_properties) Section: +```${#readJson('{"development": "dev", "staging": "stage", "production":"prod"}')[parameters.environment]}``` +Ternary Operator +``` ? : ``` +Simple example: +```${ true ? "True text" : "False text" }``` +Example (in the text of a Manual Judgement stage), where ```Get Service``` is a “Find Artifacts From Resource (Manifest)” that looks at a Kubernetes ```service``` object: +```The loadBalancer ingress is ${ #stage("Get Service")["outputs"]["manifest"]["status"]["loadBalancer"].containsKey("ingress") ? "ready" : "not ready" }.``` +Whitelisted Java Classes +Some Java classes are available for use in SpEL expressions (see [Spinnaker Reference Docs](https://www.spinnaker.io/reference/pipeline/expressions/#whitelisted-java-classes))For example, generating the current date in MM-dd-yyyy format: +```${new java.text.SimpleDateFormat("MM-dd-yyyy").format(new java.util.Date())}``` +Sometimes you may want to call a static method. For example, to generate a UUID: +``` ${T(java.util.UUID).randomUUID() }``` + diff --git a/kb/armory/General/ssh-keys-in-terraformer.md b/kb/armory/General/ssh-keys-in-terraformer.md new file mode 100644 index 0000000000..ac68b8b8a2 --- /dev/null +++ b/kb/armory/General/ssh-keys-in-terraformer.md @@ -0,0 +1,56 @@ +--- +title: SSH Keys in Terraformer +--- + +## Introduction +This document will show you how to install SSH Keys into the Terraformer container. The process requires modifications to the deployment of Terraformer running in Kubernetes.For information about what causes this issue, see the following Kubernetes issue: [https://github.com/kubernetes/kubernetes/issues/2630](https://github.com/kubernetes/kubernetes/issues/2630)The workaround is based on the following post from Stack Overflow: [https://stackoverflow.com/a/57908921](https://stackoverflow.com/a/57908921) +**Note: **Users running 2.20.x+ should be considering using Terraform Named Profiles instead +[https://docs.armory.io/docs/spinnaker/terraform-enable-integration/#named-profiles](https://docs.armory.io/docs/spinnaker/terraform-enable-integration/#named-profiles)  + +## Prerequisites +Terraformer installed in the Armory Spinnaker cluster. The SSH Key should already be created and added as a Deploy Key to the Git repository. + +## Instructions +### Create the Secret +On local workstation, create a directory and place the SSH Key and any other required authentication information inside: +* Create the directory:```mkdir ssh```* ``````Copy the SSH Key:```cp $SSH_KEY_FILE ssh/id_rsa```* Copy any other authentication information that's needed:```cp $GOOGLE_APPLICATION_CREDENTIALS ssh/account.json```* ``````Create a config file for SSH to ignore the known_hosts checks:```echo "StrictHostKeyChecking no" > ssh/config```* ``````Create the secret using ```kubectl```:```kubectl create secret generic spin-terraformer-sshkey -n spinnaker-system --from-file=id_rsa=ssh/id_rsa --from-file=config=ssh/config --from-file=account.json=ssh/account.json``` +In this example, we create a secret with the SSH key, a config to ignore known hosts file issues, and the GCP service account information to access the backend bucket that Terraform is configured to use. +### Update the Manifest +Next, the K8s manifest needs to be updated to include a few things. +First, update the secret and an empty directory volume that will contain the copy of the secret with the correct uid and permissions: +volumes: +- name: spin-terraformer-sshkey + secret: + defaultMode: 420 + secretName: spin-terraformer-sshkey +- name: ssh-key-tmp + emptyDir: + sizeLimit: "128k"​ + +Second, define an init container that copies the secret contents to the empty directory and set the permissions and ownership correctly. The Spinnaker user uses user id 1000: +### Adding to set the ownership of the ssh keys +initContainers: +- name: set-key-ownership + image: alpine:3.6 + command: ["sh", "-c", "cp /key-secret/* /key-spin/ && chown -R 1000:1000 /key-spin/* && chmod 600 /key-spin/*"] + volumeMounts: + - mountPath: /key-spin + name: ssh-key-tmp + - mountPath: /key-secret + name: spin-terraformer-sshkey​ + +Mount the (not so) empty directory into the Terraformer container at the ```/home/spinnaker/.ssh``` location: +volumeMounts: +- mountPath: /home/spinnaker/.ssh + name: ssh-key-tmp​ + +Finally, add the following snippet to the envs to get the GCP service account to work for the S3 bucket: +- env: + - name: GOOGLE_APPLICATION_CREDENTIALS + value: /home/spinnaker/.ssh/account.json​ +This isn't necessary for the SSH Keys, but it completes the example. + + +### Result +After the modifications are in place, and Terraformer has time to redeploy via the replica set, Terraform should be able to clone Git repositories via SSH as long as the repository has the SSH Key added as a deploy key. Halyard shouldn't overwrite these changes, but it would be good to back this up. + diff --git a/kb/armory/General/stacktrace-error-appears-on-the-functions-display.md b/kb/armory/General/stacktrace-error-appears-on-the-functions-display.md new file mode 100644 index 0000000000..6f72b35284 --- /dev/null +++ b/kb/armory/General/stacktrace-error-appears-on-the-functions-display.md @@ -0,0 +1,23 @@ +--- +title: Stacktrace Error appears on the Functions Display +--- + +## Issue +When a new application with a single function is created, the Deck UI will show a Stacktrace error as follows:The Stacktrace (for example) shows the following: +TypeError: Cannot destructure property 'application' of 'this.state' as it is null. + at FunctionActions_FunctionActions.render (https://preprod.spinnaker.armory.io/3.js:112515:203011) + at gi (https://preprod.spinnaker.armory.io/3.js:222406:192) + at fi (https://preprod.spinnaker.armory.io/3.js:222405:224) + at Rj (https://preprod.spinnaker.armory.io/3.js:222487:490) + at Qj (https://preprod.spinnaker.armory.io/3.js:222470:265) + at Kj (https://preprod.spinnaker.armory.io/3.js:222470:194) + at yj (https://preprod.spinnaker.armory.io/3.js:222463:172) + at https://preprod.spinnaker.armory.io/3.js:222347:115 + at push.exports.unstable_runWithPriority (https://preprod.spinnaker.armory.io/3.js:222554:467) +The issue occurs frequently with AWS Lambda, preventing Lambda details from loading.  + +## Cause + +The following PR addresses the issue:[https://github.com/spinnaker/deck/pull/9199](https://github.com/spinnaker/deck/pull/9199) + + diff --git a/kb/armory/General/stage-returns-error-that-fargate-only-supports-network-mode-awsvpc-even-though-it-is-configured.md b/kb/armory/General/stage-returns-error-that-fargate-only-supports-network-mode-awsvpc-even-though-it-is-configured.md new file mode 100644 index 0000000000..e4d10bad51 --- /dev/null +++ b/kb/armory/General/stage-returns-error-that-fargate-only-supports-network-mode-awsvpc-even-though-it-is-configured.md @@ -0,0 +1,11 @@ +--- +title: Stage Returns Error that Fargate Only Supports Network Mode awsvpc Even Though it is Configured +--- + +## Issue +When using ECS containers with Fargate, the following error message may occur, even through the stage was set up to use awsvpc. +```Fargate only supports network mode 'awsvpc'. (Service: AmazonECS; Status Code:400; Error Code: ClientException; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)``` + +## Cause +Both stage and the service definition for the container(s) need to be set up to use awsvpc + diff --git a/kb/armory/General/stage-times-out-during-implementation,-need-to-extend-stage-timeout.md b/kb/armory/General/stage-times-out-during-implementation,-need-to-extend-stage-timeout.md new file mode 100644 index 0000000000..715f0b3933 --- /dev/null +++ b/kb/armory/General/stage-times-out-during-implementation,-need-to-extend-stage-timeout.md @@ -0,0 +1,10 @@ +--- +title: Stage Times Out During Implementation, Need to Extend Stage Timeout +--- + +## Issue +When implementing a stage, the stage will timeout due to the length of the process.  This can happen often with items such as the Bake Stage and Orca, or Terraformer + +## Cause +Default timeout is set, and is not long enough to allow for the processes being implemented by the stage to complete.  As a result, the stage ends prematurely + diff --git a/kb/armory/General/storing-application-secrets-in-vault-for-use-in-spinnaker-pipeline..md b/kb/armory/General/storing-application-secrets-in-vault-for-use-in-spinnaker-pipeline..md new file mode 100644 index 0000000000..4fc6a696a6 --- /dev/null +++ b/kb/armory/General/storing-application-secrets-in-vault-for-use-in-spinnaker-pipeline..md @@ -0,0 +1,7 @@ +--- +title: Storing application secrets in vault for use in Spinnaker pipeline. +--- + + +Application secrets should not be passed through Spinnaker or any other deployment tool as this is not safe from a security standpoint. If the tool is breached you now have all applications secrets that were passed through the pipeline exposed. The only things that should ever be passed through a deployment tool are location and/or references to the secret.The best practice for application secrets is for the application to fetch the secret itself during application startup. For VMs this is during the VM bootstrap or application startup process. For Kubernetes you would usually use do this using an init-container, sidecar, or both.For Vault here are some resources on how to get this working for Kubernetes:[https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar/](https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar/)[https://banzaicloud.com/blog/inject-secrets-into-pods-vault-revisited/](https://banzaicloud.com/blog/inject-secrets-into-pods-vault-revisited/)[https://itnext.io/dynamic-vault-secrets-agent-sidecar-on-kubernetes-cc0ce3e54a94](https://itnext.io/dynamic-vault-secrets-agent-sidecar-on-kubernetes-cc0ce3e54a94)For AWS Secrets Manager and Vault see the following:[https://www.godaddy.com/engineering/2019/04/16/kubernetes-external-secrets/](https://www.godaddy.com/engineering/2019/04/16/kubernetes-external-secrets/)[https://github.com/godaddy/kubernetes-external-secrets](https://github.com/godaddy/kubernetes-external-secrets) + diff --git a/kb/armory/General/stuck-pipeline---orca-with-mysql-editing-the-database.md b/kb/armory/General/stuck-pipeline---orca-with-mysql-editing-the-database.md new file mode 100644 index 0000000000..5790051e77 --- /dev/null +++ b/kb/armory/General/stuck-pipeline---orca-with-mysql-editing-the-database.md @@ -0,0 +1,10 @@ +--- +title: Stuck Pipeline - Orca with MySQL Editing the Database +--- + +## Issue +After going through and attempting other [resolutions for stuck pipelines](https://kb.armory.io/s/article/Cancel-a-stuck-pipeline), the pipeline may still remain and still continue to exist. Therefore, it is then necessary to take a look at the SQL backend and touch the database to remove the stuck execution + +## Cause +The execution is in such a state that executions from outside of Orca (e.g. the Swagger UI fix) cannot resolve the state of the pipeline. As a result, direct intervention on the MySQL Database backend for Orca is necessary. + diff --git a/kb/armory/General/support-portal-user-registration.md b/kb/armory/General/support-portal-user-registration.md new file mode 100644 index 0000000000..24d50eaf46 --- /dev/null +++ b/kb/armory/General/support-portal-user-registration.md @@ -0,0 +1,19 @@ +--- +title: Support Portal User Registration +--- + + +In order to submit cases to Armory, review cases for their team or view customer-only Knowledge Base Articles, users will need to register their information on our ServiceNow Portal.  +Users can reach out to their designated **Service Portal ****Customer Administrator** within their company and have them create an account to complete the registration.  +#### What is a Service Portal Customer Administrator? +Customer Administrators are privileged customer users who have been granted additional access and management rights to the ServiceNow Portal. They have been designated by our customer's organizations as the "managers" of the account who should have elevated privileges +For more information about Customer Administrators, their abilities, and registration, please read the following article: +[https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010240](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010240) +#### Why is user registration set up with Customer Administrators? +Armory has created an "administration level" account system for our customers to allow them to have the ability to have stronger security and the ability to audit the memberships of their accounts, yet still provide enough flexibility to monitor and self service their issues.  By providing this function to certain elevated privilege users, our customers can match their internal access and audit requirements.  Customers can review the access of their users at any time. +#### What security issues does this help prevent? +For Armory Managed Customers, providing the ability to audit their user lists helps ensure that only authorized personnel are allowed to request changes to their managed environment.  +Similarly, for general support cases, Armory Support will only be providing information about a customer environment and providing assistance to modify an environment to authorized personnel. +#### What if there are many additional users that need to be added to the Service Portal? +If customers have a massive list of changes, Armory is still available to assist in creating those accounts. In this situation, we request that a list including first and last names and email addresses be provided by your Service Portal Customer Administrator to your Technical Account Manager, or Customer Success Manager. + diff --git a/kb/armory/General/swaggerui-commands-returns-4xx-errors.md b/kb/armory/General/swaggerui-commands-returns-4xx-errors.md new file mode 100644 index 0000000000..fce586ebb6 --- /dev/null +++ b/kb/armory/General/swaggerui-commands-returns-4xx-errors.md @@ -0,0 +1,10 @@ +--- +title: SwaggerUI commands returns 4xx errors +--- + +## Issue +Users may be able to authenticate and reach the Spinnaker SwaggerUI on gate, but are unable to execute most commands.  Commands return a 403/404/405 error and are unable to be successful + +## Cause +The access to Swagger may have been authenticated, but the command executed was not executed as a secure command, and is not passing the session through. + diff --git a/kb/armory/General/system-recommendations-for-kubernetes-deployments.md b/kb/armory/General/system-recommendations-for-kubernetes-deployments.md new file mode 100644 index 0000000000..37557d3a8b --- /dev/null +++ b/kb/armory/General/system-recommendations-for-kubernetes-deployments.md @@ -0,0 +1,194 @@ +--- +title: System recommendations for Kubernetes Deployments +--- + + +As the number of applications and the Kubernetes accounts scale, the resource requirements for the Spinnaker services would change.  Below is the table containing the resource requirements for base deployment of Spinnaker targets organizations with  +* 50 applications with 50 Kubernetes accounts* 250 deployments per day over 5 hour window* 30 req/s coming from browser sessions or tools* 10x burst for both pipelines and API calls. +In addition, these are the considerations taken into account +* Pipelines used to evaluate Spinnaker are simple and made of a Deploy and 2 Wait stages for stage scheduling. If the pipelines are expected to be complex, divide the supported executions by the number of non-trivial expected stages (baking, deploying) in the pipelines.* API requests simulate potential tool requests as well as user activity. * All services run with at least 2 replicas for basic availability. It is possible to run with fewer replicas but would cause potential outages. + + +**Service** + +**Replicas** + +**CPU request** + +**CPU limit** + +**Memory request** + +**Memory limits** + +Clouddriver + +2 + +2000m + +3000m + +2.0Gi + +2.5Gi + +Deck + +2 + +150m + +300m + +32Mi + +64Mi + +Dinghy + +2 + +500m + +1000m + +0.5Gi + +1.0Gi + +Echo + +2 + +500m + +1000m + +1.0Gi + +1.5Gi + +Fiat + +2 + +500m + +1000m + +0.5Gi + +1.0Gi + +Front50 + +2 + +500m + +1000m + +1.0Gi + +1.5Gi + +Gate + +2 + +750m + +1000m + +1.0Gi + +1.5Gi + +Kayenta + +2 + +500m + +1000m + +0.5Gi + +1.0Gi + +Igor + +2 + +500m + +1000m + +0.5Gi + +1.0Gi + +Orca + +2 + +1000m + +1500m + +1.0Gi + +1.5Gi + +Rosco + +2 + +500m + +1000m + +0.5Gi + +1.0Gi + +Terraformer + +2 + +500m + +1000m + +0.5Gi + +1.0Gi + +Redis + +1 + +500m + +1000m + +0.5Gi + +1.0Gi + +Total + +16300m + +28600m + +18.56Gi + +31.125Gi + + + +Please note that these are not meant to be a checklist and would require tuning the resources as considerations mentioned above may vary.  It is recommended that customers look to measure their requirements by observing environmental behaviors and increasing resources incrementally when scaling up usage of the environment.   + + diff --git a/kb/armory/General/task-definition-from-a-github-artifact-does-not-replace-task-iam-role.md b/kb/armory/General/task-definition-from-a-github-artifact-does-not-replace-task-iam-role.md new file mode 100644 index 0000000000..9c709802a9 --- /dev/null +++ b/kb/armory/General/task-definition-from-a-github-artifact-does-not-replace-task-iam-role.md @@ -0,0 +1,23 @@ +--- +title: Task definition from a Github artifact does not replace task IAM role +--- + +## Issue +An organization deploying an application onto ECS using a task definition from a Github artifact may run into the following issue, depending on their corporate role policies.   +For security reasons, administrators may wish to limit IAM role access to maintain least-privilege access.  As a result, if users are launching an application using the same base container and definition, and use deployment tags to influence where the deployment will land, the different applications may require different IAM roles for access to the different deployment targets. +As an example, Spinnaker would discover the correct images in ECR using ```Find image from tags``` stages. However, it cannot replace the ```taskRoleArn``` within the artifact with a newly defined IAM role in Spinnaker. +  +As an example, the following ```taskRoleARN``` was set within the artifact itself.   +```"taskRoleArn": "arn:aws:iam::############:role/production/[ORG]/[ROLE]".``` +After ingesting the artifact into Spinnaker, the stage within Spinnaker then attempts to set an ```iamRole``` value.  The expectation is that the ```taskRoleArn``` defined in the artifact would be replaced, thereby allowing the image to be deployed with a differing role depending on the application. +```"iamRole": "[ROLE]". ``` +However, Spinnaker does not end up using the newly defined role, and instead continues to use the predefined ```taskRoleArn```. +  +  + +## Cause +Clouddriver currently cannot modify IAM Roles when deploying from an artifact. The specific source code can be found here.  +[https://github.com/spinnaker/clouddriver/blob/10bba2d2e18506c3eebfd2e466cf0ee1d8a4c71e/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/CreateServerGroupAtomicOperation.java#L421-L441](https://github.com/spinnaker/clouddriver/blob/10bba2d2e18506c3eebfd2e466cf0ee1d8a4c71e/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/CreateServerGroupAtomicOperation.java#L421-L441) +This is the code for creating a task definition from an artifact, whereas the following code does the expected replacement when creating a task from the wizard. +[https://github.com/spinnaker/clouddriver/blob/10bba2d2e18506c3eebfd2e466cf0ee1d8a4c71e/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/CreateServerGroupAtomicOperation.java#L307](https://github.com/spinnaker/clouddriver/blob/10bba2d2e18506c3eebfd2e466cf0ee1d8a4c71e/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/CreateServerGroupAtomicOperation.java#L307) + diff --git a/kb/armory/General/technical-support-case-statuses.md b/kb/armory/General/technical-support-case-statuses.md new file mode 100644 index 0000000000..a41effda18 --- /dev/null +++ b/kb/armory/General/technical-support-case-statuses.md @@ -0,0 +1,10 @@ +--- +title: Technical Support Case Statuses +--- + + +### Overview: +During the Case Management workflow, there are several statuses that the Case will go through prior to closure. +### Status Definitions: +**New** - When a Case is first submitted, regardless of who and where it was submitted from, it will be in the New status. This indicates that the incoming Case submission will need to be acknowledged, and the First Response Milestone will start counting down from this point onwards, until the first customer-facing response is made.**Work in Progress - **When the first customer-facing update is made, the Case status will transition to Work in Progress. A Case in the Work in Progress status indicates that there is investigation or action pending on Armory's end.**Pending Customer - **A Case in the Pending Customer status indicates that we are awaiting additional information or response from the Customer. **Customer Replied - **When a customer responds back to a Case, the status will transition to the Customer Replied status. This is an indicator for Armory Support that we need to acknowledge the customer reply and prepare to take further action on our end. **Solution Proposed - **When Armory believes that a solution or permanent workaround has been identified for the issue at hand, we will propose a solution, where it is up to the customer whether the solution provided is something that can be accepted or rejected. There will be an Accept Solution and Reject Solution button, where Accept Solution will close the Case, and Reject Solution will notify the Case Owner, and set the status back to Work in Progress. **Blocked -** The case will enter the Blocked status when we determine that the Case is blocked by a Bug, or any other kind of Blocker where we cannot progress the Case anymore. If a customer responds to a Case in the 'Blocked' status, it will go back to 'Customer Replied' and show up on our radar, so we can ensure that we respond to the question.**Closed - **The Case will enter the Closed status when the customer believes the issue has been resolved, or if Armory believes the Case is inconclusive and there is mutual agreement to bring the Case to closure. A Closed case cannot be reopened, BUT a Repeat Case can be opened from a Closed case. + diff --git a/kb/armory/General/technical-support-engineers---creating-internal-cases-to-track-project-work.md b/kb/armory/General/technical-support-engineers---creating-internal-cases-to-track-project-work.md new file mode 100644 index 0000000000..0cf9eebb58 --- /dev/null +++ b/kb/armory/General/technical-support-engineers---creating-internal-cases-to-track-project-work.md @@ -0,0 +1,40 @@ +--- +title: Technical Support Engineers - Creating Internal Cases to Track Project Work +--- + + +To properly ascertain and track internal project work, Armory Technical Support Engineers are encouraged to open cases to register their work with pre-sales customers and any internal or cross-departmental collaboration projects that may occur. +  +### Work Examples +* OSS Ninja/OSS consultation work in the OSS Slack ChannelSOW collaboration work +* Can include when working on Customer Out of Scope issues* Mini Projects for Customers* Assistance on other team projects such as Architectural reviews +Internal Projects +* ServiceNow Development* Departmental improvements such as developing new methods to spin up clusters* Developing new methods to clean up our cluster +Cross-Departmental +* Improving testing structure for Release Team* BOB restructuring with Engineering and Product + +### What is the goal +* Allow Managers to see and track project involvement and time spent.* Allow TSEs to see what projects they are working on and progress on projects and deadlines from a single location.* A low-cost method to track and close projects once completed.* It must NOT be a burden on TSEs to update. Ideal it to show their amount of "busyness." +### Using Templates/Turning on the Templates bar +We have created some templates to help facilitate a faster case creation. To access them +* Click on the Ellipses at the top bar* Select Toggle Template bar* Templates are listed at the bottom of the Browser. +###   +### How to Open and Create a Case +#### For OSS Work +This process tracks our engagements within OSS Slack for PreSales, where we are providing "one free ticket" for customers for support and consultation. +* Open a new Case in Service now. Click on [the link](https://support.armory.io/sn_customerservice_case.do?sys_id=-1&sysparm_query=active=true&sysparm_stack=sn_customerservice_case_list.do?sysparm_query=active=true&sysparm_view=case), or search for Cases - Create New in the Navigation BarFill in the case with details, or use the **Template** to help fill in some of the categories ahead of time +* Account: **Armory - OSS Ninja/OSS Channel Presales*** Contact: **OSS Query*** Assignment Group: **Armory - Technical Support*** Assigned to: **Your Name*** Customer Availability: **Hours in your working time*** Feature: Spinnaker Feature involved* Service: Spinnaker Service Involved* Armory Version: Fill in the version information we are consulting for* All other items can be left as default. Customers **should not** be added to the watch list* Short Description: - (e.g. DirecTV - Problems with Webhook)* Description: Full issue +**To use Templates to fill in** +* Open up the Template Bar* Select** OSS Ninja/Assistance Template*** Fill in the rest of the data that needs to be filled in as advised above* Save/submit the new case + +#### For Internal Project Work +This process is for issues we are working on either internally in Support or for cross-departmental topics including, but not exclusive to: +* Work on SOWs / Architecture reviews* New process/new technology documentation and discovery* Improving internal functions (e.g., AWS cost reduction, enhancing PagerDuty, etc.) +* Open a new case in Service Now. Click on [the link](https://support.armory.io/sn_customerservice_case.do?sys_id=-1&sysparm_query=active=true&sysparm_stack=sn_customerservice_case_list.do?sysparm_query=active=true&sysparm_view=case), or search for Cases - Create New in the Navigation BarFill in the case with details, or use the **Template** to help fill in some of the categories ahead of time +* Account: **Armory - Technical Support Projects*** Contact: **TSE Internal Projects*** Assignment Group: **Armory - Technical Support*** Assigned to: **Your Name*** Customer Availability: **Hours in your working time*** Feature: Spinnaker Feature involved* Service: Spinnaker Service Involved* Armory Version: Fill in the version information* All other items can be left as default. Customers **should not** be added to the watch list* Short Description: - (e.g. DirecTV - Problems with Webhook)* Description: Full issue +**To use Templates to fill in** +* Open up the Template Bar* Select **TSE - Internal Projects*** Fill in the rest of the data that needs to be filled in as advised above* Save/submit the new case + +### What to record +* It is suggested to track significant interactions, but since these are internal notes in case someone else needs to pick back up the exchange, they do not have to be "customer facing"* Please note that new issues ***Absolutely*** must be in new cases. Part of the effort is to track Support Efforts for these situations, and have a correct record of projects or OSS PreSales customer interactions* Recommended to link to any important or relevant documentation + diff --git a/kb/armory/General/terraform-'invalid-character'-error.md b/kb/armory/General/terraform-'invalid-character'-error.md new file mode 100644 index 0000000000..d5100bcf77 --- /dev/null +++ b/kb/armory/General/terraform-'invalid-character'-error.md @@ -0,0 +1,20 @@ +--- +title: Terraform 'Invalid character' Error +--- + +## Issue +When executing a Terraform Pipeline, the following error can occur upon execution.  This error can be found in the Deck UI when investigating the failure, in the ```Source``` portion of the ```Execution Details``` +Example error: +Error: Invalid character +on ses.tf line 57, in resource "aws_iam_role_policy_attachment" "eks_workers_send_email": 57: role = data.terraform_remote_state.eks.outputs.eks_cluster.worker_iam_role.name +This character is not used within the language. + +Error: Missing newline after argument +on ses.tf line 57, in resource "aws_iam_role_policy_attachment" "eks_workers_send_email": 57: role = data.terraform_remote_state.eks.outputs.eks_cluster.worker_iam_role.name +An argument definition must end with a newline. + +## Cause +2 potential causes: +* This can be due to a ```string limit error``` (e.g. it tries to load it from ```http```, store in a string, write to file, and hits a limit, for example, a character limit)* Bug where multiple references to one resource are not working correctly +**Note: **Issue is being investigated by our Engineering team.  If you are also encountering the same issue, please open a ticket so that we can associate it with the same open investigation + diff --git a/kb/armory/General/terraform-apply-stage-doesn't-output-the-full-list-of-changes.md b/kb/armory/General/terraform-apply-stage-doesn't-output-the-full-list-of-changes.md new file mode 100644 index 0000000000..fb1bad0ac3 --- /dev/null +++ b/kb/armory/General/terraform-apply-stage-doesn't-output-the-full-list-of-changes.md @@ -0,0 +1,12 @@ +--- +title: Terraform Apply Stage Doesn't Output the Full List of Changes +--- + +## Issue +Terraform Apply Stage only provides a summary of information when running the stage.  When clicking on the console output, a full list of the changes made is not provided, and only a summary of the apply is listed + +## Cause +Since the release of Terraform v0.12, this output is actually expected behavior. The Terraform Apply Stage is an application of the command +```terraform apply -auto-approve``` +When running this stage, the output matches running the same command on your computer.  To check, run the above on your computer, and the following will be the output, matching what you will see in the Armory Spinnaker Console by clicking on the Pipeline, and checking the Console Output for the stage. + diff --git a/kb/armory/General/terraform-deployment-using-armory-spinnaker.md b/kb/armory/General/terraform-deployment-using-armory-spinnaker.md new file mode 100644 index 0000000000..180a0b7710 --- /dev/null +++ b/kb/armory/General/terraform-deployment-using-armory-spinnaker.md @@ -0,0 +1,15 @@ +--- +title: Terraform Deployment Using Armory Spinnaker +--- + +## Introduction +Looking for a quick and easy way to deploy Terraform using Armory Spinnaker? The following video walks you through a real-life example of deployments from Plan to Apply to Destroy, with a manual judgement on the plan stage thrown in for good measure. + +## Prerequisites +N/A + +## Instructions +Lee Faus walks you through a real-world example of how you can use the Armory Provisioning with Terraform plugin with Spinnaker to deploy your application and dependent infrastructure resources in real-time. + + + diff --git a/kb/armory/General/terraform-maximum-file-size-error.md b/kb/armory/General/terraform-maximum-file-size-error.md new file mode 100644 index 0000000000..07b056afa5 --- /dev/null +++ b/kb/armory/General/terraform-maximum-file-size-error.md @@ -0,0 +1,10 @@ +--- +title: Terraform Maximum File Size Error +--- + +## Issue +If an organization uses Terraform to a very large degree, it's possible to run into a bug where Terraform can actually reach a limit per file where it will no longer read instructions  The error can be a red herring for an invalid character issue. Due to caching, it’s actually quite likely that trying to run this TF file locally may actually work, but when trying to run it on any cloud services or pulling it as an artifact from Git will fail. Users may have tried to run ```od``` on the file to inspect it for strange characters at which point users will see errors on a specific line at the end of the file that doesn’t seem to go away even if manually changed. + +## Cause +Terraform has limits on sizes of files of **32KB or larger** will not be read correctly. This error is very new and it is unsure if it's a built in limitation that can be 'fixed' or if it's a hardcoded problem that will need better workarounds. More investigation is underway. + diff --git a/kb/armory/General/terraform-named-profiles-are-unable-to-use-environment-variables-to-store-secrets.md b/kb/armory/General/terraform-named-profiles-are-unable-to-use-environment-variables-to-store-secrets.md new file mode 100644 index 0000000000..91f816b350 --- /dev/null +++ b/kb/armory/General/terraform-named-profiles-are-unable-to-use-environment-variables-to-store-secrets.md @@ -0,0 +1,10 @@ +--- +title: Terraform Named Profiles are Unable to use Environment Variables to Store Secrets +--- + +## Issue +Attempting to use Environmental Variables to store secrets (e.g. ```TF_VAR_```) with Named Profiles creates an issue where the credentials cannot be processed. An error is returned that ```No Valid Credentials Found``` is provided as an error message + +## Cause +Named Profiles in Terraformer does not currently support Environment Variables + diff --git a/kb/armory/General/terraformer-does-not-clone-properly-from-bitbucket.md b/kb/armory/General/terraformer-does-not-clone-properly-from-bitbucket.md new file mode 100644 index 0000000000..c5d581da85 --- /dev/null +++ b/kb/armory/General/terraformer-does-not-clone-properly-from-bitbucket.md @@ -0,0 +1,10 @@ +--- +title: Terraformer does not clone properly from Bitbucket +--- + +## Issue +Cloning files to S3 buckets using the Terraform Stage(s) results in unexpected behavior, such as corrupted files in S3 upon being cloned. + +## Cause +This is due to the previous iteration of source code regarding how files were cloned in Terraform.  + diff --git "a/kb/armory/General/terraformer-error-exec--\"terraform\"--executable-file-not-found-in-$path.md" "b/kb/armory/General/terraformer-error-exec--\"terraform\"--executable-file-not-found-in-$path.md" new file mode 100644 index 0000000000..0e29a2cd7a --- /dev/null +++ "b/kb/armory/General/terraformer-error-exec--\"terraform\"--executable-file-not-found-in-$path.md" @@ -0,0 +1,11 @@ +--- +title: Terraformer Error exec- "terraform"- executable file not found in $PATH +--- + +## Issue +When running Terraformer stage, the following error appears +```exec: “terraform”: executable file not found in $PATH``` + +## Cause +A version of terraform was not selected in the stage, or was left as ```SYSTEM_DEFINED``` + diff --git a/kb/armory/General/terraformer-named-profiles-with-gcp.md b/kb/armory/General/terraformer-named-profiles-with-gcp.md new file mode 100644 index 0000000000..3e517ae17b --- /dev/null +++ b/kb/armory/General/terraformer-named-profiles-with-gcp.md @@ -0,0 +1,42 @@ +--- +title: Terraformer Named Profiles with GCP +--- + +## Introduction +Armory has implemented the Named Profiles concepts within Spinnaker +[https://docs.armory.io/armory-enterprise/armory-admin/terraform-enable-integration/#named-profiles](https://docs.armory.io/armory-enterprise/armory-admin/terraform-enable-integration/#named-profiles) +However, customers may also be hoping to use Named Profiles with Google Cloud Platform.  Armory's Terraformer Named Profiles solution can work with GCP credentials as well. + +## Prerequisites +Please ensure Terraformer is enabled +[https://docs.armory.io/armory-enterprise/armory-admin/terraform-enable-integration/](https://docs.armory.io/armory-enterprise/armory-admin/terraform-enable-integration/) + +## Instructions +#### Attaining the JSON Key File +In order to define credentials, users or admins should have a ```.json``` file with the credentials in a pre-defined location. Their credentials can be attained by following the first section the ```adding credentials``` section in Terraform's knowledge base: [https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/getting_started#adding-credentials](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/getting_started#adding-credentials) +Customers will need to download the JSON key file and store it.  They can do so either in a Credentials Manager or in Kubernetes Secrets.   +Once the file is in the appropriate place, it can be referenced by defining the secrets as a token [https://docs.armory.io/armory-enterprise/armory-admin/secrets/](https://docs.armory.io/armory-enterprise/armory-admin/secrets/) +#### In Halyard +Customers using Halyard should look to make the following adjustments in the ```.hal/default/profiles``` directory by creating or editing their ```terraformer-local.yml``` file.  +Please ensure the kind ```name``` is ```GOOGLE_CREDENTIALS``` so that the information can be interpreted properly.  The ```value``` should include the secrets location as defined thorough [https://docs.armory.io/armory-enterprise/armory-admin/secrets/](https://docs.armory.io/armory-enterprise/armory-admin/secrets/) (For example, ```encrypted:k8s!n:spin-secrets!k:GCPTestCredentials.json```) +- name: gcp-credentials + variables: + - kind: static + options: + name: GOOGLE_CREDENTIALS + value: +#### In Operator +Customers using Operator should place the following information under ```spec.spinnakerConfig.profiles.terraformer``` +Please ensure the kind ```name``` is ```GOOGLE_CREDENTIALS``` so that the information can be interpreted properly.  The ```value``` should include the secrets location as defined thorough [https://docs.armory.io/armory-enterprise/armory-admin/secrets/](https://docs.armory.io/armory-enterprise/armory-admin/secrets/) (For example, ```encrypted:k8s!n:spin-secrets!k:GCPTestCredentials.json```) +spec: + spinnakerConfig: + profiles: + terraformer: + - name: gcp-credentials + variables: + - kind: static + options: + name: GOOGLE_CREDENTIALS + value: +####   + diff --git a/kb/armory/General/terraformer-only-takes-input-from-a-single-file-from-github.md b/kb/armory/General/terraformer-only-takes-input-from-a-single-file-from-github.md new file mode 100644 index 0000000000..2a5c0ebbaf --- /dev/null +++ b/kb/armory/General/terraformer-only-takes-input-from-a-single-file-from-github.md @@ -0,0 +1,10 @@ +--- +title: Terraformer Only Takes Input from a Single File from Github +--- + +## Issue +To compile Terraform code using Terraformer, it is usually necessary to pull in multiple files together as a part of the compilation.  However, when executing the Plan or Apply stage Terraformer is only executing on a single file and is not looking at the repo as a whole. + +## Cause +The Artifact that was set up was done using GitHub instead of using the GitRepo selection.  The environment will need to be configured with GitRepo settings before continuing.[Artifact Configuration for GitRepo in Operator](https://docs.armory.io/docs/installation/operator-reference/artifact/#gitrepo)[Artifact Configuration for GitRepo in Halyard](https://spinnaker.io/reference/artifacts-with-artifactsrewrite/types/git-repo/) + diff --git a/kb/armory/General/terraformer-stages-stop-running-jobs-with-a-'timeout'-error-logging-enabled.md b/kb/armory/General/terraformer-stages-stop-running-jobs-with-a-'timeout'-error-logging-enabled.md new file mode 100644 index 0000000000..c13d8edfd6 --- /dev/null +++ b/kb/armory/General/terraformer-stages-stop-running-jobs-with-a-'timeout'-error-logging-enabled.md @@ -0,0 +1,27 @@ +--- +title: Terraformer Stages Stop Running Jobs with a 'timeout' error (Logging Enabled) +--- + +## Issue +Users may find that their Terraformer Stages show an error with an ```Exception (Monitor Run Terraform) timeout``` in the Spinnaker Console while having Terraform Logging Enabled + + +Customers also have ```spec.spinnakerConfig.profiles.terraformer.logging.remote.enabled``` value set to ```true``` in their Spinnaker Configuration.   +spec: + spinnakerConfig: + profiles: + terraformer: + logging: + remote: + enabled: true + +or set to ```true``` in Halyard in the hal config profiles directory e.g. (```~/.hal/default/profiles/```) in the ```terraform-local.yml``` file +logging: + remote: + enabled: true + +## Cause +The logging and metrics for Terraformer is enabled and set to an endpoint which is unreachable. +[https://docs.armory.io/docs/armory-admin/terraform-enable-integration/#logging-and-metrics](https://docs.armory.io/docs/armory-admin/terraform-enable-integration/#logging-and-metrics) + + diff --git a/kb/armory/General/terraformerversion-doesn't-render-in-pipeline-json-by-default.md b/kb/armory/General/terraformerversion-doesn't-render-in-pipeline-json-by-default.md new file mode 100644 index 0000000000..e091ff8176 --- /dev/null +++ b/kb/armory/General/terraformerversion-doesn't-render-in-pipeline-json-by-default.md @@ -0,0 +1,11 @@ +--- +title: terraformer Version doesn't render in pipeline JSON by default +--- + +## Issue +When creating a terraform pipeline, the 'terraformerVersion' doesn't render in pipeline JSON by default. Users have to manually select the dropdown and choose the version even though the default is picked. + + +## Cause +A bug in Deck causes this issue, and has since been resolved. + diff --git "a/kb/armory/General/testing-x509-certificates-subject-filtering-with-agent\342\200\250.md" "b/kb/armory/General/testing-x509-certificates-subject-filtering-with-agent\342\200\250.md" new file mode 100644 index 0000000000..3784b2becb --- /dev/null +++ "b/kb/armory/General/testing-x509-certificates-subject-filtering-with-agent\342\200\250.md" @@ -0,0 +1,10 @@ +--- +title: Testing X509 Certificates Subject Filtering with Agent
 +--- + +## Issue +Organizations using Armory Agent and X509 Certificates may run into an issue where it appears that subject filtering isn’t working.[https://docs.armory.io/docs/armory-agent/agent-mtls/](https://docs.armory.io/docs/armory-agent/agent-mtls/) This will typically come up during a testing phase.  + +## Cause +This is a red herring - This is because there are ***grpc endpoints*** that don’t go through the application layer security filtering, rather just informational that has no security risk. + diff --git a/kb/armory/General/the-role-of-helm-and-spinnaker.md b/kb/armory/General/the-role-of-helm-and-spinnaker.md new file mode 100644 index 0000000000..03b72e0783 --- /dev/null +++ b/kb/armory/General/the-role-of-helm-and-spinnaker.md @@ -0,0 +1,33 @@ +--- +title: The Role of HELM and Spinnaker +--- + + +Question: +*This is a Q&A pulled from the Spinnaker Community Slack team, which [we encourage you to join here](http://join.spinnaker.io/).* +Thank you for helping me understand the roles of Helm and Spinnaker. +As you might have guessed, I am very new to Helm. Currently, we are running our services in ```Docker Swarm``` and we’re doing a POC on ```Kubernetes + Spinnaker + Helm```. Each of our service GitHub repo has a ```Dockerfile``` which has ```CMD```s and ```RUN```s tasks, how can I translate those in ```HELM Charts``` ? +Example: +FROM nginx + +COPY build /usr/share/nginx/html +COPY entrypoint.sh / +COPY nginx-default.conf /etc/nginx/conf.d/default.conf + +RUN apt-get update && apt-get install -y curl + +HEALTHCHECK --interval=30s --timeout=5s \ + CMD curl -fkL http://localhost:8080/health || exit 1 + +RUN chmod +x /entrypoint.sh +ENTRYPOINT ["/entrypoint.sh"] +CMD ["nginx", "-g", "daemon off;"] +EXPOSE 80 +EXPOSE 443 +How and where can I write this in HELM Charts? + +Answer: +You may be getting the role of the Dockerfile and the Helm chart confused. Your Dockerfile is supposed to define the way you application runs and the environment required for it to run. When built, the Dockerfile produces a Docker image which is then run on Kubernetes. A Helm chart defines all of the Kubernetes resources needed to run your application. For example, a ```Deployment``` is used to take your Docker image, run it (w/ multiple replicas) and make sure it stays available. You would then use a Helm chart to template this ```Deployment``` so that it makes it easier to deploy the same thing across multiple environments. +We made a video explaining how it all fits together + + diff --git a/kb/armory/General/the-timeout-of-bake-stage-does-not-respect-the-timeout-in-bake-stage-configuration.md b/kb/armory/General/the-timeout-of-bake-stage-does-not-respect-the-timeout-in-bake-stage-configuration.md new file mode 100644 index 0000000000..ac4f7a4bdd --- /dev/null +++ b/kb/armory/General/the-timeout-of-bake-stage-does-not-respect-the-timeout-in-bake-stage-configuration.md @@ -0,0 +1,14 @@ +--- +title: The timeout of Bake Stage does not respect the timeout in Bake Stage configuration +--- + +## Issue +Users may notice that although the configuration of ```Fail Stage after a specific amount of time``` for a bake stage in Spinnaker is set to 4.5 hours:The bake stage does not respect the set timeout limit.  It would instead expire before that limit (in this example, 4 hours), although the baking process continues in the environment. The UI would show the following: + + + +## Cause +The option ```Fail stage after a specific amount of time``` forces the stage to fail if its running time exceeds a specific length. +By default, Spinnaker can use sensible timeouts dependent on the stage type and the operations the stage needs to perform at runtime. These defaults can vary based on chosen configuration. +As such, the timeout of bake stage not matching the timeout in the configuration in Spinnaker UI is expected behaviour, as the timeout of the stage may exceed the set maximum time in the Spinnaker Manifest.   + diff --git a/kb/armory/General/troubleshoot-baking-helm-3-charts.md b/kb/armory/General/troubleshoot-baking-helm-3-charts.md new file mode 100644 index 0000000000..82f4b5d082 --- /dev/null +++ b/kb/armory/General/troubleshoot-baking-helm-3-charts.md @@ -0,0 +1,11 @@ +--- +title: Troubleshoot Baking Helm 3 Charts +--- + +## Issue +After enabling Helm in Spinnaker, Helm repository was also added. The Helm account is visible, but unable to see any chart as it is stuck with a busy/loading status as per the following screenshot: + + +## Cause +Spinnaker recognizing the Helm URL format in a specific format. + diff --git a/kb/armory/General/troubleshooting-gate-deck-cors-issues.md b/kb/armory/General/troubleshooting-gate-deck-cors-issues.md new file mode 100644 index 0000000000..bfd2caecc4 --- /dev/null +++ b/kb/armory/General/troubleshooting-gate-deck-cors-issues.md @@ -0,0 +1,27 @@ +--- +title: Troubleshooting Gate/Deck CORS Issues +--- + +## Introduction +If you’re trying to hook up Spinnaker Gate and Deck and using different hostnames/domains for Gate and Deck, and you’re experiencing CORS issues (where Gate is refusing connections from Deck), you may need to modify Gate to support CORS from your Deck FQDN. + + +## Prerequisites +N/A + +## Instructions +For example, if you’re using ```[gate.domain.com](http://gate.domain.com/)``` for your Gate, and ```[spinnaker.domain.com](http://spinnaker.domain.com/)``` for your Deck, then go into your halconfig, ```.hal/config```, into the relevant security section of your Halyard deployment configuration, and add the ```corsAccessPattern``` field, like this: +deploymentConfigurations: +- name: default +... ... + security: + apiSecurity: + ssl: + enabled: false + overrideBaseUrl: https://gate.domain.com + corsAccessPattern: https://spinnaker.domain.com + uiSecurity: + ssl: + enabled: false + overrideBaseUrl: https://spinnaker.domain.com + diff --git "a/kb/armory/General/trying-to-turn-on-vault-results-in-a-\"missing-client-token-error.md" "b/kb/armory/General/trying-to-turn-on-vault-results-in-a-\"missing-client-token-error.md" new file mode 100644 index 0000000000..665c17499a --- /dev/null +++ "b/kb/armory/General/trying-to-turn-on-vault-results-in-a-\"missing-client-token-error.md" @@ -0,0 +1,15 @@ +--- +title: Trying to turn on Vault results in a Missing Client Token Error +--- + +## Issue +When trying to enable Vault on Spinnaker, it results in this error +SpinnakerService validation failed: +error fetching vault token - error logging into vault using kubernetes auth: Error making API request. +```URL: PUT "https://vault-prod.[CLIENTURL].com/v1/auth/eks/armory-spinnaker-prod/login''' +Code: 500. Errors: +* namespace not authorized + +## Cause +Operator and go-yaml-tools with namespace support do not currently support dedicated secrets containers using Vault Enterprise. + diff --git a/kb/armory/General/unable-to-add-ecs-cluster-to-spinnaker-not-authorized.md b/kb/armory/General/unable-to-add-ecs-cluster-to-spinnaker-not-authorized.md new file mode 100644 index 0000000000..fe1beab837 --- /dev/null +++ b/kb/armory/General/unable-to-add-ecs-cluster-to-spinnaker-not-authorized.md @@ -0,0 +1,174 @@ +--- +title: Unable to add ECS cluster to Spinnaker (not authorized) +--- + +## Issue +When using a cross account AWS role to access an ECS cluster, the user is getting the following issue:Looks like this is an RBAC related error:```You are not authorized to perform this operation```The configuration changes were successfully applied but in CloudDriver but the user is still seeing this error. +clouddriver 2020-08-27 22:43:23.719 ERROR 1 --- [ main o.s.boot.SpringApplication : Application run failed +clouddriver org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'basicAmazonDeployDescription': Unsatisfied dependency expressed through field 'regionScopedProviderFactory'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'regionScopedProviderFactory': Unsatisfied dependency expressed through field 'clusterProviders'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'amazonClusterProvider' defined in URL [jar:file:/opt/clouddriver/lib/clouddriver-aws-GCSFIX.jar!/com/netflix/spinnaker/clouddriver/aws/provider/view/AmazonClusterProvider.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cacheView' defined in class path resource [com/netflix/spinnaker/clouddriver/cache/CacheConfig.class]: Unsatisfied dependency expressed through method 'cacheView' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'catsModule' defined in class path resource [com/netflix/spinnaker/config/SqlCacheConfiguration.class]: Unsatisfied dependency expressed through method 'catsModule' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'netflixAmazonCredentials' defined in class path resource [com/netflix/spinnaker/clouddriver/aws/security/AmazonCredentialsInitializer.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.util.List: Factory method 'netflixAmazonCredentials' threw exception; nested exception is com.amazonaws.services.ec2.model.AmazonEC2Exception: You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation; Request ID: f0e6c419-049b-4759-9a21-e76d6145d35a) +clouddriver at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:643) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:130) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1422) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:594) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:893) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141) ~[spring-boot-2.2.4.RELEASE.jar:2.2.4.RELEASE +clouddriver at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747) ~[spring-boot-2.2.4.RELEASE.jar:2.2.4.RELEASE +clouddriver at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) ~[spring-boot-2.2.4.RELEASE.jar:2.2.4.RELEASE +clouddriver at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-2.2.4.RELEASE.jar:2.2.4.RELEASE +clouddriver at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:140) ~[spring-boot-2.2.4.RELEASE.jar:2.2.4.RELEASE +clouddriver at org.springframework.boot.builder.SpringApplicationBuilder$run$0.call(Unknown Source) ~[na:na +clouddriver at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) ~[groovy-2.5.10.jar:2.5.10 +clouddriver at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115) ~[groovy-2.5.10.jar:2.5.10 +clouddriver at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127) ~[groovy-2.5.10.jar:2.5.10 +clouddriver at com.netflix.spinnaker.clouddriver.Main.main(Main.groovy:78) ~[clouddriver-web-GCSFIX.jar:na +clouddriver Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'regionScopedProviderFactory': Unsatisfied dependency expressed through field 'clusterProviders'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'amazonClusterProvider' defined in URL [jar:file:/opt/clouddriver/lib/clouddriver-aws-GCSFIX.jar!/com/netflix/spinnaker/clouddriver/aws/provider/view/AmazonClusterProvider.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cacheView' defined in class path resource [com/netflix/spinnaker/clouddriver/cache/CacheConfig.class]: Unsatisfied dependency expressed through method 'cacheView' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'catsModule' defined in class path resource [com/netflix/spinnaker/config/SqlCacheConfiguration.class]: Unsatisfied dependency expressed through method 'catsModule' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'netflixAmazonCredentials' defined in class path resource [com/netflix/spinnaker/clouddriver/aws/security/AmazonCredentialsInitializer.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.util.List: Factory method 'netflixAmazonCredentials' threw exception; nested exception is com.amazonaws.services.ec2.model.AmazonEC2Exception: You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation; Request ID: f0e6c419-049b-4759-9a21-e76d6145d35a) +clouddriver at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:643) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:130) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1422) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:594) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1304) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1224) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver ... 22 common frames omitted +clouddriver Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'amazonClusterProvider' defined in URL [jar:file:/opt/clouddriver/lib/clouddriver-aws-GCSFIX.jar!/com/netflix/spinnaker/clouddriver/aws/provider/view/AmazonClusterProvider.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cacheView' defined in class path resource [com/netflix/spinnaker/clouddriver/cache/CacheConfig.class]: Unsatisfied dependency expressed through method 'cacheView' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'catsModule' defined in class path resource [com/netflix/spinnaker/config/SqlCacheConfiguration.class]: Unsatisfied dependency expressed through method 'catsModule' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'netflixAmazonCredentials' defined in class path resource [com/netflix/spinnaker/clouddriver/aws/security/AmazonCredentialsInitializer.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.util.List: Factory method 'netflixAmazonCredentials' threw exception; nested exception is com.amazonaws.services.ec2.model.AmazonEC2Exception: You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation; Request ID: f0e6c419-049b-4759-9a21-e76d6145d35a) +clouddriver at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1358) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1204) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1522) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1486) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveMultipleBeans(DefaultListableBeanFactory.java:1375) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1262) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1224) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver ... 35 common frames omitted +clouddriver Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cacheView' defined in class path resource [com/netflix/spinnaker/clouddriver/cache/CacheConfig.class]: Unsatisfied dependency expressed through method 'cacheView' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'catsModule' defined in class path resource [com/netflix/spinnaker/config/SqlCacheConfiguration.class]: Unsatisfied dependency expressed through method 'catsModule' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'netflixAmazonCredentials' defined in class path resource [com/netflix/spinnaker/clouddriver/aws/security/AmazonCredentialsInitializer.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.util.List: Factory method 'netflixAmazonCredentials' threw exception; nested exception is com.amazonaws.services.ec2.model.AmazonEC2Exception: You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation; Request ID: f0e6c419-049b-4759-9a21-e76d6145d35a) +clouddriver at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:538) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1304) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1224) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver ... 51 common frames omitted +clouddriver Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'catsModule' defined in class path resource [com/netflix/spinnaker/config/SqlCacheConfiguration.class]: Unsatisfied dependency expressed through method 'catsModule' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'netflixAmazonCredentials' defined in class path resource [com/netflix/spinnaker/clouddriver/aws/security/AmazonCredentialsInitializer.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.util.List: Factory method 'netflixAmazonCredentials' threw exception; nested exception is com.amazonaws.services.ec2.model.AmazonEC2Exception: You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation; Request ID: f0e6c419-049b-4759-9a21-e76d6145d35a) +clouddriver at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:538) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1304) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1224) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver ... 65 common frames omitted +clouddriver Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'netflixAmazonCredentials' defined in class path resource [com/netflix/spinnaker/clouddriver/aws/security/AmazonCredentialsInitializer.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.util.List: Factory method 'netflixAmazonCredentials' threw exception; nested exception is com.amazonaws.services.ec2.model.AmazonEC2Exception: You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation; Request ID: f0e6c419-049b-4759-9a21-e76d6145d35a) +clouddriver at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:655) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:635) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1522) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1486) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveMultipleBeans(DefaultListableBeanFactory.java:1375) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1262) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1224) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver ... 79 common frames omitted +clouddriver Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.util.List: Factory method 'netflixAmazonCredentials' threw exception; nested exception is com.amazonaws.services.ec2.model.AmazonEC2Exception: You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation; Request ID: f0e6c419-049b-4759-9a21-e76d6145d35a) +clouddriver at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:650) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver ... 98 common frames omitted +clouddriver Caused by: com.amazonaws.services.ec2.model.AmazonEC2Exception: You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation; Request ID: f0e6c419-049b-4759-9a21-e76d6145d35a) +clouddriver at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1799) ~[aws-java-sdk-core-1.11.764.jar:na +clouddriver at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1383) ~[aws-java-sdk-core-1.11.764.jar:na +clouddriver at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1359) ~[aws-java-sdk-core-1.11.764.jar:na +clouddriver at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139) ~[aws-java-sdk-core-1.11.764.jar:na +clouddriver at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796) ~[aws-java-sdk-core-1.11.764.jar:na +clouddriver at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[aws-java-sdk-core-1.11.764.jar:na +clouddriver at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[aws-java-sdk-core-1.11.764.jar:na +clouddriver at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[aws-java-sdk-core-1.11.764.jar:na +clouddriver at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[aws-java-sdk-core-1.11.764.jar:na +clouddriver at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[aws-java-sdk-core-1.11.764.jar:na +clouddriver at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[aws-java-sdk-core-1.11.764.jar:na +clouddriver at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:25045) ~[aws-java-sdk-ec2-1.11.764.jar:na +clouddriver at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:25012) ~[aws-java-sdk-ec2-1.11.764.jar:na +clouddriver at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:25001) ~[aws-java-sdk-ec2-1.11.764.jar:na +clouddriver at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeRegions(AmazonEC2Client.java:13170) ~[aws-java-sdk-ec2-1.11.764.jar:na +clouddriver at com.amazonaws.services.ec2.AmazonEC2Client.describeRegions(AmazonEC2Client.java:13141) ~[aws-java-sdk-ec2-1.11.764.jar:na +clouddriver at com.netflix.spinnaker.clouddriver.aws.security.DefaultAWSAccountInfoLookup.listRegions(DefaultAWSAccountInfoLookup.java:116) ~[clouddriver-aws-GCSFIX.jar:na +clouddriver at com.netflix.spinnaker.clouddriver.aws.security.config.CredentialsLoader$1.get(CredentialsLoader.java:110) ~[clouddriver-aws-GCSFIX.jar:na +clouddriver at com.netflix.spinnaker.clouddriver.aws.security.config.CredentialsLoader$1.get(CredentialsLoader.java:94) ~[clouddriver-aws-GCSFIX.jar:na +clouddriver at com.netflix.spinnaker.clouddriver.aws.security.config.CredentialsLoader$Lazy.get(CredentialsLoader.java:324) ~[clouddriver-aws-GCSFIX.jar:na +clouddriver at com.netflix.spinnaker.clouddriver.aws.security.config.CredentialsLoader.initRegions(CredentialsLoader.java:145) ~[clouddriver-aws-GCSFIX.jar:na +clouddriver at com.netflix.spinnaker.clouddriver.aws.security.config.CredentialsLoader.load(CredentialsLoader.java:249) ~[clouddriver-aws-GCSFIX.jar:na +clouddriver at com.netflix.spinnaker.clouddriver.aws.security.config.CredentialsLoader$load.call(Unknown Source) ~[na:na +clouddriver at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) ~[groovy-2.5.10.jar:2.5.10 +clouddriver at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115) ~[groovy-2.5.10.jar:2.5.10 +clouddriver at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127) ~[groovy-2.5.10.jar:2.5.10 +clouddriver at com.netflix.spinnaker.clouddriver.aws.security.DefaultAmazonAccountsSynchronizer.synchronize(DefaultAmazonAccountsSynchronizer.groovy:46) ~[clouddriver-aws-GCSFIX.jar:na +clouddriver at com.netflix.spinnaker.clouddriver.aws.security.DefaultAmazonAccountsSynchronizer$synchronize.call(Unknown Source) ~[na:na +clouddriver at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) ~[groovy-2.5.10.jar:2.5.10 +clouddriver at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115) ~[groovy-2.5.10.jar:2.5.10 +clouddriver at com.netflix.spinnaker.clouddriver.aws.security.AmazonCredentialsInitializer.netflixAmazonCredentials(AmazonCredentialsInitializer.groovy:71) ~[clouddriver-aws-GCSFIX.jar:na +clouddriver at com.netflix.spinnaker.clouddriver.aws.security.AmazonCredentialsInitializer$$EnhancerBySpringCGLIB$$a647d3fe.CGLIB$netflixAmazonCredentials$5() ~[clouddriver-aws-GCSFIX.jar:na +clouddriver at com.netflix.spinnaker.clouddriver.aws.security.AmazonCredentialsInitializer$$EnhancerBySpringCGLIB$$a647d3fe$$FastClassBySpringCGLIB$$32d70a7d.invoke() ~[clouddriver-aws-GCSFIX.jar:na +clouddriver at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver at com.netflix.spinnaker.clouddriver.aws.security.AmazonCredentialsInitializer$$EnhancerBySpringCGLIB$$a647d3fe.netflixAmazonCredentials() ~[clouddriver-aws-GCSFIX.jar:na +clouddriver at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na +clouddriver at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na +clouddriver at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na +clouddriver at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na +clouddriver at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE +clouddriver ... 99 common frames omitted +clouddriver stream closed + +## Cause +Looking at the following from the Stacktrace:```com.netflix.spinnaker.clouddriver.aws.security.DefaultAWSAccountInfoLookup```The following issue is being tracked on Github:[https://github.com/spinnaker/clouddriver/blob/master/clouddriver-aws/src/main/groovy/com/netflix/spinnaker/clouddriver/aws/security/DefaultAWSAccountInfoLookup.java#L116](https://github.com/spinnaker/clouddriver/blob/master/clouddriver-aws/src/main/groovy/com/netflix/spinnaker/clouddriver/aws/security/DefaultAWSAccountInfoLookup.java#L116)When looking at the ```listRegions``` call, it's being referenced very early. The documents here outline more about the issue:[https://docs.armory.io/docs/spinnaker-install-admin-guides/add-aws-account-iam/#instance-role-part-3-creating-a-managing-account-iam-policy-in-your-primary-aws-account](https://docs.armory.io/docs/spinnaker-install-admin-guides/add-aws-account-iam/#instance-role-part-3-creating-a-managing-account-iam-policy-in-your-primary-aws-account)The above doc was used to see what roles the primary AWS account had to have.Summary:The cross account (worker node) has an 'assume role' assigned which gives the correct permissions but before it can assume the role, we saw that it tries to get the access to describe region access, and availability zones. It does not have permissions to these without the assume role. + diff --git a/kb/armory/General/unable-to-add-kubernetes-cloud-provider-with-a-secret-engine.md b/kb/armory/General/unable-to-add-kubernetes-cloud-provider-with-a-secret-engine.md new file mode 100644 index 0000000000..eea0c923eb --- /dev/null +++ b/kb/armory/General/unable-to-add-kubernetes-cloud-provider-with-a-secret-engine.md @@ -0,0 +1,16 @@ +--- +title: Unable to Add Kubernetes Cloud Provider with a Secret Engine +--- + +## Issue +When adding a Cloud Provider Account using a secret, the SpinnakerServices validator will throw an error stating that there isn't sufficient permissions to access the cluster: ```The kubeconfig does not have explicit permissions to the target cluster's resources```. +for: "STDIN": admission webhook "webhook-spinnakerservices-v1alpha2.spinnaker.armory.io" denied the request: +SpinnakerService validation failed: +Validator for account 'spinnaker' detected an error: + error listing namespaces in account "spinnaker": + Get https://$CLUSTER_URL_HERE/api/v1/namespaces: exec: exit status 255 + + +## Cause +The kubeconfig does not have explicit permissions to the target cluster's resources, and this will need to be defined. + diff --git a/kb/armory/General/unable-to-configure-permissions-on-a-spinnaker-application-with-a-postgresql-database.md b/kb/armory/General/unable-to-configure-permissions-on-a-spinnaker-application-with-a-postgresql-database.md new file mode 100644 index 0000000000..21f720d3a8 --- /dev/null +++ b/kb/armory/General/unable-to-configure-permissions-on-a-spinnaker-application-with-a-postgresql-database.md @@ -0,0 +1,13 @@ +--- +title: Unable to configure permissions on a Spinnaker application with a PostgreSQL database +--- + +## Issue +With authentication and authorization enabled, when a user selects a role for a new or existing Spinnaker application, and presses the ```Create``` or ```Update``` button - an error similar to the following comes up on the form: +```jOOQ; bad SQL grammar [insert into applications (id, body, created_at, last_modified_at, last_modified_by, is_deleted) values (?, ?, ?, ?, ?, ?) on duplicate key update body =?, last_modified_at = ?, last_modified_by = ?, is_deleted = ? --executionId: user: ; nested exception is org.postgressql/util.PSQLException: ERROR: syntax error at or near "duplicate" Position 132``` +This prevents the user from configuring application permissions. + +## Cause +Database ```dialect``` settings for either or all of the following services may not have been configured: Front50, Orca and Clouddriver.  +If not explicitly set, Spinnaker defaults to ***MySQL dialect*** syntax. + diff --git a/kb/armory/General/unable-to-deploy-after-moving-aws-account-into-profiles.md b/kb/armory/General/unable-to-deploy-after-moving-aws-account-into-profiles.md new file mode 100644 index 0000000000..605153c081 --- /dev/null +++ b/kb/armory/General/unable-to-deploy-after-moving-aws-account-into-profiles.md @@ -0,0 +1,12 @@ +--- +title: Unable to Deploy After Moving AWS Account into Profiles +--- + +## Issue +When trying to deploy after making changes to your AWS Account definitions in Halyard, Spinnaker users may run into this error: +```Exception ( Create Server Group ) credentials not found (name: default, names: [aws_account_name])``` + +## Cause +If the AWS accounts from Halyard are moved into *Profiles* or changed from using Halyard Config to using Spring Profiles for account configuration, there is a property that also has to be set in the Orca profile (```orca-profile.yml```). +The Orca default Bake profile entry will also need to be added. + diff --git a/kb/armory/General/unable-to-deploy-spinnaker-after-changing-operator-and-halyard-versions.md b/kb/armory/General/unable-to-deploy-spinnaker-after-changing-operator-and-halyard-versions.md new file mode 100644 index 0000000000..b65bdd429c --- /dev/null +++ b/kb/armory/General/unable-to-deploy-spinnaker-after-changing-operator-and-halyard-versions.md @@ -0,0 +1,34 @@ +--- +title: Unable to deploy Spinnaker after changing Operator and Halyard versions +--- + +## Issue +After upgrading operator to version 1.2.7  and halyard to version 1.12.1, it is noticed that Spinnaker deployments do not go through. Below errors may be seen on operator and halyard logs +#### Operator container logs +``` +{"level":"error","ts":1626238705.074147,"logger":"controller-runtime.controller","msg":"Reconciler +error","controller":"spinnakerservice-controller","request":"spinnaker-operator/spinnaker","error":"got halyard response status 405, +response: Request method 'POST' not supported","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/opt/spinnaker- +operator/build/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller. +(*Controller).reconcileHandler\n\t/opt/spinnaker-operator/build/vendor/sigs.k8s.io/controller- +runtime/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller. +(*Controller).processNextWorkItem\n\t/opt/spinnaker-operator/build/vendor/sigs.k8s.io/controller- +runtime/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller. +(*Controller).worker\n\t/opt/spinnaker-operator/build/vendor/sigs.k8s.io/controller- +runtime/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/opt/spinnaker- +operator/build/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/opt/spinnaker- +operator/build/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/opt/spinnaker- +operator/build/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} +``` + +#### Halyard container logs +``` +2021-07-14 04:56:24.022 INFO 1 --- [nio-8064-exec-1] o.s.web.servlet.DispatcherServlet : Completed initialization in 28 ms +2021-07-14 04:56:55.631 WARN 1 --- [nio-8064-exec-9] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.web.HttpRequestMethodNotSupportedException: Request method 'POST' not supported] +2021-07-14 04:56:56.678 WARN 1 --- [nio-8064-exec-1] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.web.HttpRequestMethodNotSupportedException: Request method 'POST' not supported] +2021-07-14 04:56:57.711 WARN 1 --- [nio-8064-exec-2] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.web.HttpRequestMethodNotSupportedException: Request method 'POST' not supported] +``` + +## Cause +Operator pod basically is a multi-container pod that has both operator and halyard containers running within it. As seen on Halyard logs, Halyard version 1.12.1 is unable to handle the request from Operator and thus resulting in ```405 responses``` in the Operator logs. This is noticed with this particular version combination of Operator and Halyard. + diff --git a/kb/armory/General/unable-to-deploy-to-a-new-kubernetes-cluster-to-spinnaker.md b/kb/armory/General/unable-to-deploy-to-a-new-kubernetes-cluster-to-spinnaker.md new file mode 100644 index 0000000000..c9dbc04a8c --- /dev/null +++ b/kb/armory/General/unable-to-deploy-to-a-new-kubernetes-cluster-to-spinnaker.md @@ -0,0 +1,16 @@ +--- +title: Unable to Deploy to a New Kubernetes Cluster to Spinnaker +--- + +## Issue +Customers may see a variety of errors when attempting to add a new Kubernetes Cluster into Spinnaker. +Exception ( Monitor Deploy ) + Failed to read [configMap] from IDM-staging: error: You must be logged in to the server (Unauthorized) +The cluster appears in the Spinnaker Console UI, but users cannot deploy to the cluster because of various errors. +  +  + +## Cause +The reason for the situation and errors are likely to fall into one of the following situations:  +* kubeconfig file configuration * The token in the kubeconfig file does not match the token in the secrets* How does the role or cluster role configuration look like* How does the role/cluster role binding configuration look like* The path to the kubeconfig file in the SpinnakerService configuration * Spinnakerservice configuration typos/formatting + diff --git a/kb/armory/General/unable-to-evaluate-spel-variables-in-an-email-notification.md b/kb/armory/General/unable-to-evaluate-spel-variables-in-an-email-notification.md new file mode 100644 index 0000000000..d3a93ae067 --- /dev/null +++ b/kb/armory/General/unable-to-evaluate-spel-variables-in-an-email-notification.md @@ -0,0 +1,11 @@ +--- +title: Unable to evaluate SpEL variables in an email notification +--- + +## Issue +When configuring an email notification to be sent from a pipeline's stage, users can be referring to a variable from the execution context in the textbox for the email's body. +The received email does not evaluate the variable, and instead displays the literal string of the SpEL expression. + +## Cause +The textbox in the email notification window does **not** evaluate variables by design. + diff --git a/kb/armory/General/unable-to-list-kubernetes-endpoints-in-namespace-when-running-clouddriver-with-agent.md b/kb/armory/General/unable-to-list-kubernetes-endpoints-in-namespace-when-running-clouddriver-with-agent.md new file mode 100644 index 0000000000..1ae99265a5 --- /dev/null +++ b/kb/armory/General/unable-to-list-kubernetes-endpoints-in-namespace-when-running-clouddriver-with-agent.md @@ -0,0 +1,30 @@ +--- +title: Unable to list kubernetes Endpoints in namespace when running clouddriver with Agent +--- + +## Issue +Customers may notice that when the Agent Plugin is configured to use the Kubernetes cluster over Redis, Clouddriver shows the below errors during start-up. +2022-02-18 17:13:10.275 INFO --- [ main] c.n.spinnaker.cats.sql.cache.SqlCache : [] Configured for pie +2022-02-18 17:13:10.276 INFO --- [ main] c.n.spinnaker.cats.sql.cache.SqlCache : [] Configured for kubernetes +2022-02-18 17:13:10.276 INFO --- [ main] c.n.spinnaker.cats.sql.cache.SqlCache : [] Configured for com.netflix.spinnaker.clouddriver.sql.SqlProvider +2022-02-18 17:13:10.276 INFO --- [ main] c.n.spinnaker.cats.sql.cache.SqlCache : [] Configured for com.netflix.spinnaker.clouddriver.sql.SqlProvider +2022-02-18 17:13:10.276 INFO --- [ main] c.n.spinnaker.cats.sql.cache.SqlCache : [] Configured for com.netflix.spinnaker.clouddriver.docker.registry.provider.DockerRegistryProvider +2022-02-18 17:13:10.276 INFO --- [ main] c.n.spinnaker.cats.sql.cache.SqlCache : [] Configured for com.netflix.spinnaker.clouddriver.core.provider.CoreProvider +2022-02-18 17:13:10.277 INFO --- [ main] c.n.spinnaker.cats.sql.cache.SqlCache : [] Configured for kubesvc +2022-02-18 17:13:10.295 INFO --- [ main] o.s.s.c.ThreadPoolTaskScheduler : [] Initializing ExecutorService +2022-02-18 17:13:10.596 ERROR --- [ main] .k.s.o.c.k.KubernetesClouddriverRegistry : [] >>>>>>> Unable to list kubernetes Endpoints in namespace spinnaker-stage to discover clouddriver instances. Agent will NOT work if running more than one clouddriver replica! +io.kubernetes.client.openapi.ApiException: + at io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:979) + at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:895) + at io.armory.kubesvc.services.ops.cluster.kubernetes.KubernetesClouddriverRegistry$1.list(KubernetesClouddriverRegistry.java:245) + at io.armory.kubesvc.services.ops.cluster.kubernetes.KubernetesClouddriverRegistry.canList(KubernetesClouddriverRegistry.java:263) + at io.armory.kubesvc.services.ops.cluster.kubernetes.KubernetesClouddriverRegistry.start(KubernetesClouddriverRegistry.java:127) + at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) + at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) + at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + at java.base/java.lang.reflect.Method.invoke(Method.java:566) + + +## Cause +The above error is seen in Clouddriver logs because Clouddriver cannot connect to the other Clouddriver pods immediately when starting up. The Clouddriver Agent Plugin communicates with all Clouddriver pods using the Kubernetes endpoint object. Hence, the Service Account with which the Clouddriver runs should have the RBACs defined to access the endpoints on the cluster. + diff --git a/kb/armory/General/unable-to-retrigger-a-an-existing-pipeline-from-spinnaker-ui.md b/kb/armory/General/unable-to-retrigger-a-an-existing-pipeline-from-spinnaker-ui.md new file mode 100644 index 0000000000..448484100d --- /dev/null +++ b/kb/armory/General/unable-to-retrigger-a-an-existing-pipeline-from-spinnaker-ui.md @@ -0,0 +1,14 @@ +--- +title: Unable to retrigger a an existing pipeline from Spinnaker UI +--- + +## Issue +Users may run into a situation where they may be unable to re-trigger a pipeline through the Spinnaker UI. It is noticed that the UI is not responsive when such pipelines are triggered. +  + +## Cause +An application in Armory Enterprise Spinnaker is created through one of the following methods +* Using Spinnaker UI where they create the application using the Deck web console.* Invoke the Front50 API directly.* Use Armory's Pipeline As Code feature (Dinghy). +The ***Application names are generally expected to be with lowercase***. Even if they are to be created with uppercase through the UI, the Deck and the Gate Spinnaker services convert them to lowercase before they are persisted by Front50. +However if the applications with their names in uppercase are created using Front50 API or Dinghy, they get persisted as they are without any validations or conversions. When pipelines belonging to such applications are triggered through the Spinnaker UI, it ***fails to find the application*** as the original application created is with uppercase. + diff --git a/kb/armory/General/unable-to-see-the-accounts-listed-when-using-the-app-engine-plugin-in-a-pipeline.md b/kb/armory/General/unable-to-see-the-accounts-listed-when-using-the-app-engine-plugin-in-a-pipeline.md new file mode 100644 index 0000000000..803291be0b --- /dev/null +++ b/kb/armory/General/unable-to-see-the-accounts-listed-when-using-the-app-engine-plugin-in-a-pipeline.md @@ -0,0 +1,28 @@ +--- +title: Unable to see the accounts listed when using the App Engine plugin in a pipeline +--- + +## Issue +Service accounts do not show up in the server group. The drop down does not list the account, as seen in the following: + +The Clouddriver logs may not show any access errors.  The following errors may be seen in the Clouddriver logs upon restarting the Clouddriver pod: +.s.c.a.s.AppengineCredentialsInitializer : Could not load account custhub-preprod-cc6a6465-aes for App Engine + +com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden +``` +{ + "code" : 403, + "errors" : [ { + "domain" : "global", + "message" : "The caller does not have permission", + "reason" : "forbidden" + } ], + "message" : "The caller does not have permission", + "status" : "PERMISSION_DENIED" +} +``` + +## Cause +The ```403 errors``` may be lost in the logs until the Clouddriver pod is restarted. This is because an AppEngine account with no access doesn’t stop Clouddriver from starting unlike other accounts that are set up in Clouddriver (e.g. AWS, Kubernetes, etc). +AppEngine accounts are ***validated only on startup***. If the account doesn’t have access, then Clouddriver discards it and doesn’t try to validate again, and so, the errors will only show upon restart.  + diff --git a/kb/armory/General/unexpected-stale-read-warning-level-messages-in-front50-logs.md b/kb/armory/General/unexpected-stale-read-warning-level-messages-in-front50-logs.md new file mode 100644 index 0000000000..4837750b21 --- /dev/null +++ b/kb/armory/General/unexpected-stale-read-warning-level-messages-in-front50-logs.md @@ -0,0 +1,14 @@ +--- +title: Unexpected stale read warning-level messages in Front50 logs +--- + +## Issue +Warning-level log messages may appear in a user's Front50 logs that look something along the lines of: + +```| 2021-10-14 15:23:23.548 WARN 1 --- [pool-6-thread-3] c.n.s.f.m.s.DefaultServiceAccountDAO : Unexpected stale read for 9135a6cb-d088-4cde-82c2-bfc354d4abfd@managed-service-account (current: Mo |``` +This may also manifest in the form of applications, pipelines, notifications or execution history not loading at all, or displaying incorrectly in the Spinnaker UI. + +## Cause +Since Front50 stores and serves metadata out of an in-memory cache, there may have been or currently is an outage with the currently configured object store. +This may also occur after migrating the Spinnaker instance to another (for example, a Disaster Recovery) location, but not moving Operator's object store configuration to the replicated object store. + diff --git a/kb/armory/General/uninstalling-armory-spinnaker.md b/kb/armory/General/uninstalling-armory-spinnaker.md new file mode 100644 index 0000000000..18222eb15a --- /dev/null +++ b/kb/armory/General/uninstalling-armory-spinnaker.md @@ -0,0 +1,44 @@ +--- +title: Uninstalling Armory Spinnaker +--- + +## Introduction +The following information explains how uninstall Armory Spinnaker from an environment. + +## Prerequisites +Administration access + +## Instructions +**Uninstall Armory Spinnaker installed using Armory Operator** +All Armory Spinnaker components are installed inside the namespace specified by the user during installation. To uninstall Armory Spinnaker, simply delete the namespace using the following commands: + +```kubectl -n delete spinnakerservice spinnaker``` + +More information in the link below: +[https://docs.armory.io/docs/installation/armory-operator/op-manage-spinnaker/#delete-armory-enterprise](https://docs.armory.io/docs/installation/armory-operator/op-manage-spinnaker/#delete-armory-enterprise) +  +Please note that resources deployed with Armory Spinnaker may continue to remain even after removal. Admins should seek to remove all pipelines and resources, or transfer their management before removing Spinnaker. +  +**Uninstall Armory Spinnaker installed using Halyard** +If Halyard was installed on Linux/MacOS, run the following commands: +  + +```hal deploy clean``` + +  + +```sudo ~/.hal/uninstall.sh``` + +  +If Halyard was installed using Docker, simple delete the running halyard Docker container using the following commands: +  + +```docker rm halyard``` + +  +More information in the link below: +[https://spinnaker.io/docs/setup/install/halyard/#uninstall-halyard-from-debianubuntu-or-macos](https://spinnaker.io/docs/setup/install/halyard/#uninstall-halyard-from-debianubuntu-or-macos) +[https://spinnaker.io/docs/setup/install/halyard/#uninstall-halyard-from-docker](https://spinnaker.io/docs/setup/install/halyard/#uninstall-halyard-from-docker) +  +Please note that resources deployed with Armory Spinnaker may continue to remain even after removal. Admins should seek to remove all pipelines and resources, or transfer their management before removing Spinnaker. + diff --git a/kb/armory/General/unlock-the-databasechangeloglock-table.md b/kb/armory/General/unlock-the-databasechangeloglock-table.md new file mode 100644 index 0000000000..83e7fcca26 --- /dev/null +++ b/kb/armory/General/unlock-the-databasechangeloglock-table.md @@ -0,0 +1,36 @@ +--- +title: Unlock the DATABASECHANGELOGLOCK Table +--- + +## Introduction +There may be instances where the Spinnaker pods may be crashing and there are errors in the log showing something similar to: +```Error executing SQL UPDATE orca.DATABASECHANGELOGLOCK SET `LOCKED` = 1, LOCKEDBY = 'spin-orca-6755dcb97b-psszz (10.42.5.22)', LOCKGRANTED = '2020-10-16 15:34:11.084' WHERE ID = 1 AND `LOCKED` = 0: Lock wait timeout exceeded; try restarting transaction``` +Or +```2020-04-22 15:47:14.200 INFO 1 --- [ main] liquibase.executor.jvm.JdbcExecutor : SELECT `LOCKED` FROM clouddriver.DATABASECHANGELOGLOCK WHERE ID=1``` +The lock can be caused because Liquibase reads from the [DATABASECHANGELOG table](https://docs.liquibase.com/concepts/basic/databasechangelog-table.html) to determine which changeset*s* need to run, If Liquibase does not exit cleanly, the lock row may be left as locked. The solution is to clear out the current lock. + +## Prerequisites +Run the following command to confirm what the status of the table is in: +```select * from DATABASECHANGELOGLOCK;``` +If the LOCKED field is set to 1, then the table is currently locked as Liquibase is running against this database.  You can also see what pod has caused the issue, but often, the pod has already been replaced.An example of the output would be: ++----+--------+---------------------+-------------------------------------------------+ +| ID | LOCKED | LOCKGRANTED | LOCKEDBY | ++----+--------+---------------------+-------------------------------------------------+ +| 1 | | 2020-04-18 12:34:08 | spin-clouddriver-7xx7x09x97-x9xc2 (10.10.0.20) | ++----+--------+---------------------+-------------------------------------------------+ + + +## Instructions +* Connect to the SQL serverRun the command: +```UPDATE DATABASECHANGELOGLOCK SET LOCKED=0, LOCKGRANTED=null, LOCKEDBY=null;​``` +***** Make take some time for the table to unlock *** ** This can take a few minutesNote: Also the command ```liquibase releaseLocks``` will run ```UPDATE DATABASECHANGELOGLOCK SET LOCKED=0``` +You can then run the select command again to confirm. +```select * from DATABASECHANGELOGLOCK;​``` +Which should provide the following table: ++----+--------+-------------+----------+ +| ID | LOCKED | LOCKGRANTED | LOCKEDBY | ++----+--------+-------------+----------+ +| 1 | | NULL | NULL | ++----+--------+-------------+----------+​ + + diff --git a/kb/armory/General/updates-of-previously-cached-gitrepo-fails-on-artifact-fetches.md b/kb/armory/General/updates-of-previously-cached-gitrepo-fails-on-artifact-fetches.md new file mode 100644 index 0000000000..88be29b102 --- /dev/null +++ b/kb/armory/General/updates-of-previously-cached-gitrepo-fails-on-artifact-fetches.md @@ -0,0 +1,21 @@ +--- +title: Updates of Previously Cached gitrepo Fails on Artifact Fetches +--- + +## Issue +Git binary is used for the git clone operations in Spinnaker.If you have caching turned on in Spinnaker, Git will reject updates to a previously cloned repo with the following error message:  +`error.message:[sh, -c, git pull] failed. Error: +hint: You have divergent branches and need to specify how to reconcile them. +hint: You can do so by running one of the following commands sometime before +hint: your next pull: hint: hint: git config pull.rebase false # merge +hint: git config pull.rebase true # rebase +hint: git config pull.ff only # fast-forward only hint: +hint: You can replace "git config" with "git config --global" to set a default +hint: preference for all repositories. You can also pass --rebase, --no-rebase, +hint: or --ff-only on the command line to override the configured default per +hint: invocation. fatal: Need to specify how to reconcile divergent branches. +Note that the Git Repo Caching option requires a change to the configuration to be enabled and is not on by default.  + +## Cause +As indicated in the error message, once caching is turned on, it is a requirement for divergent branches to have pre-defined specifications for reconciling them.   + diff --git a/kb/armory/General/updating-spinnaker-stops-dinghy-from-updating-existing-pipelines.md b/kb/armory/General/updating-spinnaker-stops-dinghy-from-updating-existing-pipelines.md new file mode 100644 index 0000000000..ca8578c6f2 --- /dev/null +++ b/kb/armory/General/updating-spinnaker-stops-dinghy-from-updating-existing-pipelines.md @@ -0,0 +1,11 @@ +--- +title: Updating Spinnaker Stops Dinghy From Updating Existing Pipelines +--- + +## Issue +After updating Dinghy to 2.18.x (from 2.17.x), pre-existing pipelines cease to function. Note: New pipelines are unaffected and will function as usual.   + +## Cause +The following change was implemented in 2.18.1 to Dinghy:[https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-18-1/#dinghy-open-core---60cb2f53b0247e](https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-18-1/#dinghy-open-core---60cb2f53b0247e)Code Change:[https://github.com/armory/dinghy/commit/1de073e38fbd4de5f1b27216981dc69a19dbece8](https://github.com/armory/dinghy/commit/1de073e38fbd4de5f1b27216981dc69a19dbece8) +As advised in the update, there is a change in how the master branch is referenced and stored as a key in REDIS.  Previously, the ```refs/heads/``` path, was added to the key stored in REDIS, and this was since removed from the keys for cleaner operation. + diff --git a/kb/armory/General/upgrade-from-oss-to-armory-spinnaker.md b/kb/armory/General/upgrade-from-oss-to-armory-spinnaker.md new file mode 100644 index 0000000000..582e7c8496 --- /dev/null +++ b/kb/armory/General/upgrade-from-oss-to-armory-spinnaker.md @@ -0,0 +1,14 @@ +--- +title: Upgrade from OSS to Armory Spinnaker +--- + +## Introduction +Interested in moving to Armory Spinnaker?  This article will walk you through the steps for migrating from OSS Spinnaker to Armory Spinnaker + +## Prerequisites +N/A + +## Instructions +This page has been recategorized and moved to our Documents Library: +[https://docs.armory.io/spinnaker-install-admin-guides/upgrade-oss-to-armory/](https://docs.armory.io/spinnaker-install-admin-guides/upgrade-oss-to-armory/)  + diff --git a/kb/armory/General/upgrade-spinnaker-using-armory-halyard.md b/kb/armory/General/upgrade-spinnaker-using-armory-halyard.md new file mode 100644 index 0000000000..37a4dd3274 --- /dev/null +++ b/kb/armory/General/upgrade-spinnaker-using-armory-halyard.md @@ -0,0 +1,7 @@ +--- +title: Upgrade Spinnaker Using Armory Halyard +--- + + +This page has been recategorized and moved to our Documents Library:[https://spinnaker.io/docs/setup/install/upgrades/](https://spinnaker.io/docs/setup/install/upgrades/) + diff --git a/kb/armory/General/upgrade-to-spinnaker-1.20.x+-2.20.x+-causes-errors-as-pipelines-deploy-to-unavailable-namespace.md b/kb/armory/General/upgrade-to-spinnaker-1.20.x+-2.20.x+-causes-errors-as-pipelines-deploy-to-unavailable-namespace.md new file mode 100644 index 0000000000..48dc74cc17 --- /dev/null +++ b/kb/armory/General/upgrade-to-spinnaker-1.20.x+-2.20.x+-causes-errors-as-pipelines-deploy-to-unavailable-namespace.md @@ -0,0 +1,17 @@ +--- +title: Upgrade to Spinnaker (1.20.x+/2.20.x+) Causes Errors as Pipelines Deploy to Unavailable Namespace +--- + +## Issue +After upgrading Spinnaker completed successfully from 1.19.x/2.19.x or below to 1.20.x/2.20.x or above, attempts to deploy pipelines causes errors.  The following is an example of the error +```Deploy failed: Error from server (NotFound): error when creating "STDIN": namespaces "xxxxxxxxxxxxx" not found``` +The namespace is not a part of any declarations within your halconfig, etc.    +This error can also be applicable/appear where Spinnaker will attempt to deploy to the current namespace it occupies (basically what it defaults to) if the following is true: +* ```*kubeconfig*``` doesn't define any namespace* Manifest doesn't define any namespace* Override namespace in stage definition is not enabled* Deployment targets a different cluster than the one where spinnaker lives + +## Cause +Spinnaker OSS change in 1.20.x (2.20.x Armory version):[https://github.com/spinnaker/spinnaker/issues/5731](https://github.com/spinnaker/spinnaker/issues/5731) +A bugfix that shipped with Spinnaker 1.20 fixes an issue in versions 1.19 and earlier, where Spinnaker would attempt to read the default namespace in the context using a faulty JSON path.As a result of this error, Spinnaker was always falling back to the ```default``` namespace. As a result of this change, Spinnaker is now correctly interpreting the default namespace declared in the kubeconfig fileAnother possibility is that Spinnaker can also end up falling back in the namespace specified in: +```/var/run/secrets/kubernetes.io/serviceaccount/namespace``` +in absence of any namespace specified in the kubeconfig file + diff --git a/kb/armory/General/upgrade-using-armory-operator-is-not-working--got-halyard-response-status-405,-response--request-method-'post'-not-supported.md b/kb/armory/General/upgrade-using-armory-operator-is-not-working--got-halyard-response-status-405,-response--request-method-'post'-not-supported.md new file mode 100644 index 0000000000..b9641fafc8 --- /dev/null +++ b/kb/armory/General/upgrade-using-armory-operator-is-not-working--got-halyard-response-status-405,-response--request-method-'post'-not-supported.md @@ -0,0 +1,19 @@ +--- +title: Upgrade using Armory Operator is not working- got halyard response status 405, response- Request method 'POST' not supported +--- + +## Issue +When attempting to upgrade Armory Spinnaker using Armory Operator, the upgrade fails with the below logging. +  +Operator Container: +```{"level":"error","ts":1626238705.074147,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"spinnakerservice-controller","request":"spinnaker-operator/spinnaker","error":"got halyard response status 405, response: Request method 'POST' not supported","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/opt/spinnaker-operator/build/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/spinnaker-operator/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/spinnaker-operator/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/opt/spinnaker-operator/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/opt/spinnaker-operator/build/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/opt/spinnaker-operator/build/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/opt/spinnaker-operator/build/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}``` +  +Halyard Container: +2021-07-14 04:56:24.022 INFO 1 --- [nio-8064-exec-1] o.s.web.servlet.DispatcherServlet : Completed initialization in 28 ms +2021-07-14 04:56:55.631 WARN 1 --- [nio-8064-exec-9] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.web.HttpRequestMethodNotSupportedException: Request method 'POST' not supported] +2021-07-14 04:56:56.678 WARN 1 --- [nio-8064-exec-1] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.web.HttpRequestMethodNotSupportedException: Request method 'POST' not supported] +2021-07-14 04:56:57.711 WARN 1 --- [nio-8064-exec-2] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.web.HttpRequestMethodNotSupportedException: Request method 'POST' not supported] + +## Cause +This is caused by a mismatch of versions between the Halyard and Operator container. + diff --git a/kb/armory/General/upgrade-your-kustomize-version.md b/kb/armory/General/upgrade-your-kustomize-version.md new file mode 100644 index 0000000000..69716f8bbc --- /dev/null +++ b/kb/armory/General/upgrade-your-kustomize-version.md @@ -0,0 +1,45 @@ +--- +title: Upgrade Your Kustomize Version +--- + +## Introduction +It may become necessary to update the Kustomize binary in Rosco to a newer version.  Below are steps for how to go through the upgrade process + +## Prerequisites +N/A + +## Instructions +To change the Kustomize binary used by Rosco, mount a volume and use an ```initContainer``` to replace the original binary.Perform the following steps: +* Create the file following file if it does not already exist: ```~/.hal/default/service-settings/rosco.yml```Add the following volume information to the file: + kubernetes: + volumes: + - id: shared-volume + type: emptyDir + mountPath: /packer/kustomize​ + + +`````` + + + +* This example uses an ```_emptyDir_``` type for the volume type.* The ```mounthPath``` is the directory where the existing Kustomize binary is located. +In the ```~/.hal/config``` file, add an ```initContainer```: + +```````````` +initContainers: + rosco: + - name: kustomize-binary + image: tutum/curl:latest + command: ["/bin/sh", "-c"] + args: ["curl -s -L https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv3.5.4/kustomize_v3.5.4_linux_amd64.tar.gz | tar xvz -C /tmp"] + volumeMounts: + - name: shared-volume + mountPath: /tmp +```````````` + +`````` +* Replace the URL in the **args** attribute with the URL for the Kustomize version that you want to use. In the example, the URL points to version ```3.5.4```.* Set the ```name``` for the ```volumeMounts``` to the same value as the volume’s ID in the ```rosco.yml``` file from step 2. +* Apply the changes: ```hal deploy apply```* Enter into the Rosco pod: ```kubectl exec -it bash```.* ``````Go to the Kustomize directory: ```cd /packer/kustomize```. If you enter the ```ls``` command, you should see the new binary.Check the Kustomize version: ```./kustomize version```.The command returns information similar to the following: +```{Version:kustomize/v3.5.4 GitCommit:3af514fa9f85430f0c1557c4a0291e62112ab026 BuildDate:2020-01-11T03:12:59Z GoOs:linux GoArch:amd64}``` + + diff --git a/kb/armory/General/upgrading-an-eks-cluster.md b/kb/armory/General/upgrading-an-eks-cluster.md new file mode 100644 index 0000000000..73bb299442 --- /dev/null +++ b/kb/armory/General/upgrading-an-eks-cluster.md @@ -0,0 +1,103 @@ +--- +title: Upgrading an EKS Cluster +--- + +## Introduction +EKS clusters can be upgraded directly from the AWS cluster to upgrade the master.  Below is some information about how to go about making those changes to your environment and some considerations to make before doing so + +## Prerequisites +Before updating the Control plane, check the version of your Kubernetes cluster and worker node.  If your worker node version is older than your current Kubernetes version, then you must update your worker node first then only proceed with updating your control plane. +Check Kubernetes Control Plane version and worker node from the following command: +```kubectl version --short``` +```kubectl get nodes``` +This will give you the node(s) and their version. +Since Pod Security Policy(PSP) admission controller is enabled by default from 1.13 and later versions of Kubernetes, we need to make sure that proper pod security policy is in place, before updating the Kubernetes version on the Control Plane. +Check the default security policy using the command below: +```kubectl get psp eks.privileged``` +If you get any server error, then you must install PSP. +Make sure to read and understand the following AWS documentation on best practices:[Planning Kubernetes Upgrades with Amazon EKS](https://aws.amazon.com/blogs/containers/planning-kubernetes-upgrades-with-amazon-eks/)[Amazon EKS - Updating a Cluster](https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html) +Please also note that there may also be documented changes from Kubernetes to be aware of.  For example, the following documentation from Kubernetes outlines a potential issue for those upgrading to v1.21 when using HashiCorp Vault[https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-issuer-discovery](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-issuer-discovery) +Note: When the master goes down for the upgrade, deployments, services, etc. continue to work as expected. However, anything that requires the Kubernetes API stops working. This means ```kubectl``` stops working, applications that use the Kubernetes API to get information about the cluster stop working, and basically, you can’t make any changes to the cluster while it is being upgraded. +  + +## Instructions +### Upgrade Master +This can be achieved directly from AWS console, under ```EKS``` section. There’s a ```Upgrade cluster version``` button that can be used to upgrade the master. +If you are using Terraform scripts for managing your cluster, the [version](https://www.terraform.io/docs/providers/aws/r/eks_cluster.html#version) can be specified in ```aws_eks_cluster``` resource: +``` +resource "aws_eks_cluster" "aws-eks" { + name = "${var.cluster-name}" + role_arn = "${aws_iam_role.aws-eks-cluster.arn}" + + version = "${var.cluster-master-version}" + + vpc_config { + security_group_ids = ["${aws_security_group.aws-eks-cluster.id}"] + subnet_ids = ["${aws_subnet.aws-eks-subnet-1.id}", "${aws_subnet.aws-eks-subnet-2.id}"] + } + + depends_on = [ + "aws_iam_role_policy_attachment.aws-eks-cluster-AmazonEKSClusterPolicy", + "aws_iam_role_policy_attachment.aws-eks-cluster-AmazonEKSServicePolicy", + "aws_iam_role.aws-eks-cluster" + ] +} +``` + +*Effect: Doing this will only upgrade the master, but EKS worker nodes will not be affected and will continue running normally.* + +### Upgrade EKS Worker Nodes +In this step you actually change the AMI used for the worker nodes in EKS launch configuration, and then double the number of nodes in the cluster in their Auto Scaling Group configuration. If the original cluster had 3 nodes, you increase that to 6 nodes: +If you are using Terraform scripts, these settings are available under [image_id](https://www.terraform.io/docs/providers/aws/r/launch_configuration.html#image_id) of ```aws_launch_configuration``` resource: +```` +resource "aws_launch_configuration" "aws-eks" { + associate_public_ip_address = false + iam_instance_profile = "${aws_iam_instance_profile.aws-eks-node.name}" + image_id = "${data.aws_ami.eks-worker.id}" + instance_type = "${var.ec2-instance-type}" + name_prefix = "${var.cluster-name}-node" + security_groups = ["${aws_security_group.aws-eks-node.id}"] + user_data_base64 = "${base64encode(local.aws-eks-node-userdata)}" + + lifecycle { + create_before_destroy = true + } + + root_block_device { + volume_size = "50" + } +} +```` + +And Auto Scaling Group size is available under [max_size and desired_capacity](https://www.terraform.io/docs/providers/aws/r/autoscaling_group.html#desired_capacity) of ```aws_autoscaling_group``` resource: +```` +resource "aws_autoscaling_group" "aws-eks" { + desired_capacity = "${var.desired-ec2-instances}" + launch_configuration = "${aws_launch_configuration.aws-eks.id}" + max_size = "${var.max-ec2-instances}" + min_size = "${var.min-ec2-instances}" + name = "${var.cluster-name}-asg" + vpc_zone_identifier = ["${aws_subnet.aws-eks-subnet-1.id}", "${aws_subnet.aws-eks-subnet-2.id}"] + + tag { + key = "Name" + value = "${var.cluster-name}" + propagate_at_launch = true + } + + tag { + key = "kubernetes.io/cluster/${var.cluster-name}" + value = "owned" + propagate_at_launch = true + } + + depends_on = ["aws_eks_cluster.aws-eks"] + +} +```` +*Effect: Cluster size will be doubled, new worker nodes will be created with the new version, and old worker nodes will still be running and available with the old version.* +  +### Scale Down Auto Scaling Group +After the new nodes are correctly registered and available in kubernetes, it’s time to decrease the Auto Scaling Group to its original size. +*Effect: All pods in the old nodes will be automatically moved to the new nodes by kubernetes.* After Spinnaker pods are restarted, you should be able to continue using Spinnaker as usual. + diff --git a/kb/armory/General/upgrading-from-2.19.x-to-2.20.x-error-authenticating-against-eks-clusters-heptio-authenticator-aws.md b/kb/armory/General/upgrading-from-2.19.x-to-2.20.x-error-authenticating-against-eks-clusters-heptio-authenticator-aws.md new file mode 100644 index 0000000000..715820f45c --- /dev/null +++ b/kb/armory/General/upgrading-from-2.19.x-to-2.20.x-error-authenticating-against-eks-clusters-heptio-authenticator-aws.md @@ -0,0 +1,15 @@ +--- +title: Upgrading from 2.19.x to 2.20.x Error Authenticating Against EKS Clusters (heptio-authenticator-aws) +--- + +## Issue +Upon upgrading from 2.19.x to 2.20.x, ```heptio-authenticator-aws``` errors appear in CloudDriver LogsThe following type of error can be found in the CloudDriver Logs +2020-07-09 09:38:57.221 WARN 1 --- [0.0-7002-exec-5] c.n.s.c.k.v.c.ManifestController : Failed to read manifest + +com.netflix.spinnaker.clouddriver.kubernetes.v2.op.job.KubectlJobExecutor$KubectlException: Failed to read ingress from [podname]: Unable to connect to the server: getting credentials: exec: exec: "heptio-authenticator-aws": executable file not found in $PATH + + at com.netflix.spinnaker.clouddriver.kubernetes.v2.op.job.KubectlJobExecutor.get(KubectlJobExecutor.java:355) ~[clouddriver-kubernetes-v2-GCSFIX.jar:na] + +## Cause +Heptio authenticator changed names within Kubernetes SIG as per:[https://github.com/kubernetes-sigs/aws-iam-authenticator/pull/108](https://github.com/kubernetes-sigs/aws-iam-authenticator/pull/108)Armory used to symlink heptio to the iam authenticator, but this was removed to preserve the original naming practice, as of 2.20.x + diff --git a/kb/armory/General/use-case-for-aws-sticky-sessions.md b/kb/armory/General/use-case-for-aws-sticky-sessions.md new file mode 100644 index 0000000000..259c9b3699 --- /dev/null +++ b/kb/armory/General/use-case-for-aws-sticky-sessions.md @@ -0,0 +1,11 @@ +--- +title: Use case for AWS Sticky Sessions +--- + +## Issue +When using a **Red/Black (Blue/Green) deployment strategy **all new AWS EC2 instances are added to the same load balancer target group and traffic equally distributes across new and old versions, then the old version deletes as expected.Then the following happens: +* An instance from the new Auto Scaling Group (ASG) becomes healthy.* The load balancer sends traffic to it.* It responds with the link to a JavaScript file.* The load balancer chooses an instance from the old ASG and sends traffic to it, but this instance does not have this script.* Users get ```404 errors``` in their browser. + +## Cause +The sessions on the load balancer drop, causing the errors. + diff --git a/kb/armory/General/use-custom-images-with-halyard-public-and-private-docker-registry.md b/kb/armory/General/use-custom-images-with-halyard-public-and-private-docker-registry.md new file mode 100644 index 0000000000..30306a99d1 --- /dev/null +++ b/kb/armory/General/use-custom-images-with-halyard-public-and-private-docker-registry.md @@ -0,0 +1,29 @@ +--- +title: Use Custom Images with Halyard (Public and Private Docker Registry) +--- + +## Introduction +Occasionally, it may make sense to update the Spinnaker Kubernetes deployments created by Halyard with a custom Docker image. This can be done through a custom service setting. + +## Prerequisites +Docker images in a Public or Private Registry + +## Instructions +### Using a Custom Docker Image (Public Registry) +Here is how you would achieve this: identify the service that you’re modifying, and create a corresponding file in ```.hal//service-settings/.yml``` with the relevant ```artifactId```. +For example, if I want to use a custom Docker image for my Deck, I could create the following file: +```.hal//service-settings/deck.ym```l +```artifactId: docker.io/armory/deck:;``` +Then, when you go to deploy your update (```hal deploy apply```), this setting should propagate to ```.hal//staging/spinnaker.ym```l, in ```services.deck.artifactId```, and get deployed to the cluster. +More generically, individual settings in ```spinnaker.yml``` can be overridden with files in ```.hal//service-settings/.yml```, and additional yaml files can be propagated to containers by placing them in ```.hal//profiles/-local.yml``` + +### Private Docker Registry +If necessary, you may need to put your images in a private docker registry (due to corporate restrictions around using public registries like Docker Hub ([docker.io](http://docker.io/))).  In that case you will need to configure credentials for kubernetes to authenticate and successfully pull the image from a private repo.  In order to do so, you will create a **secret** that contains the credentials and configure ```ImagePullSecrets```. +First start by creating a **secret** of type ```docker-registry``` as shown below:  +```kubectl create secret docker-registry regcred --docker-server= --docker-username= --docker-password= --docker-email=``` +If necessary, refer to the kubernetes guide on for different methods of creating the secret:  [https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials) +Next, to configure ```ImagePullSecrets``` you will need to also add the following section the ```.yml``` created earlier.  +kubernetes: + imagePullSecrets: + - regcred # or the name of your secret created above + diff --git a/kb/armory/General/username-read-error-when-executing-terraform-stages-with-bitbucket-hosted-artifacts.md b/kb/armory/General/username-read-error-when-executing-terraform-stages-with-bitbucket-hosted-artifacts.md new file mode 100644 index 0000000000..27eb0de314 --- /dev/null +++ b/kb/armory/General/username-read-error-when-executing-terraform-stages-with-bitbucket-hosted-artifacts.md @@ -0,0 +1,11 @@ +--- +title: Username read error when executing Terraform stages with BitBucket hosted artifacts +--- + +## Issue +When executing a Terraform plan or apply, the stage fails and an error similar to the below displayed: +```failed to stage directory for terraform execution: There was a problem downloading an artifact type: git/repo, reference: https://bitbucket.org// - failed to fetch artifact. Status 500 - git clone --branch --depth 1 https://bitbucket.org// failed. Error: Cloning into ''... fatal: could not read Username for 'https://bitbucket.org': No such device or address Output:``` + +## Cause +Assuming that the credentials, artifacts and gitRepo have been configured per [Enable the Terraform Integration Stage in Armory Enterprise](https://docs.armory.io/plugins/terraform/install/armory-cd/), the problem is caused by the BitBucket repository being set to private whilst having to use a username/password combination for authentication.  + diff --git a/kb/armory/General/users-are-unable-to-unlock-a-spinnaker-pipeline-through-the-console---unlock-via-ui-unchecked.md b/kb/armory/General/users-are-unable-to-unlock-a-spinnaker-pipeline-through-the-console---unlock-via-ui-unchecked.md new file mode 100644 index 0000000000..05694bd8fc --- /dev/null +++ b/kb/armory/General/users-are-unable-to-unlock-a-spinnaker-pipeline-through-the-console---unlock-via-ui-unchecked.md @@ -0,0 +1,12 @@ +--- +title: Users are Unable to Unlock a Spinnaker Pipeline through the Console - Unlock via UI unchecked +--- + +## Issue +Customers may find that the options to unlock a stage may no longer be visible in the UI after locking a stage.  This issue may be because when locking a pipeline, a user chose to uncheck the ```Unlock via UI``` option + +  + +## Cause +As outlined, the only way to now unlock the pipeline is via the Spinnaker API.  Customers will then need to send a command via the API to Spinnaker to unlock the pipeline. + diff --git a/kb/armory/General/users-unable-to-execute-or-create-pipelines-due-to-large-headers-on-api-calls-to-orca,-front50-.md b/kb/armory/General/users-unable-to-execute-or-create-pipelines-due-to-large-headers-on-api-calls-to-orca,-front50-.md new file mode 100644 index 0000000000..8c945376a7 --- /dev/null +++ b/kb/armory/General/users-unable-to-execute-or-create-pipelines-due-to-large-headers-on-api-calls-to-orca,-front50-.md @@ -0,0 +1,22 @@ +--- +title: Users unable to execute or create pipelines due to large headers on API calls to Orca, front50 +--- + +## Issue +It may be noticed that certain users who are mapped to a large number of user groups are unable to create pipelines or applications or even execute a pipeline. Below errors may be seen on the gate logs  +**Error when creating a pipeline** +2021-12-22 16:55:17.179 ERROR 1 --- [0.0-8084-exec-3] c.n.s.k.w.e.GenericExceptionHandlers : Bad Request: com.netflix.spinnaker.gate.controllers.PipelineController$PipelineException: Pipeline save operation did not succeed: 01FQHH31DF5N6V14D079DJAHGF (status: TERMINAL) +2021-12-22 16:55:17.193 ERROR 1 --- [0.0-8084-exec-3] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] threw exception +2021-12-22 16:55:17.179 ERROR 1 --- [0.0-8084-exec-3] c.n.s.k.w.e.GenericExceptionHandlers : Bad Request: com.netflix.spinnaker.gate.controllers.PipelineController$PipelineException: Pipeline save operation did not succeed: 01FQHH31DF5N6V14D079DJAHGF (status: TERMINAL) +**Error when triggering a pipeline** +2021-12-22 14:17:51.408 WARN 1 --- [.0-8084-exec-10] c.n.s.k.w.e.GenericExceptionHandlers : Handled error in generic exception handler + +com.netflix.spinnaker.kork.web.exceptions.NotFoundException: Pipeline not found (id: 01FQH82MK14WWNEPR8TYQBVXYW) + at com.netflix.spinnaker.gate.controllers.PipelineController.getPipeline(PipelineController.groovy:112) + at com.netflix.spinnaker.gate.controllers.PipelineController$$FastClassBySpringCGLIB$$e4bc6ec4.invoke() + at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) + at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687) + +## Cause +When a pipeline is triggered, there is a cascade of api calls starting from ```deck->gate->echo->orca``` . When investigated at various levels, it can be seen that orca returns a ```HTTP response 400``` when it is unable to handle a request. The same might happen with Front50 for creating a pipeline or application. This is because, the default header size for Tomcat is only 8KB. If this limit is exceeded, Orca, Front50, Rosco will timeout and throw ```400 errors``` as it is unable to parse the entire header.  + diff --git a/kb/armory/General/using-`find-artifacts-from-resource-manifest`-overwrites-rendered-artifacts.md b/kb/armory/General/using-`find-artifacts-from-resource-manifest`-overwrites-rendered-artifacts.md new file mode 100644 index 0000000000..432c34f7da --- /dev/null +++ b/kb/armory/General/using-`find-artifacts-from-resource-manifest`-overwrites-rendered-artifacts.md @@ -0,0 +1,11 @@ +--- +title: Using `Find Artifacts From Resource (Manifest)` overwrites rendered artifacts +--- + +## Issue +This occurs when there is a pipeline that uses ```Find Artifacts From Resource (Manifest)``` to find details of the currently running deployment.However, when using the ```Find Artifacts From Resource (Manifest)``` stage, it overrides any pod images with the one that is currently in production. +Example issue: [https://github.com/spinnaker/spinnaker/issues/6506](https://github.com/spinnaker/spinnaker/issues/6506) + +## Cause +There was no specification available to opt-out of this artifact binding. Without this, it will always bind the artifact by ```default```. + diff --git a/kb/armory/General/using-custom-resource-status-plugin-and-scale-agent-leads-to-an-attempt-was-made-to-call-a-method-that-does-not-exist-error.md b/kb/armory/General/using-custom-resource-status-plugin-and-scale-agent-leads-to-an-attempt-was-made-to-call-a-method-that-does-not-exist-error.md new file mode 100644 index 0000000000..355bc2c739 --- /dev/null +++ b/kb/armory/General/using-custom-resource-status-plugin-and-scale-agent-leads-to-an-attempt-was-made-to-call-a-method-that-does-not-exist-error.md @@ -0,0 +1,21 @@ +--- +title: Using Custom Resource Status Plugin and Scale Agent leads to An attempt was made to call a method that does not exist Error +--- + +## Issue +Armory customers that may be using the [Custom Resource Status Plugin](https://docs.armory.io/plugins/plugin-k8s-custom-resource-status/) (2.0.3) and encounter the error below.  The following error will prevent Armory CDSH from starting. +An attempt was made to call a method that does not exist. The attempt was made from the following location: +io.armory.plugin.config.CustomResourceStatusPluginConfiguration.injectDefaultHandler(CustomResourceStatusPluginConfiguration.kt:17) +The following method did not exist: +'void com.netflix.spinnaker.clouddriver.kubernetes.description.GlobalResourcePropertyRegistry.setDefaultHandler(com.netflix.spinnaker.clouddriver.kubernetes.op.handler.KubernetesUnregisteredCustomResourceHandler)' +The method's class, com.netflix.spinnaker.clouddriver.kubernetes.description.GlobalResourcePropertyRegistry, is available from the following locations: +jar:file:/opt/clouddriver/lib/clouddriver-kubernetes-2023.05.30.19.45.40.release-1.30.x.jar!/com/netflix/spinnaker/clouddriver/kubernetes/description/GlobalResourcePropertyRegistry.class +The class hierarchy was loaded from the following locations: +com.netflix.spinnaker.clouddriver.kubernetes.description.GlobalResourcePropertyRegistry: file:/opt/clouddriver/lib/clouddriver-kubernetes-2023.05.30.19.45.40.release-1.30.x.jar +Action: +Correct the classpath of your application so that it contains a single, compatible version of com.netflix.spinnaker.clouddriver.kubernetes.description.GlobalResourcePropertyRegistry +2023-08-10T17:53:39.125317254Z + +## Cause +[Custom Resource Status Plugin](https://docs.armory.io/plugins/plugin-k8s-custom-resource-status/) (2.0.3) has a dependency on a Clouddriver change, and customers will need to have the newer version of Clouddriver, either as a hotfix image, or with Armory CDSH 2.30.3+, or 2.28.7+ + diff --git a/kb/armory/General/using-external-artifacts-as-pipeline-parameters.md b/kb/armory/General/using-external-artifacts-as-pipeline-parameters.md new file mode 100644 index 0000000000..76faed91f7 --- /dev/null +++ b/kb/armory/General/using-external-artifacts-as-pipeline-parameters.md @@ -0,0 +1,17 @@ +--- +title: Using external artifacts as pipeline parameters +--- + +## Introduction +As an organization, a user requires pipelines to be able to read the latest "Approved" version instead of just the latest "Built" version. This is a useful feature when dev or staging environments create a stable pipeline that must be reproduced in Prod.Example: "Approved" can mean that an organization has manually checked a pipeline's success (such as the latest stable docker version to use) on a non production Spinnaker.  This information is valuable for automation and efficiency. + +## Prerequisites +An organization may have need to move marked/tagged information external to Spinnaker such as Docker Container Registries, Artifacts, URLs, or any such string information. +In order to achieve this, an organization must have a working version of Spinnaker as well as an externally called resource. In this example, we are using Docker Container IDs in order to pass the ID from a manual judgement stage to an automated stage. + + +## Instructions +In this example, we will take a manual test pipeline that confirms whether a docker image is stable or not. This docker image ID will then be passed onto a Production Spinnaker that runs off of a cronjob.  +* On the Manual Test pipeline, an organization will need to add a stage which includes adding a tag to the specific docker image. This is to create a value that an organization can reference for this process. It can be something like ```Approved``` or a more unique value that an organization will recognize. It may needed to add an additional stage ahead of this removing the tag from any previous Docker images so there aren't multiple docker images that have the tag. * On the pipeline on the production environment, an organization will need to add a ```find-tag``` stage which searches for the unique value you created in the previous step.* Once you have that unique ID, an organization can assign the docker image to only use the ```Approved``` tag vs just using the latest. This stage would search for the latest approved tag and use that docker image vs just going by the 'latest'.  +This allows companies to use a pre-prod or staging Spinnaker instance to approve Docker or other Values in a safe environment, and pass that information to a production environment. + diff --git a/kb/armory/General/using-openshift-with-spinnaker.md b/kb/armory/General/using-openshift-with-spinnaker.md new file mode 100644 index 0000000000..e6bdd0684b --- /dev/null +++ b/kb/armory/General/using-openshift-with-spinnaker.md @@ -0,0 +1,11 @@ +--- +title: Using Openshift with Spinnaker +--- + + +Question: +*This is a Q&A pulled from the Spinnaker Community Slack team, which [we encourage you to join here](http://join.spinnaker.io/).* +Does anybody know if there are any plans to support Openshift? I doubt that Google would be taking this on, but curious whether any RedHat devs might be working on this? +Answer: +We have a few customers who use Spinnaker with openshift. You can use it by hooking it up to the kuberentes apis and treat it like kubernetes. + diff --git a/kb/armory/General/using-pipelines-as-code-to-enable-aws-auto-scaling-group-metrics.md b/kb/armory/General/using-pipelines-as-code-to-enable-aws-auto-scaling-group-metrics.md new file mode 100644 index 0000000000..f40ca80afd --- /dev/null +++ b/kb/armory/General/using-pipelines-as-code-to-enable-aws-auto-scaling-group-metrics.md @@ -0,0 +1,32 @@ +--- +title: Using Pipelines as Code to Enable AWS Auto Scaling Group Metrics +--- + +## Introduction +In the UI, we can enable AWS Auto Scaling Group (ASG) Metrics by clicking on individual clusters within an Application, scrolling down in the slide out to the Advanced Settings sub menu and opening the "Edit Advanced Settings" dialog. When clicking inside the "Enabled Metrics" field, we are presented with a number of metrics that correlate to the metrics in the AWS ASG Monitoring console: + +Details on what the metrics report can be found in the [AWS EC2 Auto Scaling Metics Documentation.](https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-instance-monitoring.html#available-cloudwatch-metrics) +This process needs to be done for every ASG created as part of an Application so is not scalable where ASG's are being recreated frequently and the metrics are required for operational reasons. This can, however, be achieved by enabling the metrics in a dinghyfile. + +## Prerequisites +* Dinghy setup per the [Using Dinghy documentation](https://docs.armory.io/docs/spinnaker-user-guides/using-dinghy/)* Application clusters being defined and deployed as part of a pipeline + +## Instructions +Under the cluster definition, locate or insert the ```enabledMetrics``` key: +``` + "stages": [ + { + "clusters": [ + { + + }, + "cloudProvider": "aws", + "cooldown": 10, + "copySourceCustomBlockDeviceMappings": false, + "delayBeforeDisableSec": 0, + "delayBeforeScaleDownSec": "0", + "ebsOptimized": true, + "enabledMetrics": [], +``` ​ +* Insert the metrics required as a ***comma separated list*** into the ```enabledMetrics``` field. The metric names to insert here are mirrored in the UI i.e. ```GroupMaxSize```, ```GroupMinSize``` etc. + diff --git a/kb/armory/General/using-s3-as-a-backend-for-front50-settings,-suggestions,-and-considerations.md b/kb/armory/General/using-s3-as-a-backend-for-front50-settings,-suggestions,-and-considerations.md new file mode 100644 index 0000000000..d5a3ee75b7 --- /dev/null +++ b/kb/armory/General/using-s3-as-a-backend-for-front50-settings,-suggestions,-and-considerations.md @@ -0,0 +1,43 @@ +--- +title: Using S3 as a Backend for Front50 (Settings, Suggestions, and Considerations) +--- + +## Introduction +Customers using S3 as a Front50 Backend should only do so for non-production environments/test environments.  This is due to multiple factors, such as the inability of S3 to scale with large environments and the nature of S3 being designed for Data Storage, not as a Database.   +Per the guidance provided in OSS documentation, consistency-related issues have plagued deployments using S3. +[https://spinnaker.io/docs/setup/productionize/persistence/front50-sql/#why-sql](https://spinnaker.io/docs/setup/productionize/persistence/front50-sql/#why-sql) +Customers should also take into consideration the following: +* Running Front50 with an S3 backend can be inefficient, as additional resources are required for consistent performance, either in the form of additional ReplicaSets or additional resources (horizontal or vertical scaling)* Higher thresholds may be required for an environment to function correctly, possibly loosening restrictions so that the environment can work without issue.* Resource restraints may cause the Front50 container to crash and restart consistently. +**Therefore, customers must move to MySQL as a backend whenever performance, scale, or reliability becomes a factor.  ** + +## Prerequisites +N/A + +## Instructions +In the case where admins have chosen to use S3 in an environment that does not have scaling/reliability/performance requirements but would like to, administrators will want to look to add the following changes to their Environment +* -XX:MaxRAMPercentage=75%* readinessProbe.periodSeconds = 120* readinessProbe.timeoutSeconds = 120* spinnaker.s3.maxkeys = 30000 +This can be accomplished by editing the```SpinnakerService.yml``` file in their Environment to allow the environment to function effectively +spec: + spinnakerConfig: + profiles: + front50: + spinnaker: + s3: + enabled: true + maxKeys: 30000 + service-settings: + front50: + env: + JAVA_OPTS: "-XX:InitialRAMPercentage=50 -XX:MaxRAMPercentage=75" + kubernetes: + readinessProbe: + failureThreshold: 3 + httpGet: + path: /health + port: 7002 + scheme: HTTPS + initialDelaySeconds: 60 + periodSeconds: 120 + successThreshold: 1 + timeoutSeconds: 120 + diff --git a/kb/armory/General/using-the-kubernetes-v2-patch-manifest-stage.md b/kb/armory/General/using-the-kubernetes-v2-patch-manifest-stage.md new file mode 100644 index 0000000000..2968f3e00e --- /dev/null +++ b/kb/armory/General/using-the-kubernetes-v2-patch-manifest-stage.md @@ -0,0 +1,8 @@ +--- +title: Using the Kubernetes V2 Patch Manifest Stage +--- + + +In this video, Ethan Rogers demonstrates the Kubernetes V2 Patch Manifest stage using a practical example. By using Spinnaker to empower engineers to perform operational tasks, like putting applications into maintenance mode, operations teams allow engineering teams to move faster. + + diff --git a/kb/armory/General/using-travis-ci-with-spinnaker.md b/kb/armory/General/using-travis-ci-with-spinnaker.md new file mode 100644 index 0000000000..522e055ca9 --- /dev/null +++ b/kb/armory/General/using-travis-ci-with-spinnaker.md @@ -0,0 +1,29 @@ +--- +title: Using Travis CI with Spinnaker +--- + + +Question: +Can I use Travis CI with Spinnaker? +Answer: +Yes you can! Just like Jenkins, Spinnaker has a native integration with Travis. You can easily use it to trigger deployments once CI builds finish. However, unlike with Jenkins, you *cannot* use Travis to run arbitrary jobs. This is more of a limitation imposed by Travis than a limitation of Spinnaker itself. +Configuring a Travis Master: +[Igor](https://github.com/spinnaker/igor) is the Spinnaker service responsible for interacting with external CI systems like Jenkins and Travis. So, in order to configure a Travis Master, you’ll need to add the following configuration to ```igor-local.yml```. +travis: + enabled: true + # Travis names are prefixed with travis- inside igor. + masters: + - name: ci # This will show as travis-ci inside spinnaker. + baseUrl: https://travis-ci.org + address: https://api.travis-ci.org + githubToken: a-github-token + regexes: + - /Upload https?:\/\/.+\/(.+\.(deb|rpm))/ + +You can configure multiple Travis masters by adding more entries under the ```masters``` key. +The ```regexes``` key is used to parse build information out of the Travis build log using the ```art``` CLI. This information is then used by Spinnaker pipelines to inform Bake stages which package versions should be installed during a bake stage. +Finally, in order to ensure that Igor is enabled, add the following to ```spinnaker-local.yml```: +services: + igor: + enabled: true + diff --git a/kb/armory/General/utilizing-kustomize-of-remote-base-in-a-different-git-repo.md b/kb/armory/General/utilizing-kustomize-of-remote-base-in-a-different-git-repo.md new file mode 100644 index 0000000000..035a111430 --- /dev/null +++ b/kb/armory/General/utilizing-kustomize-of-remote-base-in-a-different-git-repo.md @@ -0,0 +1,46 @@ +--- +title: Utilizing Kustomize of remote base in a different Git Repo +--- + +## Introduction +The following instructions may be followed to utilize multiple git repositories to fetch Kustomize files under "Bake Manifest" stage.  +To set up Kustomize with a different GitRepo, it is essential that the environment that will be doing the baking have the credentials/access for the other GitRepos.  This access is needed in order to clone the repo in the process. +The initial repo sent to Rosco by Clouddriver, using the credentials CloudDriver has access to.  Rosco receives the data about the repo, but at this point, Rosco itself doesn’t have credentials. +The ```Bake (Manifest)``` process works the same way. However, once the artifact is downloaded from Clouddriver, the Kustomize file has another definition contained within it that will point it to the different repository. The Kustomize build runs on Rosco, and so the SSH keys need to be mounted on the Rosco pod since Rosco will be accessing the files, not CloudDriver. + +## Prerequisites +The following pre-reqs should be taken into consideration: +* Credentials for the remote repository should be created and made available.  In this example, we will be using SSH.  The SSH keys in use must be added to any repos that Kustomize will attempt to access and this is out scope of this KB article.* These credentials should be loaded into a kubernetes secret named ```ssh-credentials``` and contain ```id_rsa``` and ```id_rsa.pub``` entries with the data of the ssh credentials. + +## Instructions +The following configuration may be added to your Spinnaker service once the SSH credentials are stored in a Kubernetes secret. +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker +spec: + spinnakerConfig: + config: + deploymentEnvironment: + initContainers: + rosco: + - name: ssh-initializer + image: busybox + command: ["/bin/sh", "-c", "cp /opt/ssh-credentials/* /home/spinnaker/.ssh && chmod 600 /home/spinnaker/.ssh/* && chown 100 /home/spinnaker/.ssh/*"] + volumeMounts: + - name: shared-volume + mountPath: /opt/ssh-credentials + - name: ssh-credentials + mountPath: /home/spinnaker/.ssh + service-settings: + rosco: + kubernetes: + volumes: + - name: shared-volume + mountPath: /opt/ssh-credentials + - id: ssh-credentials + type: emptyDir + mountPath: /home/spinnaker/.ssh + +This will mount the credentials into Rosco, so that it will have access to the remote repositories via SSH Git URLs. **A major note about security:** It should be noted that ***Rosco access is shared by all users of Spinnaker***.  This means that any user of Spinnaker will be able to use Kustomize to fetch from the accessible repositories for baking purposes.  This could impact other bake systems like Helm or Packer that may use local executions. + diff --git a/kb/armory/General/validator-*validate.halvalidator-detected-a-fatal-error.md b/kb/armory/General/validator-*validate.halvalidator-detected-a-fatal-error.md new file mode 100644 index 0000000000..8f4a7518e8 --- /dev/null +++ b/kb/armory/General/validator-*validate.halvalidator-detected-a-fatal-error.md @@ -0,0 +1,15 @@ +--- +title: Validator *validate.halValidator detected a fatal error +--- + +## Issue +After upgrading Operator to version 1.4.1+ and Halyard to 1.12.0+, the following errors are seen on the Operator logs. It should be noted there were no changes to the Canary configs before and after the upgrade. +``` +{"level":"info","ts":.....,"logger":"spinvalidate","msg":"Validator *validate.halValidator detected a fatal error"} +{"level":"error","ts":...,"logger":"spinvalidate","msg":"\nSpinnakerService validation failed:\njson: cannot unmarshal number into Go struct field PrometheusCanaryServiceIntegration.metadataCachingIntervalMS of type bool\n +``` +Despite the removal of the field ```metadataCachingIntervalMS``` from the canary configs, the error remains in the Operator logs. + +## Cause +Armory Engineering is aware of the issue. This has been identified to be an issue while parsing the configuration for Prometheus within Operator. It is a bug introduced when migrating the validators to operator. + diff --git a/kb/armory/General/viewing-pipelines-as-json-and-developing-dinghy-json-code.md b/kb/armory/General/viewing-pipelines-as-json-and-developing-dinghy-json-code.md new file mode 100644 index 0000000000..e10f73499e --- /dev/null +++ b/kb/armory/General/viewing-pipelines-as-json-and-developing-dinghy-json-code.md @@ -0,0 +1,15 @@ +--- +title: Viewing Pipelines as JSON and Developing Dinghy JSON Code +--- + +## Introduction +Viewing pipelines as JSON is a useful method to look at the actual code defined for a particular pipeline.  It can also be used in troubleshooting and providing Armory Support a copy of the Pipeline structure.Customers may also find it useful to look at a standard Pipeline JSON before defining it in Dinghy, as the code can mostly be lifted straight from the Pipeline JSON + +## Prerequisites +N/A + +## Instructions +Customers can easily find the Pipeline JSON code by doing the following +* Access Spinnaker through the Console UI* Navigate to the Pipeline in question and click on it* Click on ```Pipeline Actions``` in the Upper left corner* Click on ```Edit as JSON``` or ```Show JSON``` depending on if the pipeline is locked + + diff --git a/kb/armory/General/vpcs-and-subnets-not-populating-in-the-drop-down-menu-in-the-ui.md b/kb/armory/General/vpcs-and-subnets-not-populating-in-the-drop-down-menu-in-the-ui.md new file mode 100644 index 0000000000..d489a4be12 --- /dev/null +++ b/kb/armory/General/vpcs-and-subnets-not-populating-in-the-drop-down-menu-in-the-ui.md @@ -0,0 +1,31 @@ +--- +title: VPCs and subnets not populating in the drop down menu in the UI +--- + +## Issue +A Role for Spinnaker is created to assume with ```PowerUserAccess``` permission and ```iam:ListServerCertificates``` & ```iam:PassRole``` actions in AWS. +In the account that Spinnaker resides in (i.e., the AWS account that owns the EKS cluster where Spinnaker is installed), an IAM Policy is created with permissions to assume all of the Managed Roles. An example of the role arn that is created in the Target Account has the following permissions +``` +{ + "Version": "2012-10-18", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": [ + "elasticloadbalancing:...", + "iam:PassRole", + "ec2:*" + ], + "Resource": "*" + } + ] +} +``` +An AWS user is created that Spinnaker would use authenticate with the AWS Account. The trust relationships are edited to allow the created user entity to assume the role. +It is confirmed that the IAM Roles and Policies have been configured correctly. The trusted relationship is given to the Spinnaker role arn. However, despite this, the Spinnaker UI does not display the VPC subnets and other drop down entries of the AWS Account.  + +## Cause +The accounts are being picked up by Spinnaker but, the VPCs and subnets aren't populating in the drop down menu in the UI due to the VPC and Subnets not being tagged correctly.  +The following Github issue explains the reason for tagging in more detail: [https://github.com/spinnaker/spinnaker/issues/5768](https://github.com/spinnaker/spinnaker/issues/5768) + diff --git a/kb/armory/General/vulnerability-management-policy.md b/kb/armory/General/vulnerability-management-policy.md new file mode 100644 index 0000000000..94433ec2a9 --- /dev/null +++ b/kb/armory/General/vulnerability-management-policy.md @@ -0,0 +1,47 @@ +--- +title: Vulnerability Management Policy +--- + + +*This document was derived directly from the official Armory Vulnerability Management Policy and Vulnerability Standard Operating procedure. A more recent version may exist. Contact Armory for the latest official version.* +### Overview +Armory recognizes the importance of finding and addressing software defects for ourselves, our customers, and the Spinnaker community. Our vulnerability management functions are key to identifying risks in our software and systems. +### Purpose +Security vulnerabilities are inherent in computing systems and applications. Armory must limit the risk associated with the exploitation of published vulnerabilities by implementing a vulnerability management system in an effective, systematic, and repeatable manner with measurements taken to confirm its success. This policy provides guidance for implementing vulnerability and patch management protocols. +### Scope +The information contained herein applies to automated and manual systems including those managed or hosted by third parties on behalf of Armory, Inc. (the “Company”). This policy pertains to Information Assets, regardless of form or format, used in support of the business and applies to employees, consultants, agents, vendors, and other independent contractors who manage or use the Company’s Information Assets. Collectively these resources shall be referred to as “Personnel” or individually as “Users”. +### Policy +Armory shall deploy a repeatable, continuous process to identify and mitigate security vulnerabilities within systems, applications, and networks. This implementation must include a process to identify, evaluate, and incorporate system updates and patches to ensure known security weaknesses are addressed in a timely and effective manner. Only trained administrators shall update production systems once technical solutions are authorized by the appropriate level of management. +### Approved Tools & Scan Frequencies +Armory currently uses Aqua Security to scan images for vulnerabilities. At this time, AquaSec scans are the only scans that Armory accepts, as we have the capability of rerunning scans to ensure the resolution of identified vulnerabilities.   +Armory runs Aqua Security scans on code throughout the continuous integration (CI) process and scans are run for all new releases. +### Third Party Reports +We do not accept scan reports generated by third parties, including customers, at this time. Upon request, Armory will provide the AquaSec scan results from the most recent release, pending a signed NDA or confidentiality agreement.  +Unregistered or zero-day vulnerabilities identified by third parties should be sent to [security@armory.io](mailto: security@armory.io) for triage and response. Whenever possible, [security@armory.io](mailto: security@armory.io)​ should be included in any correspondence regarding a reported vulnerability. +### **Prioritization** +Newly detected vulnerabilities are evaluated using the [Common Vulnerability Scoring System Calculator](https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator) to recalculate the CVSS score based on applicable environmental and temporal considerations. Vulnerabilities are then assessed against our defined vulnerability taxonomy to determine the severity. High findings are evaluated by Armory Engineers for applicability. For applicable high-severity CVEs, a ticket is created, and the CVE is tracked through resolution. +### Vulnerability Taxonomy +Armory uses the Common Vulnerability Scoring System (CVSS 3.0) for all Common Vulnerabilities and Exposures (CVE) provided by the National Vulnerability Database to determine the severity of vulnerabilities.  +SeverityDescriptionHigh +CVSS 8.0 - 10.0, ​and​ may be readily compromised with publicly available malware or exploits. +Medium +CVSS 6.0 - 7.9,​ is not​ actively exploited, ​and​ no known exploit has been made publicly available. +Low +CVSS 0.0 - 5.9, ​and​ may be mitigated with reasonable efforts, ​is​ mitigated through other established controls, or​ is unable to be mitigated due to normal operations. + +### Disclosure +Public disclosure and customer notification will be coordinated by the account managers and customer success teams as necessary.  +**Note:​ **If a new CVE record needs to be created, Armory will coordinate these efforts with the Spinnaker Security SIG. +### Limitations +Armory employees are not allowed to conduct scans or penetration tests of Armory systems that they are not directly responsible for. Third parties like AWS may have additional instructions which should be followed prior to performing any security tests against these systems. +### Revision History +**Date****Description****Author**2023-06-29Updated and Reviewed +Shannon Smith, Security Compliance Manager +2022-03-01Updated and Reviewed +Shannon Smith, Security Compliance Manager +2021-04-01Updated and Reviewed +Shannon Smith, Security Compliance Manager +2021-04-01Reviewed and Approved +Andrew Backes, Head of Engineering + + diff --git a/kb/armory/General/waitforclustershrinktask-of-stage-shrinkcluster-times-out-after-30-minutes.md b/kb/armory/General/waitforclustershrinktask-of-stage-shrinkcluster-times-out-after-30-minutes.md new file mode 100644 index 0000000000..f10d153a51 --- /dev/null +++ b/kb/armory/General/waitforclustershrinktask-of-stage-shrinkcluster-times-out-after-30-minutes.md @@ -0,0 +1,16 @@ +--- +title: WaitForClusterShrinkTask of stage shrinkCluster times out after 30 minutes +--- + +## Issue +An environment may experience pipeline fails at a deploy stage with with the following error message: +```WaitForClusterShrinkTask of stage shrinkCluster timed out after 30 minutes 5 seconds. pausedDuration: 0 seconds, elapsedTime: 30 minutes 5 seconds, timeoutValue: 30 minutes``` +Upon checking Clouddriver logs, the following error can be identified: +```021-05-03 11:05:16.863 ERROR 1 --- [0.0-7002-exec-4] n.s.f.s.FiatAccessDeniedExceptionHandler : Encountered exception while processing request GET:/......``` +  + +## Cause +This issue can be traced back to timeout configurations. For example, in this instance the timeouts for Clouddriver had been set at 30 minutes. +  +  + diff --git a/kb/armory/General/what-spinnaker-services-can-be-affected-apply-opa-policy.md b/kb/armory/General/what-spinnaker-services-can-be-affected-apply-opa-policy.md new file mode 100644 index 0000000000..d69f09b8f8 --- /dev/null +++ b/kb/armory/General/what-spinnaker-services-can-be-affected-apply-opa-policy.md @@ -0,0 +1,18 @@ +--- +title: What Spinnaker Services can be Affected Apply OPA Policy +--- + +## Introduction +When designing OPA policies, customers can write a policy to affect a Spinnaker Service, but it cannot evaluate and provide feedback for that service. +This is because OPA policies can only be applied to some core Spinnaker services + +## Prerequisites +OPA and Policy Engine should be enabled. [https://docs.armory.io/armory-enterprise/armory-admin/policy-engine/policy-engine-enable/policy-engine-plug-enable/](https://docs.armory.io/armory-enterprise/armory-admin/policy-engine/policy-engine-enable/policy-engine-plug-enable/) + +## Instructions +Policies for Policy Engine can be enabled for the following services: +* Gate* Front50* CloudDriver* Orca +OPA allows admins to set rules and policies and evaluate whether the policy is met or not on a Pass-Fail basis.  +As an example, a policy applied on Front50 will affect whether a pipeline can be saved.   For more Policy Engine information, please visit our Docs site +[https://docs.armory.io/armory-enterprise/armory-admin/policy-engine/policy-engine-use/example-policies/](https://docs.armory.io/armory-enterprise/armory-admin/policy-engine/policy-engine-use/example-policies/) + diff --git a/kb/armory/General/what-to-consider-when-using-terraformer-with-the-provisioner,-local-exec.md b/kb/armory/General/what-to-consider-when-using-terraformer-with-the-provisioner,-local-exec.md new file mode 100644 index 0000000000..2cbfa0c879 --- /dev/null +++ b/kb/armory/General/what-to-consider-when-using-terraformer-with-the-provisioner,-local-exec.md @@ -0,0 +1,16 @@ +--- +title: What to Consider When Using Terraformer with the Provisioner, Local-Exec +--- + +## Introduction +In a local environment, users would usually include the ```local-exec``` provisioner to run operations on their local machine.  +[https://www.terraform.io/docs/provisioners/local-exec.html](https://www.terraform.io/docs/provisioners/local-exec.html) +This is usually because they have other operations like doing a Git Pull Request from a repository, or running an **Ansible / Chef / Puppet** script from the same system.  In a Spinnaker environment, this needs to be adjusted + +## Prerequisites +N/A + +## Instructions +Since most users/customers run their Spinnaker environment either in Kubernetes (e.g. EKS) or in containerized services (e.g. ECS) in the cloud, it will mean a change needs to be made to the scripts that are pre-existing. Because the Terraformer service is not running on an instance like EC2 or a VM where additional modules can be installed, users should instead look to leverage the Terraform provisioner, ```remote-exec```*. *[https://www.terraform.io/docs/provisioners/remote-exec.html](https://www.terraform.io/docs/provisioners/remote-exec.html)  Since ```local-exec``` invokes a local executable, it would in essence be executing the command on the container/pod, which is local to Terraformer. As an example re-working the Terraform Code and using Spinnaker, instead of running the Git Pull Request and subsequent Ansible / Chef / Puppet execution on the same container with ```local-exec,``` a separate instance should be set up to execute what is required. +One possible way would be to bake and then deploy an instance in Spinnaker with the services you need installed and then use that instance in Terraformer's ```remote-exec``` provisioner to execute the reset of your Terraform Code.While it may be possible to add services to your existing Pod/Container, in general, Armory does not recommend for our customers to make modifications to add additional services to the preexisting Spinnaker containers/pods as they can be removed upon upgrade or re-deployment. + diff --git a/kb/armory/General/when-deploying-ecs,-subnets-and-vpc-will-not-populate-in-the-ui.md b/kb/armory/General/when-deploying-ecs,-subnets-and-vpc-will-not-populate-in-the-ui.md new file mode 100644 index 0000000000..2f84993a31 --- /dev/null +++ b/kb/armory/General/when-deploying-ecs,-subnets-and-vpc-will-not-populate-in-the-ui.md @@ -0,0 +1,11 @@ +--- +title: When deploying ECS, Subnets and VPC will not populate in the UI +--- + +## Issue +An organization using ECS as a deployment target may run into an issue where, when creating a deployment, the Subnets and Load Balancers defined will not populate in the drop-down menus of Spinnaker.  + +## Cause +There is a known issue with ECS populating Subnet and Load Balancer information upon the creation of a deployment. +This issue is due to a bug. Armory is looking into this issue and will update this article and release notes when a fix is completed.  + diff --git a/kb/armory/General/when-using-aws-iam-authenticator,-fail-to-read-namespaces-and-failing-to-deploy-to-k8s-clusters.md b/kb/armory/General/when-using-aws-iam-authenticator,-fail-to-read-namespaces-and-failing-to-deploy-to-k8s-clusters.md new file mode 100644 index 0000000000..2a631f6ebb --- /dev/null +++ b/kb/armory/General/when-using-aws-iam-authenticator,-fail-to-read-namespaces-and-failing-to-deploy-to-k8s-clusters.md @@ -0,0 +1,18 @@ +--- +title: When using aws-iam-authenticator, fail to read namespaces and failing to deploy to k8s clusters +--- + +## Issue +For Kubernetes installations that use ```aws-iam-authenticator``` for authenticating to the Kubernetes cluster, it has been noticed that upgrading Spinnaker version from **v2.21.4** to **v2.23.5 (or later)** results in Spinnaker unable to deploy across namespaces. Exceptions similar to below are seen on the Clouddriver logs. + +``` +2021-03-26 13:37:17.325 ERROR 1 --- [ Thread-95] c.n.s.c.k.s.KubernetesCredentials : Could not list namespaces for account XXXXXX: Failed to read [namespace] from : error: You must be logged in to the server (Unauthorized) +2021-03-26 13:37:17.455 ERROR 1 --- [ Thread-96] c.n.s.c.k.s.KubernetesCredentials : Could not list namespaces for account XXXXXX Failed to read [namespace] from : error: You must be logged in to the server (Unauthorized) +2021-03-26 13:37:20.010 ERROR 1 --- [ Thread-97] c.n.s.c.k.s.KubernetesCredentials : Could not list namespaces for account XXXXXXXXXX: Failed to read [namespace] from : Error from server (Forbidden): namespaces is forbidden: User "system:node:XXXXXXXX" cannot list resource "namespaces" in API group "" at the cluster scope +2021-03-26 13:37:55.067 ERROR 1 --- [ Thread-98] c.n.s.c.k.s.KubernetesCredentials : Could not list namespaces for account XXXXXXXX: Failed to read [namespace] from : Error from server (Forbidden): namespaces is forbidden: User "system:node:XXXXXXXX" cannot list resource "namespaces" in API group "" at the cluster scope +``` + +## Cause +It has been identified that the Clouddriver version **2.21.4** has been using a higher version of ```aws-iam-authenticator``` (0.5.0+) when compared to the version 2.23.5. The version of ```aws-iam-authenticator``` has been downgraded from 0.5.0 to 0.4.0 to overcome the latency issues that Clouddriver has when invoking a call to the Kubernetes cluster.   The complete release notes has can be found at: [https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-23-0/#armory-clouddriver---2221122324](https://docs.armory.io/docs/release-notes/rn-armory-spinnaker/armoryspinnaker_v2-23-0/#armory-clouddriver---2221122324) + + diff --git a/kb/armory/General/where-does-spinnaker-begin-and-jenkins-end.md b/kb/armory/General/where-does-spinnaker-begin-and-jenkins-end.md new file mode 100644 index 0000000000..4e51a8789a --- /dev/null +++ b/kb/armory/General/where-does-spinnaker-begin-and-jenkins-end.md @@ -0,0 +1,10 @@ +--- +title: Where Does Spinnaker Begin and Jenkins End +--- + + +### Question: +Where does Spinnaker begin and Jenkins end? +### Answer: +Spinnaker is not a build server. It was never designed that way. While in theory Spinnaker could implement its own service that acts as a build server, there is no need to do this. Spinnaker takes advantage of the existing Jenkins ecosystem and uses Jenkins behinds the scenes. Spinnaker was designed from the ground up to be a cloud deployment tool combining Continuous Integration and Continuous Delivery. This means that instead of just executing arbitrary tasks, it has first class support for cloud concepts. + diff --git a/kb/armory/General/while-disabling-policy-engine,-orca-continually-advises-that-it-cannot-communicate-with-opa-server.md b/kb/armory/General/while-disabling-policy-engine,-orca-continually-advises-that-it-cannot-communicate-with-opa-server.md new file mode 100644 index 0000000000..e9e015ce54 --- /dev/null +++ b/kb/armory/General/while-disabling-policy-engine,-orca-continually-advises-that-it-cannot-communicate-with-opa-server.md @@ -0,0 +1,12 @@ +--- +title: While Disabling Policy Engine, Orca Continually Advises that it Cannot Communicate with OPA Server +--- + +## Issue +When disabling and removing Policy Engine, the following error continues to occur in Orca logs. There are however, no issues with deployments or executions of pipelines +2020-11-05 07:36:17.726 ERROR 1 --- [ handlers-15] io.armory.spinnaker.opa.PolicyEnforcer : [johnsmith@abc.com] Policy Error: Unable to communicate with OPA Server +[...] + +## Cause +Not all configurations were removed, and so references still exist to check for the OPA server. + diff --git a/kb/armory/General/while-using-armory-arm-cli,-cannot-refer-to-pipelineid.md b/kb/armory/General/while-using-armory-arm-cli,-cannot-refer-to-pipelineid.md new file mode 100644 index 0000000000..736125834f --- /dev/null +++ b/kb/armory/General/while-using-armory-arm-cli,-cannot-refer-to-pipelineid.md @@ -0,0 +1,15 @@ +--- +title: While using Armory ARM-CLI, Cannot Refer to PipelineID +--- + +## Issue +While using Armory ARM-CLI (```arm dinghy render```) to perform an evaluation of the DinghyFile offline, if the DinghyFile contains a ```pipelineId``` function, it produces the following error: +level=error msg=Failed to execute buffer +... +level=error msg=Parsing dinghyfile failed +.... +error calling pipelineID: runtime error: invalid memory address or nil pointer dereference + +## Cause +```pipelineID``` lookup was not implemented before ARM-CLI 1.6.0 + diff --git a/kb/armory/How To's/how-to-add-a-custom-ca-for-operator-and-vault.md b/kb/armory/How To's/how-to-add-a-custom-ca-for-operator-and-vault.md new file mode 100644 index 0000000000..d0b502cbcd --- /dev/null +++ b/kb/armory/How To's/how-to-add-a-custom-ca-for-operator-and-vault.md @@ -0,0 +1,91 @@ +--- +title: How to add a custom CA for Operator and Vault +--- + +## Introduction +When Spinnaker attempts to fetch a Vault token, you encounter the following error: + +```error fetching vault token - error logging into vault using kubernetes auth: Put https://spinnaker-vault.vault.svc.cluster.local:8200/v1/auth/kubernetes/login: x509: certificate signed by unknown authority``` +```This happens because you are using a custom CA and the Operator.``` + + +## Prerequisites +Access to the relevant certificates in PEM format.   + +## Instructions +Resolving this issue involves the following steps: +* Copy the original ```cert.pem``` file from ```/etc/ssl/cert.pem``` +* Append the custom ca for Vault +* Mount the ```cert.pem``` similar to the ```/etc/ssl/certs/java/cacerts``` + +## Copy the System trust/key store +Create a temporary copy of the system’s trust/key store and import your internal certificate. If you’re on a Mac, the Java truststore will be located at ```$(/usr/libexec/java_home/)/jre/lib/security/cacerts```. It will be different on Linux. + +``` +$ mkdir /tmp/custom-trust-store +$ cp {path-to-cacerts} /tmp/custom-trust-store +$ keytool import -alias custom-ca -keystore /tmp/custom-trust-store/cacerts -file {your-internal-certificate} +``` + +### Halyard + +Make a copy of ```/etc/ssl/cert.pem``` and append the internal certificate to your copy of ```cert.pem```. *Note: If you do not have a copy of the internal certificate - you can run the following command to retrieve it manually:* +```openssl s_client -showcerts -connect host:port |openssl x509 -outform PEM >mycertfile.pem``` +Use ```kubectl``` to create a Secret from your copy of ```cacerts``` or ```cert.pem```. +```$ kubectl create secret generic -n {your-spinnaker-namespace} internal-trust-store --from-file /tmp/custom-trust-store/cacerts``` + + +### Operator + +Copy ```cert.pem``` from the Operator to a local directory: + +```kubectl cp -n spinnaker-operator -c spinnaker-operator spinnaker-operator-xxxxx:etc/ssl/cert.pem .``` + +Append your custom cert to ```cert.pem```: + +```cat myCA.crt >> cert.pem``` + +Append any other additional certs: + +```cat myVault.crt >> cert.pem``` + +Create a Kubernetes secret for ```cert.pem```: + + +```kubectl create secret generic spinop-certs -n spinnaker-operator --from-file cert.pem``` + + +This secret then gets mounted into Spinnaker Operator. + +## Mounting Secrets to the Operator deployment +Add volume and volume mounts to Spinnaker Operator: + +``` +spec: + serviceAccountName: spinnaker-operator + containers: + - name: spinnaker-operator + image: armory/armory-operator:0.4.0 + ... + volumeMounts + - mountPath: /etc/ssl + name: internal-cert-pem + - name: halyard + image: armory/halyard-armory:operator-0.4.x + imagePullPolicy: IfNotPresent + ... + volumeMounts: + - mountPath: /etc/ssl/certs/java + name: internal-trust-store + volumes: + - name: internal-trust-store + secret: + defaultMode: 420 + secretName: internal-trust-store + - name: internal-cert-pem + secret: + defaultMode: 420 + secretName: internal-cert-pem +``` + + diff --git a/kb/armory/How To's/how-to-add-a-custom-smtp-server-to-send-notifications.md b/kb/armory/How To's/how-to-add-a-custom-smtp-server-to-send-notifications.md new file mode 100644 index 0000000000..d9a65c5e45 --- /dev/null +++ b/kb/armory/How To's/how-to-add-a-custom-smtp-server-to-send-notifications.md @@ -0,0 +1,38 @@ +--- +title: How to Add a Custom SMTP Server to Send Notifications +--- + +## Introduction +The [spinnaker.io](http://spinnaker.io/) docs go over basic configuration for how to setup notifications within Spinnaker however, it does not cover the use case of using a custom SMTP server and how to configure it. + +## Prerequisites +Have an existing SMTP server to use to send notifications. + +## Instructions +The following provides examples about adding the modifications to the deployment manifest in Halyard and Operator1. To enable notifications, add the below snippet to your ```echo.yml``` file if you are using Halyard. In Operator, this should go in the ```profiles-patch.yml``` file under the echo section.Note that the properties section of your server configuration should match your server set up.  These can usually be found from an administrator or from the information from the public site.  Below are some examples for gmail.In addition, if the SMTP server does not have a user for authentication, **the auth option needs to be set to false  and the username/password fields should be left blank or removed**. +``` + mail: + enabled: true + from: # Name of the email + spring: + mail: + host: # Typically this is something like smtp.gmail.com but, an IP address will also work + username: # Enter the SMTP server's user email + password: # Enter the SMTP server's user password + port: 587 # Match the port depending on the protocol + properties: + mail: + smtp: + auth: true + starttls: + enable: true + transport: + protocol: smtp + debug: true +``` +2. Add the below snippet to the ```settings-local.js``` file +``` + window.spinnakerSettings.notifications = window.spinnakerSettings.notifications || {}; + window.spinnakerSettings.notifications.email = window.spinnakerSettings.notifications.email || {}; + window.spinnakerSettings.notifications.email.enabled = true; +``` diff --git a/kb/armory/How To's/how-to-add-additional-users-to-an-organization.md b/kb/armory/How To's/how-to-add-additional-users-to-an-organization.md new file mode 100644 index 0000000000..52043d5fe7 --- /dev/null +++ b/kb/armory/How To's/how-to-add-additional-users-to-an-organization.md @@ -0,0 +1,23 @@ +--- +title: How to Add Additional Users to an Organization +--- + +## Introduction +Customers may want to invite additional users to deploy in their environment(s).  In order to do so, they will need to first complete their [initial registration](https://docs.armory.io/cd-as-a-service/setup/get-started/#how-to-get-started-using-armory-cd-as-a-service).  It is recommended that customer consider completing an initial set up before adding additional users + +## Prerequisites +One person from the team should initiate the [registration for an account](https://docs.armory.io/cd-as-a-service/setup/get-started/#how-to-get-started-using-armory-cd-as-a-service).  Once they have done so, they will be able to add all additional users to the cloud account for the environment(s). +  + +## Instructions +Once one user has registered, then navigate to the 'Users' section of the configuration tab, and 'invite' other users to their organization. +* Sign in to the [Armory Cloud Console](https://console.cloud.armory.io/) +* Click on configuration in the top header -> Under Access Management in the left Navigation, click on Users -> Click on Invite Users in the Upper Right corner +* A screen will pop open asking to fill in the ```Name``` and ```Email``` fields for the person that you are inviting +* The person invited will receive an email invitation.  If they do not receive the invite, we suggest that the user should check their ```spam/junk folders```, for the missing email.   +* The new user will then need to follow up the sign up instructions to gain access to the same environment +  +Note: It is recommended that customers from the same organization plan their structure carefully.  Armory cannot merge two organizations together once they are separated.  For example, if two teams within the same organization sign up for two separate CDaaS organization accounts, they ***will not be able to merge the organizations at a later date.*** They would have to in effect, delete one organization and add the existing environments and users to the remaining organization.   + +Note 2: Armory cannot manage the users within a CDaaS account without expressed written confirmation and consent from the organization in question. This may result in a  confirmation from our employees by phone or Zoom call.   + diff --git a/kb/armory/How To's/how-to-audit-for-empty-roles-in-fiat-using-redis-mysql.md b/kb/armory/How To's/how-to-audit-for-empty-roles-in-fiat-using-redis-mysql.md new file mode 100644 index 0000000000..fccff14dcd --- /dev/null +++ b/kb/armory/How To's/how-to-audit-for-empty-roles-in-fiat-using-redis-mysql.md @@ -0,0 +1,91 @@ +--- +title: How to Audit for Empty Roles in FIAT using Redis/MySQL +--- + +## Introduction +Administrators may be interested in the information in FIAT for some auditing of the roles that have been defined.  This process can sometimes help Administrators when they would like to do a quick check to ensure that roles are defined according to their policy and identify Empty Roles that may be in FIAT. +The process is much more streamlined when using MySQL, but administrators can use the default Redis storage and perform a search. + +## Prerequisites +Authentication and Authorization should be set up in the Spinnaker environment:[https://spinnaker.io/docs/reference/architecture/authz_authn/](https://spinnaker.io/docs/reference/architecture/authz_authn/) +Access to the MySQL Database to make queries. (Optional) +Access to the Redis Database to make queries. +Armory CDSH 2.27.2+ + +## Instructions +Administrators should consider configuring MySQL as a backend for FIAT for better performance than alternatives and provides ease of search. +[MySQL Process](#mysql) | [Redis Process](#redis) + +### MySQL Process +#### Setting Up FIAT MySQL Backend +Administrators can set the following for the FIAT profile in their SpinnakerConfig: +``` +spec: + spinnakerConfig: + profiles: + fiat: + permissionsRepository: + redis: + enabled: false + sql: + enabled: true + sql: + enabled: true + connectionPools: + default: + # additional connection pool parameters are available here, + # for more detail and to view defaults, see: + # [https://github.com/spinnaker/kork/blob/master/kork-sql/src/main/kotlin/com/netflix/spinnaker/kork/sql/config/ConnectionPoolProperties.kt](https://github.com/spinnaker/kork/blob/master/kork-sql/src/main/kotlin/com/netflix/spinnaker/kork/sql/config/ConnectionPoolProperties.kt) + default: true + jdbcUrl: jdbc:mysql:// + user: + password: + migration: + jdbcUrl: jdbc:mysql:// + user: + password: + redis: + enabled: false +``` +The ```fiat_user``` and ```fiat_permission``` tables will then show up in the MySQL DB under the table ```fiat```, along with several other tables. +  +### Querying for Empty Roles in MySQL +* Administrators can then log in to the database and run +```select * from fiat.fiat_user where id not in (select fiat_user_id from fiat.fiat_permission);``` +* Once queried, customers can make changes to the users so that roles are assigned to the flagged users.* It is then recommended that Administrators run the check in the database again.  + +### Redis Process +#### Querying for Empty Roles in Redis +Looking at entries for Empty Roles in Redis can be a little more complicated, but there is a reduction in setup time +Locate the Redis key with the user list, with `````` being the URL (e.g., templocation-redis.tc9ztb.ng.0001.euw1.cache.amazonaws.com)and `````` +```redis-cli -h -p smembers "spinnaker:fiat:users"``` + +Locate the Redis role keys as well, with `````` being the URL (e.g., templocation-redis.tc9ztb.ng.0001.euw1.cache.amazonaws.com)and `````` +```redis-cli -h -p keys spinnaker:fiat:permissions*:*:roles``` +Note that we used a wildcard to cover the instance where v1 and v2 permissions exist, such as: +``` +"spinnaker:fiat:permissions:USER1:roles" +"spinnaker:fiat:permissions-v2:USER2:roles" +``` + Truncate, unique, sort, and diff the list to find a list of users and roles.  For example +``` +"test.abc.io" +"dacccff0-c022-4567-b9ab-19b9356d102c@temp-service-account" +"*" +"082555d5-703c-4233-ad39-2031b33a6c41@temp-service-account" +"__unrestricted_user__" +"ab64440d-3806-4286-ab5b-28843ccaac3e@temp-service-account" +``` + +Use that list to add appropriate roles to the users/service accounts.  It is possible to check the accounts lists to see if they are being used in any pipelines or the environment +Administrators can find if one of the service accounts is being used by curling the background and comparing the returned list with the information gathered from the Redis queries above + +```curl -X GET -k https://127.0.0.1:8080/serviceAccounts |jq``` + +Administrators can also query their Front50 to see if any pipelines are using the flagged users.  + It is recommended for [large-scale and production environments that administrators use Front50 with MySQL for persistent storage](https://spinnaker.io/docs/setup/productionize/persistence/front50-sql/#why-sql).  + The query customers can run is: +```select id, name, json_extract(json_extract(body, '$.triggers[0]'), '$.runAsUser') from pipelines;``` + +* Once the changes have been made, it is recommended to re-verify that all the ```spinnaker:fiat:permissions*:*:roles``` keys are not empty + diff --git a/kb/armory/How To's/how-to-change-spinnaker's-login-session-timeout.md b/kb/armory/How To's/how-to-change-spinnaker's-login-session-timeout.md new file mode 100644 index 0000000000..7b22ab4395 --- /dev/null +++ b/kb/armory/How To's/how-to-change-spinnaker's-login-session-timeout.md @@ -0,0 +1,21 @@ +--- +title: How to Change Spinnaker's Login Session Timeout +--- + +## Introduction +While using Spinnaker, some users may want to extend the login sessions. The logins are controlled by a Gate property (*server.session.timeout-in-seconds*). By default the property to 3600 seconds which is an hour.  + +## Prerequisites +N/A + +## Instructions +1. Edit the gate-local.yml located in the Halyard Profile config yaml (please create if it is not already created):```**.hal//profiles/gate-local.yml**```In operator, in your configuration yaml (e.g. SpinnakerService.yml) at:```**spec.spinnakerConfig.profiles.gate**``` +e.g. +``` +spec: + spinnakerConfig: + profiles: + gate: +``` +2. Add the property and the new value in seconds```***server.session.timeout-in-seconds: ***```3. Save changes and redeploy Spinnaker  + diff --git a/kb/armory/General/how-to-cleanup-old-executions-in-orca-mysql-database.md b/kb/armory/How To's/how-to-cleanup-old-executions-in-orca-mysql-database.md similarity index 100% rename from kb/armory/General/how-to-cleanup-old-executions-in-orca-mysql-database.md rename to kb/armory/How To's/how-to-cleanup-old-executions-in-orca-mysql-database.md diff --git a/kb/armory/General/how-to-cleanup-rosco-bakes-older-than-specific-days-from-redis.md b/kb/armory/How To's/how-to-cleanup-rosco-bakes-older-than-specific-days-from-redis.md similarity index 100% rename from kb/armory/General/how-to-cleanup-rosco-bakes-older-than-specific-days-from-redis.md rename to kb/armory/How To's/how-to-cleanup-rosco-bakes-older-than-specific-days-from-redis.md diff --git a/kb/armory/How To's/how-to-debug-missing-agent-k8s-accounts.md b/kb/armory/How To's/how-to-debug-missing-agent-k8s-accounts.md new file mode 100644 index 0000000000..1d280dec7a --- /dev/null +++ b/kb/armory/How To's/how-to-debug-missing-agent-k8s-accounts.md @@ -0,0 +1,47 @@ +--- +title: How to debug missing Agent K8s accounts +--- + +## Introduction +This document is addressing a recurring problem with our customers for a while now. +With multiple root causes possible, the goal of this document is to underline the best practices when debugging the Armory Scale Agent issues for our customers. + +## Prerequisites +* Scale Agent configured.* Spinnaker configured with FIAT. + +## Instructions +Agent K8s accounts can be configured in two ways: +* with a kubeconfig file* with a service account +If the kubeconfig file is mangled/broken or the service account does not exist, the agent will not pick up the config, and the accounts will not appear in the UI. +A good starting point, if possible, is to emulate the connection to the K8s cluster with the kubeconfig file itself to validate the kubeconfig file via the CLI. +Kubeconfig files are not usually stored locally, so a prerequisite is to first get a copy from the location where it is stored (AWS S3, Secrets Manager, etc.) +```kubectl --kubeconfig /custom/path/kube.config get pods``` +This is a small differential test to check the validity of the Kubeconfig file.If the syntax of the kubeconfig file is not valid, the agent will not pick up the k8s account, and the above command should fail. +Alternatively, if the account is provisioned with a service account in FIAT, we can easily check the permissions and existence of the service accounts by following the steps described here: [Fiat Deep Dive](https://www.notion.so/Fiat-Deep-Dive-fa22c0ea51174585859310eed9b72b5c)If the above investigation is inconclusive, we can leverage the endpoints of the services starting from (1) Scale Agent, (2) Clouddriver and then (3) Gate +**Scale Agent:** +Access the below endpoint on the Scale Agent to see if the account is onboarded successfully on the agent. +``````curl -kv [http://localhost:8082/accounts/](http://localhost:8082/accounts/)`````` +```The output shall look similar to this:``` +**Clouddriver:** +If the account is found to be registered and available on the Scale Agent endpoint above, the next step is to check the endpoints exposed by the clouddriver agent plugins. The endpoint in the command below should help with listing the agents and the accounts these agents hold: +``````curl -kv [http://localhost:7002/armory/agents](http://localhost:7002/armory/agents)`````` +The output of the above API shall look something like below + +**Gate:**Once the accounts show up in the above endpoints, the next step is to check that Gate endpoints as shown below: +```GATE_URL/credentials``` +This will return a JSON with all the provisioned accounts successfully ingested. +Example: + +  +If the name of the Kubernetes account defined is not present, then the problem is at the configuration level. +If the account is present in the credentials endpoint but has the tag ```authorized: false``` then the problem is at the permissions level provided by OAuth for the current logged in user. +Usually the permissions for the account do not match the FIAT permissions on the service account. +To get a list of allowed accounts for the current user alongside more information about the current logged in user we can try to check the Gate endpoint: +```http://GATE_URL/auth/user``` +  +A sample user account endpoint JSON:  + +  +  +  + diff --git a/kb/armory/How To's/how-to-deploy-a-hotfix-version-for-a-particular-spinnaker-service.md b/kb/armory/How To's/how-to-deploy-a-hotfix-version-for-a-particular-spinnaker-service.md new file mode 100644 index 0000000000..c45ca320b0 --- /dev/null +++ b/kb/armory/How To's/how-to-deploy-a-hotfix-version-for-a-particular-spinnaker-service.md @@ -0,0 +1,40 @@ +--- +title: How to Deploy a Hotfix Version for a Particular Spinnaker service +--- + +## Introduction +The Spinnaker environment is up and running, but there is a need to deploy and install a customized (hotfix) service rather than upgrading the entire Spinnaker platform.This article advises about how to deploy a hotfix to particular Spinnaker services. + +## Prerequisites +Please plan ahead for any additional affects a hotfix may cause.  Please also note that the hotfix may not be completely be tested with compatibility for all Spinnaker versions, unlike a versioned release, which has gone through testing. + +## Instructions +In order to install the hotfix, the service image will need to be swapped out with the hotfix in *service-settings*. +#### In Halyard +For example, when using Halyard and in this example, a Dinghy hotfix needs to be installed, an entry will need to be made in the following service setting path (if the file doesn't already exist, please create it):  ```.``````hal//service-settings/dinghy.yml``` +artifactId: + +The `````` input parameter contains the docker image path with its tag name. For example: +```artifactId: armory/dinghy:2.21.4``` +After that, run the ```hal deploy apply ```command to apply the changes.For other particular services, please add different ```.yml``` files in the same service settings directory (```.hal//service-settings/```*```)```* and follow the same approach.e.g. ```clouddriver.yml```, ```kayenta.yml```, etc. +Once the affected pods boot up successfully, please verify it by using the ```kubectl describe pods ``` command to check if the container within this pod has the correct hotfix image version. +#### In Operator +In a similar scenario, when using Operator, please locate the `````` section in ```SpinnakerService.yml``` file, and add the ```artifactId``` YAML content of the corresponding service's service-setting, then apply the changes. +As an example, if making the change to Dinghy to match version ```2.21.4``` of the service, in ```spec.spinnakerConfig.service-settings``` +spec: + spinnakerConfig: + service-settings: + clouddriver: {} + deck: {} + dinghy: + artifactId: armory/dinghy:2.21.4 + echo: {} + fiat: {} + front50: {} + gate: {} + igor: {} + kayenta: {} + orca: {} + rosco: {} +Once the affected pods boot up successfully, please verify it by using the ```kubectl describe pods ``` command to check if the container within this pod has the correct hotfix image version. + diff --git a/kb/armory/How To's/how-to-deploy-encrypted-ebs-volumes.md b/kb/armory/How To's/how-to-deploy-encrypted-ebs-volumes.md new file mode 100644 index 0000000000..743b6cc7f7 --- /dev/null +++ b/kb/armory/How To's/how-to-deploy-encrypted-ebs-volumes.md @@ -0,0 +1,103 @@ +--- +title: How to deploy encrypted EBS volumes +--- + +## Introduction +Although no option currently exists in the UI to deploy encrypted EBS volumes (see [Cannot Define EBS Volumes When Deploying Baked AMI](https://support.armory.io/support?sys_kb_id=90991176dba0b81079f53ec8f4961907&id=kb_article_view&sysparm_rank=1&sysparm_tsqueryId=f4336852dbb3f010ac71c26c299619fe)), it is possible through the Deploy stage JSON. + +## Prerequisites +IAM permissions and appropriate role(s) to allow the use of AWS KMS. + +## Instructions +Please note that all rules regarding volume encryption (such as not being able to encrypt an already created EBS volume) follow AWS' encryption rules.  +### Scenario 1 - Deploy non-root encrypted EBS volumes +In this scenario we want to deploy an EC2 instance using a non-encrypted AMI whilst attaching an encrypted EBS non-root volume. The end result should look like this in the AWS console: + +First, go to AWS KMS and either generate a new key or grab the ID for one to use. +In the Deploy stage of a pipeline, click "Edit stage as JSON" and insert the ```blockDevices``` section as below: +``` +{ + "clusters": [ + { + "account": "myAwsAccount", + "application": "myApp", + "availabilityZones": { + "eu-west-1": [ + "eu-west-1a", + "eu-west-1b", + "eu-west-1c" + ] + }, + "blockDevices": [ + { + "deleteOnTermination": true, + "deviceName": "xvdb", + "encrypted": true, <---- + "kmskeyid": "KMS_KEY_ID", <---- + "size": 50, + "volumeType": "gp2" + } + ], + "capacity": { + "desired": 1, + "max": 1, + "min": 1 + }, + "cloudProvider": "aws", + + ...REST_OF_CONFIG... + + ], + "useAmiBlockDeviceMappings": false, <---- + } + ], + "name": "Deploy", + "type": "deploy" +} +``` +Ensure "encryption" is set to ```true``` and enter the KMS key ID from earlier in the ```kmskeyid``` field. ```useAmiBlockDeviceMappings``` should be set to ```false``` otherwise the instance will deploy with the unencrypted root volume but the encrypted volume will not be created. +Execute the deployment.  +  +**Scenario 2 - Deploy an EC2 instance with an encrypted root volume** +In this scenario we are creating an EC2 instance with an encrypted root volume only: + +**IMPORTANT**: the AMI being deployed must be encrypted, that is, ```sda1``` or the root volume should have the encrypted flag set in the AMI Block Device details. It is beyond the scope of this KB to go into how to do that, but for testing purposes a non-encrypted AMI was copied in the AWS console with the ```Encrypt target EBS snapshots``` option selected. The AMI details should like like this: + +The copied and now encrypted AMI is then referenced as the Base AMI in the Bake stage prior to deployment: + +This may not be necessary depending on how images are created and how deployments are run. When the deployment runs, the root volume is encrypted because the EBS snapshot from the AMI was also encrypted. +**NOTE**: the size, type etc. values are dictated by the AMI, and cannot be overwritten during the deployment. +  +**Scenario 3 - Deploy an EC2 instance with encrypted root and non-root EBS volumes** + +Combine scenarios 1 and 2 but with two caveats: +1. AWS will override whatever key put in the ```kmskeyid``` field in favor of the one used to encrypt the AMI initially. In this case the ```kmskeyid``` field does not even appear to be required and the deployment will complete by using the AMI encryption key.  +2. ```useAmiBlockDeviceMappings``` should be set to ```false```.  +  + +**Common Issues** + +*Volume stays in "attaching" status* +- typo in the KMS key ID or the incorrect key is being specified + +*The instance(s) launch with a non-encrypted root volume but my encrypted volume is not created or attached* +- set ```useAmiBlockDeviceMappings``` to ```false``` + +*```sda1``` specified to be encrypted but it is not* +- check the AMI being used. If ```sda1``` is not flagged as encrypted in the device details of the AMI, it will not be encrypted when deployed. Expected AWS behavior. + +**IMPORTANT**: ```useAmiBlockDeviceMappings``` = Spinnaker will use the block device mappings from the selected AMI when deploying a new server group. If instances are deploying but the EBS volumes are not being created try flipping this value. + +*When trying to deploy an encrypted non-root volume, Scaling Activity shows an error like "Launching a new EC2 instance. Status Reason: The device 'sda1' is used in more than one block-device mapping. Launching EC2 instance failed."* +- It may be that ```sda1``` was specified as the ```deviceName``` but have ```useAmiBlockDeviceMappings``` set to false. Change ```useAmiBlockDeviceMappings``` to true or use a different deviceName like ```xvdb```. Check the AMI Block Devices to see what is already being used as ```sda1``` will already be in use in most cases. + +*When trying to deploy an encrypted root volume from an encrypted AMI, Scaling Activity shows an error like "Launching a new EC2 instance. Status Reason: The device 'sda1' is used in more than one block-device mapping. Launching EC2 instance failed."* +- ```sda1``` is likely being used already from the AMI + +*When trying to deploy an encrypted non-root EBS value with an encrypted root volume, the instance is created with the encrypted root volume but the non-root-volume is not created.* +- ```useAmiBlockDeviceMappings``` should be set to false to add additional encrypted volumes else only the mappings on the AMI are used. + +*When trying to deploy an encrypted non-root EBS value with an encrypted root volume, the value for kmskeyid is ignored.* +- Believed to be expected behavior on the AWS side + + diff --git a/kb/armory/How To's/how-to-deploy-spinnaker-with-configurations-stored-in-configmaps-instead-of-secrets.md b/kb/armory/How To's/how-to-deploy-spinnaker-with-configurations-stored-in-configmaps-instead-of-secrets.md new file mode 100644 index 0000000000..bee30507b7 --- /dev/null +++ b/kb/armory/How To's/how-to-deploy-spinnaker-with-configurations-stored-in-configmaps-instead-of-secrets.md @@ -0,0 +1,67 @@ +--- +title: How To deploy Spinnaker with Configurations stored in Configmaps instead of Secrets +--- + +## Introduction +Up until now, Spinnaker configuration is stored by default as Kubernetes Secrets.  +In order to help some customers with compliance and to ensure that configurations are stored in the expected location, Armory has adjusted code to provide the option to store the configuration in Kubernetes ConfigMaps instead, for those deploying using  Armory Operator. +This article shows how to deploy Spinnaker with this option enabled. + +## Prerequisites +This feature is available starting in ```armory-operator: 1.8.0``` + +## Instructions +Below are the steps to enable the function +* Ensure your operator deployment is using the correct image  **armory-operator: 1.8.0** or greater ([https://hub.docker.com/r/armory/armory-operator/tags](https://hub.docker.com/r/armory/armory-operator/tags))Within the the halyard configmap named ```halyard-config-map```, set ```halyard.configSourceType``` to ```configMap``` instead of the default ```secret```. +data: + halyard.yml: | + halyard: + halconfig: + directory: /Users/spinnaker/.hal + configSourceType: configMap + spinnaker: + config: + input: + bucket: halconfig + region: us-west-2 + secrets: + vault: + enabled: false + url: https://vault.url + path: example + role: example + authMethod: KUBERNETES +  +Ensure that the configmap is mounted on the halyard pod +apiVersion: apps/v1 +kind: Deployment +spec: + replicas: 1 + selector: + matchLabels: + name: spinnaker-operator + template: + metadata: + creationTimestamp: null + labels: + name: spinnaker-operator + spec: + volumes: + - name: halyard-config-map + configMap: + name: halyard-config-map + defaultMode: 420 + containers: + - name: halyard + image: armory/halyard-armory:1.12.0-76b471c-operator + ports: + - containerPort: 8064 + protocol: TCP + resources: {} + volumeMounts: + - name: halyard-config-map + mountPath: /opt/spinnaker/config +#......manifest file is truncated for brevity + +If the Spinnaker environment is already deployed, it will need to be redeploy for the changes to take effect on the pods + diff --git a/kb/armory/How To's/how-to-determine-the-correct-amount-of-clouddriver-caching-agents-to-run.md b/kb/armory/How To's/how-to-determine-the-correct-amount-of-clouddriver-caching-agents-to-run.md new file mode 100644 index 0000000000..a5a3553356 --- /dev/null +++ b/kb/armory/How To's/how-to-determine-the-correct-amount-of-clouddriver-caching-agents-to-run.md @@ -0,0 +1,129 @@ +--- +title: How to determine the correct amount of Clouddriver Caching Agents to run +--- + +## Introduction +Having too many Caching Agents running at once with undersized resources can cause performance degradation but having too few can cause situations where items do not get cached at all due to a lack of caching cycles. + +This is a basic overview of how to determine the current count of running Caching Agents and identify if the count may be causing performance issues. +Below are the manual steps to execute to accomplish the task. Alternatively, the script at the end of the document can be used instead. + +## Prerequisites +* Ability to port-forward the Clouddriver pod - required* Access to the Operator Service Manifest or Halyard config files - optional + +## Instructions +  +Determine the number of running Clouddriver replicas from the Spinnaker configuration or via kubectl: +```NUM_REPLICAS=$(kubectl get deploy spin-clouddriver -o=jsonpath='{.spec.replicas}')``` +Port forward the Clouddriver pod and then curl the ```spinsvc``` introspection for the currently running agents: +```NUM_RUNNING_AGENTS=$(curl -k -s -XGET "http://spin-clouddriver:7002/cache/introspection" | jq '.[] | .id' | wc -l)``` +Open the ```SpinnakerConfig``` file and locate the ```max-concurrent-agents``` value under ```spec.spinnakerConfig.profiles.clouddriver.sql.agent```.The value should be reviewed.By default, 100 is the set value.  See our [Documentation for more information](https://docs.armory.io/docs/armory-admin/caching-agents-config/#sql-global-caching-agents-configuration). + +``` +spec: + spinnakerConfig: + profiles: + clouddriver: + sql: + agent: + max-concurrent-agents: +``` +Or from the CLI: +```max_concurrent_agents=$(kubectl get spinsvc -o jsonpath={'.items[] .spec.spinnakerConfig.profiles.clouddriver.sql.agent.max-concurrent-agents'})``` +Determine how many caching replicas are running via ```kubectl``` or tool of choice.   +```CACHING_AGENTS=$(( NUM_REPLICAS * max-concurrent-agents ))``` +For Example, 7 replicas with a ```max-concurrent-agents``` value of 1000 = 7000 caching agents. +```CACHING_UTILIZATION=$(( CACHING_AGENTS / NUM_RUNNING_AGENTS * 100 ))``` +* If, after taking a few readings throughout the day, this value is always high (***85%+***), the lack of caching agents may be causing a bottleneck. +These values need to be balanced to ensure optimum performance. +  +### Script +Paste the below script into a file and execute with the following options: +* ```./script.sh``` and provide inputs as requested, or* ```./script ``` + +``` +Click to expand code +#!/usr/bin/env bash + + +f_set_vars() { + ### Variables + GR='\e[32m' + RS='\e[0m' + + ### Port-forward variables for svc/spin-clouddriver + if [ $# -eq 3 ]; then + SERVICE_NAME="$1" + NAMESPACE="$2" + LOCAL_PORT="$3" + else + read -p "Enter SERVICE_NAME (default: spin-clouddriver): " INPUT_SERVICE_NAME + SERVICE_NAME="${INPUT_SERVICE_NAME:-spin-clouddriver}" + + read -p "Enter NAMESPACE (default: spinnaker): " INPUT_NAMESPACE + NAMESPACE="${INPUT_NAMESPACE:-spinnaker}" + + read -p "Enter LOCAL_PORT (default: 7002): " INPUT_LOCAL_PORT + LOCAL_PORT="${INPUT_LOCAL_PORT:-7002}" + fi +} + + +f_port_fwd() { + ### Port-forward Clouddriver in the background and get PID + kubectl port-forward -n ${NAMESPACE} svc/${SERVICE_NAME} ${LOCAL_PORT} & PORT_FORWARD_PID=$! + echo -e "PORT_FORWARD_PID is: \t[${GR}${PORT_FORWARD_PID}${RS}]" + + ### Allow the port-forward to complete + sleep 3 +} + + +f_get_val() { + ### Get current values from Clouddriver + NUM_REPLICAS=$(kubectl get deploy spin-clouddriver -o=jsonpath='{.spec.replicas}') + NUM_RUNNING_AGENTS=$(curl -k -s -XGET "http://localhost:7002/cache/introspection" | jq '.[] | .id' | wc -l | xargs) + concurrent_agents_setting=$(kubectl get spinsvc -o jsonpath='{.items[] .spec.spinnakerConfig.profiles.clouddriver.sql.agent.max-concurrent-agents}') + max_concurrent_agents=${concurrent_agents_setting:=100} + CACHING_AGENTS=$(( NUM_REPLICAS * max_concurrent_agents )) + + ### Calculate current utilization + CACHING_UTILIZATION=$(echo "scale=2; ${CACHING_AGENTS} / ${NUM_RUNNING_AGENTS} * 100" | bc) + + printf "\n˹-----------------------------------˺\n" + printf "|%-25s %-9s|\n" "Name" "| Value" + printf "˫-----------------------------------˧\n" + printf "|%-25s | ${GR}%-7d${RS}|\n" "NUM_REPLICAS" "${NUM_REPLICAS}" + printf "|%-25s | ${GR}%-7d${RS}|\n" "NUM_RUNNING_AGENTS" "${NUM_RUNNING_AGENTS}" + #printf "|%-25s | ${GR}%-7d${RS}|\n" "concurrent_agents_setting" "${concurrent_agents_setting}" + printf "|%-25s | ${GR}%-7d${RS}|\n" "max_concurrent_agents" "${max_concurrent_agents}" + printf "|%-25s | ${GR}%-7d${RS}|\n" "CACHING_AGENTS" "${CACHING_AGENTS}" + printf "|%-25s | ${GR}%-7.2f${RS}|\n" "CACHING_UTILIZATION" "${CACHING_UTILIZATION}" + printf "˻-----------------------------------˼\n" +} + + +cleanup() { + ### Cleanup function to kill port-forwarding PID and exit + # Kill the port-forwarding process + echo -e "\n\nRemoving port-forwarding for Clouddriver, PID ${GR}${PORT_FORWARD_PID}${RS}\n" + kill "${PORT_FORWARD_PID}" + exit +} + + +f_trap() { + # Trap signals to ensure cleanup is executed when the script finishes + trap cleanup INT TERM EXIT + exit +} + + +f_set_vars $@ +f_port_fwd +f_get_val +f_trap + +Execution: +``` + diff --git a/kb/armory/General/how-to-disable-security-authn-and-authz-for-spinnaker-migration-testing-.md b/kb/armory/How To's/how-to-disable-security-authn-and-authz-for-spinnaker-migration-testing-.md similarity index 100% rename from kb/armory/General/how-to-disable-security-authn-and-authz-for-spinnaker-migration-testing-.md rename to kb/armory/How To's/how-to-disable-security-authn-and-authz-for-spinnaker-migration-testing-.md diff --git a/kb/armory/How To's/how-to-enable-an-artifactory-helm-repo-trigger-from-jfrog-artifactory.md b/kb/armory/How To's/how-to-enable-an-artifactory-helm-repo-trigger-from-jfrog-artifactory.md new file mode 100644 index 0000000000..d99c8345ea --- /dev/null +++ b/kb/armory/How To's/how-to-enable-an-artifactory-helm-repo-trigger-from-jfrog-artifactory.md @@ -0,0 +1,52 @@ +--- +title: How to enable an Artifactory Helm repo trigger from jFrog Artifactory +--- + +## Introduction +As part of an automated pipeline trigger strategy, customers would like to use a Helm chart residing in a jFrog Artifactory account to serve as the execution trigger. + +## Prerequisites +A ```helm``` repository in the defined Artifactory account, with at least one Helm chart deployed to it. +  + +## Instructions +Configure the repository in Spinnaker, according to the official [Repository Config](https://docs.armory.io/armory-enterprise/installation/armory-operator/op-manifest-reference/repository/#artifactory) documentation, under ```spec.spinnakerConfig.config``` - example below: +``` +repository: + artifactory: + enabled: true + searches: + - name: helm-artifactory-repo + baseUrl: https://.jfrog.io/artifactory + permissions: + READ: [] + WRITE: [] + repo: default-helm-local + repoType: helm + username: + password: ​ +``` + +Note: The ```permissions``` block refers to Fiat users. The example above is a bare-bone Spinnaker baseline with no authentication/authorization configured, so if the environment has defined users, ensure that they are explicitly specified. +Enable artifact support in Spinnaker, according to the official [Artifact Config](https://docs.armory.io/armory-enterprise/installation/armory-operator/op-manifest-reference/artifact/#helm) documentation, under ```spec.spinnakerConfig.config.artifacts``` - example below: +``` +helm: + enabled: true + accounts: + - name: helm-artifactory-repo + repository: https://.jfrog.io/artifactory/default-helm-local + username: + password: ​ +``` +Igor will receive the list of accounts from Clouddriver, but a property in Igor needs to be set in order to enable the trigger, under ```spec.spinnakerConfig.profiles``` + +``` +spec: + spinnakerConfig: + profiles: + igor: + helm: + enabled: true​ +``` +* Apply changes, and if this is the first time configuring an automated pipeline trigger - observe that the Igor pod comes up successfully. If it doesn't, refer to the following Knowledge Base article: [Igor pod not running or starting up when configuring pipeline triggers](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010551).* In the pipeline, in the ```Configuration``` section, under ```Automated Triggers``` , add a new trigger of type ```Helm Chart``` . Select the helm account configured above, then define a new artifact.* In the ```Expected Artifact``` pop-up, under the ```Expected Artifact``` section, once again select the helm account, and select the artifact (Helm chart) from the ```Name``` drop-down menu.* Test the trigger, to ensure it is working as intended + diff --git a/kb/armory/How To's/how-to-find-the-tam-account-executive-sales-engineer-solutions-architect-for-an-account.md b/kb/armory/How To's/how-to-find-the-tam-account-executive-sales-engineer-solutions-architect-for-an-account.md new file mode 100644 index 0000000000..8c4fb455ba --- /dev/null +++ b/kb/armory/How To's/how-to-find-the-tam-account-executive-sales-engineer-solutions-architect-for-an-account.md @@ -0,0 +1,30 @@ +--- +title: How to Find the TAM/Account Executive/Sales Engineer/Solutions Architect for an Account +--- + + +You may need to contact the Technical Account Manager for any of the customers at Armory.  This can be because they are the official liaison for the company, or they have better visibility with the contacts in the company. + +To get the record of truth with regards to which TAM is servicing a customer first use Salesforce.  If all else fails, you can either contact @tams or go to the appropriate Slack Channel for the Company [+How are Armory’s Slack Channels Organized?](https://paper.dropbox.com/doc/71hfYqPYoBiyNRq9QMsic)  + + +## Salesforce Account Lookup + + +* Log in to Salesforce via Okta* Click on the Apps grid icon in the upper left (3x3 dots)* Search for **Opportunities*** Click on the result* Search for the customer that you want to look up using the **Search the List** field* Look for the **Customer Name** with the “Land” or “Closed Won” marker.  The other ones are either pilots or lost bids.  **Do not look them up using the** **“stage”**** column*** Click on the opportunityClick on the Details Tab and you will see a list of the personnel assigned.   +* Opportunity Owner (Account Executive)* Sales Engineer* Technical Account Manager* Solutions Architect + +  +  +* When you are done, close the tabs + + + +## View the assigned personnel via Salesforce Report + +In addition, you can also create a customized report that summarizes and pulls the data from all accounts in to a table. There is a pre-existing report at: +[https://armory.lightning.force.com/lightning/r/Report/00O5x000007mqwqEAA/view](https://armory.lightning.force.com/lightning/r/Report/00O5x000007mqwqEAA/view) + + +  + diff --git "a/kb/armory/How To's/how-to-fix-tls-error-\"reason--extension-5-should-not-be-presented-in-certificate_request.\".md" "b/kb/armory/How To's/how-to-fix-tls-error-\"reason--extension-5-should-not-be-presented-in-certificate_request.\".md" new file mode 100644 index 0000000000..d58a8c32a8 --- /dev/null +++ "b/kb/armory/How To's/how-to-fix-tls-error-\"reason--extension-5-should-not-be-presented-in-certificate_request.\".md" @@ -0,0 +1,11 @@ +--- +title: How to fix TLS error "Reason- extension (5) should not be presented in certificate_request." +--- + +## Issue +When Spinnaker attempts to connect to any client services behind a TLS endpoint, you receive the following error: +```Reason: extension (5) should not be presented in certificate_request``` + +## Cause +It may be related to a Java and Go bug mentioned here: [https://github.com/golang/go/issues/35722#issuecomment-557209799](https://github.com/golang/go/issues/35722#issuecomment-557209799). + diff --git a/kb/armory/How To's/how-to-get-a-list-of-all-executions-of-a-given-application-via-api-calls.md b/kb/armory/How To's/how-to-get-a-list-of-all-executions-of-a-given-application-via-api-calls.md new file mode 100644 index 0000000000..ffb2589eba --- /dev/null +++ b/kb/armory/How To's/how-to-get-a-list-of-all-executions-of-a-given-application-via-api-calls.md @@ -0,0 +1,20 @@ +--- +title: How to get a list of all executions of a given application via API calls +--- + +## Introduction +The following information explains how to get a list of all executions of a given application via API calls in Spinnaker.  + +## Prerequisites +A running Spinnaker instance with a known Gate URL is need to interact with the Spinnaker API. + +## Instructions +A sample curl command to retrieve a list of an application's pipeline executions is shown below: +```curl -k 'https:///applications//pipelines?limit=&statuses=' ​``` +The RESTful API Swagger UI can be accessed using the following URL format: +```http(s):///swagger-ui.html​``` +* The default ```getPipelinesUsingGET``` API method is set to 10 executions if a limit of type ```**int**``` isn't specified. Note that it is advised to set a number higher than the expected output intended. The resulting executions can ten be filtered out using a tool like **jq** for example. +  +More information in the link below: +[https://spinnaker.io/docs/reference/api/docs.html#api-Applicationcontroller-getPipelinesUsingGET](https://spinnaker.io/docs/reference/api/docs.html#api-Applicationcontroller-getPipelinesUsingGET) + diff --git a/kb/armory/General/how-to-investigate-pipeline-failures-and-execution-issues-with-end-users.md b/kb/armory/How To's/how-to-investigate-pipeline-failures-and-execution-issues-with-end-users.md similarity index 99% rename from kb/armory/General/how-to-investigate-pipeline-failures-and-execution-issues-with-end-users.md rename to kb/armory/How To's/how-to-investigate-pipeline-failures-and-execution-issues-with-end-users.md index c05077b9c4..0742a99689 100644 --- a/kb/armory/General/how-to-investigate-pipeline-failures-and-execution-issues-with-end-users.md +++ b/kb/armory/How To's/how-to-investigate-pipeline-failures-and-execution-issues-with-end-users.md @@ -4,11 +4,14 @@ title: How to Investigate Pipeline Failures and Execution Issues with End Users ## Introduction Administrators of a Spinnaker environment may find it challenging to locate the specific errors surrounding a pipeline failure in a Spinnaker environment.  This issue is exasperated as an environment scales due to the amount of data provided when scaling Spinnaker.  + For example, as the user base and executions increase, customers may need to scale each Spinnaker service so that there are multiple replicas to handle the number of queries. [https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010521](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010521) + The scaling of services can lead to multiple data points that need to be analyzed and can increase the amount of data to be parsed by a factor of the replicas. ## Prerequisites Administrators should look to create Monitoring and Logging repositories with an aggregator like NewRelic, Google Logging, Datadog, Splunk, etc., to search and locate data effectively. + Below is a KB article explaining the benefits of Monitoring and Logging, as well as some tips to start [https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010370](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010370) ***For this article, it is assumed that logging and monitoring are enabled.*** @@ -27,7 +30,10 @@ The URL for the permalink (next to the ```View as JSON``` link) will contain the ### Using the Execution ID to location the Execution Data in the logs After customers have found the execution ID, they can then use that data to find and trace back the execution details through their logs to trace the execution path and the time when the execution happened Administrators can do so by generating a query that searches through their logs for the ```executionID``` under the ```jsonPayload.message``` parameter.  + By running the query, admins can see the services involved with the pipeline that was triggered and when the execution occurred.  The data is tied to the executionID as long as the ```echo``` service is available and engaged.  Admins may still need to dive deeper into individual service logs, (e.g., CloudDriver), but they will have a starting point for the time when the execution is occurring. + Administrators can then use this data to look at individual services during the execution and trace back issues to their source.  This data can then be combined with monitoring of the environment to get a complete picture of the Spinnaker environment as the execution occurred. + We recommend administrators use our [knowledge bases](https://kb.armory.io) to assist themselves with resolving the issue or [opening a case](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010136) if they require further assistance.  diff --git a/kb/armory/How To's/how-to-iterate-complex-structures-in-pipelines-as-code-dictionary-list-sprig-functions.md b/kb/armory/How To's/how-to-iterate-complex-structures-in-pipelines-as-code-dictionary-list-sprig-functions.md new file mode 100644 index 0000000000..4a176cdc92 --- /dev/null +++ b/kb/armory/How To's/how-to-iterate-complex-structures-in-pipelines-as-code-dictionary-list-sprig-functions.md @@ -0,0 +1,67 @@ +--- +title: How to Iterate Complex Structures in Pipelines as Code - Dictionary or List Sprig functions +--- + +## Introduction +Pipelines as Code (PaC) is a pivotal feature, enabling intricate Spinnaker pipelines' rapid and efficient provisioning.  By strategically utilizing Dinghyfiles, we can apply the same PaC code across multiple applications, thereby significantly curtailing development timelines and bolstering the efficiency of Continuous Deployment workflows. +As complexity mounts, Dinghyfiles evolve into sophisticated constructs, warranting a comprehensive understanding of their potential.  This article exemplifies the extensive capabilities of iterating over intricate data types during pipeline provisioning. + +## Prerequisites +* Pipelines as Code plugin/service available in Spinnaker (Installation and configuration instructions available [here](https://docs.armory.io/plugins/pipelines-as-code/install/armory-cd/)) +* ARM-CLI used for debugging purposes and validation - [ARM-CLI](https://docs.armory.io/plugins/pipelines-as-code/arm-cli/) + +## Instructions +In this example, we are configuring a pipeline with two distinct stages designated for the Development (Dev) and Quality Assurance (QA) environments. +Both stages share a common definition to optimize resource utilization. This is achieved by employing a ```wait module``` for stage provisioning. To maintain code abstraction and reduce the Dinghyfile's size, module parameters are provisioned from a parameter dictionary, which is iterated using a List aggregator. + +For simplicity, we will utilize a wait module. + +1. The wait module is defined as follows: + +``` +{ + "name": "{{ var "waitname" ?: "defaultname" }}", + "type": "wait", + "waitTime": {{ var "waitTime" ?: 100 }} +} +``` + +2. As depicted above, the module expects two variables: ```waitname``` and ```waitTime```. In the absence of these variables, ```defaultname``` is assumed for ```waitname```, and ```100``` for ```waitTime```. + +3. Within the main Dinghyfile, the code appears as follows: + +```` +{ + "application": "dictrangeexample", + "pipelines": [ + { + "name": "Loop Example", + "application": "dictrangeexample", + "stages": [ + {{ $dictdev := dict "waitTime" "11" "name" "dev" }} + {{ $dictqa := dict "waitTime" "21" "name" "qa" }} + {{ $myenvs := list $dictdev $dictqa }} + {{ $count := 1 }} + {{ range $myenvs }} + {{ module "dinghy-modules/wait.stage.module" "waitname" ( get . "name" ) "waitTime" ( get . "waitTime" ) }} + {{ if ne $count (len $myenvs) }} + {{ $count = add $count 1 }} + , + {{ end }} + {{ end }} + ] + } + ] +} +```` + +4. We initialize two dictionary variables, one holding parameters for the Dev environment stage and the other for the QA environment stage. Subsequently, we consolidate the two dictionaries into a new list variable for iterative processing. +We also establish a counter to track the current iteration index to maintain clarity. Once the dictionaries are unified into a list, we iterate over the list structure using the 'range' construct, as demonstrated above. + +5. Within the loop, we dynamically render the wait module with the dictionary parameters, utilizing the syntax ```(get . "field")```. Here, ```.``` denotes the current iterated object, and ```get``` retrieves the specified field, in this case, ```name``` and ```waitTime```. + + +6. In this example, we have successfully generated a pipeline with customized parameters for two stages, targeting distinct environments. + +Please note that while the ARM-CLI 2.3.0 may flag the above syntax as potentially problematic, Spinnaker successfully interprets the code and provisions the pipeline. + diff --git a/kb/armory/How To's/how-to-limit-logs-to-information-to-specific-logs-by-finding-the-absolute-path.md b/kb/armory/How To's/how-to-limit-logs-to-information-to-specific-logs-by-finding-the-absolute-path.md new file mode 100644 index 0000000000..5eb3918e2b --- /dev/null +++ b/kb/armory/How To's/how-to-limit-logs-to-information-to-specific-logs-by-finding-the-absolute-path.md @@ -0,0 +1,30 @@ +--- +title: How to Limit Logs to Information to Specific Logs by Finding the Absolute Path +--- + +## Introduction +Administrators may, at some point, want to limit or reduce their logs in Spinnaker to specific logs.  This may aid them in the following situations: +* Controlling specific logs can prevent high CPU and Memory when using debug mode from the root level.* Reduce the amount of non-immediately relevant info logs by changing them only to show ERROR/WARN logs.* Easy to track.* Not overwhelming the console or aggregation tool +  +  + +## Prerequisites +Access to a Running Spinnaker InstanceAccess to change the Spinnaker Configuration + +## Instructions +Below is an example log, where we may want to limit to WARN/ERROR messagesWhen looking at the list of logs, admins can see that the logs are using a short path in the console, which makes it harder to know how to limit this specific log. The reason is when changing log mode, we need to specify the ```full path```: +```c.n.s.c.sql.event.SqlEventCleanupAgent``````s.k.p.u.r.r.RemotePluginInfoReleaseCache``` ← will show an example of this log``` i.a.k.a.r.ClouddriverAgentCleanup``` +There are two options to find an absolute path for logs: +* Looking in the ```lib folder``` for the logs class* Configuring advanced logs with ```logback``` -> will automatically show the full path of each log +In this KB, we will be focusing on the first option.In the example below, we will do it for Clouddriver: +* We exec into the Clouddriver pod* We will navigate to the lib folder by using the following command: ```cd /opt/clouddriver/lib/```* In the ```lib``` folder, we will have a lot of different ```jar``` files.* We will use the following command to narrow the list of possible jar files for the specific log we are looking for: ```find | grep plugin```. We are using grep with the word* because we can see the word ```plugin``` in the log sample.The command above will provide us with the following .jar files:  +./armory-commons-plugins-3.13.5.jar +./kork-plugins-7.169.1.jar +./kork-plugins-api-7.169.1.jar +./kork-plugins-spring-api-7.169.1.jar +./spring-plugin-core-1.2.0.RELEASE.jar +./spring-plugin-metadata-1.2.0.RELEASE.jar +  +*  In some cases, it will be easier to locate to correct ```.jar``` file, and in other cases, we will search for the absolute path in all of them*  If we are checking the first two letters initial log example, ```s.k```, we know the first latter will be Spinnaker, and based on the ```.jar``` list, the second is ```Kork```*  We will use the following command to search for the log absolute path inside the Kork .jar files:  ```jar tvf ./kork-plugins-7.169.1.jar | grep RemotePluginInfoReleaseCache```*  After running the above command in step 9, the following output will be displayed:```com/netflix/spinnaker/kork/plugins/update/release/remote/RemotePluginInfoReleaseCache.class```* We will use the full path we found in step 9, like the example below:```        logging:``````          level:``````            com.netflix.spinnaker.kork.plugins.updates.release.remote.RemotePluginInfoReleaseCache: WARN``` +The short path log is ```s.k.p.u.r.r.RemotePluginInfoReleaseCache ```The full path log is ```com.netflix.spinnaker.kork.plugins.updates.release.remote.RemotePluginInfoReleaseCache``` + diff --git a/kb/armory/How To's/how-to-modify-custom-readiness,-liveness-and-startup-probes-http,-tcp-socket,-exec.md b/kb/armory/How To's/how-to-modify-custom-readiness,-liveness-and-startup-probes-http,-tcp-socket,-exec.md new file mode 100644 index 0000000000..45936c0622 --- /dev/null +++ b/kb/armory/How To's/how-to-modify-custom-readiness,-liveness-and-startup-probes-http,-tcp-socket,-exec.md @@ -0,0 +1,65 @@ +--- +title: How to Modify Custom Readiness, Liveness and Startup Probes (HTTP, TCP socket, Exec) +--- + +## Introduction +As a best practice, customers should be looking to set up various probes to detect service failures.   +* The ```kubelet``` uses ***liveness probes*** to know when to restart a container.  For example, liveness probes could catch a deadlock, where an application is running but unable to make progress.  Restarting a container in such a state can help to make the application more available despite bugs.* The ```kubelet``` uses ***readiness probes*** to know when a container is ready to start accepting traffic.  A Pod is considered ready when all of its containers are ready.  One use of this signal is to control which Pods are used as backends for Services.  When a Pod is not ready, it is removed from Service load balancers.* The ```kubelet``` uses ***startup probes*** to know when a container application has started.  If such a probe is configured, it disables liveness and readiness checks until it succeeds, ensuring those probes don't interfere with the application startup.  The setting can be used to adopt liveness checks on slow starting containers, avoiding them getting killed by the kubelet before they are up and running. +  + +## Prerequisites +* Kubenetes cluster (EKS,AKS,GKE etc..)* Spinnaker manifest access +  + +## Instructions +Below is an example of how to modify readiness, liveness, and startup probes and how Clouddriver will interact with a kubernetes cluster.  Administrators should feel welcome to adjust the values of the probes to suit their needs.   +apiVersion: spinnaker.armory.io/v1alpha2 +``` +kind: SpinnakerService +metadata: + name: spinnaker +spec: + spinnakerConfig: + service-settings: + clouddriver: + kubernetes: + livenessProbe: + httpGet: + port: 7002 + path: /health + scheme: http + initialDelaySeconds: 60 + periodSeconds: 10 + timeoutSeconds: 1 + successThreshold: 1 + failureThreshold: 3 + readinessProbe: + tcpSocket: + port: 7002 + initialDelaySeconds: 60 + periodSeconds: 10 + timeoutSeconds: 1 + successThreshold: 1 + failureThreshold: 3 + startupProbe: + exec: + command: + - wget + - --no-check-certificate + - --spider + - -q + - http://localhost:7002/health + initialDelaySeconds: 60 + periodSeconds: 10 + timeoutSeconds: 1 + successThreshold: 1 + failureThreshold: 3 + echo: {} + fiat: {} + front50: {} + gate: {} + igor: {} + kayenta: {} + orca: {} + rosco: {} +``` diff --git a/kb/armory/How To's/how-to-override-spinnaker's-default-signer-encryption-algorithm-for-s3.md b/kb/armory/How To's/how-to-override-spinnaker's-default-signer-encryption-algorithm-for-s3.md new file mode 100644 index 0000000000..1fd624c103 --- /dev/null +++ b/kb/armory/How To's/how-to-override-spinnaker's-default-signer-encryption-algorithm-for-s3.md @@ -0,0 +1,27 @@ +--- +title: How to override Spinnaker's default signer encryption algorithm for S3 +--- + +## Introduction +If you aren't using the default S3 signer encryption algorithm, Front50 and Clouddriver may not be able to connect to the S3 provider. As of Armory 2.26, it's possible to specify the signer algorithm used with S3 in Front50 and Clouddriver's custom profile. +You need to make this change if you're seeing errors that look like this: +```An error occurred (InvalidDigest) when calling the PutObject operation: StorageFabric: UNSIGNED-PAYLOAD PUT is not allowed``` +Related PR: [spinnaker/clouddriver#5307](https://github.com/spinnaker/clouddriver/pull/5307) + +## Prerequisites +Armory Spinnaker v2.26+ + +## Instructions +To override the default S3 signer algorithm in Spinnaker, we need to make adjustments to custom profiles for Front50 and Clouddriver: +Add the following to ```front50-local.yml``` and ```clouddriver-local.yml``` replacing ```S3SignerType``` with the signer type that the service should use to connect to S3: +``` + s3: + enabled: true + accounts: + - name: s3 + signerOverride: S3SignerType​ +``` + +More information can be found in the AWS docs: +* [Signing and authenticating REST requests in S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTAuthentication.html)* [getSignerOverride() in the AWS SDK](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html#getSignerOverride--) + diff --git a/kb/armory/How To's/how-to-parse-terraform-plan-status-code-for-your-spinnaker-pipeline-logic.md b/kb/armory/How To's/how-to-parse-terraform-plan-status-code-for-your-spinnaker-pipeline-logic.md new file mode 100644 index 0000000000..52a53e50b9 --- /dev/null +++ b/kb/armory/How To's/how-to-parse-terraform-plan-status-code-for-your-spinnaker-pipeline-logic.md @@ -0,0 +1,17 @@ +--- +title: How to parse Terraform Plan Status Code for your Spinnaker Pipeline Logic +--- + +## Introduction +When using Armory's Terraform Plan stage - you can use SpEL (Pipeline Expression) to retrieve the exitcode.  +* 0 = Succeeded with empty diff (no changes)* 1 = Error* 2 = Succeeded with non-empty diff (changes present) +```${#stage("Terraform Plan").outputs.status.code}``` +The pipeline expression can be used in a "Check Preconditions" stage in order to assert a particular value for the status code before proceeding or terminating the pipeline. Alternatively, the expression can be used in the "Execution Options" under a "Manual Judgment" stage so you can bypass the Manual Judgement stage when you want to automatically apply a change. + +## Prerequisites +Armory Terraformer enabled + +## Instructions +Check out this video for a demonstration: + + diff --git a/kb/armory/How To's/how-to-pass-artifacts-from-a-parent-pipeline-to-a-child-pipeline-using-matchartifacts.md b/kb/armory/How To's/how-to-pass-artifacts-from-a-parent-pipeline-to-a-child-pipeline-using-matchartifacts.md new file mode 100644 index 0000000000..f5b65fcbbd --- /dev/null +++ b/kb/armory/How To's/how-to-pass-artifacts-from-a-parent-pipeline-to-a-child-pipeline-using-matchartifacts.md @@ -0,0 +1,17 @@ +--- +title: How to Pass Artifacts from a Parent Pipeline to a Child Pipeline using matchArtifacts +--- + +## Introduction +Spinnaker allows users to pass artifacts from a parent pipeline to a child pipeline. This can be useful to compartmentalize pipelines e.g. one pipeline for baking and one for deploying.Artifacts can be passed from the parent pipeline's trigger or artifacts that are created upstream. For this example, the matchArtifacts feature will be used to pass artifacts between pipelines. + +## Prerequisites +A file/image to be used as an artifact. + +## Instructions +First, define an artifact in the parent pipeline. In this example, a helm chart from GitHub will be used as the artifact.Next, go ahead and reference this artifact in the child pipeline. +To do so, first create a new artifact in the child pipeline and set the below configurations for Match Artifact.I +t is important to note to use the **custom-artifact** account in the above example and set the type to the same type as the parent pipeline's artifact while also providing the name and the branch that the parent pipeline's artifact was found in. +If the pipeline requires passing an embedded/base64 artifact, choose the **embedded-artifact** account instead and directly reference the name of the base64 artifact as shown belowFor more information and a visual representation, please refer to the [spinnaker.io](http://spinnaker.io/) docs provided below +[https://spinnaker.io/reference/artifacts/in-pipelines/#passing-artifacts-between-pipelines](https://spinnaker.io/reference/artifacts/in-pipelines/#passing-artifacts-between-pipelines) + diff --git a/kb/armory/How To's/how-to-provide-feature-request-feedback.md b/kb/armory/How To's/how-to-provide-feature-request-feedback.md new file mode 100644 index 0000000000..656c12fc73 --- /dev/null +++ b/kb/armory/How To's/how-to-provide-feature-request-feedback.md @@ -0,0 +1,22 @@ +--- +title: How to Provide Feature Request Feedback +--- + + +Communication with our customers is an important part to making Armory Spinnaker a success, and we welcome feedback at all times.   +* Is the feature request with regards to a **feature which is not behaving as intended**? +* Is the feature request with regards to a **new feature or idea which has no work around, and is therefore blocking the success of the environment**?  +* Is the feature request with regards to a **new feature or idea which has a work around, but with an additional feature, if implemented, it will provide additional benefits to a wide range of companies**?  + +### Submitting Feature Requests for Armory CDSH or Spinnaker OSS related products and Plugins +If the issues fall outside of the Armory Compatibility Matrix ([https://docs.armory.io/docs/feature-status/armory-enterprise-matrix/](https://docs.armory.io/docs/feature-status/armory-enterprise-matrix/)), Armory may direct the request to the provider/community directly, for example, the [Spinnaker OSS channels](https://spinnaker.io/community/contributing/).  At times, the provider/community will be able to provide a more rapid response to the request at hand +* To open a feature request suggestion, we encourage customers to please [open a ticket at our support portal](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010136).   +* Armory will then assess the situation with our engineering team to determine if the resolution fits in with our product roadmap +* We will advise customers about estimates and keep them up to date about the progress of the feature requests +* Sometimes, a request may not fit in the current roadmap, but that doesn't mean it won't be part of the roadmap in the future.  We encourage all customers to please provide their feedback to us whenever possible + +### Submitting Feature Requests for Armory CDaaS related products +Customers can submit their suggested feedback directly in the Armory CDaaS Console.   + +* [Log in to the Armory CDaaS Site](https://console.cloud.armory.io/)* Click on the Help/Question Mark in the upper right hand corner* Select "Suggest Features" from the window + diff --git a/kb/armory/How To's/how-to-provision-jenkins-triggers-with-global-variables-in-pipelines-as-code-dinghy.md b/kb/armory/How To's/how-to-provision-jenkins-triggers-with-global-variables-in-pipelines-as-code-dinghy.md new file mode 100644 index 0000000000..b6fa8ffcc6 --- /dev/null +++ b/kb/armory/How To's/how-to-provision-jenkins-triggers-with-global-variables-in-pipelines-as-code-dinghy.md @@ -0,0 +1,139 @@ +--- +title: How to Provision Jenkins Triggers with Global Variables in Pipelines as Code/Dinghy +--- + +## Introduction +Pipelines as Code is a powerful feature of Spinnaker, which can deploy complex pipelines for applications reusing the same code. This article will showcase how we can provision two distinct Jenkins triggers with global variables defined in a top-level dinghyfile. + +## Prerequisites +* Pipelines as Code plugin/service available in Spinnaker (Installation and configuration instructions available [here](https://docs.armory.io/plugins/pipelines-as-code/install/armory-cd/))* ARM-CLI used for debugging purposes and validation - [ARM-CLI](https://docs.armory.io/plugins/pipelines-as-code/arm-cli/)* Jenkins configured in Spinnaker: [JenkinsCI](https://spinnaker.io/docs/setup/other_config/ci/jenkins/) + +## Instructions +In this article, we want to provision a simple pipeline with two Jenkins triggers and an Evaluate Variables stage that ingests the Jenkins trigger parameters, all using Pipelines as Code. + +The Jenkins triggers are based on global variables corresponding to the Jenkins job names we want to trigger the pipeline. +The top-level dinghyfile would look like the below example.  +In the example: +* We initialized two global variables ```configmap_trigger``` and ```application_trigger``` which we pass on to the Jenkins trigger submodule* Both triggers are enabled* ```jenkins_Main``` is the name of the Jenkins account configured in Spinnaker +``` +{ + "application": "DynamicJenkinsTriggers", + "globals": { + "save_app_on_update": true, + "application": "DynamicJenkinsTriggers", + "application_trigger" : "application_backend", + "configmap_trigger": "configmap_backend", + "jenkins_Main": "Jenkins_master" + }, + "pipelines": [ + { + "application": "DynamicJenkinsTriggers", + "locked": { + "allowUnlockUi": true, + "description": "", + "ui": true + }, + "name": "PipelineWithTriggers", + "stages": [ + { + "failOnFailedExpressions": true, + "isNew": true, + "name": "Evaluate Variables", + "type": "evaluateVariables", + "variables": [ + { + "key": "JenkinsJOB", + "value": "${trigger.job}" + } + ] + } + ], + "triggers": [ + {{ module "dinghy-modules/stage.automatedTriggers.module" + "type" "jenkins" + "job" (var "application_trigger") + "master" (var "jenkins_Main") + "enabled" "true" + }}, + {{ module "dinghy-modules/stage.automatedTriggers.module" + "type" "jenkins" + "job" (var "configmap_trigger") + "master" (var "jenkins_Main") + "enabled" "true" + }}] + }]} +``` +The Sub-module used would look like the following definition below. +``` +{ + "name": "{{ var "name" ?: "Build in Jenkins" }}", + "enabled" : {{ var "enabled" ?: "true" }}, + "type": "jenkins", + "refId": "{{ var "refId" }}", + "requisiteStageRefIds": {{ var "requisiteStageRefIds" ?: [] }}, + "master": "{{ var "master" }}", + "job": "{{ var "job" }}", + "parameters": {{ var "parameters" ?: {} }}, + "propertyFile": "{{ var "propertyFile" }}", + "expectedArtifacts": {{ var "expectedArtifacts" ?: [] }} +} +``` +Upon rendering the pipeline in Spinnaker, we will end up with the following pipeline JSON: +``` +{ + "keepWaitingPipelines": false, + "lastModifiedBy": "anonymous", + "limitConcurrent": false, + "locked": { + "allowUnlockUi": true, + "ui": true + }, + "parallel": false, + "schema": "1", + "spelEvaluator": "", + "stages": [ + { + "failOnFailedExpressions": true, + "isNew": true, + "name": "Evaluate Variables", + "type": "evaluateVariables", + "variables": [ + { + "key": "JenkinsJOB", + "value": "${trigger.job}" + } + ] + } + ], + "triggers": [ + { + "enabled": true, + "expectedArtifacts": [], + "job": "application_backend", + "master": "Jenkins_master", + "name": "Build in Jenkins", + "parameters": {}, + "propertyFile": "", + "refId": "", + "requisiteStageRefIds": [], + "type": "jenkins" + }, + { + "enabled": true, + "expectedArtifacts": [], + "job": "configmap_backend", + "master": "Jenkins_master", + "name": "Build in Jenkins", + "parameters": {}, + "propertyFile": "", + "refId": "", + "requisiteStageRefIds": [], + "type": "jenkins" + } + ], + "updateTs": "1697023404864" +} +``` +* We evaluate the ```trigger.job``` parameter in the Evaluate Variable stage so we can use this value further down the pipeline. If we run the corresponding jobs in Jenkins, the pipeline will get triggered, as seen in the below image: +  + diff --git a/kb/armory/How To's/how-to-provision-unique-reference-ids-for-sub-module-stages-in-pipelines-as-code-dinghy-using-uuidv4.md b/kb/armory/How To's/how-to-provision-unique-reference-ids-for-sub-module-stages-in-pipelines-as-code-dinghy-using-uuidv4.md new file mode 100644 index 0000000000..6fd79a51d4 --- /dev/null +++ b/kb/armory/How To's/how-to-provision-unique-reference-ids-for-sub-module-stages-in-pipelines-as-code-dinghy-using-uuidv4.md @@ -0,0 +1,120 @@ +--- +title: How to provision unique Reference IDs for sub-module stages in Pipelines as Code (Dinghy) using UUIDv4 +--- + +## Introduction +Pipelines as Code (PaC) represent a pivotal capability, facilitating swift and efficient provisioning of intricate Spinnaker pipelines. By judicious employment of Dinghyfiles, users can seamlessly apply the same PaC code across diverse applications, significantly reducing development timelines and enhancing the efficacy of Continuous Deployment workflows. +A common problem using PaC is in provisioning unique reference ID's for stages. In this article, admins and users can learn to leverage the power of the UUID function to provision unique identifiers for each stage. + +## Prerequisites +* Pipelines as Code plugin/service available in Spinnaker (installation and configuration instructions available [here](https://docs.armory.io/plugins/pipelines-as-code/install/armory-cd/))* ARM-CLI used for debugging purposes and validation - [ARM-CLI](https://docs.armory.io/plugins/pipelines-as-code/arm-cli/) + +## Instructions +This example will generate two stages for two environments by parsing a list. +* Each stage will have a unique reference ID passed on as a sub-module variable using the UUID function.The Wait stage sub-module looks like this: +``` +{ + "name": "{{ var "waitname" ?: "defaultname" }}", + "type": "wait", + "refId": "{{ var "refId" ?: "defaultrefId" }}", + "waitTime": {{ var "waitTime" ?: 100 }} +} +``` + +The module is then used in the top-level ```dinghyfile```. The ```dinghyfile``` looks like this: +``` +{ + "application": "uuidexample", + "pipelines": [ + { + "name": "uuid", + "application": "uuid", + "stages": [ + {{ $listdev := list "10" "dev" }} + {{ $listqa := list "20" "qa" }} + {{ $myenvs := list $listdev $listqa }} + {{ $count := 1 }} + {{ range $myenvs }} + {{ module "dinghy-modules/wait.stage.module" "waitname" ( index . 1 ) "waitTime" ( index . 0 ) "refId" uuidv4 }} + {{ if ne $count (len $myenvs) }} + {{ $count = add $count 1 }} + , + {{ end }} + {{ end }} + ] + } + ] + } +``` +* In the above example, administrators and users can aggregate a list of parameters into another list, which can be iterated upon and then pass variables to the sub-module. We pass the ```uuidv4``` function as a ```refId``` variable value.The final pipelines after it has been rendered will look like this: +``` +{ + "application": "uuidexample", + "pipelines": [ + { + "name": "uuid", + "application": "uuid", + "stages": [ + + + + + + { + "name": "dev", + "type": "wait", + "refId": "63dbd819-53b4-4312-acd0-dc839cba2aa7", + "waitTime": 10 +} + + + + , + + + { + "name": "qa", + "type": "wait", + "refId": "d9bf16a3-b0d0-4760-9c8d-f9a84fcd5a0b", + "waitTime": 20 +} + + + + ] + } + ] + } +``` + +We can easily reference the stages by having unique identifiers for them downstream. In the following example, we manually created a wait stage that depends on the previous two stages by using the reference IDs in the requisiteStageRefIds field +The stages section will look like this :  +``` +"stages": [ + { + "name": "dev", + "refId": "63dbd819-53b4-4312-acd0-dc839cba2aa7", + "type": "wait", + "waitTime": 10 + }, + { + "name": "qa", + "refId": "d9bf16a3-b0d0-4760-9c8d-f9a84fcd5a0b", + "type": "wait", + "waitTime": 20 + }, + { + "name": "final", + "refId": "1", + "requisiteStageRefIds": [ + "63dbd819-53b4-4312-acd0-dc839cba2aa7", + "d9bf16a3-b0d0-4760-9c8d-f9a84fcd5a0b" + ], + "type": "wait" + } + ] +``` +  +  +Please note that the ARM-CLI verification tool (v2.3.0) will report the syntax as incorrect, but Spinnaker will successfully render the pipeline on a Pull Request. + diff --git a/kb/armory/How To's/how-to-return-all-effective-permissions-authorizations-for-a-user-through-redis-fiat.md b/kb/armory/How To's/how-to-return-all-effective-permissions-authorizations-for-a-user-through-redis-fiat.md new file mode 100644 index 0000000000..48c77725f3 --- /dev/null +++ b/kb/armory/How To's/how-to-return-all-effective-permissions-authorizations-for-a-user-through-redis-fiat.md @@ -0,0 +1,37 @@ +--- +title: How to Return All Effective Permissions/Authorizations for a User Through Redis/Fiat +--- + +## Introduction +In some situations, it can be useful to investigate and audit a user's effective permissions and authorizations.  This can be accomplished in multiple ways.  One of the ways is to query Redis via FiatFor example, in the cases Redis is being used with Spinnaker, these permissions are stored under the following keys ```spinnaker:fiat:permissions::``` and stored as a hash with the following info: +* ```key``` = name of the resource (e.g. name of the app)* ```value``` = ```{"name": , "permissions": { : [list of roles] } }``` + +## Prerequisites +Fiat (authorization) service should enabled and running properly, as well as Redis + +## Instructions +1. Exec into Redis pod (```kubectl exec -it sh```) +2. Once inside the Redis pod, run the command ```redis-cli```. +3. Then, once inside the redis-cli shell, run the below command to return the stored hash: + +```HGETALL spinnaker:fiat:permissions::applications``` + +HGETALL is a Redis command returns all fields and values of the hash stored at ```key```.  + +For example, on __unrestricted_user__: +``` +HGETALL spinnaker:fiat:permissions:__unrestricted_user__:applications +"app1" +"{\n \"name\" : \"app1\",\n \"permissions\" : { }\n}" +"ncecs" +"{\n \"name\" : \"app2\",\n \"permissions\" : { }\n}" +"cam" +"{\n \"name\" : \"app3\",\n \"permissions\" : { }\n}" +``` +Another example, poll the permissions for user "brian": +``` +127.0.0.1:6379> HGETALL spinnaker:fiat:permissions:brian:applications +"BRIANAPPLICATION" +"{\n \"name\" : \"BRIANAPPLICATION\",\n \"permissions\" : {\n \"READ\" : [ \"baz\" ],\n \"WRITE\" : [ \"baz\" ],\n \"EXECUTE\" : [ \"baz\" ]\n },\n \"details\" : {\n \"instancePort\" : 80,\n \"lastModifiedBy\" : \"brian\",\n \"createTs\" : \"1586446405209\",\n \"cloudProviders\" : \"aws,kubernetes\",\n \"providerSettings\" : {\n \"aws\" : {\n \"useAmiBlockDeviceMappings\" : false\n }\n },\n \"updateTs\" : \"1603129132000\",\n \"user\" : \"brian\",\n \"dataSources\" : {\n \"disabled\" : [ ],\n \"enabled\" : [ \"canaryConfigs\" ]\n },\n \"email\" : \"brian@armory-test.io\",\n \"trafficGuards\" : [ ]\n }\n}" +127.0.0.1:6379> +``` diff --git a/kb/armory/How To's/how-to-scale-igor-replicas.md b/kb/armory/How To's/how-to-scale-igor-replicas.md new file mode 100644 index 0000000000..e09d1fdb32 --- /dev/null +++ b/kb/armory/How To's/how-to-scale-igor-replicas.md @@ -0,0 +1,37 @@ +--- +title: How to Scale Igor Replicas +--- + +## Introduction +***Igor*** facilitates the use of Jenkins in Spinnaker pipelines (*a pipeline can be triggered by a Jenkins job or invoke a Jenkins job*).  Customers may need to scale Igor for redundancy purposes and provide additional resources for their environment. + +## Prerequisites +Permissions to make administrative changes to the Spinnaker instance + +## Instructions +#### In Halyard +To scale number of Igor replicas, the following setting must be added to Igor’s profile in hal config profiles directory e.g. (```~/.hal/default/profiles/```) in the ```igor-local.yml``` file.  If the file exists, please add the line to the file, otherwise, please create the file in the directory. + +```locking.enabled: true``` + + + +`````` +After this is enabled, you can following the below to scale replicas:[https://spinnaker.io/docs/reference/halyard/component-sizing/#replicas](https://spinnaker.io/docs/reference/halyard/component-sizing/#replicas) +  +#### In Operator + +Here’s what it looks like in Operator.  The settings should be placed under ```spec.spinnakerConfig.profiles.igor``` +apiVersion: spinnaker.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker +spec: + spinnakerConfig: + profiles: + igor: + locking.enabled: true # this must be set in order for Igor's number of replicas to scale to something greater than 1 +  +`````` +After this is enabled, you can following the below to scale replicas:[https://spinnaker.io/docs/reference/halyard/component-sizing/#replicas](https://spinnaker.io/docs/reference/halyard/component-sizing/#replicas) + diff --git a/kb/armory/General/how-to-set-application-features-with-dinghy.md b/kb/armory/How To's/how-to-set-application-features-with-dinghy.md similarity index 99% rename from kb/armory/General/how-to-set-application-features-with-dinghy.md rename to kb/armory/How To's/how-to-set-application-features-with-dinghy.md index 565c991ec1..782e19a9f9 100644 --- a/kb/armory/General/how-to-set-application-features-with-dinghy.md +++ b/kb/armory/How To's/how-to-set-application-features-with-dinghy.md @@ -10,6 +10,7 @@ Dinghy has multiple features that allow users to edit and compartmentalize piece ## Instructions The available options can be found in the list below. Please note that multiple options can be set in a single ```dinghyfile```. +``` Available Options: serverGroups = Clusters executions = Pipelines @@ -17,7 +18,8 @@ loadBalancers = Load Balancers securityGroups = Firewalls canaryConfigs = Canary Configs (This option is only available if you have Canary setup) functions = Functions (This option is only available if you have the Lambda Plugin installed) -  +``` + When enabling/disabling these options in Dinghy, it can be set in the ```spec.dataSources.enabled``` or ```spec.dataSources.disabled``` section. Please find the example ```dinghyfile``` below with an example configuration that enables ```canaryConfigs``` and ```serverGroups``` and disables ```loadBalancers```. ***Please note*** that when setting Application Attributes, it is recommended that the ```globals.save_app_on_update``` option is set to ```true``` within the ```dinghyfile``` (see in the below example) The reasoning behind this is because when the ```dinghyfile``` is pushed into Spinnaker, it will retroactively update the Application. If this option is not set, Dinghy may have unexpected results. The configuration file below has an example of how to set this. diff --git a/kb/armory/General/how-to-set-up-least-privilege-access-with-fiat.md b/kb/armory/How To's/how-to-set-up-least-privilege-access-with-fiat.md similarity index 96% rename from kb/armory/General/how-to-set-up-least-privilege-access-with-fiat.md rename to kb/armory/How To's/how-to-set-up-least-privilege-access-with-fiat.md index a378ef6f66..02c8b95a51 100644 --- a/kb/armory/General/how-to-set-up-least-privilege-access-with-fiat.md +++ b/kb/armory/How To's/how-to-set-up-least-privilege-access-with-fiat.md @@ -4,13 +4,16 @@ title: How to Set up Least Privilege Access with Fiat ## Introduction Customers may find that they wish to set up FIAT with least privileges in order to limit access. + To start, it is important to note that in FIAT, if customers do not define any rules for the service, and the FIAT service is nevertheless enabled, access defaults full permissions for all users.  Therefore, in order to secure using FIAT, at least one definition needs to be provided so that security is defined and applied. + Another concept to understand is that security within FIAT lies within two Spinnaker micro services. * Definitions within FIAT control the general outline of access to the Spinnaker console.  The access granted is provided on the ***application level only*** and cannot be created granularly on the pipeline level, or on a larger scale on the project level* Definitions for access within CloudDriver determine whether a particular user is able to access cloud resources; whether to read what is in the cloud resources or to execute and make changes to the resources With those concepts in mind, we can talk about the steps to creating tighter access in FIAT ## Prerequisites -[Authentication should also be already set up and established](https://spinnaker.io/setup/security/authentication/).  Authentication is used to confirm the role/group a particular user and will be used to provide a set of access rights via FIAT Authorization.  Establish which accounts will be ```Administration Accounts``` that have full access rights and the name of their role/group must be done before proceeding. +[Authentication should also be already set up and established](https://spinnaker.io/setup/security/authentication/).  +Authentication is used to confirm the role/group a particular user and will be used to provide a set of access rights via FIAT Authorization.  Establish which accounts will be ```Administration Accounts``` that have full access rights and the name of their role/group must be done before proceeding. [Authorization (FIAT)](https://docs.armory.io/docs/overview/fiat-permissions-overview/) should already be established and enabled before hand.  This should be tested and the pod should be running. ## Instructions @@ -22,6 +25,7 @@ An external service (e.g. OKTA) is being used for the authentication ### Phase 1: Creating Access for Administrators The below yaml code is an example of the changes that would be implemented in the FIAT section of the Operator SpinnakerService yaml.  In this example, a separate Kustomize file with these settings would be patched The following are some notes about the configuration:```groupMembership.service``` is ```EXTERNAL``` because OKTA is in place```admin``` roles are being provided to the two admin groups, ```spinadmin``` and ```superadmin``` because a set of users should have admin/highest/unlimited privileges or the environment risks becoming locked out.  Permissions need to be defined, or all users will have complete access as Spinnaker's default state is to allow full access unless some access is defined within FIAT +``` apiVersion: spinnaker.io/v1alpha2 kind: SpinnakerService metadata: @@ -42,7 +46,9 @@ spec: roles: - spinadmin - superadmin​ +``` It is also important to define the CloudDriver access permissions.  The permissions on the CloudDriver level/account level for each cloud service/account determines whether a user will have permissions to use CloudDriver to access resources.The example in the yaml below provides ```spinadmin```, ```superadmin```, ```prod``` and ```dev``` group members the ability to view resources within AWS.  For example, to be able to find a Target Group so that it can be used later, or discover an AMI based on its tag.However, only ```spinadmin```, ```superadmin```, and ```prod``` accounts are able to create or change resources in the cloud.  Members of the ```dev``` group will not have permissions.  Of particular note is that if this section is left blank, then any group members will be able to deploy into the AWS account.  The absence of any definition of access will mean that access will default to being available for all users.  This can be extremely important for users who are intending to bypass using the Spinnaker Console, and instead wish to use Dinghy. +``` apiVersion: spinnaker.io/v1alpha2 kind: SpinnakerService metadata: @@ -73,15 +79,17 @@ spec: - name: us-east-1 - name: eu-west-1 assumeRole: role/ArmoryExample-role - +``` * Compile and apply the changes into the environment using ```kubectl```Test and confirm the access restrictions. Check user or service account permissions for all of Spinnaker. This should return a JSON listing what roles the user/service account is in, what Spinnaker applications the user/service account has access to, what clouddriver accounts the user/service account has access to, and what build services the user/service account has access to.In the example below, ```http://spin-fiat:7003``` is the local service endpoint for FIAT, and may need to be modified for your environment.Please also replace ```$user-or-service-account​``` with the name of the user or service account +``` > export FIAT=http://spin-fiat:7003 > curl -s $FIAT/authorize/$user-or-service-account​ - +``` ### Phase 2: Provide additional permissions to other groups After administrative access has been provided to the administration groups, it is now time to add additional permissions for the rest of the teams. The below yaml code is an example of the changes that would be implemented in the FIAT section of the Operator SpinnakerService yaml.  In this example, a separate Kustomize file with these settings would be patched The following are some notes about the configuration:```fiat.restrictApplicationCreation``` is set to ```true``` in order to limit the creation of additional applications.  If users end up creating applications that do not fall under the naming conventions (e.g., in the case below, not starting with ```banana``` or ```orange```, the end user may end up creating an application that they cannot access.  Instead, application creation becomes managed by the administrators, and they will need to toggle this setting whenever an adjustment must be made```auth.permissions.source.application.front50.enabled``` is set to ```false``` in order to turn off the ability for applications to be created within Front50, from the UI.  Without setting this parameter, [permissions set within the UI](https://docs.armory.io/docs/overview/fiat-permissions-overview/#applications) will be aggregated with the permissions defined below.  This can make it harder to determine the effective permissions for a user.```Full Permissions``` are provided to the ```Admin groups``` in order to guarantee that someone can administer all applications.  This is set here to ensure that superusers can still have access in the case access is lost```Each prefix definition``` determines what applications the groups defined will have access to.  The sections below defining ```CREATE```, ```READ```, ```WRITE``` and ```EXECUTE``` determine the amount of access a group has.  Wildcards can be used so that applications with certain naming conventions will be allowed for particular groups. +``` apiVersion: spinnaker.io/v1alpha2 kind: SpinnakerService metadata: @@ -151,7 +159,9 @@ spec: - "banana" - "prod" - "devb" +``` Please note that if you wish to declare certain groups to have Administration access to override the ```restrictApplicationCreation``` property at all times, this can be accomplished by formatting the restriction like so: +``` apiVersion: spinnaker.io/v1alpha2 kind: SpinnakerService metadata: @@ -225,9 +235,10 @@ spec: - "banana" - "prod" - "devb" - +``` * Compile and apply the changes into the environment using ```kubectl```Test and confirm the access restrictions.  +``` > export FIAT=http://spin-fiat:7003 > curl -s $FIAT/authorize/$user-or-service-account​ - +``` diff --git a/kb/armory/How To's/how-to-specify-an-application-owner-with-dinghy.md b/kb/armory/How To's/how-to-specify-an-application-owner-with-dinghy.md new file mode 100644 index 0000000000..a6d61417ad --- /dev/null +++ b/kb/armory/How To's/how-to-specify-an-application-owner-with-dinghy.md @@ -0,0 +1,18 @@ +--- +title: How to specify an application Owner with Dinghy +--- + +## Introduction +When defining an application spec with Dinghy - the user would like to specify the ```Owner field```, sometimes referred to as the ```e-mail field``` in the Applications view. + +## Prerequisites +Fully configured and enabled Dinghy service: +[https://docs.armory.io/docs/armory-admin/dinghy-enable/](https://docs.armory.io/docs/armory-admin/dinghy-enable/) + +## Instructions +The parameter to set Owner field when creating an application with Dinghy is "email". It should reside within the "spec" configuration section of the Dinghyfile: +```"spec": {``` +```"email": ""``` +```},``` +[https://docs.armory.io/docs/spinnaker-user-guides/using-dinghy/#basic-format](https://docs.armory.io/docs/spinnaker-user-guides/using-dinghy/#basic-format) + diff --git a/kb/armory/How To's/how-to-switch-packer's-powershell-provisioner.md b/kb/armory/How To's/how-to-switch-packer's-powershell-provisioner.md new file mode 100644 index 0000000000..fadb97f4cb --- /dev/null +++ b/kb/armory/How To's/how-to-switch-packer's-powershell-provisioner.md @@ -0,0 +1,17 @@ +--- +title: How to Switch Packer's Powershell provisioner +--- + +## Introduction +When baking Windows AMIs, back-to-back bakes may fail on a seemingly random, intermittent basis. +Since there are [multiple open Packer issues](https://github.com/hashicorp/packer/issues?q=is%3Aissue+is%3Aopen+powershell) with how Powershell scripts and their environment variables are handled, it has been observed that switching the Powershell provisioner from ```winRM``` to ```SSH``` addresses this issue. + +## Prerequisites +Administrator access to the Spinnaker environment. + +## Instructions +Apply the following changes to the Packer config: +* In the ```variables``` block, change: ```"aws_winrm_username": null```  to:  ```"aws_ssh_username": "Administrator"```* ``````In the ```builders``` block, change: ```"communicator": "winrm"``` to: ```"communicator": "ssh"```* ``````In the ```provisioner``` block, add the following: ```"execute_command": "powershell -executionpolicy bypass \"& { if (Test-Path variable:global:ProgressPreference){$global:ProgressPreference='SilentlyContinue'};. {{.Vars}}; &'{{.Path}}'; exit $LastExitCode }\""``` +Then, in the Spinnaker configuration manifest +* Under ```virtualizationSettings```, change: ```winRmUserName``` to: ```sshUserName: Administrator```* Save and apply the changes to the environment + diff --git a/kb/armory/How To's/how-to-tag-ebs-volumes-with-spinnaker-using-launch-templates-in-aws.md b/kb/armory/How To's/how-to-tag-ebs-volumes-with-spinnaker-using-launch-templates-in-aws.md new file mode 100644 index 0000000000..755c58f495 --- /dev/null +++ b/kb/armory/How To's/how-to-tag-ebs-volumes-with-spinnaker-using-launch-templates-in-aws.md @@ -0,0 +1,25 @@ +--- +title: How to Tag EBS Volumes with Spinnaker using Launch Templates in AWS +--- + +## Introduction +Tagging is a means of categorization within a cloud environment to identify, organize, search and filter resources.  As teams scale, tagging becomes an essential part of most DevOps teams' best practices. + +In deploying AWS resources in Spinnaker, defining resources happened using Launch Configurations, but AWS has since migrated towards using Launch Templates.  (Please see the following KB Article: [https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010548](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010548)) + +## Prerequisites +Launch templates will need to be enabled in the Spinnaker environment (The steps are described in the Spinnaker OSS docs: [https://spinnaker.io/docs/setup/other_config/server-group-launch-settings/aws-ec2/launch-templates-setup/](https://spinnaker.io/docs/setup/other_config/server-group-launch-settings/aws-ec2/launch-templates-setup/))  + +It is recommended that admins ensure the environment is up-to-date, as certain features are only available with specific versions of Spinnaker. +* Customers should attempt to use Spinnaker 2.27.3+ +* Access to launch templates is available to the Spinnaker Environment. + + +## Instructions +Launch Template [tag specifications feature](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-launchtemplate-tagspecification.html) to [tag EBS volumes](https://github.com/spinnaker/clouddriver/blob/master/clouddriver-aws/src/main/java/com/netflix/spinnaker/clouddriver/aws/services/LaunchTemplateService.java#L601') is a supported feature in Spinnaker. +Since Spinnaker creates new launch templates with every server group, the EBS volumes defined in the launch templates can be tagged using the [*blockDeviceTags*](https://github.com/spinnaker/clouddriver/blob/master/clouddriver-aws/src/main/groovy/com/netflix/spinnaker/clouddriver/aws/deploy/description/BasicAmazonDeployDescription.groovy#L101)* *parameter in the [```createServerGroup``` within the launch template request](https://spinnaker.io/docs/setup/other_config/server-group-launch-settings/aws-ec2/launch-templates/#create-a-server-group-with-launch-template). +For example: `````` +```"blockDeviceTags":{ "test": "myVolume" }``` +  +By defining the value, users can tag newly created EBS volumes for the server group with application, cluster, and user-provided block device tags. The following code was added to the Spinnaker project by AWS to allow for this functionality:[https://github.com/spinnaker/clouddriver/blob/master/clouddriver-aws/src/main/java/com/netflix/spinnaker/clouddriver/aws/deploy/DefaultAmazonResourceTagger.java#L59-L64](https://github.com/spinnaker/clouddriver/blob/master/clouddriver-aws/src/main/java/com/netflix/spinnaker/clouddriver/aws/deploy/DefaultAmazonResourceTagger.java#L59-L64) + diff --git a/kb/armory/How To's/how-to-upgrade-operator.md b/kb/armory/How To's/how-to-upgrade-operator.md new file mode 100644 index 0000000000..aa404b92fd --- /dev/null +++ b/kb/armory/How To's/how-to-upgrade-operator.md @@ -0,0 +1,15 @@ +--- +title: How to Upgrade Operator +--- + +## Introduction +Armory provides Kubernetes Operators that make it easy to install, deploy, and upgrade Armory Enterprise or Spinnaker.  +The Operator contains software extensions to Kubernetes that make use of custom resources to manage applications and their components.  It is recommended that customers only upgrade the Operator version, and do not modify individual container versions within the overall Pod, which can lead to errors and conflicts. + +## Prerequisites +Environment installed with Armory Operator: [https://docs.armory.io/continuous-deployment/installation/armory-operator/op-quickstart/](https://docs.armory.io/continuous-deployment/installation/armory-operator/op-quickstart/) + +## Instructions +Administrators are recommended to read the release notes first before upgrading.  It is recommended that administrators read all release notes between their current deployed version and the version they will be upgrading to.[https://docs.armory.io/continuous-deployment/release-notes/rn-armory-operator/](https://docs.armory.io/continuous-deployment/release-notes/rn-armory-operator/) +Follow the instructions here to proceed with Upgrading the Operator when necessary.[https://docs.armory.io/continuous-deployment/installation/armory-operator/op-manage-operator/#upgrade-the-operator](https://docs.armory.io/continuous-deployment/installation/armory-operator/op-manage-operator/#upgrade-the-operator) + diff --git a/kb/armory/How To's/how-to-upload-to-aws-s3-from-armory-cdsh---spinnaker.md b/kb/armory/How To's/how-to-upload-to-aws-s3-from-armory-cdsh---spinnaker.md new file mode 100644 index 0000000000..7e74237ff0 --- /dev/null +++ b/kb/armory/How To's/how-to-upload-to-aws-s3-from-armory-cdsh---spinnaker.md @@ -0,0 +1,52 @@ +--- +title: How to upload to AWS S3 from Armory CDSH / Spinnaker +--- + +## Introduction +Although Clouddriver can download artifacts from S3 as part of a ```Deploy (Manifest)``` pipeline stage, it cannot natively upload to S3.  A ```Run Job (Manifest) stage``` can be used to upload objects to S3.  The following process example is based on a KB article describing running a generic shell script in Spinnaker ([https://support.armory.io/kb_view.do?sysparm_article=KB0010129](https://support.armory.io/kb_view.do?sysparm_article=KB0010129)).   + +## Prerequisites +Write access to an S3 bucket will be required from the Spinnaker cluster. + +## Instructions +Users looking to write to an S3 bucket will first need to create a ```Run Job (Manifest) stage``` in the pipeline.  The following is an example that should be modified to fit the parameters of the S3 bucket and will write a test message to ```file.tmp``` within the S3 Bucket. +``` +apiVersion: v1 +kind: ConfigMap +data: + script.sh: |- + echo "test file contents" > file.tmp + aws s3 cp file.tmp s3://your-bucket/your-folder/ +metadata: + name: s3-upload-configmap +--- +apiVersion: batch/v1 +kind: Job +metadata: + labels: + app: s3-upload-test + name: s3-upload-test +spec: + backoffLimit: 2 + template: + spec: + containers: + - command: + - sh + - /opt/script/script.sh + image: amazon/aws-cli:latest + name: s3-upload + volumeMounts: + - mountPath: /opt/script + name: s3-upload-configmap + readOnly: false + restartPolicy: Never + volumes: + - configMap: + name: s3-upload-configmap + name: s3-upload-configmap +``` + +The key sections are: +* ```ConfigMap.data.script.sh```: the shell commands to upload a file to s3.  In this example, we generate a test file for upload.  * ```Job.template.spec.containers.image```: This needs to be an image with AWS CLI installed.  * Users should utilize IAM roles to handle authentication if deploying to an EC2 or ECS instance.  Otherwise, they could configure AWS authentication by inserting ```aws configure``` commands in the args section before the upload.  Providing credentials within pipelines is not recommended for secure environments, as the credential data will be present in pipelines and logs.   + diff --git a/kb/armory/How To's/how-to-use-and-migrate-to-launch-templates,-migrating-from-launch-configuration.md b/kb/armory/How To's/how-to-use-and-migrate-to-launch-templates,-migrating-from-launch-configuration.md new file mode 100644 index 0000000000..1ff43e3b6a --- /dev/null +++ b/kb/armory/How To's/how-to-use-and-migrate-to-launch-templates,-migrating-from-launch-configuration.md @@ -0,0 +1,28 @@ +--- +title: How to Use and Migrate to Launch Templates, Migrating from Launch Configuration +--- + +## Introduction +By Default, Spinnaker and customers have been using [Launch Configuration from AWS](https://docs.aws.amazon.com/autoscaling/ec2/userguide/LaunchConfiguration.html) as a way for managing their Auto-Scaling Groups, as it has been the more matured solution.  However, as of 2017, Amazon has introduced the concepts of [Launch Templates](https://docs.aws.amazon.com/autoscaling/ec2/userguide/LaunchTemplates.html) and has been actively developing this method to manage resources.  +Launch templates have been available since 2.24.x, and additional supports to this feature have been added in 2.27 and the upcoming 2.28 release thanks to contributions by AWS in OSS that we leverage in Armory's Enterprise distribution. +In AWS' own documentation, they have since advised [in blog posts](https://aws.amazon.com/blogs/compute/amazon-ec2-auto-scaling-will-no-longer-add-support-for-new-ec2-features-to-launch-configurations/#:~:text=Launch%20templates%20define%20the%20steps,ve%20added%20to%20launch%20templates.) and in their documentation pages about their intention to discontinue support for new features from Launch Configurations. +AWS' own involvement in the OSS community and making changes to Spinnaker has been in the direction of adding and supporting functionality for Launch Templates only, and it is suggested that companies plan to move to this methodology as well.  +Launch templates have been available since 2.24.x, and Amazon is has added additional supports to this feature in 2.27 and the upcoming 2.28 release thanks to contributions by AWS in OSS that we leverage in our Enterprise distribution. + +## Prerequisites +Set up the environment to use Launch Templates using the following OSS documentation:[https://spinnaker.io/docs/setup/other_config/server-group-launch-settings/aws-ec2/launch-templates-setup/](https://spinnaker.io/docs/setup/other_config/server-group-launch-settings/aws-ec2/launch-templates-setup/) +  + +## Instructions +It is highly encourage all administrators enabling Launch Templates to consider a controlled rollout methodology. Please take special note of the [Rollout section near the bottom](https://spinnaker.io/docs/setup/other_config/server-group-launch-settings/aws-ec2/launch-templates-setup/#rollout-configuration), which instructs on how to phase this rollout with a whitelist set of applications to ensure a controlled rollout in the Enterprise cluster. +In general, customers should look to do the following: +Enable [Launch Templates given the above documentation](https://spinnaker.io/docs/setup/other_config/server-group-launch-settings/aws-ec2/launch-templates-setup/) in a sandbox environment +* Adjust the policies attached to the role Spinnaker relies on. See: [https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-launch-template-permissions.html](https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-launch-template-permissions.html)) +* Only whitelist a small set of applications that can be used to perform a validation +Validate that pipelines are behaving as expected, and deploying EC2 applications +* Check and review policies above are set correctly +* Make any adjustments necessary +* Widen the rollout to whitelist an entire AWS account, repeat step 2 +* Enable for all applications, regardless of account, repeat step 2 +* Roll out to Production instance in the same manner + diff --git a/kb/armory/How To's/how-to-use-dinghy-module-in-module-support.md b/kb/armory/How To's/how-to-use-dinghy-module-in-module-support.md new file mode 100644 index 0000000000..df2804885c --- /dev/null +++ b/kb/armory/How To's/how-to-use-dinghy-module-in-module-support.md @@ -0,0 +1,13 @@ +--- +title: How to use Dinghy module in module support +--- + +## Introduction +Module-in-module support is now available in Dinghy 2.24 and above. This means that users can include pipeline definitions from other modules in their Dinghyfile, and reference variables from other modules as needed. + +## Prerequisites +Dinghy 2.24 and above + +## Instructions +Check out the following YouTube video where Principal Software Engineer, Fernando Freire, walks you through how to leverage this feature available in 2.24 and above: + diff --git a/kb/armory/How To's/how-to-use-global-variables-in-dinghy-conditionals.md b/kb/armory/How To's/how-to-use-global-variables-in-dinghy-conditionals.md new file mode 100644 index 0000000000..65f38f4475 --- /dev/null +++ b/kb/armory/How To's/how-to-use-global-variables-in-dinghy-conditionals.md @@ -0,0 +1,86 @@ +--- +title: How to use Global Variables in Dinghy Conditionals +--- + +## Introduction +When defining pipeline as code, Dinghy allows both: +* Conditionals, which are conditions for when something should be rendered in the pipeline* Global variables ,which are variables defined in ```dinghyfiles``` typically used to pass information to/from modules. +These functionalities can be a cornerstone in replicating pipelines across multiple applications, environments, etc. This article will detail how to use global variables inside of dinghy conditionals to achieve a more dynamic and repeatable pipeline definition. + +## Prerequisites +* Must have Dinghy installed and a repository connected.* (Optional) ```arm-cli``` to test syntax and formatting + +## Instructions +Let's start off with a barebones dinghyfile and define a global variable inside of our dinghyfile so that we can use it later. In this example, we've defined the variable env. + +```` +{ + "application": "conditionals", + "globals": { + "env": "dev" +```` +Next, let's add the correct syntax for the conditional +```` +{ + "application": "conditionals", + "globals": { + "env": "dev" + }, + "pipelines": [ + { + "application": "conditionals", + "name": "my-pipeline-name", + "stages": [ + {{ if eq (var "env") "dev" }} +```` +Here we can see that once we are inside the go template conditional denoted by the double curly braces ```{{ }}```, we can define variables using a parenthesis and call them with the above syntax. +Once the conditional has been set, all we need to do now is set what we want to render in our pipeline. In this example, we are rendering a simple wait stage with the name bar. +```` +{ + "application": "conditionals", + "globals": { + "env": "dev" + }, + "pipelines": [ + { + "application": "conditionals", + "name": "my-pipeline-name", + "stages": [ + {{ if eq (var "env") "dev" }} +```` +We can add additional conditions, if we want to, such as below +```` +{ + "application": "conditionals", + "globals": { + "env": "dev" + }, + "pipelines": [ + { + "application": "conditionals", + "name": "my-pipeline-name", + "stages": [ + {{ if eq (var "env") "dev" }} +```` +The rendered output after pushing the example provided in step 4's ```dinghyfile``` will be as such +```` +{ + "application": "conditionals", + "globals": { + "env": "dev" + }, + "pipelines": [ + { + "application": "conditionals", + "name": "my-pipeline-name", + "stages": [ + { + "type": "wait", + "name": "bar" + } + ] + } + ] +}​ +```` + diff --git a/kb/armory/How To's/how-to-use-spot-instances-with-launch-templates-for-spinnaker-.md b/kb/armory/How To's/how-to-use-spot-instances-with-launch-templates-for-spinnaker-.md new file mode 100644 index 0000000000..35be6364d7 --- /dev/null +++ b/kb/armory/How To's/how-to-use-spot-instances-with-launch-templates-for-spinnaker-.md @@ -0,0 +1,59 @@ +--- +title: How to use Spot Instances with Launch Templates for Spinnaker +--- + +## Introduction +A spot instance is an instance that uses unused Amazon EC2 capacity at a price much lower than on-demand instances.  Spot instances are mainly utilized for non-critical use cases, as they get terminated once a higher bid is made for the instance. +For more information, please visit AWS' information on Spot Instances: [https://aws.amazon.com/ec2/spot/](https://aws.amazon.com/ec2/spot/) +AWS suggests the best use cases for Spot Instances include +* Containerized workloads* Big data and analytics* CI/CD workloads +Launch Templates are AWS's preferred suggested method for customers to use the latest AWS capabilities.  AWS encourages customers to leverage Launch Templates to use the newest Spot Instance functions and find the latest instance-type options. + +## Prerequisites +Launch templates will need to be enabled in the Spinnaker environment before being able to use Spot instances with Spinnaker +The steps are described in the Spinnaker OSS docs: [https://spinnaker.io/docs/setup/other_config/server-group-launch-settings/aws-ec2/launch-templates-setup/](https://spinnaker.io/docs/setup/other_config/server-group-launch-settings/aws-ec2/launch-templates-setup/) + +It is recommended that admins ensure the environment is up-to-date, as certain features are only available with specific versions of Spinnaker.  Customers should attempt to use Spinnaker 2.27.3+, with some features only available on 2.28.1+ +Customers should also consider implementing security, such as enabling ```IMDSv2``` + + +## Instructions +Customers seeking to implement Spot instances in Spinnaker can utilize Launch Templates.  Users can invoke the Launch Template Feature in Spinnaker using API calls.   +### Via Launch Templates - Using API calls  +Spot Instances using Launch templates allows for additional options to reduce costs by diversifying instances across purchase options and Spot allocation strategies.  AWS recommends following their best practices around Spot Instances: [https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-best-practices.html](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-best-practices.html) +For more information the capabilities of Spot Instances and ASG deployment strategies, please see the following documentation:[https://spinnaker.io/docs/setup/other_config/server-group-launch-settings/aws-ec2/launch-templates/#additional-features](https://spinnaker.io/docs/setup/other_config/server-group-launch-settings/aws-ec2/launch-templates/#additional-features) +* Spot instances, along with other instance types and strategies, can be created by making API calls to Spinnaker.  This allows users and admins to make use of AWS's more advanced strategies around deploymentsCustomers can run a cURL command as per the example below.  +``` +curl -H 'Content-Type: application/json' -d '{ "job": [ + { + "type": "createServerGroup", + "cloudProvider": "aws", + "account": "my-aws-account", + "application": "myAwsApp", + "stack": "myStack", + "credentials": "my-aws-account", + "setLaunchTemplate": true, + "subnetType":"test-subnet" + "availabilityZones": {"us-west-1": ["us-west-1a","us-west-1b","us-west-1c"]}, + "amiName": "ami-12345", + "capacity": {"desired": 3,"max": 5,"min": 3}, + "iamRole":"MyInstanceProfile", + "instanceType":"m4.large", + "requireIMDSv2": true, + "onDemandBaseCapacity":1, + "onDemandPercentageAboveBaseCapacity":50, + "spotAllocationStrategy":"capacity-optimized", + "launchTemplateOverridesForInstanceType": [ + {"instanceType":"m5.large","weightedCapacity":"1","priority": 2}, + {"instanceType":"m5.xlarge","weightedCapacity":"2","priority": 1}] + }], "application": "myAwsApp", "description": "Create New Server Group in cluster myAwsApp"}' -X POST http://my-spinnaker-gate:8084/tasks +``` + +* For more information and examples, please visit the Spinnaker OSS information [https://spinnaker.io/docs/setup/other_config/server-group-launch-settings/aws-ec2/launch-templates/#use-cases--sample-api-requests](https://spinnaker.io/docs/setup/other_config/server-group-launch-settings/aws-ec2/launch-templates/#use-cases--sample-api-requests). +  +### Via Launch Config - Deploy Strategy in the Spinnaker Console (Legacy) +* In the ```Deploy Stage``` of a pipeline, navigate to the ```Deploy configuration``` then select ```Add server group```. +* In the ```Advanced Settings``` section, you will see an option called ```Spot instances price (optional)```.  Users can input a cost threshold for a Spot Instance price.  +* Once the pipeline is executed, Spinnaker will deploy the Spot Instance with the price threshold set for it. +* Please note that options available through the Deploy Stage (as of 2.28.x) are limited since they employ Amazon Launch Configs. + diff --git a/kb/armory/How To's/how-to-use-the-reader-endpoint-for-rds-clusters-in-spinnaker-ha-mode-only.md b/kb/armory/How To's/how-to-use-the-reader-endpoint-for-rds-clusters-in-spinnaker-ha-mode-only.md new file mode 100644 index 0000000000..801bd84c0e --- /dev/null +++ b/kb/armory/How To's/how-to-use-the-reader-endpoint-for-rds-clusters-in-spinnaker-ha-mode-only.md @@ -0,0 +1,241 @@ +--- +title: How to use the Reader Endpoint for RDS Clusters in Spinnaker (HA Mode Only) +--- + +## Introduction +Providing Spinnaker with an SQL database backend has been tested to improve performance in many Spinnaker installations. This article will go over how to further configure the RDS Cluster with Spinnaker to further improve performance by enabling the Reader Endpoint for Clouddriver to use. +At this time, Spinnaker only supports the Reader Instance while Clouddriver is in HA (High Availability) Mode. The reader endpoint can be configured for a normal Clouddriver service.  However, this has not been very well tested and is not generally recommended. + +## Prerequisites +* RDS Cluster with a Reader Endpoint* Running Spinnaker* Clouddriver HA (High Availability) + +## Instructions +1. Create a new patch file with the following configuration as the base. + +``` +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker + namespace: $NAMESPACE +spec: + spinnakerConfig: + profiles: + clouddriver-ro: {} + clouddriver-ro-deck: {} + clouddriver-rw: {} + clouddriver-caching: {} +``` +We split the Clouddriver profiles here so that we can apply custom profiles to each of the Clouddriver services. + +2. Add in the SQL configuration for each of these profiles. An example of this can be seen below +``` +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker + namespace: $NAMESPACE +spec: + spinnakerConfig: + profiles: +################################################################################################################################################################ + clouddriver-ro: + sql: + enabled: true + read-only: true + taskRepository: + enabled: true + cache: + enabled: true + readBatchSize: 500 + writeBatchSize: 300 + scheduler: + enabled: true + unknown-agent-cleanup-agent: + enabled: true + connectionPools: + default: + default: true + jdbcUrl: jdbc:mysql://$SOME_URL_HERE:3306/clouddriver + user: clouddriver_service + password: $PASSWORD + maxPoolSize: 50 + tasks: + user: clouddriver_service + jdbcUrl: jdbc:mysql://$SOME_URL_HERE:3306/clouddriver + password: austinthao + migration: + user: clouddriver_migrate + jdbcUrl: jdbc:mysql://$SOME_URL_HERE:3306/clouddriver + password: $PASSWORD + redis: + enabled: false + cache: + enabled: false + scheduler: + enabled: false + taskRepository: + enabled: false +################################################################################################################################################################ + clouddriver-caching: + sql: + enabled: true + read-only: false + taskRepository: + enabled: true + cache: + enabled: true + readBatchSize: 500 + writeBatchSize: 300 + scheduler: + enabled: true + unknown-agent-cleanup-agent: + enabled: true + connectionPools: + default: + default: true + jdbcUrl: jdbc:mysql://$SOME_URL_HERE:3306/clouddriver + user: clouddriver_service + password: $PASSWORD + maxPoolSize: 50 + tasks: + user: clouddriver_service + jdbcUrl: jdbc:mysql://$SOME_URL_HERE:3306/clouddriver + password: $PASSWORD + migration: + user: clouddriver_migrate + jdbcUrl: jdbc:mysql://$SOME_URL_HERE:3306/clouddriver + password: $PASSWORD + redis: + enabled: false + cache: + enabled: false + scheduler: + enabled: false + taskRepository: + enabled: false +################################################################################################################################################################ + clouddriver-ro-deck: + sql: + enabled: true + read-only: true + taskRepository: + enabled: true + cache: + enabled: true + readBatchSize: 500 + writeBatchSize: 300 + scheduler: + enabled: true + unknown-agent-cleanup-agent: + enabled: true + connectionPools: + default: + default: true + jdbcUrl: jdbc:mysql://$SOME_URL_HERE:3306/clouddriver + user: clouddriver_service + password: $PASSWORD + maxPoolSize: 49 + tasks: + user: clouddriver_service + jdbcUrl: jdbc:mysql://$SOME_URL_HERE:3306/clouddriver + password: $PASSWORD + migration: + user: clouddriver_migrate + jdbcUrl: jdbc:mysql://$SOME_URL_HERE:3306/clouddriver + password: $PASSWORD + redis: + enabled: false + cache: + enabled: false + scheduler: + enabled: false + taskRepository: + enabled: false +################################################################################################################################################################ + clouddriver-rw: + sql: + enabled: true + read-only: false + taskRepository: + enabled: true + cache: + enabled: true + readBatchSize: 500 + writeBatchSize: 300 + scheduler: + enabled: true + unknown-agent-cleanup-agent: + enabled: true + connectionPools: + default: + default: true + jdbcUrl: jdbc:mysql://$SOME_URL_HERE:3306/clouddriver + user: clouddriver_service + password: $PASSWORD + maxPoolSize: 50 + tasks: + user: clouddriver_service + jdbcUrl: jdbc:mysql://$SOME_URL_HERE:3306/clouddriver + password: $PASSWORD + migration: + user: clouddriver_migrate + jdbcUrl: jdbc:mysql://$SOME_URL_HERE:3306/clouddriver + password: $PASSWORD + redis: + enabled: false + cache: + enabled: false + scheduler: + enabled: false + taskRepository: + enabled: false +``` + +3. Next, we'll need to set Clouddriver's read only services (```clouddriver-ro``` and ```clouddriver-ro-deck```) to read only. +``` +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker + namespace: $NAMESPACE +spec: + spinnakerConfig: + profiles: +################################################################################################################################################################ + clouddriver-ro: + read-only: true +################################################################################################################################################################ + clouddriver-ro-deck: + read-only: true +``` + +4. Finally, we can configure the reader endpoint for the default connectionPools for both ```clouddriver-ro``` and ```clouddriver-ro-deck```. Please keep in mind that the tasks connectionPool will still need the writer endpoint as it still needs to ```UPDATE``` the ```DATABASECHANGELOCK``` table. +``` +apiVersion: spinnaker.armory.io/v1alpha2 +kind: SpinnakerService +metadata: + name: spinnaker + namespace: $NAMESPACE +spec: + spinnakerConfig: + profiles: +################################################################################################################################################################ + clouddriver-ro-deck: + sql: + connectionPools: + default: + default: true + jdbcUrl: jdbc:mysql:$READER_ENDPOINT_HERE:3306/clouddriver + user: clouddriver_service + password: $PASSWORD +################################################################################################################################################################ + clouddriver-ro: + sql: + connectionPools: + default: + default: true + jdbcUrl: jdbc:mysql:$READER_ENDPOINT_HERE:3306/clouddriver + user: clouddriver_service + password: $PASSWORD +``` \ No newline at end of file diff --git a/kb/armory/How To's/how-to-use-trunc-and-split-functions-in-pipelines-as-code.md b/kb/armory/How To's/how-to-use-trunc-and-split-functions-in-pipelines-as-code.md new file mode 100644 index 0000000000..a91c5bcfd5 --- /dev/null +++ b/kb/armory/How To's/how-to-use-trunc-and-split-functions-in-pipelines-as-code.md @@ -0,0 +1,87 @@ +--- +title: How to use Trunc and Split functions in Pipelines as Code +--- + +## Introduction +Using Pipelines as Code (PaC) is an important feature that helps create Spinnaker pipelines quickly and efficiently. By using ```Dinghyfiles``` strategically, Spinnaker users can use the same PaC code for different applications, saving a lot of development time and making Continuous Deployment workflows run smoother. + +As pipelines get more complex, ```Dinghyfiles``` become more advanced, so advanced functions can help create consistency across different pipelines and their stages. This article shows users how to use ***Trunc and Split*** functions to set up pipelines in a detailed and effective way. + +## Prerequisites +* Pipelines as Code plugin/service available in Spinnaker (Installation and configuration instructions available [here](https://docs.armory.io/plugins/pipelines-as-code/install/armory-cd/))* ARM-CLI used for debugging purposes and validation - [ARM-CLI](https://docs.armory.io/plugins/pipelines-as-code/arm-cli/) + +## Instructions +In this example, we want to provision three different wait stages based on some string manipulation using ```trunc``` and ```split``` commands +``` +{ + "application": "splitexample", + "pipelines": [ + { + "application": "splitexample", + "name": "my-pipeline-name", + "stages": [ + {{ $count := 0 }} + {{ $a := trunc 11 "foo$bar$baz "}} + {{ $b := split "$" $a }} + {{ range $b }} + {{ + module "dinghy-modules/wait.stage.module" + "waitname" . + }} + {{ $count = add $count 1 }}, + {{ end }} + {{ + module "dinghy-modules/wait.stage.module" + "waitname" "Final Wait" + }} + ] +}]} +``` + +Starting from the ```"foo$bar$baz  "``` string, users need to truncate the whitespace.  This can be accomplished by using the ```trunc``` function as seen above.  After applying the function, the data will be appended to the ```"foo$bar$baz"``` string.  This string can be remapped into a list by using the ```split``` function with the ```"$"``` operator. +* Trunc accepts a numeric value as a parameter.  The value denotes the number of characters to be trimmed either from the start of the string (using positive integers) or from the end of the string (using negative integers).* Split splits a string into substrings based on a delimiter defined as a positional parameter. +The final pipeline JSON definition would look like this after all the logic has been applied:  +``` +{ + "application": "splitexample", + "pipelines": [ + { + "application": "splitexample", + "name": "my-pipeline-name", + "stages": [ + + + + + { + "name": "foo", + "type": "wait", + "waitTime": 100 +} + , + + { + "name": "bar", + "type": "wait", + "waitTime": 100 +} + , + + { + "name": "ba", + "type": "wait", + "waitTime": 100 +} + , + + { + "name": "Final Wait", + "type": "wait", + "waitTime": 100 +} + ] +}]} +``` +  +For more information on Sprig functions, please visit: [https://masterminds.github.io/sprig/strings.html](https://masterminds.github.io/sprig/strings.html) + diff --git a/kb/armory/How To's/how-to-utilize-canary-analysis-in-other-appications-within-spinnaker.md b/kb/armory/How To's/how-to-utilize-canary-analysis-in-other-appications-within-spinnaker.md new file mode 100644 index 0000000000..792a422f65 --- /dev/null +++ b/kb/armory/How To's/how-to-utilize-canary-analysis-in-other-appications-within-spinnaker.md @@ -0,0 +1,46 @@ +--- +title: How to utilize Canary analysis in other Appications within Spinnaker +--- + +## Introduction +Canary is a deployment strategy in which a change is partially rolled out, then evaluated against the current deployment (baseline) according to a set guideline.  The strategy administrators determine the guidelines that follow a pass/fail response. +Canary deployments are used to ensure that the new deployment is within the guidelines at a point in time of the deployment.  This check determines if the rest of the deployment can continue.  This evaluation uses key metrics chosen when the Canary deployment was configured. +For more information about setting up Canary analysis, please visit our Armory Docs page: +[https://docs.armory.io/armory-enterprise/spinnaker-user-guides/canary/kayenta-canary-use/](https://docs.armory.io/armory-enterprise/spinnaker-user-guides/canary/kayenta-canary-use/) +However, there may come a time when the Canary Analysis results will be used in other applications in the environment. + +## Prerequisites +Canary analysis and the Kayenta service will need to be enabled in the Spinnaker environment.  +The steps to enable it can be found here: +[https://docs.armory.io/armory-enterprise/spinnaker-user-guides/canary/kayenta-canary-use/](https://docs.armory.io/armory-enterprise/spinnaker-user-guides/canary/kayenta-canary-use/)) +  +  + +## Instructions +```App A```'s Canary config will need to be utilized within ```App B```'s canary analysis results.  + +### Enable Canary in App B +(Please note that this may already be available in the Spinnaker application) +* Login to Spinnaker +* Navigate to ```Applications``` then select ```App B``` +* Navigate to ```Config``` on the left pane of the Spinnaker application then, select ```Features``` +* Check ```Canary``` then save changes +* On the left pane of the Spinnaker application the ```Canary configs``` will now be available to ```App B``` + +#### Utilize the Canary Config from App A in a pipeline for App B  +* Navigate to a pipeline that needs to utilize the ```Canary config``` created in ```App A``` +* Select ```Configure``` the select ```Add stage``` +* In the ```Type``` option, choose ```Canary analysis``` +* A configuration section named ```Canary Analysis Configuration``` will be a part of the configuration of the stage.  In the ```Config name```, use the drop-down to select the Canary config created in App A.  + +Please note that while App B can now utilize the application, it will not be able to edit the Canary Config, unless the below changes are made to provide permissions to allow the application to make the edit. +  +#### Allow App B to edit the config from App A + +### Steps  +* Login into the Spinnaker account* Navigate to ```Applications``` then select ```App A``` +* Navigate to ```Canary configs``` then select the ```Canary config``` that ```App B``` will utilize  +* Select ```JSON``` and then ```edit``` +* Navigate to the ```Applications``` section in the **JSON** file and add ```App B``` to the listed ```Applications```. +Now ```App B``` will be able to edit the Canary config file created in ```App A``` + From ded952d2a9ac440ec1e95363d5d78b0c0995d25d Mon Sep 17 00:00:00 2001 From: joshua-armory Date: Wed, 16 Oct 2024 17:26:54 -0400 Subject: [PATCH 2/3] fixed kb titles --- ...-as-user-already-exists-status-code-409.md | 18 ---------- ...als-did-not-have-\"xxxxxx-xxxxx'-scope.md" | 33 ----------------- ...ot-register-user-as-user-already-exists.md | 18 ---------- ...rk-agent-cannot-connect-unauthenticated.md | 35 ------------------- ...ailable-request-timed-out-after-10000ms.md | 0 ...with-access-denied-to-application-error.md | 3 +- .../delete-pipeline-executions-in-redis.md | 10 +++--- ...when-list-customresourcedefinition-null.md | 5 +-- ...d-balancer-service-does-not-exist-error.md | 1 + ...ab-missing-previously-available-options.md | 0 ...ed-a-feature-that-requires-registration.md | 0 ...erver-group-names-for-cluster-are-taken.md | 0 ...cannot-read-property-label-of-undefined.md | 0 ...stages-configured-to-ignore-the-failure.md | 0 ...st-to-stabilize-unexpected-task-failure.md | 0 ...default-base-branch-from-master-to-main.md | 1 + ...-config-artifact-for-google-app-engine.md} | 2 ++ ...-json-canary-config-in-the-spinnaker-ui.md | 0 ...by-admin-access-users-when-using-run-as.md | 0 ...of-project-aurora-for-spinnaker-no-helm.md | 8 +++++ ...clouddrivers-redis-to-its-own-instance.md} | 2 ++ ...-using-api-version-networking.k8s.io-v1.md | 0 ...bel-keys-in-common-with-target-workload.md | 0 ...-on-wait-for-manifest-to-stabilize-task.md | 7 ++-- ...ot-work--error--got-map-expected-string.md | 2 ++ ...-count-metric-unavailable-in-prometheus.md | 0 ...ions-folder-with-execution-id-times-out.md | 2 ++ ...on-issue-must-be-of-type-string-integer.md | 3 +- ...spring-expression-language-spel-samples.md | 14 ++++---- ...aform-executable-file-not-found-in-path.md | 0 ...results-in-a-missing-client-token-error.md | 0 ...not-be-presented-in-certificate_request.md | 0 32 files changed, 41 insertions(+), 123 deletions(-) delete mode 100644 kb/armory/General/armory-cloud---cannot-register-user-as-user-already-exists-status-code-409.md delete mode 100644 "kb/armory/General/armory-cloud---remote-network-agent-cannot-connect-unauthenticated--provided-credentials-did-not-have-\"xxxxxx-xxxxx'-scope.md" delete mode 100644 kb/armory/General/armory-cloud-cannot-register-user-as-user-already-exists.md delete mode 100644 kb/armory/General/armory-cloud-remote-network-agent-cannot-connect-unauthenticated.md rename "kb/armory/General/clouddriver-shows-\"connection-is-not-available,-request-timed-out-after-10000ms.\".md" => kb/armory/General/clouddriver-shows-connection-is-not-available-request-timed-out-after-10000ms.md (100%) rename "kb/armory/General/custom-runjobs-intermittently-failing-with-\"access-denied-to-application\"-error.md" => kb/armory/General/custom-runjobs-intermittently-failing-with-access-denied-to-application-error.md (93%) rename "kb/armory/General/deploying-through-armory-agent-for-kubernetes-with-crd-results-in-\"error-occurred-when-list-customresourcedefinition-null\".md" => kb/armory/General/deploying-through-armory-agent-for-kubernetes-with-crd-results-in-error-occurred-when-list-customresourcedefinition-null.md (99%) rename "kb/armory/General/deployment-fails-with-\"load-balancer-service--does-not-exist\"-error.md" => kb/armory/General/deployment-fails-with-load-balancer-service-does-not-exist-error.md (99%) rename "kb/armory/General/deployments-cluster-tab-missing-\"undo-rollback\"-feature---clusters-tab-missing-previously-available-options.md" => kb/armory/General/deployments-cluster-tab-missing-undo-rollback-feature-clusters-tab-missing-previously-available-options.md (100%) rename "kb/armory/General/disable-deployment-registration-banner-message-\"you-configured-a-feature-that-requires-registration\".md" => kb/armory/General/disable-deployment-registration-banner-message-you-configured-a-feature-that-requires-registration.md (100%) rename "kb/armory/General/ecs-deployment-errors-with-\"all-server-group-names-for-cluster....are-taken\".md" => kb/armory/General/ecs-deployment-errors-with-all-server-group-names-for-cluster-are-taken.md (100%) rename "kb/armory/General/editing-application-permissions-yields-an-error-\"cannot-read-property-'label'-of-undefined\".md" => kb/armory/General/editing-application-permissions-yields-an-error-cannot-read-property-label-of-undefined.md (100%) rename "kb/armory/General/errors-don't-propagate-from-child-to-parent-pipelines-if-they-occur-in-stages-configured-to-\"ignore-the-failure\".md" => kb/armory/General/errors-dont-propagate-from-child-to-parent-pipelines-if-they-occur-in-stages-configured-to-ignore-the-failure.md (100%) rename "kb/armory/General/failed-to-read-manifest--\"exception-wait-for-manifest-to-stabilize.-unexpected-task-failure\".md" => kb/armory/General/failed-to-read-manifest-exception-wait-for-manifest-to-stabilize-unexpected-task-failure.md (100%) rename "kb/armory/General/github-changed-default-base-branch-from-\"master\"-to-\"main\".md" => kb/armory/General/github-changed-default-base-branch-from-master-to-main.md (99%) rename kb/armory/General/{groovy.lang.missingmethodexception-when-referencing-github-config-artifact-for-google-app-engine.md => groovy-missingmethodexception-when-referencing-github-config-artifact-for-google-app-engine.md} (99%) rename "kb/armory/General/infinite-ui-loop-\"save-changes\"-stuck-attempting-to-save-a-json-canary-config-in-the-spinnaker-ui.md" => kb/armory/General/infinite-ui-loop-save-changes-stuck-attempting-to-save-a-json-canary-config-in-the-spinnaker-ui.md (100%) rename "kb/armory/General/locked-pipelines-cannot-be-unlocked-except-by-admin-access-users-when-using-\"run-as\".md" => kb/armory/General/locked-pipelines-cannot-be-unlocked-except-by-admin-access-users-when-using-run-as.md (100%) rename "kb/armory/General/manual-installation-of-project-aurora-for-spinnaker\342\204\242-no-helm.md" => kb/armory/General/manual-installation-of-project-aurora-for-spinnaker-no-helm.md (99%) rename kb/armory/General/{migrate-clouddriver's-redis-to-its-own-instance.md => migrate-clouddrivers-redis-to-its-own-instance.md} (98%) rename "kb/armory/General/no-ingress-is-supported-when-using-api-version-\"networking.k8s.io-v1\".md" => kb/armory/General/no-ingress-is-supported-when-using-api-version-networking.k8s.io-v1.md (100%) rename "kb/armory/General/pipeline-execution-error-\"service-selector-must-have-no-label-keys-in-common-with-target-workload\".md" => kb/armory/General/pipeline-execution-error-service-selector-must-have-no-label-keys-in-common-with-target-workload.md (100%) rename "kb/armory/General/pipeline-stuck-on-\"wait-for-manifest-to-stabilize\"-task.md" => kb/armory/General/pipeline-stuck-on-wait-for-manifest-to-stabilize-task.md (72%) rename "kb/armory/General/resource-requests-on-custom-stages-does-not-work--error--got-\"map\",-expected-\"string\".md" => kb/armory/General/resource-requests-on-custom-stages-does-not-work--error--got-map-expected-string.md (99%) rename "kb/armory/General/rosco's-\"bakescompleted_seconds_count\"-metric-unavailable-in-prometheus.md" => kb/armory/General/rosco-bakescompleted-seconds-count-metric-unavailable-in-prometheus.md (100%) rename "kb/armory/General/spel-expression-issue--must-be-of-type-string--\"integer\".md" => kb/armory/General/spel-expression-issue-must-be-of-type-string-integer.md (72%) rename "kb/armory/General/terraformer-error-exec--\"terraform\"--executable-file-not-found-in-$path.md" => kb/armory/General/terraformer-error-exec-terraform-executable-file-not-found-in-path.md (100%) rename "kb/armory/General/trying-to-turn-on-vault-results-in-a-\"missing-client-token-error.md" => kb/armory/General/trying-to-turn-on-vault-results-in-a-missing-client-token-error.md (100%) rename "kb/armory/How To's/how-to-fix-tls-error-\"reason--extension-5-should-not-be-presented-in-certificate_request.\".md" => kb/armory/How To's/how-to-fix-tls-error-reason-extension-5-should-not-be-presented-in-certificate_request.md (100%) diff --git a/kb/armory/General/armory-cloud---cannot-register-user-as-user-already-exists-status-code-409.md b/kb/armory/General/armory-cloud---cannot-register-user-as-user-already-exists-status-code-409.md deleted file mode 100644 index fd35ab3706..0000000000 --- a/kb/armory/General/armory-cloud---cannot-register-user-as-user-already-exists-status-code-409.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Armory Cloud - Cannot Register User as User Already Exists (status code 409) ---- - -## Issue -**Early Access****As part of participating in an early access program, you will receive information from Armory that is not yet publicly available and is subject to your NDA or confidentiality obligations with Armory.** -Administrators attempting to register new users in their Armory Cloud instance may find that they receive an error when attempting to add them through the console.  The message advises: ```Request failed with status code 409: The user already exists: Error ID: [uuid of instance]``` -Customers cannot find the user in their user list, and if the user in question attempts to log in, they are able to, but do not see anything, or have a blank login.  Even after resetting passwords, they still see the same results. -  - - -## Cause -The main cause is because a user has been associated with an incorrect account.  This will mainly happen as a result of a user attempting to register an account and signed up in the [Armory Cloud signup page](https://console.cloud.armory.io/signup), and did not provide the exact spelling of the Company as registered with Armory.  -For example, the company could be registered as ```ACME``` with Armory, but the person entered ```ACME Inc``` instead -As a result, the user is registered with Armory, and cannot be re-invited, and if attempting to sign in, they will not see any data because the company does not exist within Armory records -  - - diff --git "a/kb/armory/General/armory-cloud---remote-network-agent-cannot-connect-unauthenticated--provided-credentials-did-not-have-\"xxxxxx-xxxxx'-scope.md" "b/kb/armory/General/armory-cloud---remote-network-agent-cannot-connect-unauthenticated--provided-credentials-did-not-have-\"xxxxxx-xxxxx'-scope.md" deleted file mode 100644 index 856cf168ec..0000000000 --- "a/kb/armory/General/armory-cloud---remote-network-agent-cannot-connect-unauthenticated--provided-credentials-did-not-have-\"xxxxxx-xxxxx'-scope.md" +++ /dev/null @@ -1,33 +0,0 @@ ---- -title: Armory Cloud - Remote Network Agent cannot connect (UNAUTHENTICATED- provided credentials did not have "xxxxxx-xxxxx' scope) ---- - -## Issue -**Early Access****As part of participating in an early access program, you will receive information from Armory that is not yet publicly available and is subject to your NDA or confidentiality obligations with Armory.** -Admins may find that the Remote Network Agent pods are unable to start successfully.   -Upon investigation, the following types of errors can be found in the logs of Remote Network Agent pods.  The specific scope that is shown below, ```connect:agentHub``` may differ in the customer environment -``` -[2022-02-11 00:00:02,118] [INFO] An error occurred, attempting to re-connect with hub -[2022-02-11 00:00:07,122] [INFO] The Remote Network Agent is connected to Armory Cloud Agent Hub waiting for requests -[2022-02-11 00:00:07,134] [ERROR] Error while receiving messages from the hub -io.grpc.StatusRuntimeException: UNAUTHENTICATED: provided credentials did not have 'connect:agentHub' scope - at io.grpc.Status.asRuntimeException(Status.java:535) - at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478) - at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463) - at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:427) - at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:460) - at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:557) - at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:69) - at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:738) - at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:717) - at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) - at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) - at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) - at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) - at java.base/java.lang.Thread.run(Thread.java:833) -[2022-02-11 00:00:07,134] [WARN] Agent exited while loop without shutting down channel, telling channel to shutdown now -[2022-02-11 00:00:07,135] [INFO] An error occurred, attempting to re-connect with hub -``` -## Cause -A change in scope can be found if an environment hasn't updated Agent in a while, or the environment is using an out-of-date version of Agent.  A scope change is just one of the reasons why the following error could occur.      - diff --git a/kb/armory/General/armory-cloud-cannot-register-user-as-user-already-exists.md b/kb/armory/General/armory-cloud-cannot-register-user-as-user-already-exists.md deleted file mode 100644 index fd35ab3706..0000000000 --- a/kb/armory/General/armory-cloud-cannot-register-user-as-user-already-exists.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Armory Cloud - Cannot Register User as User Already Exists (status code 409) ---- - -## Issue -**Early Access****As part of participating in an early access program, you will receive information from Armory that is not yet publicly available and is subject to your NDA or confidentiality obligations with Armory.** -Administrators attempting to register new users in their Armory Cloud instance may find that they receive an error when attempting to add them through the console.  The message advises: ```Request failed with status code 409: The user already exists: Error ID: [uuid of instance]``` -Customers cannot find the user in their user list, and if the user in question attempts to log in, they are able to, but do not see anything, or have a blank login.  Even after resetting passwords, they still see the same results. -  - - -## Cause -The main cause is because a user has been associated with an incorrect account.  This will mainly happen as a result of a user attempting to register an account and signed up in the [Armory Cloud signup page](https://console.cloud.armory.io/signup), and did not provide the exact spelling of the Company as registered with Armory.  -For example, the company could be registered as ```ACME``` with Armory, but the person entered ```ACME Inc``` instead -As a result, the user is registered with Armory, and cannot be re-invited, and if attempting to sign in, they will not see any data because the company does not exist within Armory records -  - - diff --git a/kb/armory/General/armory-cloud-remote-network-agent-cannot-connect-unauthenticated.md b/kb/armory/General/armory-cloud-remote-network-agent-cannot-connect-unauthenticated.md deleted file mode 100644 index 73fa194559..0000000000 --- a/kb/armory/General/armory-cloud-remote-network-agent-cannot-connect-unauthenticated.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -title: Armory Cloud - Remote Network Agent cannot connect (UNAUTHENTICATED) ---- - -## Issue -**Early Access****As part of participating in an early access program, you will receive information from Armory that is not yet publicly available and is subject to your NDA or confidentiality obligations with Armory.** -Admins may find that the Remote Network Agent pods are unable to start successfully.   -Upon investigation, the following types of errors can be found in the logs of Remote Network Agent pods.  The specific scope that is shown below, ```connect:agentHub``` may differ in the customer environment - -``` -[2022-02-11 00:00:02,118] [INFO] An error occurred, attempting to re-connect with hub -[2022-02-11 00:00:07,122] [INFO] The Remote Network Agent is connected to Armory Cloud Agent Hub waiting for requests -[2022-02-11 00:00:07,134] [ERROR] Error while receiving messages from the hub -io.grpc.StatusRuntimeException: UNAUTHENTICATED: provided credentials did not have 'connect:agentHub' scope - at io.grpc.Status.asRuntimeException(Status.java:535) - at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478) - at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463) - at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:427) - at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:460) - at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:557) - at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:69) - at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:738) - at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:717) - at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) - at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) - at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) - at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) - at java.base/java.lang.Thread.run(Thread.java:833) -[2022-02-11 00:00:07,134] [WARN] Agent exited while loop without shutting down channel, telling channel to shutdown now -[2022-02-11 00:00:07,135] [INFO] An error occurred, attempting to re-connect with hub -``` - -## Cause -A change in scope can be found if an environment hasn't updated Agent in a while, or the environment is using an out-of-date version of Agent.  A scope change is just one of the reasons why the following error could occur.      - diff --git "a/kb/armory/General/clouddriver-shows-\"connection-is-not-available,-request-timed-out-after-10000ms.\".md" b/kb/armory/General/clouddriver-shows-connection-is-not-available-request-timed-out-after-10000ms.md similarity index 100% rename from "kb/armory/General/clouddriver-shows-\"connection-is-not-available,-request-timed-out-after-10000ms.\".md" rename to kb/armory/General/clouddriver-shows-connection-is-not-available-request-timed-out-after-10000ms.md diff --git "a/kb/armory/General/custom-runjobs-intermittently-failing-with-\"access-denied-to-application\"-error.md" b/kb/armory/General/custom-runjobs-intermittently-failing-with-access-denied-to-application-error.md similarity index 93% rename from "kb/armory/General/custom-runjobs-intermittently-failing-with-\"access-denied-to-application\"-error.md" rename to kb/armory/General/custom-runjobs-intermittently-failing-with-access-denied-to-application-error.md index f0478a0fa6..fc32a2a7d3 100644 --- "a/kb/armory/General/custom-runjobs-intermittently-failing-with-\"access-denied-to-application\"-error.md" +++ b/kb/armory/General/custom-runjobs-intermittently-failing-with-access-denied-to-application-error.md @@ -12,5 +12,6 @@ This is a known issue that has been addressed in Armory Spinnaker 2.26.1, specif The issue stems from Orca passing an application name to Clouddriver to ask for permissions, however no such application actually exists. In order to confirm this indeed is the case for a particular execution failure, look at the Clouddriver log for the following error: ```FiatAccessDeniedExceptionHandler : Encountered exception while processing request GET:/{}applications/``` -And confirm `````` is not an application that exists in your environment. + +And confirm it is not an application that exists in your environment. diff --git a/kb/armory/General/delete-pipeline-executions-in-redis.md b/kb/armory/General/delete-pipeline-executions-in-redis.md index da3e811e4c..540df2804b 100644 --- a/kb/armory/General/delete-pipeline-executions-in-redis.md +++ b/kb/armory/General/delete-pipeline-executions-in-redis.md @@ -12,11 +12,11 @@ Both pieces of information can be found in the execution details.  Navigate to The pipeline id will be labeled ```id``` and will be one of the first lines. The execution id will be ```execution_id``` and will be towards the end. #### Via the URL of Resources in the Console -The URL for the pipeline when editing the pipeline from the console also contains the ```Pipeline ID``````/#/applications//executions/configure/****``` -The URL for the permalink (next to the source location above) will contain the information about the ```Pipeline Execution ID``````/#/applications//executions/details/****?stage=0&step=0&details=``` +The URL for the pipeline when editing the pipeline from the console also contains the ```Pipeline ID`/#/applications//executions/configure/****``` +The URL for the permalink (next to the source location above) will contain the information about the ```Pipeline Execution ID /#/applications//executions/details/****?stage=0&step=0&details=``` #### Via Orca Logs The ```Pipeline Execution ID``` can also be found within the logs of the Orca instance, by running -``````kubectl logs -n deploy/spin-orca -f `````` +```kubectl logs -n deploy/spin-orca -f ``` Sometimes, locating the execution ID can be difficult unless it can be identified from any other executions that are running at the time ## Instructions @@ -25,7 +25,7 @@ Display all executions from this pipeline: 2. For each execution that you want to remove: ```zrem pipeline:executions:{pipeline_id} {execution_id}​``` 3. Delete the execution: +```` del pipeline:${execution_id} del orchestration:${execution_id} ​ - - +```` \ No newline at end of file diff --git "a/kb/armory/General/deploying-through-armory-agent-for-kubernetes-with-crd-results-in-\"error-occurred-when-list-customresourcedefinition-null\".md" b/kb/armory/General/deploying-through-armory-agent-for-kubernetes-with-crd-results-in-error-occurred-when-list-customresourcedefinition-null.md similarity index 99% rename from "kb/armory/General/deploying-through-armory-agent-for-kubernetes-with-crd-results-in-\"error-occurred-when-list-customresourcedefinition-null\".md" rename to kb/armory/General/deploying-through-armory-agent-for-kubernetes-with-crd-results-in-error-occurred-when-list-customresourcedefinition-null.md index 950d57c0e6..58a1a6d6b4 100644 --- "a/kb/armory/General/deploying-through-armory-agent-for-kubernetes-with-crd-results-in-\"error-occurred-when-list-customresourcedefinition-null\".md" +++ b/kb/armory/General/deploying-through-armory-agent-for-kubernetes-with-crd-results-in-error-occurred-when-list-customresourcedefinition-null.md @@ -4,7 +4,8 @@ title: Deploying through Armory Agent for Kubernetes with CRD results in "Error ## Issue Customers may find that Kubernetes custom resource definition (CRD) deployments fail with the below error when deployed through the Armory Agent for Kubernetes -``` + +```` exception="io.armory.kubesvc.services.ops.executor.KubesvcRuntimeException: Error occurred when List customResourceDefinition/null in : (401: namespace default has not been authorized in the configuration of account xxxxxxxxxxx) at io.armory.kubesvc.util.OperationUtils.perform(OperationUtils.java:79) at io.armory.kubesvc.services.ops.executor.KubesvcExecutor.list(KubesvcExecutor.java:254) @@ -133,7 +134,7 @@ exception="io.armory.kubesvc.services.ops.executor.KubesvcRuntimeException: Erro at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:829)" -``` +```` ## Cause This is identified as an issue with the ```Armory Agent plugin ```if the version used is below ```0.8.26```, ```0.9.18```, or ```0.10.2```.  Below is the plugin matrix for corresponding Spinnaker versions that had the issue Armory Enterprise (Spinnaker) VersionAgent plugin Version2.25.x (1.25.x)0.8.262.26.x (1.26.x)0.9.182.27.x (1.27.x)0.10.2 diff --git "a/kb/armory/General/deployment-fails-with-\"load-balancer-service--does-not-exist\"-error.md" b/kb/armory/General/deployment-fails-with-load-balancer-service-does-not-exist-error.md similarity index 99% rename from "kb/armory/General/deployment-fails-with-\"load-balancer-service--does-not-exist\"-error.md" rename to kb/armory/General/deployment-fails-with-load-balancer-service-does-not-exist-error.md index b5648b299c..95eac96672 100644 --- "a/kb/armory/General/deployment-fails-with-\"load-balancer-service--does-not-exist\"-error.md" +++ b/kb/armory/General/deployment-fails-with-load-balancer-service-does-not-exist-error.md @@ -13,6 +13,7 @@ This is a known bug of Kubernetes, Kustomize, and even Helm. There is another Kn [https://kb.armory.io/s/article/Load-balancer-service-svc-spinnaker-demo-does-not-exist](https://kb.armory.io/s/article/Load-balancer-service-svc-spinnaker-demo-does-not-exist) Additionally, there is a very similar issue on Github. [https://github.com/spinnaker/spinnaker/issues/5040](https://github.com/spinnaker/spinnaker/issues/5040) + Kubernetes has trouble viewing manifests when deployment strategies are used, thus causing errors.  diff --git "a/kb/armory/General/deployments-cluster-tab-missing-\"undo-rollback\"-feature---clusters-tab-missing-previously-available-options.md" b/kb/armory/General/deployments-cluster-tab-missing-undo-rollback-feature-clusters-tab-missing-previously-available-options.md similarity index 100% rename from "kb/armory/General/deployments-cluster-tab-missing-\"undo-rollback\"-feature---clusters-tab-missing-previously-available-options.md" rename to kb/armory/General/deployments-cluster-tab-missing-undo-rollback-feature-clusters-tab-missing-previously-available-options.md diff --git "a/kb/armory/General/disable-deployment-registration-banner-message-\"you-configured-a-feature-that-requires-registration\".md" b/kb/armory/General/disable-deployment-registration-banner-message-you-configured-a-feature-that-requires-registration.md similarity index 100% rename from "kb/armory/General/disable-deployment-registration-banner-message-\"you-configured-a-feature-that-requires-registration\".md" rename to kb/armory/General/disable-deployment-registration-banner-message-you-configured-a-feature-that-requires-registration.md diff --git "a/kb/armory/General/ecs-deployment-errors-with-\"all-server-group-names-for-cluster....are-taken\".md" b/kb/armory/General/ecs-deployment-errors-with-all-server-group-names-for-cluster-are-taken.md similarity index 100% rename from "kb/armory/General/ecs-deployment-errors-with-\"all-server-group-names-for-cluster....are-taken\".md" rename to kb/armory/General/ecs-deployment-errors-with-all-server-group-names-for-cluster-are-taken.md diff --git "a/kb/armory/General/editing-application-permissions-yields-an-error-\"cannot-read-property-'label'-of-undefined\".md" b/kb/armory/General/editing-application-permissions-yields-an-error-cannot-read-property-label-of-undefined.md similarity index 100% rename from "kb/armory/General/editing-application-permissions-yields-an-error-\"cannot-read-property-'label'-of-undefined\".md" rename to kb/armory/General/editing-application-permissions-yields-an-error-cannot-read-property-label-of-undefined.md diff --git "a/kb/armory/General/errors-don't-propagate-from-child-to-parent-pipelines-if-they-occur-in-stages-configured-to-\"ignore-the-failure\".md" b/kb/armory/General/errors-dont-propagate-from-child-to-parent-pipelines-if-they-occur-in-stages-configured-to-ignore-the-failure.md similarity index 100% rename from "kb/armory/General/errors-don't-propagate-from-child-to-parent-pipelines-if-they-occur-in-stages-configured-to-\"ignore-the-failure\".md" rename to kb/armory/General/errors-dont-propagate-from-child-to-parent-pipelines-if-they-occur-in-stages-configured-to-ignore-the-failure.md diff --git "a/kb/armory/General/failed-to-read-manifest--\"exception-wait-for-manifest-to-stabilize.-unexpected-task-failure\".md" b/kb/armory/General/failed-to-read-manifest-exception-wait-for-manifest-to-stabilize-unexpected-task-failure.md similarity index 100% rename from "kb/armory/General/failed-to-read-manifest--\"exception-wait-for-manifest-to-stabilize.-unexpected-task-failure\".md" rename to kb/armory/General/failed-to-read-manifest-exception-wait-for-manifest-to-stabilize-unexpected-task-failure.md diff --git "a/kb/armory/General/github-changed-default-base-branch-from-\"master\"-to-\"main\".md" b/kb/armory/General/github-changed-default-base-branch-from-master-to-main.md similarity index 99% rename from "kb/armory/General/github-changed-default-base-branch-from-\"master\"-to-\"main\".md" rename to kb/armory/General/github-changed-default-base-branch-from-master-to-main.md index 69e5f23275..98523a6470 100644 --- "a/kb/armory/General/github-changed-default-base-branch-from-\"master\"-to-\"main\".md" +++ b/kb/armory/General/github-changed-default-base-branch-from-master-to-main.md @@ -4,6 +4,7 @@ title: GitHub Changed (default) Base Branch from "master" to "main" ## Issue [As of Oct 1st, 2020, Github has changed their (default) base branches from "master" to "main"](https://github.com/github/renaming) .  Any new repos created by users will now use "main" as the (default) base branch.**Existing repos are unaffected.** + Artifacts and dinghyfiles coming from GitHub will need to be adjusted by adding additional configuration, until updates to Spinnaker are released to accommodate these changes.  ## Cause diff --git a/kb/armory/General/groovy.lang.missingmethodexception-when-referencing-github-config-artifact-for-google-app-engine.md b/kb/armory/General/groovy-missingmethodexception-when-referencing-github-config-artifact-for-google-app-engine.md similarity index 99% rename from kb/armory/General/groovy.lang.missingmethodexception-when-referencing-github-config-artifact-for-google-app-engine.md rename to kb/armory/General/groovy-missingmethodexception-when-referencing-github-config-artifact-for-google-app-engine.md index 149e2fa3f5..7526e5eb8c 100644 --- a/kb/armory/General/groovy.lang.missingmethodexception-when-referencing-github-config-artifact-for-google-app-engine.md +++ b/kb/armory/General/groovy-missingmethodexception-when-referencing-github-config-artifact-for-google-app-engine.md @@ -4,8 +4,10 @@ title: groovy.lang.MissingMethodException when referencing github config artifac ## Issue ```groovy.lang.MissingMethodException``` error occurs when referencing Github config artifact for Google App Engine (GAE) deploy. The config artifacts were of instance of HashMap not an Artifact object.The following is an example of the error found within Spinnaker +``` Exception ( Create Server Group ) No signature of method: com.netflix.spinnaker.orca.pipeline.util.ArtifactUtils.getBoundArtifactForStage() is applicable for argument types: (com.netflix.spinnaker.orca.pipeline.model.Stage, null, HashMap) values: [Stage {id='', executionId=''}, ...] Possible solutions: getBoundArtifactForStage(com.netflix.spinnaker.orca.pipeline.model.Stage, java.lang.String, com.netflix.spinnaker.kork.artifacts.model.Artifact) +``` ## Cause OSS bug:[https://github.com/spinnaker/spinnaker/issues/5836](https://github.com/spinnaker/spinnaker/issues/5836)The config artifacts were returning a HashMap format instead of an artifact format, within the Orca service diff --git "a/kb/armory/General/infinite-ui-loop-\"save-changes\"-stuck-attempting-to-save-a-json-canary-config-in-the-spinnaker-ui.md" b/kb/armory/General/infinite-ui-loop-save-changes-stuck-attempting-to-save-a-json-canary-config-in-the-spinnaker-ui.md similarity index 100% rename from "kb/armory/General/infinite-ui-loop-\"save-changes\"-stuck-attempting-to-save-a-json-canary-config-in-the-spinnaker-ui.md" rename to kb/armory/General/infinite-ui-loop-save-changes-stuck-attempting-to-save-a-json-canary-config-in-the-spinnaker-ui.md diff --git "a/kb/armory/General/locked-pipelines-cannot-be-unlocked-except-by-admin-access-users-when-using-\"run-as\".md" b/kb/armory/General/locked-pipelines-cannot-be-unlocked-except-by-admin-access-users-when-using-run-as.md similarity index 100% rename from "kb/armory/General/locked-pipelines-cannot-be-unlocked-except-by-admin-access-users-when-using-\"run-as\".md" rename to kb/armory/General/locked-pipelines-cannot-be-unlocked-except-by-admin-access-users-when-using-run-as.md diff --git "a/kb/armory/General/manual-installation-of-project-aurora-for-spinnaker\342\204\242-no-helm.md" b/kb/armory/General/manual-installation-of-project-aurora-for-spinnaker-no-helm.md similarity index 99% rename from "kb/armory/General/manual-installation-of-project-aurora-for-spinnaker\342\204\242-no-helm.md" rename to kb/armory/General/manual-installation-of-project-aurora-for-spinnaker-no-helm.md index b4de2ccc79..44e0dfd54d 100644 --- "a/kb/armory/General/manual-installation-of-project-aurora-for-spinnaker\342\204\242-no-helm.md" +++ b/kb/armory/General/manual-installation-of-project-aurora-for-spinnaker-no-helm.md @@ -21,6 +21,7 @@ Project Aurora for Spinnaker™ requires that Argo Rollouts Controller 1.x or la For information about how to install Argo Rollouts, see [Controller Install](https://argoproj.github.io/argo-rollouts/installation/#controller-installation)[ation](https://argoproj.github.io/argo-rollouts/installation/#controller-installation) in the Argo documentation. ## ****Configure permissions Create a ```ClusterRole```, ```ServiceAccount```, and ```ClusterRoleBinding``` for the Remote Network Agent by applying the following manifest to the ```aurora``` namespace: +``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: @@ -135,8 +136,11 @@ roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: spin-cluster-role +``` + ## ****Configure the Remote Network Agent For information about adding accounts, see the ```kubernetes.accounts[]``` options in the [Agent Configuration documentation](https://docs.armory.io/docs/armory-agent/agent-options/#configuration-options). +``` apiVersion: v1 kind: ConfigMap metadata: @@ -156,8 +160,11 @@ data: verify: true kubernetes: accounts: [] +``` + ## ****Deploy the Remote Network Agent Apply the following Remote Network Agent deployment manifest to the namespace created on the target cluster for the Agent (```aurora``` for the examples on this page): +``` apiVersion: apps/v1 kind: Deployment metadata: @@ -220,6 +227,7 @@ spec: # secret: # defaultMode: 420 # secretName: kubeconfigs-secret +``` ## Next steps After completing the above steps, return to [Get Started with Project Aurora for Spinnaker™](https://docs.armory.io/docs/installation/aurora-install/) and continue from [Verify the Agent Deployment](https://docs.armory.io/docs/installation/aurora-install/#verify-the-agent-deployment). diff --git a/kb/armory/General/migrate-clouddriver's-redis-to-its-own-instance.md b/kb/armory/General/migrate-clouddrivers-redis-to-its-own-instance.md similarity index 98% rename from kb/armory/General/migrate-clouddriver's-redis-to-its-own-instance.md rename to kb/armory/General/migrate-clouddrivers-redis-to-its-own-instance.md index 1db6196a9f..39019854ea 100644 --- a/kb/armory/General/migrate-clouddriver's-redis-to-its-own-instance.md +++ b/kb/armory/General/migrate-clouddrivers-redis-to-its-own-instance.md @@ -4,7 +4,9 @@ title: Migrate Clouddriver's Redis To Its Own Instance * Take a snapshot of your existing Redis cluster* Create a new cluster with the snapshot from the previous step. This will include all the keys from the previous cluster that you can clear out later.Configure Clouddriver to use the new cluster. In your ```/opt/spinnaker/config/clouddriver-local.yml``` add the following: +``` redis: connection: redis://${YOUR_REDIS_HOST}:6379​ +``` * Redeploy your Spinnaker instance diff --git "a/kb/armory/General/no-ingress-is-supported-when-using-api-version-\"networking.k8s.io-v1\".md" b/kb/armory/General/no-ingress-is-supported-when-using-api-version-networking.k8s.io-v1.md similarity index 100% rename from "kb/armory/General/no-ingress-is-supported-when-using-api-version-\"networking.k8s.io-v1\".md" rename to kb/armory/General/no-ingress-is-supported-when-using-api-version-networking.k8s.io-v1.md diff --git "a/kb/armory/General/pipeline-execution-error-\"service-selector-must-have-no-label-keys-in-common-with-target-workload\".md" b/kb/armory/General/pipeline-execution-error-service-selector-must-have-no-label-keys-in-common-with-target-workload.md similarity index 100% rename from "kb/armory/General/pipeline-execution-error-\"service-selector-must-have-no-label-keys-in-common-with-target-workload\".md" rename to kb/armory/General/pipeline-execution-error-service-selector-must-have-no-label-keys-in-common-with-target-workload.md diff --git "a/kb/armory/General/pipeline-stuck-on-\"wait-for-manifest-to-stabilize\"-task.md" b/kb/armory/General/pipeline-stuck-on-wait-for-manifest-to-stabilize-task.md similarity index 72% rename from "kb/armory/General/pipeline-stuck-on-\"wait-for-manifest-to-stabilize\"-task.md" rename to kb/armory/General/pipeline-stuck-on-wait-for-manifest-to-stabilize-task.md index 277e87f483..bacf934ab9 100644 --- "a/kb/armory/General/pipeline-stuck-on-\"wait-for-manifest-to-stabilize\"-task.md" +++ b/kb/armory/General/pipeline-stuck-on-wait-for-manifest-to-stabilize-task.md @@ -7,8 +7,11 @@ When executing a pipeline the ```Wait For Manifest To Stabilize``` stage gets ## Cause This can be caused by a few different reasons.  In order to narrow down the potential root cause, please check the following: -* If the execution was initiated by *non-admin* user and when an *admin* user starts the the pipeline, it gets stuck * If manually triggering the pipeline shows the same behavior -If there is a difference in behavior when checking the above, then the issue may be due to the moniker naming convention. What happens is that the request to fetch a manifest from Clouddriver does two permission checks: +* If the execution was initiated by *non-admin* user and when an *admin* user starts the the pipeline, it gets stuck  +* If manually triggering the pipeline shows the same behavior +* If there is a difference in behavior when checking the above, then the issue may be due to the moniker naming convention. + +What happens is that the request to fetch a manifest from Clouddriver does two permission checks: * Does the user have permission to the account in question?Does the user have permission to the app specified on the manifest's app * In cases where a ```moniker.app``` is explicitly specified on the manifest, then this is the app that is used.* In cases where there is no ```moniker.app``` specified, then the app is determined by parsing using ```frigga``` as per [this article](https://github.com/Netflix/frigga/blob/d24ab6f96529827d7d99e2e3b23e3417f8435cf6/src/main/java/com/netflix/frigga/Names.java#L95). diff --git "a/kb/armory/General/resource-requests-on-custom-stages-does-not-work--error--got-\"map\",-expected-\"string\".md" b/kb/armory/General/resource-requests-on-custom-stages-does-not-work--error--got-map-expected-string.md similarity index 99% rename from "kb/armory/General/resource-requests-on-custom-stages-does-not-work--error--got-\"map\",-expected-\"string\".md" rename to kb/armory/General/resource-requests-on-custom-stages-does-not-work--error--got-map-expected-string.md index 0c1ee24d81..cba5a28b08 100644 --- "a/kb/armory/General/resource-requests-on-custom-stages-does-not-work--error--got-\"map\",-expected-\"string\".md" +++ b/kb/armory/General/resource-requests-on-custom-stages-does-not-work--error--got-map-expected-string.md @@ -5,6 +5,7 @@ title: Resource requests on custom stages does not work- Error- got "map", expec ## Issue An organization may want to set a ```resource request (CPU)``` on jobs for custom stages.  Improper settings can result in an error similar to the one below:```Create failed: error: error validating “STDIN”: error validating data: [ValidationError(Job.spec.template.spec.containers[0].resources.requests.cpu): invalid type for io.k8s.apimachinery.pkg.api.resource.Quantity: got “map”, expected “string”,``` An example manifest snippet is provided below. +``` spec: restartPolicy: Never containers: @@ -14,6 +15,7 @@ An example manifest snippet is provided below. resources: ##### I ADDED THIS TO REQUEST RESOURCES requests: ##### I ADDED THIS TO REQUEST RESOURCES cpu: 100m ##### I ADDED THIS TO REQUEST RESOURCES +``` The above manifest will not work and results in the error: ```resources.requests.cpu > invalid type > got “map”, expected “string”```*,* These settings can cause organizations to encounter situations where resources are not being used effectively or efficiently. diff --git "a/kb/armory/General/rosco's-\"bakescompleted_seconds_count\"-metric-unavailable-in-prometheus.md" b/kb/armory/General/rosco-bakescompleted-seconds-count-metric-unavailable-in-prometheus.md similarity index 100% rename from "kb/armory/General/rosco's-\"bakescompleted_seconds_count\"-metric-unavailable-in-prometheus.md" rename to kb/armory/General/rosco-bakescompleted-seconds-count-metric-unavailable-in-prometheus.md diff --git a/kb/armory/General/searching-applications-folder-with-execution-id-times-out.md b/kb/armory/General/searching-applications-folder-with-execution-id-times-out.md index ce0721d50e..4e9a4b7934 100644 --- a/kb/armory/General/searching-applications-folder-with-execution-id-times-out.md +++ b/kb/armory/General/searching-applications-folder-with-execution-id-times-out.md @@ -4,12 +4,14 @@ title: Searching Applications folder with Execution ID times out ## Issue When calling ```**/applications/{application_name}/executions/search with an execution_id**```,** **the calls time out for some pipelines. This time out error makes it difficult to find specific executions as needed. Example Search Query +```` curl --location --request GET 'http://{spinnaker-gate}/applications/{application_name}/executions/search?eventId={some_uuid}' \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --header 'Cache-Control: no-cache' \ --header 'Authorization: Basic {token}' \ --header 'Cookie: SESSION={session_id}' +```` ## Cause The number of executions for the queried pipeline can grow so large that it can reach the default timeout limit and return a failure. This is expected behavior for a scaled up Spinnaker that has had many pipelines put through it. diff --git "a/kb/armory/General/spel-expression-issue--must-be-of-type-string--\"integer\".md" b/kb/armory/General/spel-expression-issue-must-be-of-type-string-integer.md similarity index 72% rename from "kb/armory/General/spel-expression-issue--must-be-of-type-string--\"integer\".md" rename to kb/armory/General/spel-expression-issue-must-be-of-type-string-integer.md index fd01126f5a..d197005fc8 100644 --- "a/kb/armory/General/spel-expression-issue--must-be-of-type-string--\"integer\".md" +++ b/kb/armory/General/spel-expression-issue-must-be-of-type-string-integer.md @@ -3,7 +3,8 @@ title: SpEL expression issue- Must be of type string- "integer" --- ## Issue -Error noticed: ```spec.endpoints.metricRelabelings.replacement in body must be of type string: "integer"```This error indicates that replacement expects a string instead of an integer.  In the CRD some values may contain Regex expressions like ```${}```.  These characters in Spinnaker are used to evaluate SPEL expressions.  There is a way to escape these characters but doesn't work for numbers at this time. +Error noticed: ```spec.endpoints.metricRelabelings.replacement in body must be of type string: "integer"```This error indicates that replacement expects a string instead of an integer.  In the CRD some values may contain Regex expressions like ```${}```.  +These characters in Spinnaker are used to evaluate SPEL expressions.  There is a way to escape these characters but doesn't work for numbers at this time. ## Cause Spinnaker is evaluating the expression and returning an integer causing the error. diff --git a/kb/armory/General/spring-expression-language-spel-samples.md b/kb/armory/General/spring-expression-language-spel-samples.md index 6d01ccba29..6af4600575 100644 --- a/kb/armory/General/spring-expression-language-spel-samples.md +++ b/kb/armory/General/spring-expression-language-spel-samples.md @@ -5,7 +5,8 @@ title: Spring Expression Language (SpEL) samples ## Introduction Spring Expression Language (SpEL) is a powerful tool that can help customers with a variety of tasks around pipelines.  This can include passing information to subsequent stages and transforming information that was provided from previous stages. Customers wanting to see some larger examples of how SpEL can be leveraged, can take a look at the following KB articles that provide some real-life situations where SpEL helps solve some common issues from customers: -* Passing Rosco-Packer information to subsequent stages: [https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010305](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010305)* Using SpEL to add Target Groups in a Deployment Pipeline for AWS CloudFormation: [https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010280](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010280) +* Passing Rosco-Packer information to subsequent stages: [https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010305](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010305) +* Using SpEL to add Target Groups in a Deployment Pipeline for AWS CloudFormation: [https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010280](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010280)   Below are some common usage examples of using in Evaluate Variables stage and Check Preconditions stages @@ -35,15 +36,13 @@ Evaluate Variables Configuration. Pulls the parameters called ```a1``` from the ``````${parameters.a1}`````` - -###   +  ### Sample 4: Call parameter all values Evaluate Variables Configuration. Pulls the parameter all values from the stage ``````${parameters.values()}`````` - -###   +  ### Sample 5: Use java class  Evaluate Variables Configuration.  Implementing ```Java class``` for evaluations.  @@ -58,7 +57,6 @@ Evaluate Variables Configuration.  Implementing ```Java class``` for evaluation ``````${new java.lang.String(a).substring(1,8)}`````` -###   ### Sample 6: Use Helper functions @@ -98,7 +96,7 @@ Evaluate Variables Configuration captures the webhook stage values specified.  ``````${#stage("WebhookBefore").context.webhook.body}​`````` -###   + ### Sample 8: Capture the Size of Map of JSON @@ -109,7 +107,7 @@ Evaluate Variables Configuration.  Size evaluates the size of the list and retu ``````${alreadyDeployedBeforeRun.size()}​`````` -###   + ### Sample 9: Use toString().contains() diff --git "a/kb/armory/General/terraformer-error-exec--\"terraform\"--executable-file-not-found-in-$path.md" b/kb/armory/General/terraformer-error-exec-terraform-executable-file-not-found-in-path.md similarity index 100% rename from "kb/armory/General/terraformer-error-exec--\"terraform\"--executable-file-not-found-in-$path.md" rename to kb/armory/General/terraformer-error-exec-terraform-executable-file-not-found-in-path.md diff --git "a/kb/armory/General/trying-to-turn-on-vault-results-in-a-\"missing-client-token-error.md" b/kb/armory/General/trying-to-turn-on-vault-results-in-a-missing-client-token-error.md similarity index 100% rename from "kb/armory/General/trying-to-turn-on-vault-results-in-a-\"missing-client-token-error.md" rename to kb/armory/General/trying-to-turn-on-vault-results-in-a-missing-client-token-error.md diff --git "a/kb/armory/How To's/how-to-fix-tls-error-\"reason--extension-5-should-not-be-presented-in-certificate_request.\".md" b/kb/armory/How To's/how-to-fix-tls-error-reason-extension-5-should-not-be-presented-in-certificate_request.md similarity index 100% rename from "kb/armory/How To's/how-to-fix-tls-error-\"reason--extension-5-should-not-be-presented-in-certificate_request.\".md" rename to kb/armory/How To's/how-to-fix-tls-error-reason-extension-5-should-not-be-presented-in-certificate_request.md From 162420ec652e0298d5a413f3c510b0638b48dcc2 Mon Sep 17 00:00:00 2001 From: joshua-armory Date: Wed, 16 Oct 2024 18:22:07 -0400 Subject: [PATCH 3/3] fixed kb titles --- kb/armory/General/cancel-a-stuck-pipeline.md | 16 +++++- ...spring-expression-language-spel-samples.md | 53 ++++++++----------- 2 files changed, 37 insertions(+), 32 deletions(-) diff --git a/kb/armory/General/cancel-a-stuck-pipeline.md b/kb/armory/General/cancel-a-stuck-pipeline.md index c5d81763e5..8c073b5a18 100644 --- a/kb/armory/General/cancel-a-stuck-pipeline.md +++ b/kb/armory/General/cancel-a-stuck-pipeline.md @@ -11,9 +11,11 @@ When a pipeline is stuck and cannot be cancelled via the UI, the following are s Both pieces of information can be found in the execution details.  Navigate to the pipeline, and go to **Execution Details** -> **Source** The pipeline id will be labeled ```id``` and will be one of the first lines. The execution id will be ```execution_id``` and will be towards the end. + #### Via the URL of Resources in the Console The URL for the pipeline when editing the pipeline from the console also contains the ```Pipeline ID``````/#/applications//executions/configure/****``` The URL for the permalink (next to the source location above) will contain the information about the ```Pipeline Execution ID``````/#/applications//executions/details/****?stage=0&step=0&details=``` + #### Via Orca Logs The ```Pipeline Execution ID``` can also be found within the logs of the Orca instance, by running ```kubectl logs -n deploy/spin-orca -f ``` @@ -22,15 +24,25 @@ Sometimes, locating the execution ID can be difficult unless it can be identifie ## Instructions ### Via Swagger UI * Navigate to ```/swagger-ui.html```(You can find your gate endpoint, as an example, by running ```kubectl get svc``` if you are using Spinnaker on kubernetes or kubernetes cloud environments.  For more information about exposing endpoints, please look here: [https://docs.armory.io/docs/spinnaker/exposing-spinnaker/](https://docs.armory.io/docs/spinnaker/exposing-spinnaker/))* From there you can navigate to the pipeline controller and ```/pipelines/{id}/cancel```* Once there, add the ```pipeline execution ID``` from the source of the pipeline and execute the cancel* A ```200 response``` means the pipeline was successfully cancelled* Check back in your Deck UI to confirm the cancellation -## + ### If Using Redis and Need to Remove Pipeline from Execution The following are commands that can be run in **redis-cli** that can be used to manage and remove pipeline executions.  The commands should be run in order, once the ```Pipeline ID``` and ```Pipeline Execution ID``` has been located  -**Command****Function***zrange pipeline:executions:{pipeline_id} 0 -1*this displays all executions from this pipeline for each execution that you want to remove*zrem pipeline:executions:{pipeline_id} {execution_id}*unlink the execution from pipeline_config_id*del pipeline:${execution_id}*delete the execution pipeline*del orchestration:${execution_id}*delete the execution orchestration +**Command****Function** +```zrange pipeline:executions:pipeline_id 0 -1``` +*this displays all executions from this pipeline for each execution that you want to remove* +```zrem pipeline:executions:pipeline_id execution_id``` +*unlink the execution from pipeline_config_id +*del pipeline:execution_id +*delete the execution pipeline +*del orchestration:execution_id +*delete the execution orchestration   The following KB article advises about how to put all the commands together to complete the deletion:[https://kb.armory.io/s/article/Delete-Pipeline-Executions-in-Redis](https://kb.armory.io/s/article/Delete-Pipeline-Executions-in-Redis) + ### If Using MySQL and Need to Remove Pipeline from Execution The following KB article advises about how to remove a stuck pipeline from MySQL: [https://kb.armory.io/s/article/Stuck-Pipeline-Orca-with-MySQL-Editing-the-Database](https://kb.armory.io/s/article/Stuck-Pipeline-Orca-with-MySQL-Editing-the-Database) + ### ORCA Zombie Please read the following about resolving ORCA zombie executions.  The documentation outlines how to determine if it is an ORCA zombie or not, and what to do to remediate the issue[https://spinnaker.io/guides/runbooks/orca-zombie-executions/](https://spinnaker.io/guides/runbooks/orca-zombie-executions/)  diff --git a/kb/armory/General/spring-expression-language-spel-samples.md b/kb/armory/General/spring-expression-language-spel-samples.md index 6af4600575..3556aae794 100644 --- a/kb/armory/General/spring-expression-language-spel-samples.md +++ b/kb/armory/General/spring-expression-language-spel-samples.md @@ -5,7 +5,9 @@ title: Spring Expression Language (SpEL) samples ## Introduction Spring Expression Language (SpEL) is a powerful tool that can help customers with a variety of tasks around pipelines.  This can include passing information to subsequent stages and transforming information that was provided from previous stages. Customers wanting to see some larger examples of how SpEL can be leveraged, can take a look at the following KB articles that provide some real-life situations where SpEL helps solve some common issues from customers: + * Passing Rosco-Packer information to subsequent stages: [https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010305](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010305) + * Using SpEL to add Target Groups in a Deployment Pipeline for AWS CloudFormation: [https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010280](https://support.armory.io/support?id=kb_article_view&sysparm_article=KB0010280)   Below are some common usage examples of using in Evaluate Variables stage and Check Preconditions stages @@ -19,14 +21,14 @@ Pipeline sample: a job stage followed by an evaluate stage  Evaluate Variables Configuration, provides the ```status``` value of the stage as the output -``````${#stage('Job 4390')['status']}​`````` +```${#stage('Job 4390')['status']}​``` ### Sample 2: Capture Job Stage Output Value Evaluate Variables Configuration, provides the particular output from the previous stage. Outputs the sample named ```a``` from the stage. -``````${#stage("Job 4390").outputs.a}`````` +```${#stage("Job 4390").outputs.a}```   @@ -35,27 +37,25 @@ Evaluate Variables Configuration, provides the particular output from the previo Evaluate Variables Configuration. Pulls the parameters called ```a1``` from the stage -``````${parameters.a1}`````` +```${parameters.a1}```   ### Sample 4: Call parameter all values Evaluate Variables Configuration. Pulls the parameter all values from the stage -``````${parameters.values()}`````` +```${parameters.values()}```   ### Sample 5: Use java class  Evaluate Variables Configuration.  Implementing ```Java class``` for evaluations.  -``````${new java.lang.String(parameters.a1).contains("a")}`````` - - +```${new java.lang.String(parameters.a1).contains("a")}``` -``````${new java.lang.String(a).indexOf(':')}`````` +```${new java.lang.String(a).indexOf(':')}``` -``````${new java.lang.String(a).substring(1,8)}`````` +```${new java.lang.String(a).substring(1,8)}``` ### Sample 6: Use Helper functions @@ -63,8 +63,7 @@ Evaluate Variables Configuration.  Implementing ```Java class``` for evaluation Evaluate Variables Configuration. Simple values output transformations. ```alphanumerical``` command returns only A-z and 0-9 values from the parameter ```a1```. - -``````${#alphanumerical('a1')}`````` +```${#alphanumerical('a1')}```   @@ -72,7 +71,7 @@ Evaluate Variables Configuration. Simple values output transformations.     -``` +```` {"a":1,"b":2} @@ -86,16 +85,14 @@ ${#readJson(map01)}​ ${#deployedServerGroups()} ${deployedServerGroup[0].serverGroup} - ``` + ```` ### Sample 7: Capture Webhook Stage Output Values Pipeline sample: a webhook stage followed by an evaluate stage  Evaluate Variables Configuration captures the webhook stage values specified.  In the example, it is the ```WebhookBefore``` stage and the output values. -``````${#stage("WebhookBefore").context.webhook.body}​`````` - - +```${#stage("WebhookBefore").context.webhook.body}​``` ### Sample 8: Capture the Size of Map of JSON @@ -114,19 +111,19 @@ Evaluate Variables Configuration.  Size evaluates the size of the list and retu Evaluate Variables Configuration. Checks to see if the string is contained within the value.  -``````${alreadyDeployedBeforeRun.toString().contains(deployedServerGroupName)}`````` +```${alreadyDeployedBeforeRun.toString().contains(deployedServerGroupName)}```   ### Sample 10: Perform Calculations Evaluate Variables Configuration.  -``` - -${afterDeployServerGroup.size()} -${sizeOfAfter-sizeOfAlready} -${delta > 0} +```` +afterDeployServerGroup.size() +sizeOfAfter-sizeOfAlready +delta > 0 +```` ###   ### Sample 11: Use expression in Check Preconditions stage @@ -135,17 +132,13 @@ Define Check Preconditions Edit Precondition to use Expression: - +```` Expression: -``````${!errorCreated}`````` +errorCreated -``` +````   ### Reference: [Use the Spring Expression Language (SpEL) in Spinnaker Pipelines](https://docs.armory.io/armory-enterprise/spinnaker-user-guides/expression-language/) [Pipeline Expressions Guide](https://spinnaker.io/docs/guides/user/pipeline/expressions/) -[Pipeline Expression Reference](https://spinnaker.io/docs/reference/pipeline/expressions/) - - - - +[Pipeline Expression Reference](https://spinnaker.io/docs/reference/pipeline/expressions/) \ No newline at end of file