[Fleet] Move Fleet Setup to `start` lifecycle #117552

kpollich · 2021-11-04T16:11:52Z

Summary

Move Fleet setup from an API request on every Fleet page-load to the Fleet plugin's server-side start lifecycle method.

For reference, the current Fleet Setup process performs the following steps (see https://github.com/elastic/kibana/blob/main/x-pack/plugins/fleet/server/services/setup.ts):

setupFleet()
If Fleet setup is already pending, return a pending status, else createSetupSideEffects()
Grab packages, policies, and outputs from kibana.yml
ensurePreconfiguredOutputs() and run settingsSetup
- ensurePreconfiguredOuputs writes or updates output settings to the ingest-outputs saved object type
- settingsSetup ensures that any of Fleet's default settings are properly set
ensureDefaultOutput()
- Ensures that a default output it set either by config in kibana.yml or by creating one
If agentIdVerificationEnabled is set in config, then ensureFleetGlobalEsAssets
- Installs Fleet's default component template and ingest pipeline if they do not already exist
ensurePreconfiguredPackagesAndPolicies - Install preconfigured packages + policies based on
- Packages listed in kibana.yml
- DEFAULT_PACKAGES hard coded in Fleet
- AUTO_UPDATE_PACKAGES hard coded in Fleet
- Handle upgrades for managed package policies (upgradeManagedPackagePolicies) for any managed package w/ keepPoliciesUpToDate: true set
cleanPreconfiguredOutputs
- Deletes any outputs that were previously preconfigured and no longer are
ensureDefaultEnrollmentAPIKeysExists
ensureFleetServerAgentPoliciesExists

Checklist

Call setupFleet from Fleet plugin's server-side start lifecycle
Install any preconfigured packages

x-pack/plugins/fleet/public/applications/fleet/app.tsx

kpollich · 2021-11-08T14:42:07Z

@elasticmachine merge upstream

elasticmachine · 2021-11-08T16:54:44Z

Pinging @elastic/fleet (Team:Fleet)

kpollich · 2021-11-08T16:55:12Z

@elasticmachine merge upstream

joshdover · 2021-11-08T16:59:22Z

x-pack/plugins/fleet/server/plugin.ts

+        new Promise<void>(async (resolve, reject) => {
+          try {
+            await startFleetServerSetup();
+            await setupFleet(
+              new SavedObjectsClient(core.savedObjects.createInternalRepository()),
+              core.elasticsearch.client.asInternalUser
+            );
+
+            resolve();
+          } catch (error) {
+            reject(error);
+          }


I think we should move this promise constructor outside the fleetSetupCompleted function so that it isn't called each time fleetSetupCompleted is called and instead just return the already-constructed Promise. Also what is calling fleetSetupCompleted? I don't think we should rely on this being called for setup to run.

Also more of a nit, but I don't think you need the Promise constructor and you can just use an async function directly:

const fleetSetup = (async () { await startFleetServerSetup(); await setupFleet(...); })();

Fleet doesn't call fleetSetupCompleted on its own, but the security_solution plugin does rely on it to handle some of their dependent setup logic here:

kibana/x-pack/plugins/security_solution/server/plugin.ts

Lines 368 to 381 in 828a579

// Migrate artifacts to fleet and then start the minifest task after that is done

plugins.fleet.fleetSetupCompleted().then(() => {

migrateArtifactsToFleet(savedObjectsClient, artifactClient, logger).finally(() => {

logger.info('Dependent plugin setup complete - Starting ManifestTask');

if (this.manifestTask) {

this.manifestTask.start({

taskManager,

});

} else {

logger.error(new Error('User artifacts task not available.'));

}

});

});

Agreed on restructuring this - it doesn't need a Promise constructor anymore.

Fleet doesn't call fleetSetupCompleted on its own, but the security_solution plugin does rely on it to handle some of their dependent setup logic here

I still don't think we should rely on security solution to call this. In fact, we or they may remove this in 8.0 as part of removing our BWC code for pre-GA fleet. We can still support this API for now by initiating the Fleet setup directly in the start function before the return statement, but just don't await the promise and instead return it from fleetSetupCompleted().

Moved this logic out of fleetSetupCompleted so we don't need to rely on security solution calling it.

x-pack/plugins/fleet/server/plugin.ts

kpollich · 2021-11-09T13:14:45Z

x-pack/plugins/fleet/server/plugin.ts

+      .then(() => {
+        logger.info('Fleet setup completed');
+      })
+      .catch((error) => {


catching errors here seems to squash the UnhandledPromiseRejection errors that were breaking various tests in CI. 🤞

kpollich · 2021-11-09T13:45:30Z

@elasticmachine merge upstream

kpollich · 2021-11-11T19:40:44Z

@elasticmachine merge upstream

nchaulet

🚀

x-pack/plugins/fleet/server/plugin.ts

kpollich · 2021-11-11T20:25:03Z

Sample debug log output on Kibana boot:

joshdover · 2021-11-15T10:18:04Z

x-pack/plugins/fleet/server/plugin.ts

+    const fleetSetupPromise = (async () => {
+      try {
+        logger.info('Beginning fleet setup');
+
+        const { nonFatalErrors } = await setupFleet(
+          new SavedObjectsClient(core.savedObjects.createInternalRepository()),
+          core.elasticsearch.client.asInternalUser
+        );
+
+        if (nonFatalErrors.length > 0) {
+          logger.info('Encountered non fatal errors during Fleet setup');
+          formatNonFatalErrors(nonFatalErrors).forEach((error) =>
+            logger.info(JSON.stringify(error))
+          );
+        }
+
+        logger.info('Fleet setup completed');
+      } catch (error) {
+        logger.warn('Fleet setup failed');
+        logger.warn(error);
+      }
+    })();


Should we move this logging logic into setupFleet itself so we get the same logging when running manually via the API?

Yes I think that's a good idea.

I moved all the logs except for what's in our catch into setupFleet in 76138ec

kpollich · 2021-11-15T13:55:43Z

@elasticmachine merge upstream

kibana-ci · 2021-11-15T15:24:14Z

💛 Build succeeded, but was flaky

Test Failures

[job] [logs] Default Firefox Tests / InfraOps App Metrics UI Home page with metrics present change color palette

Metrics [docs]

✅ unchanged

History

💔 Build #6634 failed 76138ec
💚 Build #6398 succeeded 76e4829
💔 Build #6385 failed f1f9fc6
💔 Build #6376 failed 95475d4
💚 Build #6324 succeeded ede4a18
💔 Build #6257 failed 3c019ba

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kpollich

* Call setup on fleet start, remove API calls * Fix unused import * Revert removal of setup API call * Restructor fleetSetupCompleted promise * Add logging + handle setup failures * Restructure logging to mix of debug/info * Maybe fix failing tests * Try fixing tests again * Fix another dashboard test * Re-add output logs after merge * Log non-fatal errors during Fleet setup on boot * Don't rely on fleetSetupCompleted to be called * Fix failing test * Track fleet setup status to avoid double calls * Use IIFE in place of Promise ctor * Remove unnecessary fleetSetupStatus value * Move non-error logs into setupFleet method * Remove unused formatNonFatalErrors import Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-11-15T15:36:33Z

💚 Backport successful

Status	Branch	Result
✅	8.0

This backport PR will be merged automatically after passing CI.

* Call setup on fleet start, remove API calls * Fix unused import * Revert removal of setup API call * Restructor fleetSetupCompleted promise * Add logging + handle setup failures * Restructure logging to mix of debug/info * Maybe fix failing tests * Try fixing tests again * Fix another dashboard test * Re-add output logs after merge * Log non-fatal errors during Fleet setup on boot * Don't rely on fleetSetupCompleted to be called * Fix failing test * Track fleet setup status to avoid double calls * Use IIFE in place of Promise ctor * Remove unnecessary fleetSetupStatus value * Move non-error logs into setupFleet method * Remove unused formatNonFatalErrors import Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kyle Pollich <kyle.pollich@elastic.co>

* Call setup on fleet start, remove API calls * Fix unused import * Revert removal of setup API call * Restructor fleetSetupCompleted promise * Add logging + handle setup failures * Restructure logging to mix of debug/info * Maybe fix failing tests * Try fixing tests again * Fix another dashboard test * Re-add output logs after merge * Log non-fatal errors during Fleet setup on boot * Don't rely on fleetSetupCompleted to be called * Fix failing test * Track fleet setup status to avoid double calls * Use IIFE in place of Promise ctor * Remove unnecessary fleetSetupStatus value * Move non-error logs into setupFleet method * Remove unused formatNonFatalErrors import Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Call setup on fleet start, remove API calls

3f504b9

kpollich added WIP Work in progress v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team auto-backport Deprecated - use backport:version if exact versions are needed v8.1.0 labels Nov 4, 2021

kpollich self-assigned this Nov 4, 2021

kpollich requested a review from joshdover November 4, 2021 16:42

joshdover reviewed Nov 4, 2021

View reviewed changes

x-pack/plugins/fleet/public/applications/fleet/app.tsx Outdated Show resolved Hide resolved

kpollich added 4 commits November 4, 2021 15:23

Fix unused import

ff41a00

Revert removal of setup API call

7f5330c

Merge branch 'main' into 111858-fleet-setup-on-boot

b377871

Merge branch 'main' into 111858-fleet-setup-on-boot

539e6a3

Merge branch 'main' into 111858-fleet-setup-on-boot

ee899c3

kpollich marked this pull request as ready for review November 8, 2021 16:54

kpollich requested a review from a team as a code owner November 8, 2021 16:54

kpollich changed the title ~~[Fleet] [WIP] Move Fleet Setup to start lifecycle~~ [Fleet] Move Fleet Setup to start lifecycle Nov 8, 2021

Merge branch 'main' into 111858-fleet-setup-on-boot

28c6fbf

joshdover self-requested a review November 8, 2021 16:55

joshdover reviewed Nov 8, 2021

View reviewed changes

kpollich added 2 commits November 8, 2021 12:05

Restructor fleetSetupCompleted promise

252cf54

Add logging + handle setup failures

b5d900b

kpollich commented Nov 9, 2021

View reviewed changes

x-pack/plugins/fleet/server/plugin.ts Show resolved Hide resolved

kpollich commented Nov 9, 2021

View reviewed changes

Merge branch 'main' into 111858-fleet-setup-on-boot

6c2f05e

Track fleet setup status to avoid double calls

95475d4

kibanamachine and others added 2 commits November 11, 2021 14:40

Merge branch 'main' into 111858-fleet-setup-on-boot

f1f9fc6

Use IIFE in place of Promise ctor

c937d62

nchaulet approved these changes Nov 11, 2021

View reviewed changes

nchaulet reviewed Nov 11, 2021

View reviewed changes

x-pack/plugins/fleet/server/plugin.ts Outdated Show resolved Hide resolved

Remove unnecessary fleetSetupStatus value

76e4829

joshdover reviewed Nov 15, 2021

View reviewed changes

kpollich added 3 commits November 15, 2021 07:32

Merge branch 'main' into 111858-fleet-setup-on-boot

1c4a465

Move non-error logs into setupFleet method

76138ec

Remove unused formatNonFatalErrors import

59190ba

elastic deleted a comment from kibanamachine Nov 15, 2021

Merge branch 'main' into 111858-fleet-setup-on-boot

697f8be

joshdover approved these changes Nov 15, 2021

View reviewed changes

kpollich merged commit ec504d6 into elastic:main Nov 15, 2021

kpollich deleted the 111858-fleet-setup-on-boot branch November 15, 2021 15:33

kibanamachine mentioned this pull request Nov 15, 2021

[8.0] [Fleet] Move Fleet Setup to start lifecycle (#117552) #118554

Merged

juliaElastic mentioned this pull request Nov 30, 2021

Failing test: Jest Integration Tests.src/core/server/saved_objects/migrations/integration_tests - migration from 7.13 to 7.14+ with many failed action_tasks filters out all outdated action_task_params and action tasks #118626

Closed

jlind23 mentioned this pull request Jun 21, 2023

Remove KibanaSetup call as it is not needed anymore elastic/elastic-agent#2910

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Move Fleet Setup to `start` lifecycle #117552

[Fleet] Move Fleet Setup to `start` lifecycle #117552

kpollich commented Nov 4, 2021 •

edited

Loading

kpollich commented Nov 8, 2021

elasticmachine commented Nov 8, 2021

kpollich commented Nov 8, 2021

joshdover Nov 8, 2021 •

edited

Loading

kpollich Nov 8, 2021

joshdover Nov 9, 2021

kpollich Nov 11, 2021

kpollich Nov 9, 2021

kpollich commented Nov 9, 2021

kpollich commented Nov 11, 2021

nchaulet left a comment

kpollich commented Nov 11, 2021

joshdover Nov 15, 2021

kpollich Nov 15, 2021

kpollich Nov 15, 2021

kpollich commented Nov 15, 2021

kibana-ci commented Nov 15, 2021

kibanamachine commented Nov 15, 2021

	// Migrate artifacts to fleet and then start the minifest task after that is done
	plugins.fleet.fleetSetupCompleted().then(() => {
	migrateArtifactsToFleet(savedObjectsClient, artifactClient, logger).finally(() => {
	logger.info('Dependent plugin setup complete - Starting ManifestTask');

	if (this.manifestTask) {
	this.manifestTask.start({
	taskManager,
	});
	} else {
	logger.error(new Error('User artifacts task not available.'));
	}
	});
	});

[Fleet] Move Fleet Setup to start lifecycle #117552

[Fleet] Move Fleet Setup to start lifecycle #117552

Conversation

kpollich commented Nov 4, 2021 • edited Loading

Summary

Checklist

kpollich commented Nov 8, 2021

elasticmachine commented Nov 8, 2021

kpollich commented Nov 8, 2021

joshdover Nov 8, 2021 • edited Loading

Choose a reason for hiding this comment

kpollich Nov 8, 2021

Choose a reason for hiding this comment

joshdover Nov 9, 2021

Choose a reason for hiding this comment

kpollich Nov 11, 2021

Choose a reason for hiding this comment

kpollich Nov 9, 2021

Choose a reason for hiding this comment

kpollich commented Nov 9, 2021

kpollich commented Nov 11, 2021

nchaulet left a comment

Choose a reason for hiding this comment

kpollich commented Nov 11, 2021

joshdover Nov 15, 2021

Choose a reason for hiding this comment

kpollich Nov 15, 2021

Choose a reason for hiding this comment

kpollich Nov 15, 2021

Choose a reason for hiding this comment

kpollich commented Nov 15, 2021

kibana-ci commented Nov 15, 2021

💛 Build succeeded, but was flaky

Test Failures

Metrics [docs]

History

kibanamachine commented Nov 15, 2021

💚 Backport successful

[Fleet] Move Fleet Setup to `start` lifecycle #117552

[Fleet] Move Fleet Setup to `start` lifecycle #117552

kpollich commented Nov 4, 2021 •

edited

Loading

joshdover Nov 8, 2021 •

edited

Loading