Get tests running in FIPS environment

[donoghuc]

Release notes

[rn:skip]

What does this PR do?

This PR is broken up into commits that track the sub tasks in https://github.com/elastic/ingest-dev/issues/5048. Many of the test failures in the integration test suite for running under fips mode are actually due to tools required by tests being missing in the runtime. Similarly the tests assume they are not run as root user. This PR fixes those issues by adding required packages and configuring a non root user. By fixing those issues some of the other more test specific failures can be seen. By collecting the modifications in a single PR we can show incremental progress on total test failures and eventually get to fully green.

Related issues

Closes https://github.com/elastic/ingest-dev/issues/5074
Closes https://github.com/elastic/ingest-dev/issues/5088
Closes https://github.com/elastic/ingest-dev/issues/5071
Closes https://github.com/elastic/ingest-dev/issues/5072
Closes https://github.com/elastic/ingest-dev/issues/5069
Closes https://github.com/elastic/ingest-dev/issues/5073 (fixed in e825357 by disabling debug logging)
Closes https://github.com/elastic/ingest-dev/issues/5107

[donoghuc]

Tracking with https://github.com/elastic/ingest-dev/issues/5074

[donoghuc]

Need to sort out what is hanging in CI, but for FIPS part one locally on commit 71a886b i'm down to just these failures!

    Failures:

      1) Beat Input Without TLS successfully send events
         Failure/Error: @fixture = Fixture.new(__FILE__)

         RuntimeError:
           /logstash/qa/integration/services/filebeat_setup.sh FAILED with exit status pid 952 exit 127
         Shared Example Group: "send events" called from ./specs/beats_input_spec.rb:98
         # ./services/service.rb:36:in `setup'
         # ./framework/fixture.rb:82:in `block in setup_services'
         # ./framework/fixture.rb:81:in `setup_services'
         # ./framework/fixture.rb:47:in `initialize'
         # ./specs/beats_input_spec.rb:28:in `block in <main>'
         # ./rspec.rb:36:in `<main>'

      2) Beat Input With TLS Server auth successfully send events
         Failure/Error: @fixture = Fixture.new(__FILE__)

         RuntimeError:
           /logstash/qa/integration/services/filebeat_setup.sh FAILED with exit status pid 952 exit 127
         Shared Example Group: "send events" called from ./specs/beats_input_spec.rb:131
         # ./services/service.rb:36:in `setup'
         # ./framework/fixture.rb:82:in `block in setup_services'
         # ./framework/fixture.rb:81:in `setup_services'
         # ./framework/fixture.rb:47:in `initialize'
         # ./specs/beats_input_spec.rb:28:in `block in <main>'
         # ./rspec.rb:36:in `<main>'

      3) Beat Input With TLS Mutual auth successfully send events
         Failure/Error: @fixture = Fixture.new(__FILE__)

         RuntimeError:
           /logstash/qa/integration/services/filebeat_setup.sh FAILED with exit status pid 952 exit 127
         Shared Example Group: "send events" called from ./specs/beats_input_spec.rb:162
         # ./services/service.rb:36:in `setup'
         # ./framework/fixture.rb:82:in `block in setup_services'
         # ./framework/fixture.rb:81:in `setup_services'
         # ./framework/fixture.rb:47:in `initialize'
         # ./specs/beats_input_spec.rb:28:in `block in <main>'
         # ./rspec.rb:36:in `<main>'

      4) Install and run java plugin successfully runs a pipeline with an installed Java plugins
         Got 2 failures:

         4.1) Failure/Error: expect(execute.stderr_and_stdout).to match(/Installation successful/)

                expected "Using bundled JDK: /logstash/build/logstash-8.18.0-SNAPSHOT/jdk\nERROR: Something went wrong when installing logstash-input-java_input_example, message: Received fatal alert: handshake_failure\n" to match /Installation successful/
                Diff:
                @@ -1 +1,2 @@
                -/Installation successful/
                +Using bundled JDK: /logstash/build/logstash-8.18.0-SNAPSHOT/jdk
                +ERROR: Something went wrong when installing logstash-input-java_input_example, message: Received fatal alert: handshake_failure

              # ./specs/install_java_plugin_spec.rb:53:in `block in <main>'
              # /logstash/build/qa/integration/vendor/jruby/3.1.0/gems/logstash-devutils-2.6.2-java/lib/logstash/devutils/rspec/spec_helper.rb:47:in `block in <main>'
              # ./rspec.rb:36:in `<main>'

         4.2) Failure/Error: expect(removal.stderr_and_stdout).to match(/Successfully removed #{plugin_name}/)

                expected "Using bundled JDK: /logstash/build/logstash-8.18.0-SNAPSHOT/jdk\nERROR: Operation aborted, cannot remove plugin, message: This plugin has not been previously installed\n" to match /Successfully removed logstash-input-java_input_example/
                Diff:
                @@ -1 +1,2 @@
                -/Successfully removed logstash-input-java_input_example/
                +Using bundled JDK: /logstash/build/logstash-8.18.0-SNAPSHOT/jdk
                +ERROR: Operation aborted, cannot remove plugin, message: This plugin has not been previously installed

              # ./specs/install_java_plugin_spec.rb:41:in `block in <main>'
              # /logstash/build/qa/integration/vendor/jruby/3.1.0/gems/logstash-devutils-2.6.2-java/lib/logstash/devutils/rspec/spec_helper.rb:47:in `block in <main>'
              # ./rspec.rb:36:in `<main>'

    Finished in 5 minutes 12 seconds (files took 1.02 seconds to load)
    33 examples, 4 failures

    Failed examples:

    rspec './specs/beats_input_spec.rb[1:1:1]' # Beat Input Without TLS successfully send events
    rspec './specs/beats_input_spec.rb[1:2:1:1]' # Beat Input With TLS Server auth successfully send events
    rspec './specs/beats_input_spec.rb[1:2:2:1]' # Beat Input With TLS Mutual auth successfully send events
    rspec ./specs/install_java_plugin_spec.rb:50 # Install and run java plugin successfully runs a pipeline with an installed Java plugins

    Randomized with seed 48078

[mergify]

This pull request is now in conflicts. Could you fix it @donoghuc? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b integration-tests-setup upstream/integration-tests-setup
git merge upstream/feature/fedramp-high-8.x
git push upstream integration-tests-setup

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: fe1604b

Failed CI Steps

• 🥼 Integration Tests - FIPS mode / part 1
• 🥼 Integration Tests / part 2
• 🥼 Integration Tests / part 2

History

• 💛 Build #2450 was flaky 2c6e294
• 💔 Build #2448 failed 82055e8
• 💔 Build #2437 failed 9b161d4
• 💔 Build #2416 failed ff062a9
• 💔 Build #2415 failed c5c188e
• 💔 Build #2395 failed 143eab7