Populate Elastic Instance with Simulated ULI Data

[darnjo]

In order to make the project so that any developer can test it locally, we'll need to be able to generate realistic testing data.

• To run the environment locally, see this guide: https://github.com/RESOStandards/uli-service/blob/main/docs/running-the-pilot.md
• Simulated data will be inserted into the local elastic instance in cases where someone wants to bulk-upload data to test the system. Rather than the upload data button in Create Mock for Initial Bulk Ingest #14, we would most likely have a "generate" data button that uses something like Faker to populate the records in the format shown in the ULI proposal.
• What we want is to be able to simulate the cases we know we have issues with, for example if First Initial and First Name matching is something we want to test, we'd want someone with a first name and first initial in the data set that both causes a collision and doesn't.

With the backend Elastic instance running, we'll want to have a Node/Express service that can connect to the instance and insert the simulated data. We'll want to make sure we simulate this with auth (bearer tokens):

POST /uli-service/v1/<providerUoi>/generate-test-data/[num-records]
  Authorization: Bearer <token>

We want the num-records parameter so we can load test the system with varying sizes of member data sets, up to 100,000.

We'd probably have spinner or progress indicator on the frontend to show where the generation and matching process is at.

Let's assume this will go into an index called uli-import. In order to do that, you can use the Elastic Node Client and POST data into the Elastic index, once running. See documentation on connecting to an Elastic instance.