Set up database + projects for real form testing

[dave-mills]

Similar to the setup where we can run a database seeder and a pair of commands to get our local environment to use the existing "test form" projects, we need a way to share ODK Central projects that have the real forms:

Steps:

• Make sure the platform project ID is unset, then refresh the database with the default seeders to create 4 new ODK Central projects - one linked to the 'platform' and 3 linked to the 3 new teams.

• Add the 2 real forms as XlsformTemplates. (make sure to run the queue for the import). There should be Xlsform entries created for each team automatically.

• Manually add the 'default' english Locale to each team so that their forms aren't missing labels and hints.

• Publish the 2 forms for each team

• at this point, can you share the ODK Central projects with @alex-thomson222 to make a couple of manual entries?

To make it so we can recreate the environment:

• Setup a new folder of database seeders, similar to the TestTemplates and TestOdkStuff.
  • In fact, I had originally split up those seeders into those 2 folders because I thought the TestTemplates would be fine for some purposes, but really they're best run all together. Can you move all those existing seeders into a single folder? That way, it'll be easier to keep the current ones for the test templates and the new ones for the real form templates organised.
• Copy the required media files from local storage into the test/assets/ folder, and add a command to run to copy them back into the app storage (similar to CopyMediaForTestOdkStuff).

[dan-tang-ssd]

Similar to the setup where we can run a database seeder and a pair of commands to get our local environment to use the existing "test form" projects, we need a way to share ODK Central projects that have the real forms:

Hi @dave-mills, I would like to ask some questions for clarification first.

Just to confirm:

• Um... would you indicate which database seeder file?
• a pair of commands is:
  1. php artisan db:seed
  2. php artisan db:seed TestOdkStuffSeeder

Are we going to get our local environment to use the existing "real form" projects in ODK central staging env?
So the only different thing is "test form" projects and "real form" projects?

Um... Do you have a PR that includes all those changes for easy reference?
Should I make a copy of the full set of these files and just change it from "test form" projects to "real form" projects?

[dave-mills]

@dan-tang-ssd

I assume you are familiar with how the seeders and commands work for setting up a local environment to use the shared "test form" projects on ODK Central. You can look at #172 for my description, and #173 for the PR.

In the PR, focus on:

• The Commands
• The Seeders
• The assets stored in /test/**

All the changes in app/Filament, app/Imports, app/Livewire and app/Models are just namespacing changes (moving the Xlsform and XlsformTemplate models back into the package).

To summarise the current flow:

• we can run php artisan migrate:fresh --seed to get a default setup with the 3 teams.
• we can then run php artisan db:seed TestOdkStuffSeeder to run the TestOdkStuffSeeder. That runs all the seeder files inside the 2 folders I mentioned: TestTemplates and TestOdkStuff.
• we can then run the 2 commands: CopyMediaForTestOdkStuff and UpdateXlsformVersionsFromOdkCentral to ensure we have the correct files in the correct places in storage.

Those seeders were created using the issed tablename,tablename2... etc command after doing the manual setup that I described.

Are we going to get our local environment to use the existing "real form" projects in ODK central staging env? So the only different thing is "test form" projects and "real form" projects?

We could use the existing "real form" projects in ODK Central, assuming those forms are the correct + up to date versions, but it might be cleaner to make new ones, so that we know we don't have older submissions in the wrong format etc.

Um... Do you have a PR that includes all those changes for easy reference?

You can

Should I make a copy of the full set of these files and just change it from "test form" projects to "real form" projects?

I wouldn't make a copy of all those seeder files; they were created using the iseed, and the quickest way would be to get your database setup the way it's needed and then run the iseed command for those tables; then copy those into a new folder, and create a "RealOdkFormSeeder" that does the same as

[dan-tang-ssd]

Thank you @dave-mills .

I have a better understanding now.

Um... What is the URL for ODK Central projects that have the real forms? Or... where can I find the real forms?

[dan-tang-ssd]

Hi @dave-mills , some progress update...

Thank you for your email. I have downloaded two real forms.

Before working on seeder files, I would like to test if I can create new xlsform templates for two real forms in local env first.

I deleted all tables in backend database, then run below commands to setup my local env:

• php artisan migrate
• php artisan db:seed
• php artisan db:seed TestOdkStuffSeeder
• php artisan app:copy-media-test
• php artisan app:update-xlsform-versions-from-odk-central

Now my local env can retrieve submissions from ODK central "mini forms".
I login as admin, retrieve submissions for household form and fieldwork form of "P1 Test Team 2", success.

---

In admin panel, I created new xlsform templates for mini fieldwork form and mini household form. This is to ensure new xlsform template can be created successfully.

After that, I create new xlsform templates for real fieldwork form and real household form.
I got different error for them, please see below screen shots for details.

I also tried to create new forms with real forms in ODK central staging env directly.

Project folder:
https://odk-test.stats4sdtest.online/#/projects/1606

Real fieldwork form can be created with warnings, while real household form cannot be created with errors.
Please see below screen shots for details.

---

Screen shots:

Real Fieldwork Form

• error occurred: There is an error in the XLS Form
  • error message: An error occurred while creating the draft form. The error is not an XLSForm file validation issue, but something else that might require further investigation. Please try again later or contact support if the problem persists
• not quite sure is it related to label column, excel file column F is "label::English (en)"
• xlsform_templates record created without values in ODK central related columns, e.g. odk_id, odk_draft_token, enketo_draft_id, odk_version_id, schema
• no xlsform_template_sections record created
• no error message found in laravel.log or Ray
  

---

Real Household Form

• error 504 Gateway Time-out
• xlsform_templates record created without values in ODK central related columns, e.g. odk_id, odk_draft_token, enketo_draft_id, odk_version_id, schema
• no xlsform_template_sections record created
• error "Allowed memory size of 134217728 bytes exhausted" found in laravel.log and Ray

---

ODK central staging env, Real Fieldwork Form

---

ODK central staging env, Real Household Form