Issues on Provision Tenant

[tomg-formpak]

@PoeppingT
Related to the previous questions, please see the below issues:

1. Initially the application is configured with an initial database (PostgreSQL database, which is UTF-8 encoded and in .sql format) and on Provision Tenant (tenant1) :

• instead of adding the sub-domain to the existing hosted domain, it created a duplicate hosted domain (same name as the existing one ) and inside that it created the subdomain for the tenant as a record. I.E, we are having two hosted domains in the same name in Route53 - one is already created before application configuration and the second is created at the time of tenant provision (which contains the subdomain for the tenant). Is it normal? What is the normal flow in this case?

• Tenant provision got failed on creating CF stack at "RDSBootstrapDatabase" with a status reason "Resource handler returned message: "User: arn:aws:sts::*****:assumed-role/sb-dev-onboarding-svc-role-eu-west-1/sb-dev-onboarding-events is not authorized to perform: lambda:TagResource on resource: arn:aws:lambda:eu-west-1:******:function:sb-dev-rds-bootstrap-tenant-ecda4611-main because no identity-based policy allows the lambda:TagResource action (Service: Lambda, Status Code: 403, Request ID: ******)" (RequestToken: *****, HandlerErrorCode: AccessDenied)". Please advise on this. @brtrvn

Note: It got "ROLLBACK_COMPLETE" in CF with status as"Deleted" in the Onboarding tab in the panel and also above mentioned newly created hosted domain is also deleted.

2. The Second time, I removed the initial database from the application configuration and on Provision Tenant (tenant2):

• Tenant provisioning was failed on "RecordSetAlias" with a status reason " The specified hosted zone does not exist. (Service: AmazonRoute53; Status Code: 404; Error Code: NoSuchHostedZone; Request ID: 5a2e2880-be62-4210-942b-e3ba30714422; Proxy: null)". I think this is related to the above reason. Please advise on this.

Note: It got ROLLBACK_COMPLETE in CF at "AWS SaaS Boost Tenant Onboarding" and with a status as "Provisioning" in the Onboarding tab in the panel.

3. On running the 3rd and 4th time (Tenant3 and Tenant4), the status is "Failed" and "Validating" respectively with no entry in the CF stack.

Please let me know.

[PoeppingT]

Hi @tomg-formpak , thanks for reaching out!

1. It seems that there is a misunderstanding between our users and our original intention. Were you hoping that SaaS Boost would reuse an existing Hosted Zone that you had created yourself, like the customers in DNS hosted zone should be registered when we set app config #257 ? Our code is designed to create net new Route53 Hosted Zones for domain names entered into AppConfig if there does not exist one already created by SaaS Boost. Can you describe what experience you think would be best? We recognize this is not well communicated and are looking at gathering feedback on whether we should change the current logic.
2. This bug in particular has been fixed with the merge of 5e8d435 into main. Pull the latest code to fix this.
3. The Hosted Zone does not exist error is due to a bug in our code, thanks for the report! Find more information here: Updating AppConfig deletes SaaS Boost created HostedZone #300 . We'll merge a fix in for this before releasing our final v2.0.0 release.
4. Lastly, typically the "Failed" and "Validating" statuses for Tenants are due to users not uploading a docker image to the proper ECR repository. The best way to check that is to view the CloudWatch LogGroup for the Onboarding validation workflow: sb-${env}-onboarding-validation.

[tomg-formpak]

Hi @PoeppingT, thanks for the reply.

1. I think I agree with report DNS hosted zone should be registered when we set app config #257, because "Our code is designed to create net new Route53 Hosted Zones for domain names entered into AppConfig if there does not exist one already created by SaaS Boost", this doesn't seems to be right order - because if this is the case how to add NS(name server) to domain purchase site and how to request for an SSL certificate ?? These two seem to be done before onboarding service (because SSL and domain need to be added in application configuration) and in that case, hosted zone needs to be created before the tenant onboard. Ideally, I think at the time of Provision Tenant, only the tenant domain(sub-domain) needs to be added inside the hosted domain. Please let me know.
2. I will pull the latest code for 5e8d435 and will update you.
3. Thank you.
4. I will check that.

Thank you. Just to confirm, as of now, the status of tenants on onboarding is FAILED, VALIDATING, DELETED and PROVISIONING. Are there any charges getting added to the AWS account due to these tenants as they are in inactive or failed status? I am asking this because I can see some charges added to ECC - Nat Gateway.

[PoeppingT]

Hey @tomg-formpak if there are lingering resources in AWS they will be represented by a CloudFormation stack with a name like sb-${env}-tenant-{$id}. You can view these stacks in the CloudFormation AWS Console.

The best way to delete these tenants is through the SaaS Boost Admin UI. Click to the Tenants menu on the left to open the Tenants page, click on the tenant you want to delete, then click Actions -> Delete in the top right. This will send a delete API call to the Tenants service, which should remove the Tenant CloudFormation stacks and all associated AWS resources.

If you find the CloudFormation stacks still hanging out more than 10 minutes later, you can also manually delete them through the CloudFormation console. Be sure to delete the sb-${env}-tenant-{$id}-app-${serviceName}-${id} stacks first before deleting the sb-${env}-tenant-{$id} stacks, as the former depends on resources controlled by the latter.

[tomg-formpak]

@PoeppingT @brtrvn

Thank you for the comments. I thought the issue was solved. Initially, we were able to add the initial set of database backups also (PostgreSQL database, which is UTF-8 encoded and in .sql format) in the application configuration. But couldn't provision a tenant successfully. Please see the below points:

1. Not able to load a database backup in application configuration - after I upload and submit - it's not reflected in the configuration. How to check this issue?
2. On provisioning a new tenant, there is no error in CF stacks - all three stacks were created successfully and the status was ACTIVE in the Tenant tab. But after some time, the onboarded tenant is not showing in the Tenant tab and the status is Failed in the onboarding tab. How do troubleshoot this?

I have checked Cloudwatch logs for onboarding validation logs, but I couldn't find anything and no error in CF stacks also.

Note: I have checked this or the tenant is missing from the Tenants tab after I start to access the URL and I can see some error in ECS logs related to the database, a table such as `"dataversion_table" does not exist (but there is no database loaded), so at least on accessing the tenant URL, the homepage is supposed to load, right? How to check these issues on, why tenant URL is not loading or showing 5 error ?**

Note: Hosted domain's namespace is added to the registered domain.

Also, how to configure/pass the application environment variables for the application in the ECS Fargate container via GUI? example, GITHUB_API_KEY. Do we need to manually edit the ECS task definitions for this? How to do this?

Please advise on this.

Thank you

[tomg-formpak]

Hi @PoeppingT

We are stuck at this stage now. Just for reference, please see below for the issues related mentioned above :

1. I uploaded a backup file to initialize the database, and it was loaded properly before. But due to other errors (not being able to access the tenant URL), I removed the file and tried to onboard the tenant again to see if it works, but I am getting above mentioned error n ECS logs such as `"dataversion_table" does not exist.

So now when I tried to upload the SQL file again and it's not getting uploaded.

Once I submit it, it's loading and there is no error. But when I go to the other tab and check again in the Application tab, I can't see the loaded database file.

Why is it happening and how to troubleshoot it?

Please advice.

[PoeppingT]

Hi @tomg-formpak , the way the upload of the bootstrap file currently works is a little complicated. Let me walk you through how it works.

1. You set a database bootstrap filename through the UI, which sends this in the AppConfig to the Settings Service.
2. The Settings Service will create a presigned S3 url for the SaaS Boost Resources bucket like sb-${environment}-resources-${randomString}/services/${serviceName}/bootstrap.sql if this is your first time saving the AppConfig or if you have no provisioned tenants. Settings Service returns the updated AppConfig in the response to this call.
3. The UI saves the presigned URLs returned in a map and uploads each file in response to the S3 bucket.

So there's a few things you can do to troubleshoot:

1. Can you confirm the status of the bootstrap.sql file in your S3 resources bucket? It is possible that the UI is out of sync with the S3 status.
2. Can you confirm that you have no currently provisioned tenants? We don't allow switching the database bootstrap file when there are provisioned tenants since this would create inconsistency between tenants: an anti-pattern in SaaS practices.
3. You state that this is a "backup" file: to be clear, this is only intended to be a schema object creation tool. You can look at the logs for the Lambda that runs your bootstrap file in the AWS Console for CloudWatch Logs. Look for the LogGroup with a name like sb-${environment}-rds-bootstrap-tenant-${tenantId}-${serviceName}. If there were any errors parsing or running your script they would be there.

[tomg-formpak]

Hi @PoeppingT

Thank you for the detailed response.

I can see a bootstrap.sql file under the S3 bucket, sb-${environment}-resources-${randomString}/services/${serviceName}/. Is it created with empty database initialization or only when we upload a .sql file for database initialisation? How to check whether UI is in sync with S3 or not? Can I delete the bootstrap.sql file and try again with uploading the database file, and then onboard a tenant to resolve the issue?

Note: I meant by "backup file" is that it's just an initial dataset to run the application, which is created using a backup, and in UTF-8 encoded and in .sql format

Currently, we don't have any active tenants under the Tenants tab and under the Onboarding tab, the onboarding status for tenants are as follows: Deleted (5 tenants), Failed (3 tenants), Validating (1 tenant) and Provisioning (2 tenants). On CG stacks, 2 sb-${env}-tenant-{$id} are under ROLLBACK_COMPLETE status.
To delete the tenant entries and to start fresh, what all needs to be deleted from the AWS console? - I can see 2 entries in CF-related to tenants as mentioned above and 5 entries in the S3 bucket under sb-${environment}-resources-${randomString}/tenants/
Is it a good idea to have a delete option under the Onboarding tab also - to delete tenant entries and related resources?

On checking, LogGroup with a name like sb-${environment}-rds-bootstrap-tenant-${tenantId}-${serviceName}, I can't find anything. Please see the below screenshot.

[tomg-formpak]

@PoeppingT

I have deleted the bootstrap.sql file under the S3 bucket, sb-${environment}-resources-${randomString}/services/${serviceName}/ and also tenant entries from CF stack, sb-${env}-tenant-{$id}. After that, I was able to upload the database file again. On provisioning tenant, I am getting below error under the log group, sb-${environment}-rds-bootstrap-tenant-${tenantId}-${serviceName}

This is again getting retried and showing the same error again. In CF stacks, it's stuck on InvokeRDSBootstrapDatabase with the status CREATE_IN_PROGRESS.

Please advise on this.

[brtrvn]

@tomg-formpak This looks like your SQL file timed out. The bootstrap function is given 14 1/2 minutes to execute (Lambda's timeout was 15 minutes max when we wrote this). If the function is going to timeout, we interrupt the thread and return a FAILED response back to CloudFormation so the stack doesn't hang.

In this case, the error message from that thread interruption was NULL and exposed a bug in our CloudFormation response handler. We will fix the NPE, but that won't fix why your SQL script is taking so long to execute.

[tomg-formpak]

@brtrvn

Thank you, just updating you on the testing I did after our conversation. I am waiting for more inputs from my developers, I will let you know soon.

I have used a better version of the SQL file today, which is optimized in size and data with the INSERT option (but no -- in the statement) instead of COPY. I was just trying to see if its works.

1. I could not find sb-${environment}-rds-bootstrap-tenant-${tenantId}-${serviceName} under the log group this time, too see bootstrap file related logs. When did this get generated ?
2. There is no error CF stack creation, but it's showing DEPLOYING under the Onboarding tab and ACTIVE under the Tenants tab.
3. On checking ECS logs, I found the below error, please check and let me know.

• How else to troubleshoot at this stage?
• Which logs to look into - Is sb-${environment}-rds-bootstrap-tenant-${tenantId}-${serviceName} is the only logs to check bootstrap file related issue ?
• Where to look for container-related errors, other than ECS logs?
• How to check if it's a domain or hosted domain-related issue or not, since the tenant URL shows an error?

NB: sorry for shooting a lot of questions :P just excited to work on this.

FYI @PoeppingT

Thank you

[PoeppingT]

Hi @tomg-formpak ,

1. The sb-${environment}-rds-bootstrap-tenant-${tenantId}-${serviceName} LogGroup will be created when the CloudFormation Tenant App RDS Template is executed. If you see no LogGroup check CloudFormation to verify the status of that CloudFormation stack. If you only see the tenant base stack (e.g. tenant-1a2b3c4d) but don't see any app stacks, you can check the sb-${environment}-onboarding-events LogGroup to see if Onboarding had problems creating the app stack.
2. Unfortunately the ECS logs are as far as you can get for troubleshooting your own application: those are logs coming from the Docker container you uploaded. One option is to add more debugging statements to your application to understand exactly what errors your application is seeing while booting.

It's possible that if no rds stack was created (due to errors in the Onboarding service) that no RDS instance was created, which could be leading to your JDBC service failing to initialize.

Re: your last question, by "the tenant URL shows an error" do you mean that attempting to access the Tenant's application results in an error? This is likely due to the ECS container failing to start, as above.

[tomg-formpak]

Hi @PoeppingT @brtrvn

We are facing an error now while provisioning a tenant, ERROR OnboardingService - Provisioning will exceed limits. Quota will be exceeded for service Fargate On-Demand resource count. You are currently consuming 0.0, and Service Quota is 0.0 in the sb-${environment}-onboarding-validation LogGroup. Please refer below screenshot.

Please advise me on how to resolve this issue.

Note: I have also tried deleting the SaaS boost using the installer and tried with a fresh installation (v2.0.0) to troubleshoot this but still have the same error.

Thank you

[PoeppingT]

Hi @tomg-formpak , this is a known issue with quota changes being pushed out in Fargate. 0159079 fixes this issue and we are working on a v2.0.1 release to address this. To unblock you immediately you will want to cherry-pick the aforementioned commit onto your local branch before running the installer for a new environment.

[PoeppingT]

You can now also use https://github.com/awslabs/aws-saas-boost/releases/tag/v2.0.1 instead, which includes that commit and other small bugfixes and version upgrades.

[tomg-formpak]

Hi @PoeppingT @brtrvn

Thank you for your support. Issues related to tenant provision are sorted I guess.

Before I close this discussion, please consider below points:

1. The onboarding request status is not constant, it doesn't reflect the same status as per Cloudformation. For example, as below screenshot :

• tenant 2 and 3 are deleted directly from the console - but it's stuck on "deleting status". I have also deleted the CF stack entries for those tenants.
• tenant 6 is provisioned successfully, as per CF, there is no error. But it's stuck at "deploying" status on onboarding and ACTIVE under the Tenants tab, which also does vanish from the Tenants tab after some time but stays active and in "deploying" status on the onboarding tab.

Expectation: Onboarding status and Tenant status to reflect the same as per AWS. If it can show an error or warning in detail, if anything, from log groups (onboarding validation or bootstrap rds or anything ), in the onboarding tab or tenant tab, it will be really useful from the operational point of view.

2. Clear documentation or user guides for

• Tenant onboarding will be useful and smooth, such as details on the flow or stages of onboarding with status and which log group to look for issues
• meaning of statuses as in when status changes occur
• documentation for "optional config file " at the time of onboarding a new tenant (expected variables or format or a sample file)
• where to look for parameters related to setting services in the SSM Parameter store of any issues or changes or to double-check anything.

3. Documentation for the expected bootstrap file for database initialization as in format or size or any conditions.

4. AWS S3 bucket option as the File system for tenants instead of EFS.

These are suggestions or feedback based on my experience during tenant provisioning.

Thank you
Tom

[rshettyami]

My tenant provisioning is failing with the error
#####################
The following resource(s) failed to create: [RDSInstance].
####################
Any help to further troubleshoot is appreciated.