Add performance index for token resolution

[strehle]

• add external_key to identity_provider
• add index and optional search in retrieveByIssuer
• add external_key for OAuth/OIDC and SAML IdP

Verified on DB level with explain that index is used.

Add unique index with constrains: identity zone, type, external-id, means that you can have a SAML IDP and OIDC IDP with same issuer / entityID. This can happen and should be allowed. What should be prevented to have 2 IdPs with same entityID or issuer in configuration.

To stay compatible, introduce allowAllOrigins option. Currently we loop in REST calls and check uniqueness of issuer/entityId. Admin can decide to remove this loop in some time.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/187807023

The labels on this github issue will be updated when the story is started.

[hsinn0]

I am a bit concerned about the performance impact for searching by and having a unique index on a text column. Have we done any performance testing for this change?

[strehle]

I am a bit concerned about the performance impact for searching by and having a unique index on a text column. Have we done any performance testing for this change?

yes, of course I have the explain and index is used in both DBs.
The usage of a text in an index is no real difference for postgresql , for mysql the complete text is not usable in index, so you see I had to limit the index usage....
The unique index is the runtime behavior and now visible in DB. so you cannot have 2 IdPs from type SAML with same entityID. Same with OIDC. Only one OIDC IdP with same issuer is possible. We have internal runtime checks with the loop.

The loop over all IdP is configurable because I dont want to change the behavior for all, but we will move to new one.
I created a PR to have all this new behaviors in one place , e.g. #2934 so that users of UAA can decide what type of checks they want have

[strehle]

@torsten-sap Should we limit the size of external id to maybe 512 ?

[hsinn0]

The unique index is the runtime behavior and now visible in DB.

You mean the uniqueness was already being checked in Java runtime, and we are adding the index in DB this time? So there is no added performance degradation with this change in that regard?

[strehle]

The unique index is the runtime behavior and now visible in DB.

You mean the uniqueness was already being checked in Java runtime, and we are adding the index in DB this time? So there is no added performance degradation with this change in that regard?

yes, in OIDC and SAML there are checks, but currently these checks loop. and this is a performance issuer right now ... if you have many IdP, see OIDC runtime as example
https://github.com/cloudfoundry/uaa/blob/develop/server/src/main/java/org/cloudfoundry/identity/uaa/provider/oauth/ExternalOAuthProviderConfigurator.java#L157-L168

I will change the text field into VARCHAR(512) so that mysql does not have an issue with it.
I would like to stay with the unique index because the runtime prevents several issuers / entityId also now.

[strehle]

The unique index is the runtime behavior and now visible in DB.

You mean the uniqueness was already being checked in Java runtime, and we are adding the index in DB this time? So there is no added performance degradation with this change in that regard?

here is the SAML check for duplicate
https://github.com/cloudfoundry/uaa/blob/develop/server/src/main/java/org/cloudfoundry/identity/uaa/provider/saml/SamlIdentityProviderConfigurator.java#L105-L107

That is the reason why I added the index as unique, because we have the checks in runtime are there but without a performant access and simply via select * in DB.... but this is a problem in case of many IdPs

[strehle]

The unique index is the runtime behavior and now visible in DB.

You mean the uniqueness was already being checked in Java runtime, and we are adding the index in DB this time? So there is no added performance degradation with this change in that regard?

Correct 👍