RFC0005: Reference strategy by arnau · Pull Request #11 · openregister/registers-rfcs

arnau · 2018-03-29T14:35:09Z

Context

This RFC proposes consolidating references in Registers to CURIEs only.

Guidance to review

Review the description of the reality for foreign keys and CURIEs is accurate.
Review the proposal is coherent.
Raise any doubts or concerns.

nacnudus · 2018-04-03T10:47:39Z

content/reference-strategy/index.md

+1. Compose a record set URI: `https://country.register.gov.uk/records/`
+1. Concatenate the URI with the value: `https://country.register.gov.uk/records/GB`
+
+There is a special case here were a foreign key refers to a register register


Typo, were should be where

nacnudus · 2018-04-03T10:49:43Z

content/reference-strategy/index.md

+
+
+TODO: Where does the context live? It can't be part of the register API
+because a Register could be part of multiple environments.


What is an example of a register being part of multiple environments?

The immediate situation would be a Register being part of the register.gov.uk environment and being part of a local/test environment.

Another one, artificial case right now, would be a register from NHS that is part of the NHS environment and part of ours.

Does that mean the CURIE doesn't resolve to a "single source of truth" but instead just one of many places where that truth is published?

It resolves to the source of truth you decided. Which most of the time is our environment.

@michaelabenyohai and I had a chat about another topic that led to a discussion around the possibility to encode more specific data instead of the CURIE (as per the spec) so it could be proved correct. But right now it is just a thought.

nacnudus · 2018-04-03T10:50:50Z

content/reference-strategy/index.md

+* `example:32` -> `https://example.org/32`
+
+
+TODO: Where does the context live? It can't be part of the register API


It needs to be tamper-proof though.

What do you mean?

If someone tampers with the context, then a CURIE would resolve to the wrong data.

Ah yes, it's something to keep an eye on.

michaelabenyohai

I've added a few thoughts that came to mind. I'm not disagreeing with your points, just mentioning things we might want to note that we have considered.

On a more general note, do we want to consider how this affects our current field naming rules and conventions and the difficulties we have had with this recently?

Specifically, we still have the rule that the key field of a register must have the same name as the register. This guidance instructs that these fields must no longer be created as a foreign key (which they always have previously). This in turn means that this field name can never be reused as a link from another register (which we currently do a lot). Instead, we will have to create a new field with a slightly different name, so could end up proliferating fields that come in "pairs" with similar-but-not-quite-the-same name.

I feel we could treat this as the key field should describe "the thing", whereas the curie field in the "other thing" register should describe the relationship of "the thing" to "other thing". In practice I think this could be hard though.

We've previously considered removing the rule that the key field always has the same name as the register and start calling it id (although there's no reason why it has to be the same in every register, since it is the key property of the entry that is actually mandatory to name correctly). The crux of my point is whether we need accelerate this change to make it feasible to remove foreign keys in practice?

michaelabenyohai · 2018-07-13T12:41:35Z

content/reference-strategy/index.md

+
+### Foreign Key
+
+A foreign key is a regular string that _happens_ to be an identifier in


It doesn't always have to be a string. It could also be any other datatype. Not sure if we care about that detail here.

But on that note. Is there ever any benefit in being able to say a link also has a datatype like integer?

Thinking about this, it connects with another discussion about expectations around primary key values: #18

In there I was only considering strings but your comment opens the door to more (potential) complexity.

I think we care about how flexible we must be with identifiers that can be used in CURIEs (and because historical reasons in foreign keys). In particular with CURIEs, the value needs to be encoded in terms of a URL path so foo:1 would be the same if the identifier is a string or an integer. I'm leaning towards exploring issues derived to restrict identifiers to be only a subset of UTF-8.

michaelabenyohai · 2018-07-13T12:46:27Z

content/reference-strategy/index.md

+when found in another register it acts as a foreign key. To get the URI of the
+referenced resource you have to:
+
+1. Given a fieldname `country`


This should actually be:

Given a field with name x, where the record for x in the Field register has the register field populated with value country

I.e. to be clear, the field name does not always have the same name as the foreign register to be a foreign key, though it often does.

Do you know of any cases where this is not the case?

The field allergen-group is one and the field fields is another.

I think this point still needs addressing.

Ok, changed with your suggestion.

michaelabenyohai · 2018-07-13T12:49:52Z

content/reference-strategy/index.md

+1. Compose a record set URI: `https://country.register.gov.uk/records/`
+1. Concatenate the URI with the value: `https://country.register.gov.uk/records/GB`
+
+There is a special case here where a foreign key refers to a register register


I would counter that this is not what it actually means. In the example below it still just resolves to https://register.register.gov.uk/country - but we hoped that people might then follow that to the contents of https://country.register.gov.uk/records. Though that is a problem in itself. We'd have the same problem with curies if we hadn't done the URI consolidation work.

I think I'll need a chat with you on this one.

michaelabenyohai · 2018-07-13T12:56:32Z

content/reference-strategy/index.md

+* To know a field contains a foreign key, you need to look up the register field
+in the field definition. If it is informed, the datatype is not “string” but
+“key”.
+* Only able to reference one register per field.


Is this always a problem? Or does it help consumers to know that the thing in a field will always be in the same place? Are there performance benefits knowing you don't have to check the location of every single record?

You are right in that it is not a problem in itself, just a restriction. The problem derived from this restriction is the need to have multiple fields for similar things when the situation arises. I'll think in a better way to phrase this.

michaelabenyohai · 2018-07-13T12:57:32Z

content/reference-strategy/index.md

+in the field definition. If it is informed, the datatype is not “string” but
+“key”.
+* Only able to reference one register per field.
+* Linking a full set of records is a special case.


I'd say linking a full set of records is not possible (see above).

michaelabenyohai · 2018-07-13T13:03:43Z

content/reference-strategy/index.md

+
+#### Problems
+
+* To know a field contains a foreign key, you need to look up the register field


You have the same problem with CURIEs that we don't mention below. That is you still need to look up the field definition to find out whether it is a curie or a string. "country:GB" is a valid string as well as a valid curie and they mean very different things.

Or is the point you are trying to get at that you have to check a field other than the datatype field in the field definition?

Yes, my point was around not being able to know by the datatype.

michaelabenyohai · 2018-07-13T13:10:47Z

content/reference-strategy/index.md

+This RFC accepts CURIE as the only mechanism for linking between register from
+now on given they offer the flexibility required by registers.
+
+Existing registers that use foreign keys will be maintained to avoid


Will we make a plan to evolve them to use curies?

We should, yes.

To clarify, this plan is out of scope for this RFC.

arnau · 2018-07-16T10:07:06Z

Specifically, we still have the rule that the key field of a register must have the same name as the register. This guidance instructs that these fields must no longer be created as a foreign key (which they always have previously). This in turn means that this field name can never be reused as a link from another register (which we currently do a lot). Instead, we will have to create a new field with a slightly different name, so could end up proliferating fields that come in "pairs" with similar-but-not-quite-the-same name.

I feel we could treat this as the key field should describe "the thing", whereas the curie field in the "other thing" register should describe the relationship of "the thing" to "other thing". In practice I think this could be hard though.

Very good point, I'll address it in the RFC.

arnau · 2018-07-18T08:30:15Z

@michaelabenyohai I've changed a few bits to address some of your comments, please have another read and see if it's better now.

MatMoore · 2018-07-18T13:17:04Z

The proposal makes sense to me based on what I know about how things work now.

Am I right in thinking that to actually replace the existing foreign keys with curies we first need to create the metadata log?

Also, even if we're not adding any more foreign keys, will we retain any information in the spec about the semantics of them, or will it become purely an implementation detail of specific ORJ registers? I'm just wondering whether a spec-compliant client would still be expected to resolve these kinds of links.

arnau · 2018-07-19T08:44:50Z

Am I right in thinking that to actually replace the existing foreign keys with curies we first need to create the metadata log?

Yes, in particular the schema evolution (adding fields) #5

Also, even if we're not adding any more foreign keys, will we retain any information in the spec about the semantics of them, or will it become purely an implementation detail of specific ORJ registers? I'm just wondering whether a spec-compliant client would still be expected to resolve these kinds of links.

Good point, I think the spec should, at best, mention that at some point a register had this assumption; so some sort of informative section. But a client shouldn't be expected to implement legacy assumptions.

Moreover, the current spec doesn't define foreign keys at all, so in a way we are addressing an assumption we have encoded in our implementation and clients. I opened an issue in the spec repo to address this issue: openregister/specification#84

MatMoore

The limitation of not knowing how to resolve a CURIE to a URL could be solved by including the context within the registers resource.

The expansion mechanism would then be

Given a CURIE country:GB
And a register resource
Take the curie's prefix: country
Look up the prefix in the context object of the register resource to get a URL: https://country.register.gov.uk
Add /records/ to it: https://country.register.gov.uk/records/
Add the CURIE value to it: https://country.register.gov.uk/records/GB

In theory we could provide this information through the API without storing the data in the register itself, for example by making all registers on .register.gov.uk have a context that resolves all CURIEs to .register.gov.uk.

MatMoore · 2018-08-06T14:08:49Z

content/reference-strategy/index.md

+
+Existing registers that use foreign keys will be maintained to avoid
+disruption in the registers ecosystem. An independent task will define and
+execute the migration from foreing keys to CURIEs.


there's a typo here: "foreing"

actually you could just drop this whole sentence as it doesn't say when this will happen or how.

MatMoore · 2018-08-06T14:14:30Z

content/reference-strategy/index.md

+to a register (set of records). It can be seen as the natural consequence of
+[RFC 0002: URI Consolidation](https://github.com/openregister/registers-rfcs/blob/master/content/uri-consolidation/index.md).
+
+Note that other types of references (e.g. subset of records) are out of the


"out of the scope" should be "out of scope"

arnau · 2018-08-07T08:45:43Z

The limitation of not knowing how to resolve a CURIE to a URL could be solved by including the context within the registers resource.

This is indeed the current thinking for how to handle reference resolution. It requires schema evolution to add this information to the catalog (register register).

But it's not of this RFC to say how the catalog works. There are more twists on this, happy to discuss it further.

michaelabenyohai · 2018-08-08T11:29:19Z

content/reference-strategy/index.md

 * Only able to reference one register per field.
-* Linking a full set of records is a special case.
+* You need a field for each register even if they have the same type of
+  relationship with the record (e.g. owned by, managed by).


I don't understand this point.

At all or part of it? Let me try again:

With foreign keys, the field has a local identifier so the field itself is dedicated to a single type of link (e.g. local-authority-eng). If you need to provide links to e.g. local-authority-nir as well, you need another column. Even though both have the same intention (relationship wise).

Yes that makes sense now. I think it would be good to clarify that you "need a field for each register linked to"

RFC describing the two mechanisms for linking registers and the proposal to use CURIEs from now on. Signed-off-by: Arnau Siches <arnau.siches@digital.cabinet-office.gov.uk>

Signed-off-by: Arnau Siches <arnau.siches@digital.cabinet-office.gov.uk>

arnau added rfc wip labels Mar 29, 2018

arnau requested review from Jamie-Atkinson, michaelabenyohai and nacnudus March 29, 2018 14:35

nacnudus reviewed Apr 3, 2018

View reviewed changes

arnau force-pushed the reference-strategy branch 2 times, most recently from bffede7 to dbc76ee Compare July 4, 2018 12:12

arnau removed the wip label Jul 4, 2018

michaelabenyohai reviewed Jul 13, 2018

View reviewed changes

arnau force-pushed the reference-strategy branch from e3ecace to 7afe483 Compare July 18, 2018 08:28

arnau requested a review from MatMoore July 18, 2018 08:29

MatMoore approved these changes Aug 6, 2018

View reviewed changes

michaelabenyohai reviewed Aug 8, 2018

View reviewed changes

arnau changed the title ~~Reference strategy~~ RFC0005: Reference strategy Aug 9, 2018

michaelabenyohai approved these changes Aug 13, 2018

View reviewed changes

Arnau Siches added 3 commits August 13, 2018 13:50

Add reference consolidation

0777fbd

RFC describing the two mechanisms for linking registers and the proposal to use CURIEs from now on. Signed-off-by: Arnau Siches <arnau.siches@digital.cabinet-office.gov.uk>

Reword based on @michaelabenyohai suggestions

ed8b804

Signed-off-by: Arnau Siches <arnau.siches@digital.cabinet-office.gov.uk>

Add RFC number

159b93a

Signed-off-by: Arnau Siches <arnau.siches@digital.cabinet-office.gov.uk>

arnau force-pushed the reference-strategy branch from 9d7ea58 to 159b93a Compare August 13, 2018 12:55

Add decision date

ed6a2b0

Signed-off-by: Arnau Siches <arnau.siches@digital.cabinet-office.gov.uk>

arnau merged commit ed6a2b0 into master Aug 13, 2018

arnau pushed a commit that referenced this pull request Aug 13, 2018

Merge branch 'reference-strategy'. Close #11

a723466

arnau deleted the reference-strategy branch August 13, 2018 12:59



		TODO: Where does the context live? It can't be part of the register API
		because a Register could be part of multiple environments.

		* `example:32` -> `https://example.org/32`


		TODO: Where does the context live? It can't be part of the register API


		### Foreign Key

		A foreign key is a regular string that _happens_ to be an identifier in


		#### Problems

		* To know a field contains a foreign key, you need to look up the register field

Comments

Conversation

arnau commented Mar 29, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Guidance to review

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaelabenyohai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arnau commented Jul 16, 2018

Uh oh!

arnau commented Jul 18, 2018

Uh oh!

MatMoore commented Jul 18, 2018

Uh oh!

arnau commented Jul 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MatMoore left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

arnau commented Mar 29, 2018 •

edited

Loading

arnau commented Jul 19, 2018 •

edited

Loading

MatMoore left a comment •

edited

Loading