[Bug]: Bug with start_node_reference in Google_Cloud_to_Neo4j tempalate

[AntonKetska]

Related Template(s)

Google_Cloud_to_Neo4j

Template Version

latest

What happened?

The problem is in the jobSpec, it is written in docs that we can set start_node_reference/end_node_reference as string or object docs. But actually we can't set is as object because we get error:
[SCHM-1029][$.targets.relationships[1].start_node_reference] $.targets.relationships[1].start_node_reference: object found, string expected - [SCHM-1029][$.targets.relationships[1].end_node_reference] $.targets.relationships[1].end_node_reference: object found, string expected

This is the part of specification which causes error:

{
  "relationships": [
    {
      "source": "roles",
      "name": "Acted_in",
      "type": "ACTED_IN",
      "start_node_reference": {
        "name": "Person",
        "key_mappings": [
          {
            "source_field": "personId",
            "node_property": "id"
          }
        ]
      },
      "end_node_reference": {
        "name": "Movie",
        "key_mappings": [
          {
            "source_field": "movieId",
            "node_property": "id"
          }
        ]
      },
      "write_mode": "merge",
      "node_match_mode": "match",
      "properties": [
        {
          "source_field": "role",
          "target_property": "role",
          "target_property_type": "string"
        }
      ]
    }
  ]
}

Relevant log output

com.google.cloud.teleport.v2.common.UncaughtExceptionLogger - The template launch failed.
java.lang.RuntimeException: Unable to process Neo4j job specification
	at com.google.cloud.teleport.v2.neo4j.model.helpers.JobSpecMapper.validationFailure(JobSpecMapper.java:124)
	at com.google.cloud.teleport.v2.neo4j.model.helpers.JobSpecMapper.parse(JobSpecMapper.java:57)
	at com.google.cloud.teleport.v2.neo4j.templates.GoogleCloudToNeo4j.<init>(GoogleCloudToNeo4j.java:173)
	at com.google.cloud.teleport.v2.neo4j.templates.GoogleCloudToNeo4j.main(GoogleCloudToNeo4j.java:228)
Caused by: org.neo4j.importer.v1.validation.InvalidSpecificationException: Import specification is invalid, see report below:
===============================================================================
Summary
===============================================================================
	- 3 error(s)
	- 0 warning(s)

=== Errors ===
	- [SCHM-1042][$.version] $.version: must be the constant value '1'
	- [SCHM-1029][$.targets.relationships[1].start_node_reference] $.targets.relationships[1].start_node_reference: object found, string expected
	- [SCHM-1029][$.targets.relationships[1].end_node_reference] $.targets.relationships[1].end_node_reference: object found, string expected
===============================================================================
	at org.neo4j.importer.v1.ImportSpecificationDeserializer.validate(ImportSpecificationDeserializer.java:147)
	at org.neo4j.importer.v1.ImportSpecificationDeserializer.deserialize(ImportSpecificationDeserializer.java:87)
	at org.neo4j.importer.v1.ImportSpecificationDeserializer.deserialize(ImportSpecificationDeserializer.java:75)
	at com.google.cloud.teleport.v2.neo4j.model.helpers.JobSpecMapper.parse(JobSpecMapper.java:55)
	... 2 common frames omitted

[fbiville]

Hello, this is because we have not updated the Dataflow template with the latest import-spec release.
Until that happens, you can only use node target name strings for start_node_reference and end_node_reference.
If you need to override key mappings and cannot wait, then you'll need to create a custom query target with explicit dependencies on the two node targets.

In any case, you also need to set "version": "1" at the top level.

[AntonKetska]

Thank you for the update. Could you please share if there’s an estimated timeline for when the Dataflow template will be updated to include the latest import-spec release?

[AntonKetska]

I also found 1 more issue regarding key_constraints. It is written in the docs The options key_constraints and existence_constraints require Neo4j/Aura Enterprise Edition, and do not have any effect when run against a Neo4j Community Edition installation. We are using community edition and want to have range_indexes and unique_constraints on the properties specified in key_constraints but we get error that index is redundant and as I understood key_constraints is always required. So we can't create index properly on properties specified in key_constraints.

[fbiville]

Unfortunately, Community Edition is not officially supported at the moment.
We may have fixed the problem you're describing in the latest releases.

The update to the latest release of import-spec should probably happen within the next 1.5 months.

[fbiville]

@AntonKetska The Neo4j Dataflow template now uses the latest version of import-spec, so you should be able to configure as you intended to.