timeline release of the ilum marquez ui

[jeff-99]

@thijs-s I've seen you commenting in the "official" marquez repo's issues that you would start upstreaming changes and opensource your own version of the marquez webui. Do you have a rough estimate of when we can expect these upstream changes and/or the new web-ui?

[thijs-s]

Hey @jeff-99 sorry for the late reply. We’ve shifted our focus a bit to another module (Spark declarative pipeline UI), but Marquez is still on our roadmap before the end of the year. In the meantime, the current repo is fully operational with the old UI.

It’d be great to hear your expectations. are we heading in the right direction, or way off? 🙂 the version we plan to open source will be very close to the lineage module in the Ilum data platform. You can try it here:
https://ilum.cloud/docs/get_started/ or read more about the features here: https://ilum.cloud/docs/features/lineage/

If you have ideas on how we could expand it, that would be hugely appreciated.

Thanks!

[jeff-99]

@thijs-s this is great stuff. Read through the docs and it is exactly what I'm looking for. Our requirement is to be able to display and search through column level lineage and entity relation diagrams.

So if you're looking for someone to test an early version let me know!

[thijs-s]

Awesome! Searching through the ERD is there, but the search option in column lineage is missing! so huge thanks for the feedback. I will quickly jump in and add it.

testing would be highly appreciated. The 6.6.0 version of our data platform should be released next week, and this would be the best time to test the lineage part as it will be closest to the open source version. Will let you know.

[thijs-s]

Hey @jeff-99, Happy to share that we have finalized version 6.6.1 so it would be awesome to get your feedback.

you can follow the installation guide under this link -> https://ilum.cloud/docs/get_started/

or try with this helm install command

helm repo add ilum https://charts.ilum.cloud

helm install ilum ilum/ilum \
  --set ilum-hive-metastore.enabled=true \
  --set ilum-core.metastore.enabled=true \
  --set ilum-core.metastore.type=hive \
  --set ilum-sql.enabled=true \
  --set ilum-core.sql.enabled=true \
  --set global.lineage.enabled=true

[jeff-99]

Thanks for the heads-up @thijs-s I will set aside some time next week to test it out 👍

[jeff-99]

Hi @thijs-s, I've tested the ui in the new ilumcloud version and I must say I really like the visuals. I did break it though. I loaded a datamodel from one of our projects with over a 1000 tables. The ui then becomes really unstable in multiple occassions crashing my browser window. If you like I can show you what I mean in a video call

[thijs-s]

Hey @jeff-99, we are on it to fix that. I will ping you ASAP when it's ready.

[thijs-s]

Fixed :)

I checked it on 934 nodes, and it's way better now. I'm adding some small tweaks, like contextual zoom, and then I'll create a Docker image with the update. I'll let you know within next 24h.

[thijs-s]

Hey @jeff-99, here is a new Docker image for ilum-ui ilum/dev:ui-6.6.2-RC1

you can upgrade it with

helm upgrade ilum ilum/ilum --reuse-values --set ilum-ui.image=ilum/dev:ui-6.6.2-RC1
or
kubectl set image deployment/ilum-ui ilum-ui=ilum/dev:ui-6.6.2-RC1

It should work a lot better now.

[jeff-99]

Hey @thijs-s , The update does not freeze my browser anymore, so it fixes that issue. Still I find the overview to be worse than in the Marquez UI.
See below I selected one of the tables in the datamart layer to see it's lineage.

In ilum ui it looks like this:

In marquez it looks like this

As you can see the overview in Marquez is way easier to navigate

[thijs-s]

It seems that not everything is loaded. have you tried to use load more load all button ?

[jeff-99]

@thijs-s, That does render all the parts 👍. Overall it's still hard to get an lineage overview this way as it has to render the entire graph. What I love about the marquez ui is that I can filter on a specific dataset or job and that only its upstream and downstream consumers (for X number of levels) get rendered. I can't seem to do that with the ilum-ui. Not for the lineage or the ERD's which makes navigating through the maze of lines difficult.

Also somehow I can't get the column level lineage tab to show, while I do post the column level lineage information in the API calls. See the below. Is there anything missing ?

{
  "schema": {
    "fields": [
      {
        "name": "Asset HK",
        "type": "CHAR(64)"
      },
      {
        "name": "Terminal ID",
        "type": "NVARCHAR(255)"
      },
      {
        "name": "Terminal HK",
        "type": "CHAR(64)"
      },
      {
        "name": "Asset ID",
        "type": "NVARCHAR(255)"
      },
      {
        "name": "Asset Name",
        "type": "NVARCHAR(255)"
      },
      {
        "name": "Asset Description",
        "type": "NVARCHAR(1000)"
      },
      {
        "name": "Is Active",
        "type": "BIT"
      },
      {
        "name": "Year Of Build",
        "type": "INT"
      },
      {
        "name": "Start Operation Date",
        "type": "DATE"
      },
      {
        "name": "End Operation Date",
        "type": "DATE"
      },
      {
        "name": "Structure Level ID",
        "type": "NVARCHAR(255)"
      },
      {
        "name": "Asset Structure Name",
        "type": "NVARCHAR(1000)"
      },
      {
        "name": "Asset Structure Description",
        "type": "NVARCHAR(1000)"
      },
      {
        "name": "Asset Structure Path",
        "type": "NVARCHAR(4000)"
      },
      {
        "name": "Asset Structure Name Path",
        "type": "NVARCHAR(4000)"
      },
      {
        "name": "Asset Structure Description Path",
        "type": "NVARCHAR(4000)"
      },
      {
        "name": "Asset Category",
        "type": "NVARCHAR(255)"
      },
      {
        "name": "Asset Group",
        "type": "NVARCHAR(255)"
      },
      {
        "name": "Asset Type",
        "type": "NVARCHAR(255)"
      },
      {
        "name": "Asset Unit",
        "type": "NVARCHAR(255)"
      },
      {
        "name": "Component",
        "type": "NVARCHAR(255)"
      },
      {
        "name": "Last Modified Date Time",
        "type": "DATETIME2"
      }
    ],
    "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.39.0/client/python",
    "_schemaURL": "https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/SchemaDatasetFacet"
  },
  "columnLineage": {
    "fields": {
      "Asset HK": {
        "inputFields": [
          {
            "name": "Asset_RSTSAT",
            "field": "Asset_HK",
            "namespace": "dv"
          }
        ],
        "transformationType": "DIRECT",
        "transformationDescription": "deletes load"
      }
    },
    "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.39.0/client/python",
    "_schemaURL": "https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/ColumnLineageDatasetFacet"
  }
}

[thijs-s]

I will try to address both problems. Would it be anyhow possible to get a dump of your marquez db from Postgres?

I'll try to match marquez style or improve on it. I'd love to hear any other ideas you have!

[jeff-99]

here you go marquez.sql.gz