Skip to content

Field-level lineage, from a dashboard tile to a warehouse column

Table-level lineage tells you that a dashboard touches a table. Column-level lineage tells you which dashboards break when you drop a column. Here is the difference, why the last hop is hard, and how to read a map without drowning in it.

Updated · 5 min read

What column grain adds

A column is about to be dropped, retyped or renamed. Two questions sound alike at that moment. One is which dashboards touch this table. The other is which dashboards break.

Table-level lineage answers the first one. On a wide table it returns forty dashboards. Two of them use the column, and someone now has to open forty workbooks to find those two. The report is right, and close to useless.

Column-level lineage answers the second one, and that is why the extra work of reaching that grain is worth doing.

The chain, hop by hop

  1. A dashboard tile shows a worksheet, and the worksheet uses fields.
  2. Each field is either a calculation or a direct link into a datasource.
  3. A calculation uses other fields, sometimes several layers deep, sometimes across datasources.
  4. A datasource field maps to a column in a physical table.
  5. That table has upstream tables, through views and transforms, back to wherever the data first landed.

Every hop crosses a system boundary. Each system has its own idea of what a name is. That is where the trouble lives, and it is not in the drawing of the graph.

The last hop

Two problems turn up here. Both were found by probing live warehouses, not by reading about them.

  • Names arrive dialect-quoted. A metadata API returns the full name quoted the way its own dialect quotes it. That is not a string another engine will take as it stands.
  • Case matters. Snowflake binds are case-sensitive, and an unquoted name folds to upper case. Bind a name raw and it matches zero rows.

A silent zero is the worst failure in lineage. An empty result looks just like a clean answer. A link that is connected, set up right and matching nothing reads as an environment with no dependencies at all.

The fix is to normalise before every warehouse probe, not at the end. Unquoted Snowflake segments fold to upper case. Databricks names are dequoted and lowered in the SQL itself. The full name is split into segments instead of picked apart by a string search.

Where column edges come from

Snowflake
The account’s own lineage function, distance-capped so a deep graph cannot run away. An account-usage fallback covers older accounts where that function is absent.
Databricks
system.access.column_lineage. Real column-to-column lineage kept by the platform itself, which is the strongest evidence on offer.
Palantir AIP
Link types, read from each object type in turn. They are the ontology’s own join graph, and the direction they run is the one the platform declares rather than one inferred from names.
Tableau
The Metadata API. It answers at column level when Catalogue has indexed that far, and at table level when it has not. That case has to be handled, not assumed away.
Name matching
A bridge from a datasource field to a warehouse column by name. Used only where the match is unique, and always labelled as matched.

The fourth one earns a hard look, and the product gives it one. A name-matched bridge is labelled as matched, and is never shown as catalogue data. A guess labelled a guess is useful. A guess quietly promoted to a fact ruins the whole map, including the parts that were right.

Say which grain you reached

A lineage map should state the grain it reached, and name what completed it. Catalogue may answer at table level. Column lists pulled from the warehouse then finish the job, and the map says exactly that.

This sounds like a small point of manners. It is the gap between a map you can decide from and a map you have to check by hand. A map that always claims column grain will be wrong now and then, and it will never say which times those were.

Reading a map without drowning

A full column-level graph of a real environment is hard to read all at once. Thousands of nodes is not a map. It is a texture.

The first load shows the backbone at table level. Click one card and only that card’s own grain opens: its columns, and the fields flowing from them. That is one hop, not the whole closure. The layout then refits, so what opened lands in view.

That behaviour came out of two rounds of live use, in both directions. Showing everything was tried first, and it was unusable. The answer turned out to be a local opening rather than a smaller graph: a different idea, and a better one.

What to use it for

  • Impact work before a schema change. You get the real list of dashboards hit, not the wide one.
  • Tracing a number back to its source when two dashboards disagree and both authors are sure.
  • Finding which dashboards lean on a table that nobody will admit to owning.
  • Handing someone a self-contained map they can click through, exported as one HTML file. It works with no server and no internet.

The export is worth knowing about. Governance talks happen with people who have no login and are not going to get one. A file they can open beats an invitation they will not accept.