Field-level lineage, from a dashboard tile to a warehouse column
Table-level lineage tells you that a dashboard touches a table. Column-level lineage tells you which dashboards break when you drop a column. Here is the difference, why the last hop is hard, and how to read a map without drowning in it.
A column is about to be dropped, retyped or renamed. Two questions sound alike at that moment. One is which dashboards touch this table. The other is which dashboards break.
Table-level lineage answers the first one. On a wide table it returns forty dashboards. Two of them use the column, and someone now has to open forty workbooks to find those two. The report is right, and close to useless.
Column-level lineage answers the second one, and that is why the extra work of reaching that grain is worth doing.
The chain, hop by hop
A dashboard tile shows a worksheet, and the worksheet uses fields.
Each field is either a calculation or a direct link into a datasource.
A calculation uses other fields, sometimes several layers deep, sometimes across datasources.
A datasource field maps to a column in a physical table.
That table has upstream tables, through views and transforms, back to wherever the data first landed.
Every hop crosses a system boundary. Each system has its own idea of what a name is. That is where the trouble lives, and it is not in the drawing of the graph.
The last hop
Two problems turn up here. Both were found by probing live warehouses, not by reading about them.
Names arrive dialect-quoted. A metadata API returns the full name quoted the way its own dialect quotes it. That is not a string another engine will take as it stands.
Case matters. Snowflake binds are case-sensitive, and an unquoted name folds to upper case. Bind a name raw and it matches zero rows.
A silent zero is the worst failure in lineage. An empty result looks just like a clean answer. A link that is connected, set up right and matching nothing reads as an environment with no dependencies at all.
The fix is to normalise before every warehouse probe, not at the end. Unquoted Snowflake segments fold to upper case. Databricks names are dequoted and lowered in the SQL itself. The full name is split into segments instead of picked apart by a string search.
Where column edges come from
Snowflake
The account’s own lineage function, distance-capped so a deep graph cannot run away. An account-usage fallback covers older accounts where that function is absent.
Databricks
system.access.column_lineage. Real column-to-column lineage kept by the platform itself, which is the strongest evidence on offer.
Palantir AIP
Link types, read from each object type in turn. They are the ontology’s own join graph, and the direction they run is the one the platform declares rather than one inferred from names.
Tableau
The Metadata API. It answers at column level when Catalogue has indexed that far, and at table level when it has not. That case has to be handled, not assumed away.
Name matching
A bridge from a datasource field to a warehouse column by name. Used only where the match is unique, and always labelled as matched.
The fourth one earns a hard look, and the product gives it one. A name-matched bridge is labelled as matched, and is never shown as catalogue data. A guess labelled a guess is useful. A guess quietly promoted to a fact ruins the whole map, including the parts that were right.
Say which grain you reached
A lineage map should state the grain it reached, and name what completed it. Catalogue may answer at table level. Column lists pulled from the warehouse then finish the job, and the map says exactly that.
This sounds like a small point of manners. It is the gap between a map you can decide from and a map you have to check by hand. A map that always claims column grain will be wrong now and then, and it will never say which times those were.
Reading a map without drowning
A full column-level graph of a real environment is hard to read all at once. Thousands of nodes is not a map. It is a texture.
The first load shows the backbone at table level. Click one card and only that card’s own grain opens: its columns, and the fields flowing from them. That is one hop, not the whole closure. The layout then refits, so what opened lands in view.
That behaviour came out of two rounds of live use, in both directions. Showing everything was tried first, and it was unusable. The answer turned out to be a local opening rather than a smaller graph: a different idea, and a better one.
What to use it for
Impact work before a schema change. You get the real list of dashboards hit, not the wide one.
Tracing a number back to its source when two dashboards disagree and both authors are sure.
Finding which dashboards lean on a table that nobody will admit to owning.
Handing someone a self-contained map they can click through, exported as one HTML file. It works with no server and no internet.
The export is worth knowing about. Governance talks happen with people who have no login and are not going to get one. A file they can open beats an invitation they will not accept.
A semantic layer gives a metric one home. Each tool that works out the metric reads that one home. They all get the same number. Here is what that means in practice, and what changes when you run more than one warehouse.
Four things get called governance: certification, ownership, stale content and change control. Here each one is written as a check you can run, not an aim you can state. It also covers the one change that passes a visual review and is still wrong.
An assertion is a sentence. Evidence is something a second person can check without asking you. What an audit trail has to contain, what a hash actually proves, and what to require of any tool that claims to produce one.