The Companion Program · Databricks Companion

A bronze layer the lakehouse can refine.

The lakehouse architecture is excellent at refining data through bronze, silver, and gold layers. It is only as defensible as the bronze layer it ingests. Smart Stack 3.0 governs the Microsoft source so Mosaic AI, Genie, Unity Catalog, and Lakeflow refine a bronze layer that already carries classification, provenance, and chain of custody.

Listen to the Podcast

The Problem This Companion Solves

The lakehouse architecture is one of the genuine engineering accomplishments of the past decade. Databricks has built a unified platform that brings analytics and AI workloads onto the same data foundation, eliminating the historical separation between data lakes and data warehouses. Delta Lake provides ACID transactions on object storage. Unity Catalog provides governance primitives across the workspace. Mosaic AI delivers production-grade machine learning capabilities. Genie translates natural language questions into governed analytical answers. Lakeflow orchestrates the data engineering workflows that feed all of it.

The lakehouse architecture is excellent at refining data through bronze, silver, and gold layers. Bronze tables hold raw ingested data. Silver tables hold cleaned, conformed data. Gold tables hold business-ready aggregates. The refinement model is sound. The work happens at compute scale on data that has already arrived in the lakehouse.

The architecture is only as defensible as the bronze layer it ingests. When the source is the Microsoft estate, the bronze layer inherits whatever the file shares and SharePoint environments contain. ROT contamination at the bronze layer compounds at every refinement step. Duplicates at bronze appear as separate records at silver. Superseded versions at bronze produce conflicting truth at gold. Sensitive content at bronze surfaces in AI outputs at the consumption layer. The lakehouse architecture cannot fix upstream data quality problems. It can only refine them at increasing cost.

Mosaic AI reasoning over a contaminated bronze layer produces models trained on noise that looks like signal. Genie translating natural language questions against contaminated tables produces answers that reflect the data quality of the source, not the sophistication of the platform. Unity Catalog can apply governance to what Databricks can see, but cannot itself classify content that was never classified in the source environment. The governance has to happen upstream, in the Microsoft estate, before the lakehouse ingest.

Built for the executives who own the data estate

For the CDO

The lakehouse economics pay back when the bronze layer is worthy of the platform. Mosaic AI, Genie, and Unity Catalog are all excellent at refining data. None of them can fix upstream data quality. The Trusted Data Refinery makes the Microsoft estate Databricks-ready before ingestion. Trusted Data Collections become the governed bronze layer the lakehouse architecture was designed to refine. Mosaic AI training on a governed bronze layer delivers the capability the CDO sponsored when the Databricks investment was approved.

For the CIO

Compute scales correctly when the bronze layer is governed. ROT in the bronze layer is paid for at every refinement step. Duplicates at bronze become duplicate records at silver. Superseded versions at bronze produce conflicting gold tables. Eliminating ROT upstream means lakehouse compute scales with the volume of content the platform actually has to process, not the residue of an ungoverned source. Storage and compute spend in Databricks fall proportionally to the ROT eliminated upstream.

For the CISO

The lakehouse becomes a concentration of governed and ungoverned content unless the source is classified before ingestion. Sensitive content classified inside Microsoft can resurface ungoverned in Databricks. Mosaic AI can reason over privileged communications. Genie can return answers grounded in PII nobody intended to expose. Sensitive Data Classification governs what crosses the boundary. The AI Governance Workflow controls what reaches Mosaic AI and Genie. The Microsoft tenant remains the security boundary for the entire data estate.

What the Trusted Data Toolkit Does for Databricks Companion

Trusted Data Archive

Provenance From Source to Silver Layer

Captures Exchange, M365, and Teams communications with WORM retention and evidentiary chain of custody. Customer communications, internal correspondence, regulatory threads, and contract negotiations preserved in the Microsoft tenant become available as bronze tables in the lakehouse with envelope metadata and chain of custody intact. Mosaic AI workloads training on communications data train on a defensible source.

Trusted Data Refinery

The Governed Bronze Layer

Prepares Windows File Share and SharePoint data for lakehouse ingestion by classifying, enriching, and curating it in place before it becomes a bronze table. ROT eliminated. Sensitive content classified through Sensitive Data Classification. Metadata enriched. Trusted Data Collections become the governed bronze layer the lakehouse architecture was designed to refine. Silver and gold work that follows operates on signal, not noise.

Trusted Data Portal

Lakehouse Federation With Governance Intact

Routes Trusted Data Collections through OneLake into the Databricks environment via native lakehouse federation. Chain of custody and provenance metadata travel through every transformation. The AI Governance Workflow ensures only policy-aligned data reaches Mosaic AI and Genie. Workflow Builder orchestrates lakehouse-specific governance patterns including Bronze Layer Governance Workflows and Lakehouse AI Workflows.

What Changes for the Databricks Customer

The bronze layer becomes a governed foundation, not a raw ingestion zone. Trusted Data Collections arrive classified, ROT-eliminated, metadata-enriched, and provenance-aware. The bronze tables the lakehouse refines reflect the actual operational record of the business, not the chaos of the underlying source.

Silver and gold refinement work operates on signal. Compute waste falls because the lakehouse is not refining content that should never have been ingested. Output tables produce analytical results that can be traced back through the bronze layer to the original Microsoft source. Defensibility extends from the bronze table to the gold report to the executive dashboard that consumes it.

Mosaic AI trains and infers on classified, governed Trusted Data Collections. Model outputs are traceable to provenance-verified sources. Training data quality improvements show up in inference quality improvements. The model is not getting smarter. The data is becoming trustworthy.

Genie's natural language analytical interface produces answers grounded in governed data. The user asking a question gets a response that reflects the actual operational record, not the residue of decades of ungoverned accumulation. Answers are defensible to leadership review and audit.

Unity Catalog becomes more powerful because the underlying data is healthier. The classification metadata that travels with Trusted Data Collections strengthens Unity Catalog's ability to apply governance, lineage, and access control across the lakehouse.

Govern once. Activate everywhere.