{"vulnerability": "cve-2024-2562", "sightings": [{"uuid": "021babbf-a19b-43c1-92d9-ff196cc17af0", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25625", "type": "seen", "source": "https://t.me/arpsyndicate/3674", "content": "#ExploitObserverAlert\n\nCVE-2024-25625\n\nDESCRIPTION: Exploit Observer has 1 entries in 1 file formats related to CVE-2024-25625. Pimcore's Admin Classic Bundle provides a Backend UI for Pimcore. A potential security vulnerability has been discovered in `pimcore/admin-ui-classic-bundle` prior to version 1.3.4. The vulnerability involves a Host Header Injection in the `invitationLinkAction` function of the UserController, specifically in the way `$loginUrl` trusts user input.  The host header from incoming HTTP requests is used unsafely when generating URLs. An attacker can manipulate the HTTP host header in requests to the /admin/user/invitationlink endpoint, resulting in the generation of URLs with the attacker's domain. In fact, if a host header is injected in the POST request, the $loginURL parameter is constructed with this unvalidated host header. It is then used to send an invitation email to the provided user. This vulnerability can be used to perform phishing attacks by making the URLs in the invitation links emails point to an attacker-controlled domain. Version 1.3.4 contains a patch for the vulnerability. The maintainers recommend validating the host header and ensuring it matches the application's domain. It would also be beneficial to use a default trusted host or hostname if the incoming host header is not recognized or is absent.", "creation_timestamp": "2024-02-20T18:54:30.000000Z"}, {"uuid": "9d227c91-3404-4942-93a8-da9eaa2dafcd", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25626", "type": "seen", "source": "Telegram/d8YNJ_IhzxAj-FCR4t25rOEGyCbvl13tNYDSAkUE9NYY0BBR", "content": "", "creation_timestamp": "2025-02-06T02:40:18.000000Z"}, {"uuid": "7594dcf5-6478-49ac-9452-90fdf58ac9a0", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25621", "type": "seen", "source": "https://bsky.app/profile/ferramentaslinux.bsky.social/post/3makv2j5u3c2g", "content": "", "creation_timestamp": "2025-12-22T09:07:33.350940Z"}, {"uuid": "44d10b72-4a10-4f62-83fb-c63fb5792140", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25622", "type": "seen", "source": "https://t.me/cvedetector/7690", "content": "{\n  \"Source\": \"CVE FEED\",\n  \"Title\": \"CVE-2024-25622 - h2o HTTP Server Header Inheritance Vulnerability\", \n  \"Content\": \"CVE ID : CVE-2024-25622 \nPublished : Oct. 11, 2024, 3:15 p.m. | 31\u00a0minutes ago \nDescription : h2o is an HTTP server with support for HTTP/1.x, HTTP/2 and HTTP/3. The configuration directives provided by the headers handler allows users to modify the response headers being sent by h2o. The configuration file of h2o has scopes, and the inner scopes (e.g., path level) are expected to inherit the configuration defined in outer scopes (e.g., global level). However, if a header directive is used in the inner scope, all the definition in outer scopes are ignored. This can lead to headers not being modified as expected. Depending on the headers being added or removed unexpectedly, this behavior could lead to unexpected client behavior. This vulnerability is fixed in commit 123f5e2b65dcdba8f7ef659a00d24bd1249141be. \nSeverity: 3.1 | LOW \nVisit the link for more details, such as CVSS details, affected products, timeline, and more...\",\n  \"Detection Date\": \"11 Oct 2024\",\n  \"Type\": \"Vulnerability\"\n}\n\ud83d\udd39 t.me/cvedetector \ud83d\udd39", "creation_timestamp": "2024-10-11T17:51:57.000000Z"}, {"uuid": "a325e58f-cab0-4709-9486-b5ae437de28d", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-2562", "type": "seen", "source": "https://t.me/ctinow/209913", "content": "https://ift.tt/houGLji\nCVE-2024-2562", "creation_timestamp": "2024-03-17T13:26:17.000000Z"}, {"uuid": "8bd34662-fc9e-4e8f-b328-ead06136e4d4", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25629", "type": "seen", "source": "https://t.me/ctinow/207333", "content": "https://ift.tt/TYeQiwF\nCVE-2024-25629 | c-ares up to 1.26.x Null Character ares__read_line memory corruption", "creation_timestamp": "2024-03-14T00:46:26.000000Z"}, {"uuid": "eff63462-6bd3-49d6-bdd5-3fa2d235672b", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-2562", "type": "seen", "source": "https://t.me/ctinow/209911", "content": "https://ift.tt/houGLji\nCVE-2024-2562", "creation_timestamp": "2024-03-17T13:21:52.000000Z"}, {"uuid": "03c83468-1c1f-4be4-9d25-09ea38994ac6", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25623", "type": "seen", "source": "https://t.me/ctinow/203207", "content": "https://ift.tt/TJX75zk\nCVE-2024-25623 | Mastodon up to 3.5.18/4.0.14/4.1.14/4.2.6 HTTP Header Content-Type unrestricted upload", "creation_timestamp": "2024-03-08T10:51:35.000000Z"}, {"uuid": "50ed9cf8-edbd-464b-945b-e2886ce0f5c2", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25625", "type": "seen", "source": "https://t.me/ctinow/203208", "content": "https://ift.tt/xYUlbZF\nCVE-2024-25625 | Pimcore admin-ui-classic-bundle up to 1.3.3 HTTP Header invitationLinkAction Host injection", "creation_timestamp": "2024-03-08T10:51:36.000000Z"}, {"uuid": "2b649bd1-62df-455a-8594-beba95f6bb87", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25629", "type": "seen", "source": "https://t.me/ctinow/192146", "content": "https://ift.tt/tJcTr6w\nCVE-2024-25629", "creation_timestamp": "2024-02-23T20:41:31.000000Z"}, {"uuid": "87848366-cbf1-45bc-8fa6-9ff93b60dcf3", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25625", "type": "seen", "source": "https://t.me/ctinow/187832", "content": "https://ift.tt/I0bVQN5\nCVE-2024-25625", "creation_timestamp": "2024-02-19T17:26:27.000000Z"}, {"uuid": "eeefab9d-bded-4a87-9d06-6ad41b2948dd", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25626", "type": "seen", "source": "https://t.me/ctinow/187929", "content": "https://ift.tt/HC2wgqt\nCVE-2024-25626", "creation_timestamp": "2024-02-19T21:26:31.000000Z"}, {"uuid": "8af561a0-4ecc-43a4-b7f3-a316c9d3ecaf", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25629", "type": "seen", "source": "https://t.me/ctinow/191907", "content": "https://ift.tt/C5uRzbQ\nCVE-2024-25629", "creation_timestamp": "2024-02-23T16:41:49.000000Z"}, {"uuid": "3303f317-4379-4699-9d29-48306ef43277", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25625", "type": "seen", "source": "https://t.me/ctinow/187827", "content": "https://ift.tt/I0bVQN5\nCVE-2024-25625", "creation_timestamp": "2024-02-19T17:21:59.000000Z"}, {"uuid": "0944c6a6-4f10-4208-bfd1-0e07a91c442f", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25623", "type": "seen", "source": "https://t.me/ctinow/187826", "content": "https://ift.tt/n1uQg9f\nCVE-2024-25623", "creation_timestamp": "2024-02-19T17:21:58.000000Z"}, {"uuid": "0b2294ce-e1ef-4abd-9f25-e736730912fc", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25623", "type": "seen", "source": "https://t.me/ctinow/187831", "content": "https://ift.tt/n1uQg9f\nCVE-2024-25623", "creation_timestamp": "2024-02-19T17:26:26.000000Z"}, {"uuid": "bf30d889-e62e-4c37-b916-991f73117b5c", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25629", "type": "seen", "source": "https://t.me/ctinow/191883", "content": "https://ift.tt/C5uRzbQ\nCVE-2024-25629", "creation_timestamp": "2024-02-23T16:32:07.000000Z"}, {"uuid": "16acd5d2-022e-4192-8be0-dbd318c75523", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25628", "type": "seen", "source": "https://t.me/ctinow/186662", "content": "https://ift.tt/2VpXFed\nCVE-2024-25628", "creation_timestamp": "2024-02-16T22:22:02.000000Z"}, {"uuid": "b4e3f224-66fa-4259-8262-776bb4f9e6fb", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25627", "type": "seen", "source": "https://t.me/ctinow/186661", "content": "https://ift.tt/O6lheic\nCVE-2024-25627", "creation_timestamp": "2024-02-16T22:21:58.000000Z"}, {"uuid": "4f1adc23-d72c-4cf9-842d-c02d586ad7a1", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25620", "type": "seen", "source": "https://t.me/ctinow/185308", "content": "https://ift.tt/j6ZB5OD\nCVE-2024-25620", "creation_timestamp": "2024-02-15T08:11:20.000000Z"}, {"uuid": "975b1864-ee3b-476c-b12a-3cc8ecb968b6", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25626", "type": "seen", "source": "https://t.me/ctinow/187922", "content": "https://ift.tt/HC2wgqt\nCVE-2024-25626", "creation_timestamp": "2024-02-19T21:21:24.000000Z"}, {"uuid": "53cb49be-6667-4a4a-a807-3bc5542da400", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25620", "type": "seen", "source": "https://t.me/ctinow/185168", "content": "https://ift.tt/pX6GAT8\nCVE-2024-25620", "creation_timestamp": "2024-02-15T01:26:45.000000Z"}, {"uuid": "7e72e134-2ed1-4336-b50a-b09c304a73aa", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25624", "type": "seen", "source": "https://gist.github.com/khoindq/bded28fd1242788522ce29d279b9d883", "content": "\n\n\n\nLyra \u00b7 Unified Governance & ACL Design \u2014 Complete\n\n  :root {\n    --bg: #0b0d10; --panel: #12161b; --border: #1f262e; --ink: #e6edf3;\n    --mute: #8b949e; --accent: #58a6ff; --warn: #f0883e; --err: #f85149;\n    --ok: #3fb950; --pick: #d2a8ff; --line: #30363d;\n    --mono: ui-monospace, SFMono-Regular, \"SF Mono\", Menlo, Consolas, monospace;\n  }\n  * { box-sizing: border-box; }\n  body { background: var(--bg); color: var(--ink); font: 14px/1.55 -apple-system, system-ui, sans-serif; margin: 0; padding: 32px 24px 80px; }\n  .wrap { max-width: 1200px; margin: 0 auto; }\n  h1 { font-size: 26px; margin: 0 0 4px; }\n  h2 { font-size: 19px; margin: 36px 0 12px; padding-bottom: 6px; border-bottom: 1px solid var(--line); }\n  h3 { font-size: 15px; margin: 20px 0 8px; color: var(--accent); }\n  h4 { font-size: 13px; margin: 14px 0 6px; color: var(--pick); text-transform: uppercase; letter-spacing: 0.04em; }\n  .sub { color: var(--mute); margin: 0 0 20px; font-size: 13px; }\n  .panel { background: var(--panel); border: 1px solid var(--border); border-radius: 8px; padding: 16px 18px; margin: 12px 0; }\n  .toc { columns: 2; column-gap: 32px; font-size: 13px; }\n  .toc a { color: var(--accent); text-decoration: none; display: block; padding: 2px 0; }\n  .toc a:hover { color: var(--ink); }\n  .stat { display: grid; grid-template-columns: repeat(4, 1fr); gap: 12px; margin: 16px 0 24px; }\n  .stat .card { background: var(--panel); border: 1px solid var(--border); border-radius: 8px; padding: 14px; }\n  .stat .n { font-size: 28px; font-weight: 600; color: var(--accent); }\n  .stat .l { font-size: 11px; color: var(--mute); text-transform: uppercase; letter-spacing: 0.05em; }\n  table { border-collapse: collapse; width: 100%; margin: 8px 0 16px; font-size: 13px; }\n  th, td { border: 1px solid var(--line); padding: 7px 10px; text-align: left; vertical-align: top; }\n  th { background: #161b22; font-weight: 600; color: var(--mute); font-size: 12px; text-transform: uppercase; letter-spacing: 0.04em; }\n  tr:hover td { background: #0f1318; }\n  code { font-family: var(--mono); background: #161b22; padding: 1px 5px; border-radius: 3px; font-size: 12px; }\n  pre { background: #0d1117; border: 1px solid var(--border); border-radius: 6px; padding: 12px; overflow-x: auto; font: 12px/1.55 var(--mono); }\n  .ok { color: var(--ok); } .warn { color: var(--warn); } .err { color: var(--err); } .mute { color: var(--mute); } .pick { color: var(--pick); font-weight: 600; }\n  .pill { display: inline-block; padding: 1px 7px; border-radius: 10px; font-size: 11px; border: 1px solid var(--line); background: #161b22; color: var(--mute); }\n  .pill.g { color: var(--ok); border-color: #1e3a2a; background: #0d1f15; }\n  .pill.r { color: var(--err); border-color: #3a1d1d; background: #1f0e0e; }\n  .pill.y { color: var(--warn); border-color: #3a2c1d; background: #1f1810; }\n  .pill.b { color: var(--accent); border-color: #1d3050; background: #0e1a2b; }\n  .pill.v { color: var(--pick); border-color: #3a2848; background: #1a1124; }\n  /* tree */\n  .tree { font-family: var(--mono); font-size: 12.5px; line-height: 1.7; padding: 14px 18px; background: var(--panel); border: 1px solid var(--border); border-radius: 6px; }\n  .tree ul { list-style: none; padding-left: 18px; margin: 0; border-left: 1px dashed var(--line); }\n  .tree > ul { padding-left: 0; border-left: none; }\n  .tree li { position: relative; padding: 1px 0 1px 14px; }\n  .tree li::before { content: \"\u2500\"; position: absolute; left: 0; color: var(--line); }\n  .tree .root { color: var(--pick); font-weight: 600; }\n  .tree .acc { color: var(--accent); }\n  .tree .new { color: var(--ok); font-weight: 600; }\n  .tree small { color: var(--mute); margin-left: 6px; font-size: 11px; }\n  .badge-new { background: #0d1f15; color: var(--ok); border: 1px solid #1e3a2a; padding: 0 5px; border-radius: 3px; font-size: 10px; margin-left: 4px; }\n  .grid2 { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; }\n  .pick-row td { background: #1a1124 !important; }\n  hr.sep { border: 0; border-top: 1px solid var(--line); margin: 28px 0; }\n  .num { display: inline-block; width: 18px; height: 18px; border-radius: 50%; background: var(--accent); color: var(--bg); text-align: center; font-size: 11px; font-weight: 700; line-height: 18px; margin-right: 6px; vertical-align: 2px; }\n\n\n\n\n\n\n\nLyra \u00b7 Unified Governance & ACL Design complete\n\nSingle source of truth for the Lyra Unity Catalog clone. Consolidates 7 review agents (codebase audit, Databricks UC 2026, Snowflake / BigQuery / Polaris / Cedar / Lake Formation cross-checks) into one document. Replaces v1 (review), v2 (full inventory), v3 (gap-closer) \u2014 this is the merged, paste-ready spec.\n\n\n\n  \n\n6 \u2192 1\nACL tables collapsed\n  \n\n47\nsecurables covered\n  \n\n29\nprivileges (incl. macros)\n  \n\n4-pass\nforbid \u2192 admin \u2192 owner \u2192 permit\n\n\n\n\n0 \u00b7 Table of contents\n\n\nTL;DR \u2014 the picture\n1. Current state & bugs found\n2. Decision: why one grant table\n3. Engine comparison\n4. Peer cross-check (Databricks / Snowflake / \u2026)\n5. Complete asset inventory (47)\n6. Object hierarchy tree\n7. Privilege vocabulary\n8. Cascade rules\n9. DDL \u2014 paste-ready\n10. Resolver \u2014 4 passes\n11. List-endpoint pushdown\n12. Row filters & column masks\n13. Conditions \u2014 Cedar grammar\n14. Audit invariants\n15. Federated identity\n16. Multi-tenant isolation\n17. Anti-escalation rules\n18. Cache architecture\n19. Future assets \u2014 pre-planned\n20. Migration plan\n21. Anti-patterns\n22. Sources\n\n\n\n\nTL;DR \u2014 the picture\n\n\n\nToday. Lyra has 6 parallel ACL surfaces \u2014 grants (UC), compute_cluster_acls, workspace_object_acls, job_acls, secret_acls, workspace_permission_assignments. Each has its own privilege vocabulary, its own admin-bypass logic, and its own bugs. Permission checks are scattered across 70+ handlers; several handlers skip checks entirely.\n\nTarget. One grant table over a typed principal \u2194 securable graph. One 4-pass resolver (forbid \u2192 admin \u2192 owner \u2192 permit). One privilege vocabulary mapped from legacy permission levels. Owner is a column on the securable, never a grant row. Cascade is data (securable_cascade table), not code. Conditions are Cedar expressions, not free-form JSON.\n\nCoverage. 47 active securables across 7 planes (UC data, Sharing & Marketplace, Workspace, Compute, Orchestration, Identity, Account / Secrets) plus 30+ pre-planned reservations. Every handler in the codebase maps to exactly one securable.\n\n\n\n\n1 \u00b7 Current state & bugs found\n\n\n1.1 \u2014 The 6 parallel ACL systems\n\n\n#TableSourceVocabularyCoverage\n1grantsstorage / repositories / grant.rsUC privileges (USE_CATALOG, SELECT, MODIFY, \u2026)UC data plane\n2compute_cluster_aclscluster_acl.rsCAN_ATTACH_TO \u00b7 CAN_RESTART \u00b7 CAN_MANAGECluster only\n3workspace_object_aclsworkspace_object_acl.rsCAN_READ \u00b7 CAN_RUN \u00b7 CAN_EDIT \u00b7 CAN_MANAGENotebook \u00b7 File \u00b7 Directory \u00b7 Dashboard\n4job_aclsjob.rsCAN_VIEW \u00b7 CAN_MANAGE_RUN \u00b7 IS_OWNER \u00b7 CAN_MANAGEJobs only\n5secret_aclssecret.rsREAD \u00b7 WRITE \u00b7 MANAGESecret scope\n6workspace_permission_assignmentsworkspace_permission_assignment.rsUSER \u00b7 ADMINWorkspace membership\n\n\n\n1.2 \u2014 Bugs surfaced by audit\n\n\nSeverityBugWhere\nP0workspace_bindings PATCH does not enforce metastore admincrates/lyra-api/src/workspace_bindings.rs\nP0delta_commits ignores _user: AuthUser \u2014 anyone with valid token can write commitsdelta_commits.rs:46\nP0list_volumes / list_models / list_functions have no permission filter \u2014 full enumerationvolumes.rs, models.rs, functions.rs\nP0update_metastore / list_metastore / get_metastore rely on JWT-only, no metastore-owner checkmetastore.rs\nP0/P1grant.rs:25 SELECT references non-existent metastore_id column \u2014 broken since refactorrepositories/grant.rs:25\nP1Admin-bypass logic duplicated in ~12 handlers; semantics driftgovernance/permissions.rs, jobs_permissions.rs, \u2026\nP1List endpoints fetch-then-filter at app layer (N+1 ACL queries)multiple handlers\n\n\n\n\n2 \u00b7 Decision \u2014 why one grant table\n\n\nOptionVerdictRationale\n\u2460 Keep 6 separate ACL tablesREJECTAlready produced 6 P0 bugs; admin-bypass duplicated 12\u00d7; vocabularies don't compose\n\u2461 Single grant table over typed securable graphCHOSENOne resolver, one vocabulary, predicate pushdown for list endpoints, owner-as-column avoids grant-row maintenance, cascade-as-data avoids hardcoding\n\u2462 Embed Cedar/OPA as policy enginePARTIALAdopt Cedar grammar for the optional condition column \u2014 but the 4-pass core stays SQL because list-filter pushdown beats anything an external evaluator can do\n\n\n\n\n3 \u00b7 Engine comparison\n\n\nCriterionHand-written Rust + PostgresCedar (Rust crate)OPA / regorusCasbin-rsBiscuit\nSQL list-filter pushdown\u2713 native\u2717\u2717\u2717\u2717\nLatency for single-decision<1 ms (cached)<1 ms2-5 ms<1 ms<1 ms\nGroup nestingrecursive CTEflat / templatevia regoRBAC w/ domainflat\nAudit forensicsSQL nativemanualmanualmanualcryptographic\nTenancy isolationRLS + account_idmanualmanualmanualmanual\nTotal (out of 35)3222202119\n\n\n\n\n4 \u00b7 Peer cross-check\n\n\nSystemWhat we stealWhat we don't\nDatabricks UCPrivilege vocabulary, securable types, OWNER-as-column, BROWSE/USE/MANAGE/SELECT/MODIFY/EXECUTECentralised metadata-only stance \u2014 already match\nSnowflakeFUTURE GRANTS pattern (apply to objects not yet created), row access policies as attached objectsTwo-direction privilege graph \u2014 overkill for our scale\nBigQueryColumn-level ACL = mask UDF attached to columnIAM project hierarchy \u2014 Lyra's hierarchy is shallower\nApache PolarisCatalog/Namespace/Table primitive set; PrincipalRole / CatalogRole indirectionJCasbin-style RBAC-with-domains \u2014 we use direct grants\nIceberg RESTGranular table verbs (TABLE_ADD_SNAPSHOT, \u2026) \u2014 keep as reserved future\u2014\nAWS Lake FormationLF-Tags = tag-based ABAC at scale; Lyra's GovernedTag + ASSIGN\u2014\nCedarExpression grammar for the condition columnWhole-engine \u2014 slower than SQL pushdown for lists\n\n\n\n\n5 \u00b7 Complete asset inventory (47 active)\n\n\n5.1 \u2014 UC data plane (19)\n\n\nSecurableCodeTodayNotes\nMetastoremetastore.rsgrantstop of UC tree\nCatalogcatalogs.rsgrants\u2014\nSchemaschemas.rsgrants\u2014\nTabletables.rsgrants\u2014\nStreamingTable v3tables.rs (kind discriminator)noneUC 2026 first-class; ABAC GA targets it\nView \u00b7 MaterializedViewtables.rsgrants\u2014\nVolumevolumes.rsgrants\u2014\nFunctionfunctions.rsgrants\u2014\nProcedure v3functions.rs (kind=procedure)noneUC 2026; different EXECUTE semantics\nModel \u00b7 ModelVersionmodels.rsgrants\u2014\nStorageCredentialgovernance/storage_credentials.rsgrants\u2014\nExternalLocationgovernance/external_locations.rsgrants\u2014\nConnectiongovernance/connections.rsgrants\u2014\nServiceCredential v2federationno enumadd to SecurableType\nExternalMetadata v3(reserved)noneUC 2026 external lineage objects\nDatabaseInstance / LakebaseCatalog v3(reserved)nonePostgres-as-catalog\nGovernedTag v3(reserved)noneABAC anchor\nTagPolicy v3(reserved)nonepredicate-style: \"tag PII denies SELECT\"\n\n\n\n5.2 \u2014 Sharing & Marketplace (6)\n\n\nSecurableCodeTodayNotes\nSharegovernance/shares.rsgrants\u2014\nProvidergovernance/providers.rsgrants\u2014\nRecipientgovernance/recipients.rsgrantsalso a principal kind\nCleanRoom(reserved)grants\u2014\nMarketplaceListing v3(reserved)nonequota-counted; USE_MARKETPLACE_ASSETS\nExchange v3(reserved)noneprivate listing namespace\n\n\n\n5.3 \u2014 Workspace (8 + cascade-only children)\n\n\nSecurableCodeTodayNotes\nWorkspaceaccounts/workspaces.rsworkspace_permission_assignmentsfold\nWorkspaceCatalogBindingworkspace_bindings.rsnoneprivilege on Workspace \u00d7 Catalog\nNotebooknotebooks/mod.rsworkspace_object_aclsfold\nNotebookCell \u00b7 KernelSessionnotebooks/cells.rs \u00b7 session.rscascadeparent: Notebook\nFile \u00b7 Directoryworkspace_files.rsworkspace_object_aclspath-based ancestor\nDashboarddashboards/mod.rsworkspace_object_aclsfold\nDashboard childrenDataset \u00b7 Chat \u00b7 Message \u00b7 Attachment \u00b7 Schedule \u00b7 Refresh \u00b7 Soul \u00b7 SkillOverridecascadeparent: Dashboard\n\n\n\n5.4 \u2014 Compute (3)\n\n\nSecurableCodeTodayNotes\nClusterclusters.rscompute_cluster_aclsfold\nNetwork v2accounts/networks.rsnoneaccount-level\nMCPServer v3(reserved)noneAI-Gateway on-behalf-of\n\n\n\n5.5 \u2014 Orchestration (5)\n\n\nSecurableCodeTodayNotes\nJobjobs.rsjob_aclsfold\nJobRunjobs_dispatcher.rscascadeparent: Job; no SELECT cascade\nFlow v2orchestrator.rsnonedistinct from Job\nFlowRun \u00b7 FlowStepRunorchestratorcascadeparent: Flow\nPipeline (DLT) v3(reserved)none5 perm levels: VIEW/RUN/MANAGE/IS_OWNER\n\n\n\n5.6 \u2014 Identity (4)\n\n\nSecurableCodeTodayNotes\nUserscim/*.rsprincipal onlyalso securable \u2014 MANAGE_PAT \u00b7 MANAGE_USER\nGroupscim/*.rsprincipal onlyalso securable \u2014 MANAGE_GROUP\nServicePrincipalscim/account_scope.rsprincipal onlyalso securable \u2014 MANAGE_SP\nPAT(via identity)implicit ownercascade from User\n\n\n\n5.7 \u2014 Account & Secrets (2)\n\n\nSecurableCodeTodayNotes\nAccountaccounts/mod.rsimplicit admintop-level; MANAGE_ACCOUNT\nSecretScope \u00b7 Secretsecrets.rssecret_aclsfold; secret cascades from scope\n\n\n\n\n6 \u00b7 Object hierarchy tree\n\n\n\n\n  \nAccount\n    \n\n      \n\u21b3 Workspace (formerly workspace_permission_assignments)\n        \n\n          \n\u21b3 Cluster compute_cluster_acls \u2192 grant\n          \n\u21b3 Network [v2]\n          \n\u21b3 MCPServer [v3]\n          \n\u21b3 Job job_acls \u2192 grant\n            \n\n\u21b3 JobRun \u00b7 TaskRun (cascade \u00b7 no SELECT)\n          \n          \n\u21b3 Flow [v2]\n            \n\n\u21b3 FlowRun \u00b7 FlowStepRun (cascade)\n          \n          \n\u21b3 Pipeline (DLT) [v3]\n            \n\n\u21b3 PipelineUpdate \u00b7 PipelineEvent (cascade)\n          \n          \n\u21b3 Notebook workspace_object_acls \u2192 grant\n            \n\n\u21b3 NotebookCell \u00b7 KernelSession (cascade)\n          \n          \n\u21b3 Directory \u2192 File (path ancestor)\n          \n\u21b3 Dashboard\n            \n\n\u21b3 Dataset \u00b7 Chat \u00b7 Message \u00b7 Attachment \u00b7 Schedule \u00b7 Refresh \u00b7 Soul \u00b7 SkillOverride (cascade)\n          \n          \n\u21b3 User \u00b7 Group \u00b7 ServicePrincipal (also principals)\n            \n\n\u21b3 PAT (owner-only)\n          \n        \n      \n      \n\u21b3 Metastore grants\n        \n\n          \n\u21b3 Catalog\n            \n\n              \n\u21b3 Schema\n                \n\n                  \n\u21b3 Table \u00b7 StreamingTable \u00b7 View \u00b7 MaterializedView\n                  \n\u21b3 Volume \u00b7 Function \u00b7 Procedure \u00b7 Model\n                \n              \n            \n          \n          \n\u21b3 StorageCredential \u00b7 ExternalLocation\n          \n\u21b3 Connection\n            \n\n              \n\u21b3 ForeignCatalog \u00b7 ForeignSchema \u00b7 ForeignTable (cascade)\n              \n\u21b3 ServiceCredential [v2]\n            \n          \n          \n\u21b3 Share \u00b7 Provider \u00b7 Recipient \u00b7 CleanRoom\n          \n\u21b3 MarketplaceListing \u00b7 Exchange [v3]\n          \n\u21b3 ExternalMetadata [v3]\n          \n\u21b3 DatabaseInstance [v3]\n          \n\u21b3 GovernedTag \u00b7 TagPolicy [v3]\n          \n\u21b3 SecretScope \u2192 Secret secret_acls \u2192 grant\n        \n      \n      \n\u21b3 MetastoreAssignment (Workspace \u00d7 Metastore)\n      \n\u21b3 WorkspaceCatalogBinding (Workspace \u00d7 Catalog, mode=R/RW)\n    \n  \n\n\n\n\n\n7 \u00b7 Privilege vocabulary\n\n\n7.1 \u2014 Core verbs\n\n\nPrivilegeApplies toMaps from (legacy)\nBROWSEanyCAN_READ \u00b7 CAN_VIEW \u00b7 CAN_VIEW_METADATA\nSELECTTable \u00b7 View \u00b7 MV \u00b7 StreamingTable\u2014\nMODIFYTable \u00b7 Volume \u00b7 File \u00b7 Notebook \u00b7 QueryCAN_EDIT \u00b7 CAN_MANAGE_RUN(write)\nEXECUTEFunction \u00b7 Procedure \u00b7 Model \u00b7 Job \u00b7 Flow \u00b7 Pipeline \u00b7 Notebook \u00b7 ClusterCAN_RUN \u00b7 CAN_RESTART \u00b7 CAN_QUERY\nUSECluster(attach) \u00b7 Connection \u00b7 ServiceCredential \u00b7 ExternalLocationCAN_ATTACH_TO\nUSE_CATALOG \u00b7 USE_SCHEMACatalog \u00b7 Schema\u2014\nMANAGEanyCAN_MANAGE\n(owner column)anyIS_OWNER\n\n\n\n7.2 \u2014 Specialised verbs\n\n\nPrivilegeApplies toNotes\nREAD_VOLUME \u00b7 WRITE_VOLUMEVolume \u00b7 ExternalLocationbyte-level\nREAD_FILES \u00b7 WRITE_FILESExternalLocation \u00b7 Volume\u2014\nREFRESHMV \u00b7 StreamingTable \u00b7 Pipelinerefresh w/o full MANAGE\nEXTERNAL_USE_SCHEMA \u00b7 EXTERNAL_USE_LOCATIONSchema \u00b7 ExternalLocationfor OSS engines (Iceberg-REST); excluded from ALL_PRIVILEGES\nUSE_CONNECTION \u00b7 USE_PROVIDER \u00b7 USE_RECIPIENT \u00b7 USE_SHAREConnection \u00b7 Provider \u00b7 Recipient \u00b7 Sharefederation & sharing\nUSE_MARKETPLACE_ASSETSMarketplaceListing \u00b7 Exchangesubscribe / install\nASSIGN \u00b7 CREATE_TAG \u00b7 MANAGE_TAGGovernedTag \u00b7 TagPolicytag-based ABAC\nCREATE_EXTERNAL_METADATAMetastore \u00b7 ExternalMetadata\u2014\nCREATE_*parent containersCREATE_TABLE / CREATE_SCHEMA / CREATE_FUNCTION / CREATE_VOLUME / CREATE_MODEL / CREATE_MODEL_VERSION / CREATE_FOREIGN_CATALOG\nMANAGE_PAT \u00b7 MANAGE_USER \u00b7 MANAGE_GROUP \u00b7 MANAGE_SPUser \u00b7 Group \u00b7 ServicePrincipalSCIM admin\nMANAGE_ACCOUNTAccounttop-level admin; never auto-cascades to data plane (see \u00a717)\nREAD_BINDING \u00b7 WRITE_BINDINGWorkspaceCatalogBinding\u2014\nALL_PRIVILEGES macroanyexpands to all of above except EXTERNAL_USE_*, MANAGE_PAT, MANAGE_ACCOUNT \u2014 Databricks-spec compliant\n\n\n\n\n8 \u00b7 Cascade rules \u2014 data, not code\n\nStatic parent\u2192child privilege flow rules. Resolver pass-4 JOINs securable_cascade; adding a new asset = add rows here, no Rust change.\n\nINSERT INTO securable_cascade (parent_type, child_type, privileges) VALUES\n  -- Account \u2192 Workspace/Metastore\n  ('Account',     'Workspace',           ARRAY['MANAGE','BROWSE']),\n  ('Account',     'Metastore',           ARRAY['MANAGE','BROWSE']),\n  -- Workspace \u2192 compute / orchestration / collab\n  ('Workspace',   'Cluster',             ARRAY['MANAGE','BROWSE']),\n  ('Workspace',   'Network',             ARRAY['MANAGE']),\n  ('Workspace',   'MCPServer',           ARRAY['MANAGE','BROWSE']),\n  ('Workspace',   'Job',                 ARRAY['MANAGE','BROWSE']),\n  ('Workspace',   'Flow',                ARRAY['MANAGE','BROWSE']),\n  ('Workspace',   'Pipeline',            ARRAY['MANAGE','BROWSE']),\n  ('Workspace',   'Notebook',            ARRAY['MANAGE','BROWSE']),\n  ('Workspace',   'Dashboard',           ARRAY['MANAGE','BROWSE']),\n  ('Workspace',   'Directory',           ARRAY['MANAGE','BROWSE']),\n  -- File system + collab cascades\n  ('Directory',   'File',                ARRAY['MANAGE','BROWSE','MODIFY','SELECT']),\n  ('Directory',   'Directory',           ARRAY['MANAGE','BROWSE','MODIFY','SELECT']),\n  ('Notebook',    'NotebookCell',        ARRAY['MANAGE','BROWSE','MODIFY','EXECUTE']),\n  ('Notebook',    'KernelSession',       ARRAY['MANAGE','BROWSE','EXECUTE']),\n  ('Dashboard',   'DashboardChild',      ARRAY['MANAGE','BROWSE']),\n  -- Run-style children (NO SELECT \u2014 don't leak through logs)\n  ('Job',         'JobRun',              ARRAY['MANAGE','BROWSE','EXECUTE']),\n  ('Flow',        'FlowRun',             ARRAY['MANAGE','BROWSE','EXECUTE']),\n  ('Pipeline',    'PipelineUpdate',      ARRAY['MANAGE','BROWSE','EXECUTE']),\n  -- Metastore \u2192 resources\n  ('Metastore',   'Catalog',             ARRAY['MANAGE','BROWSE']),\n  ('Metastore',   'StorageCredential',   ARRAY['MANAGE','BROWSE']),\n  ('Metastore',   'ExternalLocation',    ARRAY['MANAGE','BROWSE']),\n  ('Metastore',   'Connection',          ARRAY['MANAGE','BROWSE']),\n  ('Metastore',   'Share',               ARRAY['MANAGE','BROWSE']),\n  ('Metastore',   'Provider',            ARRAY['MANAGE','BROWSE']),\n  ('Metastore',   'Recipient',           ARRAY['MANAGE','BROWSE']),\n  ('Metastore',   'CleanRoom',           ARRAY['MANAGE','BROWSE']),\n  ('Metastore',   'MarketplaceListing',  ARRAY['MANAGE','BROWSE']),\n  ('Metastore',   'Exchange',            ARRAY['MANAGE','BROWSE']),\n  ('Metastore',   'ExternalMetadata',    ARRAY['MANAGE','BROWSE']),\n  ('Metastore',   'DatabaseInstance',    ARRAY['MANAGE','BROWSE']),\n  ('Metastore',   'GovernedTag',         ARRAY['MANAGE','BROWSE','ASSIGN']),\n  ('Metastore',   'SecretScope',         ARRAY['MANAGE','BROWSE']),\n  -- Connection \u2192 foreign objects\n  ('Connection',  'ForeignCatalog',      ARRAY['USE_CATALOG','BROWSE']),\n  ('Connection',  'ServiceCredential',   ARRAY['USE','MANAGE']),\n  -- UC tree\n  ('Catalog',     'Schema',              ARRAY['USE_CATALOG','BROWSE','MANAGE']),\n  ('Schema',      'Table',               ARRAY['USE_SCHEMA','BROWSE','SELECT','MODIFY','MANAGE']),\n  ('Schema',      'StreamingTable',      ARRAY['USE_SCHEMA','BROWSE','SELECT','REFRESH','MANAGE']),\n  ('Schema',      'View',                ARRAY['USE_SCHEMA','BROWSE','SELECT','MANAGE']),\n  ('Schema',      'MaterializedView',    ARRAY['USE_SCHEMA','BROWSE','SELECT','REFRESH','MANAGE']),\n  ('Schema',      'Volume',              ARRAY['USE_SCHEMA','BROWSE','READ_VOLUME','WRITE_VOLUME','MANAGE']),\n  ('Schema',      'Function',            ARRAY['USE_SCHEMA','BROWSE','EXECUTE','MANAGE']),\n  ('Schema',      'Procedure',           ARRAY['USE_SCHEMA','BROWSE','EXECUTE','MANAGE']),\n  ('Schema',      'Model',               ARRAY['USE_SCHEMA','BROWSE','EXECUTE','MANAGE']),\n  -- Identity\n  ('User',        'PAT',                 ARRAY['MANAGE_PAT']),\n  -- Secrets\n  ('SecretScope', 'Secret',              ARRAY['BROWSE','SELECT','MODIFY','MANAGE']);\n\n\n\n9 \u00b7 DDL \u2014 paste-ready\n\n\n9.1 \u2014 Principals (with FederatedIdentity)\n\nCREATE TYPE principal_kind AS ENUM ('User','Group','ServicePrincipal','Recipient','FederatedIdentity');\n\nCREATE TABLE principal (\n    id                  UUID PRIMARY KEY DEFAULT gen_random_uuid(),\n    account_id          UUID NOT NULL,\n    kind                principal_kind NOT NULL,\n    name                TEXT NOT NULL,\n    -- federated identity columns (NULL for native principals)\n    federated_issuer    TEXT,\n    federated_subject   TEXT,\n    federated_audience  TEXT,\n    attrs               JSONB NOT NULL DEFAULT '{}',  -- for ABAC\n    created_at          TIMESTAMPTZ NOT NULL DEFAULT now(),\n    UNIQUE (account_id, kind, name)\n);\n\nCREATE UNIQUE INDEX principal_federated_unique\n  ON principal (federated_issuer, federated_subject, federated_audience)\n  WHERE kind = 'FederatedIdentity';\n\nCREATE TABLE principal_membership (\n    parent_id   UUID NOT NULL REFERENCES principal(id) ON DELETE CASCADE,\n    member_id   UUID NOT NULL REFERENCES principal(id) ON DELETE CASCADE,\n    account_id  UUID NOT NULL,\n    PRIMARY KEY (parent_id, member_id),\n    CHECK (parent_id != member_id)\n);\nCREATE INDEX principal_membership_member ON principal_membership (member_id);\n-- depth \u2264 3 enforced at write time\n\n\n9.2 \u2014 Securables (typed graph)\n\nCREATE TABLE securable (\n    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),\n    account_id      UUID NOT NULL,\n    type            TEXT NOT NULL,                       -- enum widened (see \u00a75)\n    parent_id       UUID REFERENCES securable(id) ON DELETE CASCADE,\n    name            TEXT NOT NULL,\n    full_name       TEXT NOT NULL,                       -- e.g. \"main.sales.orders\"\n    owner_id        UUID NOT NULL REFERENCES principal(id),\n    pending_owner_id UUID REFERENCES principal(id),      -- two-phase transfer\n    pending_since   TIMESTAMPTZ,\n    metadata        JSONB NOT NULL DEFAULT '{}',\n    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),\n    UNIQUE (account_id, type, full_name)\n);\nCREATE INDEX securable_parent ON securable (parent_id);\nCREATE INDEX securable_account_type ON securable (account_id, type);\n\n\n9.3 \u2014 Grants (with valid_from / revoked_at / Cedar condition)\n\nCREATE TABLE grant (\n    id                  UUID PRIMARY KEY DEFAULT gen_random_uuid(),\n    account_id          UUID NOT NULL,\n    principal_id        UUID NOT NULL REFERENCES principal(id) ON DELETE CASCADE,\n    securable_id        UUID NOT NULL REFERENCES securable(id) ON DELETE CASCADE,\n    privilege           TEXT NOT NULL,\n    effect              TEXT NOT NULL DEFAULT 'ALLOW' CHECK (effect IN ('ALLOW','DENY')),\n    granted_by          UUID NOT NULL REFERENCES principal(id),\n    granted_at          TIMESTAMPTZ NOT NULL DEFAULT now(),\n    valid_from          TIMESTAMPTZ NOT NULL DEFAULT now(),\n    expires_at          TIMESTAMPTZ,\n    revoked_at          TIMESTAMPTZ,\n    revoked_by          UUID REFERENCES principal(id),\n    condition_cedar     TEXT,                            -- optional Cedar fragment\n    UNIQUE (account_id, principal_id, securable_id, privilege, effect)\n);\n\nCREATE INDEX grant_live_allow_idx ON grant (account_id, principal_id, securable_id)\n  WHERE effect = 'ALLOW'\n    AND revoked_at IS NULL\n    AND valid_from <= now()\n    AND (expires_at IS NULL OR expires_at > now());\n\n-- Forbid pass needs a separate index:\nCREATE INDEX grant_live_deny_idx ON grant (account_id, principal_id, securable_id)\n  WHERE effect = 'DENY' AND revoked_at IS NULL;\n\n\n9.4 \u2014 Cascade rules table\n\nCREATE TABLE securable_cascade (\n    parent_type   TEXT NOT NULL,\n    child_type    TEXT NOT NULL,\n    privileges    TEXT[] NOT NULL,\n    PRIMARY KEY (parent_type, child_type)\n);\n-- Seed rows: see \u00a78\n\n\n9.5 \u2014 Row filters & column masks (attached, not granted)\n\nCREATE TABLE row_filter_attachment (\n    table_id      UUID PRIMARY KEY REFERENCES securable(id) ON DELETE CASCADE,\n    function_id   UUID NOT NULL REFERENCES securable(id) ON DELETE RESTRICT,\n    args          TEXT[] NOT NULL DEFAULT '{}'\n);\n\nCREATE TABLE column_mask_attachment (\n    table_id      UUID NOT NULL REFERENCES securable(id) ON DELETE CASCADE,\n    column_name   TEXT NOT NULL,\n    function_id   UUID NOT NULL REFERENCES securable(id) ON DELETE RESTRICT,\n    args          TEXT[] NOT NULL DEFAULT '{}',\n    PRIMARY KEY (table_id, column_name)\n);\n-- Applied at query rewrite (DuckDB/Spark/Trino client).\n-- If user has SELECT on table but cannot EXECUTE the mask UDF \u2192 fail-closed.\n\n\n9.6 \u2014 Audit (hash-chained, SOC2/HIPAA-grade)\n\nCREATE TABLE audit_event (\n    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),\n    account_id      UUID NOT NULL,\n    request_id      UUID NOT NULL,\n    happened_at     TIMESTAMPTZ NOT NULL DEFAULT now(),\n    actor_id        UUID NOT NULL,\n    actor_session   UUID,                                -- on-behalf-of chain id\n    source_ip       INET,\n    user_agent      TEXT,\n    action          TEXT NOT NULL,                       -- GRANT/REVOKE/READ/DECISION_DENY/...\n    securable_type  TEXT NOT NULL,\n    securable_id    UUID,\n    privilege       TEXT,\n    decision        TEXT NOT NULL CHECK (decision IN ('ALLOW','DENY','ERROR')),\n    rationale       TEXT NOT NULL,                       -- \"owner pass\" / \"deny@ancestor X\"\n    before_state    JSONB,\n    after_state     JSONB,\n    prev_hash       BYTEA,\n    self_hash       BYTEA NOT NULL                       -- sha256(prev_hash || canonical_json)\n);\nCREATE INDEX audit_event_account_time ON audit_event (account_id, happened_at DESC);\nCREATE INDEX audit_event_actor       ON audit_event (account_id, actor_id, happened_at DESC);\nCREATE INDEX audit_event_securable   ON audit_event (account_id, securable_id, happened_at DESC);\n\n\n9.7 \u2014 ACL versioning (cache invalidation)\n\nCREATE TABLE acl_version (\n    id                      INT PRIMARY KEY DEFAULT 1 CHECK (id = 1),\n    grants_version          BIGINT NOT NULL DEFAULT 0,\n    membership_version      BIGINT NOT NULL DEFAULT 0,\n    ownership_version       BIGINT NOT NULL DEFAULT 0,\n    cascade_version         BIGINT NOT NULL DEFAULT 0\n);\nINSERT INTO acl_version DEFAULT VALUES;\n\n-- Bump triggers (one per table)\nCREATE FUNCTION bump_grants_version() RETURNS TRIGGER AS $$\nBEGIN\n  UPDATE acl_version SET grants_version = grants_version + 1 WHERE id = 1;\n  RETURN NEW;\nEND $$ LANGUAGE plpgsql;\nCREATE TRIGGER grant_version_bump AFTER INSERT OR UPDATE OR DELETE ON grant\n  FOR EACH STATEMENT EXECUTE FUNCTION bump_grants_version();\n-- analogous triggers for principal_membership, securable.owner_id, securable_cascade\n\n\n\n10 \u00b7 Resolver \u2014 the 4 passes\n\nDefault-deny. Order-independent. Cedar-style. Single SQL CTE returns booleans for all four passes; final decision = NOT forbid AND (admin OR owner OR permit).\n\nWITH RECURSIVE\neffective_principals AS (              -- pass 0: user + transitive groups\n    SELECT $user_id::uuid AS id\n  UNION\n    SELECT pm.parent_id FROM principal_membership pm\n      JOIN effective_principals ep ON pm.member_id = ep.id\n),\nancestors AS (                         -- self + parent chain\n    SELECT $sec_id::uuid AS id, 0 AS depth\n  UNION ALL\n    SELECT s.parent_id, a.depth + 1\n      FROM securable s JOIN ancestors a ON s.id = a.id\n     WHERE s.parent_id IS NOT NULL\n)\nSELECT\n  -- PASS 1 \u2014 explicit DENY anywhere on the ancestor chain?\n  EXISTS (SELECT 1 FROM grant g\n            JOIN ancestors a ON g.securable_id = a.id\n            JOIN effective_principals ep ON g.principal_id = ep.id\n           WHERE g.account_id = $acct\n             AND g.privilege = $priv AND g.effect = 'DENY'\n             AND g.revoked_at IS NULL) AS forbid,\n\n  -- PASS 2 \u2014 account/workspace admin?\n  EXISTS (SELECT 1 FROM grant g\n            JOIN effective_principals ep ON g.principal_id = ep.id\n           WHERE g.account_id = $acct\n             AND g.privilege = 'MANAGE_ACCOUNT'\n             AND g.effect = 'ALLOW'\n             AND g.revoked_at IS NULL) AS admin,\n\n  -- PASS 3 \u2014 securable owner (column, not grant row)?\n  EXISTS (SELECT 1 FROM securable s\n            JOIN effective_principals ep\n              ON ep.id IN (s.owner_id, s.pending_owner_id)\n           WHERE s.id = $sec_id) AS owner,\n\n  -- PASS 4 \u2014 explicit ALLOW (with cascade-rule check)?\n  EXISTS (SELECT 1 FROM grant g\n            JOIN ancestors a ON g.securable_id = a.id\n            JOIN effective_principals ep ON g.principal_id = ep.id\n            JOIN securable child  ON child.id  = $sec_id\n            JOIN securable parent ON parent.id = a.id\n       LEFT JOIN securable_cascade c\n              ON c.parent_type = parent.type AND c.child_type = child.type\n           WHERE g.account_id = $acct\n             AND g.privilege = $priv\n             AND g.effect    = 'ALLOW'\n             AND g.revoked_at IS NULL\n             AND g.valid_from <= now()\n             AND (g.expires_at IS NULL OR g.expires_at > now())\n             AND (a.depth = 0 OR $priv = ANY(c.privileges))) AS permit;\n-- decision = NOT forbid AND (admin OR owner OR permit)\n-- if condition_cedar IS NOT NULL \u2192 evaluate Cedar fragment with (principal,resource,context)\n-- \u2192 on parse/eval error: deny.\n\n\n\n11 \u00b7 List-endpoint predicate pushdown\n\nSingle-decision uses the CTE above. List endpoints use a materialized view to avoid N+1.\n\nCREATE MATERIALIZED VIEW effective_grants_mv AS\nWITH RECURSIVE eff_p AS (\n    SELECT p.id AS principal_id, p.id AS effective_id, p.account_id\n      FROM principal p WHERE p.kind IN ('User','ServicePrincipal','FederatedIdentity')\n  UNION\n    SELECT ep.principal_id, pm.parent_id, ep.account_id\n      FROM principal_membership pm\n      JOIN eff_p ep ON pm.member_id = ep.effective_id\n)\nSELECT\n    g.account_id, ep.principal_id, g.securable_id, g.privilege, g.effect\nFROM grant g\nJOIN eff_p ep ON g.principal_id = ep.effective_id\nWHERE g.revoked_at IS NULL\n  AND g.valid_from <= now()\n  AND (g.expires_at IS NULL OR g.expires_at > now());\n\nCREATE UNIQUE INDEX effective_grants_mv_pk\n  ON effective_grants_mv (account_id, principal_id, securable_id, privilege, effect);\n\n-- list-filter pattern\nSELECT s.* FROM securable s\nWHERE s.account_id = $acct AND s.type = 'Catalog'\n  AND EXISTS (\n    SELECT 1 FROM effective_grants_mv eg\n    WHERE eg.account_id   = $acct\n      AND eg.principal_id = $user\n      AND eg.securable_id IN (SELECT id FROM ancestors_of(s.id))\n      AND eg.privilege IN ('BROWSE','USE_CATALOG','MANAGE','MANAGE_ACCOUNT')\n      AND eg.effect = 'ALLOW'\n  )\n  AND NOT EXISTS (\n    SELECT 1 FROM effective_grants_mv eg\n    WHERE eg.account_id = $acct AND eg.principal_id = $user\n      AND eg.securable_id IN (SELECT id FROM ancestors_of(s.id))\n      AND eg.effect = 'DENY'\n  );\n\n-- REFRESH MATERIALIZED VIEW CONCURRENTLY effective_grants_mv;\n-- triggered by acl_version bump (debounced 500ms).\n\n\n\n12 \u00b7 Row filters & column masks\n\nSnowflake / BigQuery / Lake Formation all attach filters/masks to the table-or-column and apply at query rewrite. v3 follows that. Filters/masks themselves are SQL UDFs registered as Function securables \u2014 so the EXECUTE check on the UDF naturally enforces who can apply them.\n\n-- Attachment lookup at query plan time:\nSELECT rf.function_id, rf.args FROM row_filter_attachment rf WHERE rf.table_id = $t;\nSELECT cm.column_name, cm.function_id, cm.args FROM column_mask_attachment cm WHERE cm.table_id = $t;\n\n-- Rewrite (pseudo):\n--   SELECT a, b, c FROM t\n-- becomes\n--   SELECT mask_a(a) AS a, b, mask_c(c) AS c\n--   FROM t\n--   WHERE row_filter(b, c)\n-- The UDF evaluation invokes the resolver (EXECUTE on the function securable).\n-- If denied \u2192 deny the whole SELECT (fail-closed).\n\n\n\n13 \u00b7 Conditions \u2014 Cedar grammar\n\n-- The grant.condition_cedar column accepts a Cedar policy fragment.\n-- Examples:\nwhen { context.source_ip in ip(\"10.0.0.0/8\") }\nwhen { context.now < datetime(\"2026-12-31T00:00:00Z\") }\nwhen { principal has \"department\" && principal.department == \"finance\" }\nwhen { resource has \"tag.PII\" && resource.tag.PII == \"true\" implies\n       principal has \"training.hipaa\" && principal.training.hipaa == \"current\" }\n\n-- Resolver flow:\n-- 1. Pass-4 produces ALLOW (no condition) \u2192 check passes.\n-- 2. If condition_cedar IS NOT NULL \u2192 parse + evaluate with\n--    (principal_attrs, resource_attrs, context).\n-- 3. Parse error / eval error \u2192 DENY (fail-closed).\n-- Cedar is total + decidable, so eval is bounded.\n\n\n\n14 \u00b7 Audit invariants (SOC2 / HIPAA)\n\n\nFieldWhy\nrequest_idcorrelate w/ HTTP layer + downstream calls\nactor_id + actor_sessiondistinguish direct call vs on-behalf-of chain\nsource_ip + user_agentOWASP ASVS V7 minimum\nbefore_state + after_state (JSONB)policy snapshot for grant mutations \u2014 forensic replay\nrationalewhich pass matched: \"owner column\" / \"deny @ ancestor X\" / \"permit via group Y\"\nprev_hash + self_hashappend-only hash chain; integrity verified by periodic job\n\n\n\n\n15 \u00b7 Federated identity\n\nBeyond User/Group/SP, modern workloads carry tokens from external IDPs (GitHub Actions, Azure WIF, GCP WIF, SPIFFE). v3 models these as a principal kind with stable (issuer, sub, aud) identity.\n\n-- JWT validation path (RFC 8693 token-exchange):\n-- 1. Verify signature against issuer JWKS.\n-- 2. Extract (iss, sub, aud).\n-- 3. SELECT id FROM principal\n--    WHERE kind='FederatedIdentity'\n--      AND federated_issuer=$iss AND federated_subject=$sub AND federated_audience=$aud;\n-- 4. If not found and issuer is allow-listed \u2192 JIT-create principal row.\n-- 5. Resolver runs against returned principal_id as if it were a User.\n-- 6. audit_event.actor_session captures the token-exchange chain id.\n\n-- Policy: chain depth \u2264 3 (rejects multi-hop OBO laundering).\n\n\n\n16 \u00b7 Multi-tenant isolation\n\n-- account_id on EVERY core ACL row, even when redundant w/ FK chain.\nALTER TABLE securable             ADD COLUMN account_id UUID NOT NULL;\nALTER TABLE grant                 ADD COLUMN account_id UUID NOT NULL;\nALTER TABLE principal             ADD COLUMN account_id UUID NOT NULL;\nALTER TABLE principal_membership  ADD COLUMN account_id UUID NOT NULL;\nALTER TABLE audit_event           ADD COLUMN account_id UUID NOT NULL;\n\n-- Postgres RLS \u2014 defence in depth\nALTER TABLE grant ENABLE ROW LEVEL SECURITY;\nCREATE POLICY grant_tenant_isolation ON grant\n  USING (account_id = current_setting('app.account_id')::uuid);\n\n-- Cache key includes account_id:\n-- decision_cache: (account_id, principal_id, securable_id, privilege, grants_version)\n-- list_cache:     (account_id, principal_id, securable_type, grants_version)\n\n\n\n17 \u00b7 Anti-escalation rules\n\n\nRuleWhyEnforcement\nGrantor-holds-privilege (Postgres semantics)Stop \"MANAGE on Catalog\" laundering into SELECT on every TableMutating handler: has(actor, target, MANAGE) AND has(actor, target, privilege_being_granted) before INSERT\nTwo-phase ownership transferAvoid owner-orphaningPhase 1: prior owner sets pending_owner_id. Phase 2: new owner accepts within 7 days, else revert. During pending: both have MANAGE.\nDENY walks full ancestor chainCedar CVE-2024-25624 classPass-1 SELECT JOINs ancestors CTE, not direct grants only\nNo self-grant of MANAGE_ACCOUNTCasbin CVE-2023-26485 classBEFORE INSERT trigger rejects actor_id = principal_id AND privilege = 'MANAGE_ACCOUNT'\nToken-exchange chain depth \u2264 3OBO chains hide intentactor_session.depth tracked; reject >3 hops\nMANAGE_ACCOUNT does NOT auto-cascade to data planeAccount admins shouldn't silently see all dataPass-2 (admin) controls control-plane only; data-plane reads still need explicit BROWSE/SELECT or owner\nALL_PRIVILEGES excludes EXTERNAL_USE_*, MANAGE_PAT, MANAGE_ACCOUNTDatabricks-spec complianceMacro expansion at grant write-time\n\n\n\n\n18 \u00b7 Cache architecture\n\n\nCacheKeyInvalidated byHit rate target\nDecision(account_id, principal_id, securable_id, privilege, grants_version, ownership_version, cascade_version)any version bump>95%\nGroup closure(account_id, principal_id, membership_version)membership_version bump>99%\nList filter(account_id, principal_id, securable_type, grants_version)grants_version bump>90%\nMV refresh\u2014500ms debounce after grants_version bump\u2014\n\n\nmoka-rs with TTI (time-to-idle) of 5 min, max 100k entries per cache. Version columns are bumped by triggers so cache reads never need a separate timestamp lookup.\n\n\n\n19 \u00b7 Future assets \u2014 pre-planned\n\nLock the shape now so adding the feature later is a 1-line enum widen + 1 cascade row. No retro migration ever.\n\n\n19.1 \u2014 AI / ML\n\n\nAssetParentPrivilegesCascade\nVectorIndexSchemaSELECT \u00b7 MODIFY \u00b7 MANAGE \u00b7 BROWSEUSE_SCHEMA \u00b7 BROWSE \u00b7 MANAGE\nServingEndpointWorkspaceEXECUTE (CAN_QUERY) \u00b7 MANAGE \u00b7 BROWSEMANAGE \u00b7 BROWSE\nFeatureTableSchemaSELECT \u00b7 MODIFY \u00b7 MANAGE \u00b7 BROWSEUSE_SCHEMA \u00b7 BROWSE \u00b7 MANAGE\nOnlineTableSchemaSELECT \u00b7 MANAGE \u00b7 BROWSEUSE_SCHEMA \u00b7 BROWSE \u00b7 MANAGE\nInferenceLogServingEndpointBROWSE \u00b7 MANAGEBROWSE \u00b7 MANAGE\nAIGatewayWorkspaceUSE \u00b7 MANAGE \u00b7 BROWSEMANAGE\n\n\n\n19.2 \u2014 SQL / Compute\n\n\nAssetParentPrivileges\nSQLWarehouseWorkspaceUSE \u00b7 EXECUTE \u00b7 MANAGE \u00b7 BROWSE\nInstancePool \u00b7 ClusterPolicyWorkspaceUSE \u00b7 MANAGE \u00b7 BROWSE\nQuery \u00b7 Alert \u00b7 QueryHistoryWorkspaceSELECT/run \u00b7 MODIFY \u00b7 MANAGE \u00b7 BROWSE\n\n\n\n19.3 \u2014 Workspace / Collaboration\n\n\nAssetParentPrivileges\nRepoWorkspaceSELECT \u00b7 MODIFY \u00b7 EXECUTE \u00b7 MANAGE \u00b7 BROWSE\nApp (Lakehouse Apps) \u00b7 GenieSpaceWorkspaceUSE \u00b7 EXECUTE \u00b7 MANAGE \u00b7 BROWSE\nComment / Discussion (polymorphic)anyBROWSE \u00b7 MODIFY (own) \u00b7 MANAGE\n\n\n\n19.4 \u2014 Account / Platform\n\n\nAssetParentPrivileges\nBudget \u00b7 PrivateAccessSettings \u00b7 NetworkConnectivityConfigAccountBROWSE \u00b7 MANAGE\nIPAccessListWorkspaceBROWSE \u00b7 MANAGE\nOAuthApp \u00b7 EnterpriseAppAccountUSE \u00b7 MANAGE \u00b7 BROWSE\nAuditLogConfigAccountBROWSE \u00b7 MANAGE\nSystemSchemaMetastoreUSE_CATALOG \u00b7 USE_SCHEMA \u00b7 SELECT (admin-only by default)\n\n\n\n19.5 \u2014 Governance / Quality\n\n\nAssetParentPrivileges\nTag \u00b7 TagPolicyMetastoreBROWSE \u00b7 MANAGE \u00b7 ASSIGN\nLineage\u2014derived; visibility = AND of upstream/downstream BROWSE\nMonitorTableBROWSE \u00b7 MANAGE\nRowFilter \u00b7 ColumnMaskFunction (attached, not granted)USE \u00b7 MANAGE\n\n\n\n19.6 \u2014 Hard rules for adding any future asset\n\n\n#Rule\n1New feature = 1 enum value + 1 cascade row + parent FK on row. Never a new ACL table.\n2Owner = column on the resource table, never a grant row.\n3Reuse BROWSE/SELECT/MODIFY/EXECUTE/USE/MANAGE before inventing new verbs.\n4Empty cascade must be documented (admin-only / privacy).\n5Run-style children inherit BROWSE+EXECUTE+MANAGE only \u2014 never SELECT (don't leak through logs).\n6Polymorphic children (Comment, Tag) \u2192 parent_securable_id + JOIN securable_cascade at query time.\n7Reserve enum value at spec time, not code time.\n8Every new privilege documents whether ALL_PRIVILEGES includes it.\n\n\n\n\n20 \u00b7 Migration plan\n\n\n#PhaseStepReversible?\n1SchemaCreate new tables; widen SecurableType enum to 47 values + reservedyes (DROP)\n2Backfill UCCopy grants rows; add owner_id column on every UC tableyes\n3Backfill workspaceworkspace_object_acls + job_acls + compute_cluster_acls + secret_acls \u2192 grant rows w/ vocab translationyes\n4Backfill accountworkspace_permission_assignments \u2192 grants on Workspace securableyes\n5Materialize new securablesInsert rows in securable for Account, Workspace, Network, Flow, Pipeline, MCPServer, User, Group, SP, PAT, Binding, etc.yes\n6Cascade rulesSeed securable_cascade tableyes\n7Dual-writeMutating handlers write old + new; reads still old. 1 release.yes\n8Read cutoverReads from new; old tables read-only for 1 releaseyes (flip flag)\n9Drop legacyRemove 5 ACL tables + workspace_permission_assignmentsno \u2014 gate behind 2 stable releases\n10Enable RLS + CedarTurn on Postgres RLS on grant/securable; deploy Cedar evaluatoryes\n11MV pushdownMaterialize effective_grants_mv; flip list endpoints to use ityes\n\n\n\n\n21 \u00b7 Anti-patterns (postmortem-derived)\n\n\n#Anti-patternWhy it bitesMitigation\n1Hardcoded admin bypass in handlersDrift across 12 handlers; Lyra has 6 different bypass codepaths todaySingle resolver; admin = pass-2 only\n2Owner stored as grant rowLost grant = lost ownership; can't transfer atomicallyOwner = column on securable\n3Cascade hardcoded in RustNew asset requires Rust diff + recompilesecurable_cascade table\n4Group nesting via closure tableUpdates O(n\u00b2) on deep changes; Snowflake postmortemEdge list + recursive CTE, depth \u2264 3\n5Free-form JSON conditionsEval becomes Turing-complete; perf cliffCedar grammar (total, decidable)\n6List endpoints fetch-then-filterN+1 ACL queries; hidden 100ms latency on dashboardsMV + SQL pushdown\n7Audit log without before/afterSOC2 finding; can't replay incidentJSONB snapshots + hash chain\n8account_id implicit in FK chainSingle forgotten join = cross-tenant leakExplicit column + RLS\n9MANAGE auto-grants child SELECTQuiet escalationGrantor-holds-privilege rule\n10Owner transfer is single-phasePrior owner orphaned mid-transferTwo-phase w/ pending_owner_id\n\n\n\n\n22 \u00b7 Sources\n\n\n  \nDatabricks UC \u2014 securable objects (2026)\n  \nUC privileges reference\n  \nDLT / Lakeflow Pipeline ACLs\n  \nExternal metadata / lineage\n  \nGoverned tag permissions\n  \nMarketplace listings\n  \nLakebase UC registration\n  \nUnity AI Gateway / MCP\n  \nSnowflake row access policies\n  \nBigQuery column data masking\n  \nAWS Lake Formation LF-TBAC\n  \nApache Polaris RBAC\n  \nCedar policy language\n  \nRFC 8693 \u2014 OAuth Token Exchange\n  \nDelta Sharing protocol\n\n\n\n\n\n", "creation_timestamp": "2026-05-10T02:48:32.000000Z"}, {"uuid": "54c9e443-aa64-4189-9657-305ac59044fd", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-25624", "type": "seen", "source": "https://gist.github.com/khoindq/f145db4952e1aa3644b83c964afbf97a", "content": "\n\n\n\nData + AI Platform \u2014 Foundation Spec\n\n:root {\n  --bg:#0b0d10; --panel:#12161b; --panel2:#0f1318; --border:#1f262e; --ink:#e6edf3;\n  --mute:#8b949e; --accent:#58a6ff; --warn:#f0883e; --err:#f85149; --ok:#3fb950;\n  --pick:#d2a8ff; --line:#30363d; --gold:#e3b341;\n  --mono:ui-monospace,SFMono-Regular,\"SF Mono\",Menlo,Consolas,monospace;\n}\n*{box-sizing:border-box}\nbody{background:var(--bg);color:var(--ink);font:14px/1.55 -apple-system,system-ui,sans-serif;margin:0;padding:0}\n.layout{display:grid;grid-template-columns:240px 1fr;min-height:100vh}\nnav{position:sticky;top:0;align-self:start;height:100vh;overflow-y:auto;border-right:1px solid var(--border);background:var(--panel2);padding:16px 12px}\nnav h4{font-size:11px;color:var(--mute);text-transform:uppercase;letter-spacing:.06em;margin:14px 4px 4px}\nnav a{display:block;color:var(--ink);text-decoration:none;font-size:12px;padding:3px 6px;border-radius:4px;line-height:1.4}\nnav a:hover{background:#161b22;color:var(--accent)}\nnav a.h2{color:var(--accent);font-weight:600}\n.wrap{max-width:1080px;margin:0 auto;padding:32px 28px 96px}\nh1{font-size:28px;margin:0 0 6px}\nh2{font-size:20px;margin:36px 0 12px;padding-bottom:6px;border-bottom:1px solid var(--line);scroll-margin-top:12px}\nh3{font-size:15px;margin:22px 0 8px;color:var(--accent);scroll-margin-top:12px}\nh4{font-size:13px;margin:14px 0 6px;color:var(--pick);text-transform:uppercase;letter-spacing:.04em}\n.sub{color:var(--mute);margin:0 0 18px;font-size:13px}\n.lead{font-size:15px;line-height:1.6}\n.panel{background:var(--panel);border:1px solid var(--border);border-radius:8px;padding:14px 16px;margin:12px 0}\n.stat{display:grid;grid-template-columns:repeat(4,1fr);gap:10px;margin:14px 0 22px}\n.stat .card{background:var(--panel);border:1px solid var(--border);border-radius:8px;padding:12px}\n.stat .n{font-size:24px;font-weight:600;color:var(--accent)}\n.stat .l{font-size:10.5px;color:var(--mute);text-transform:uppercase;letter-spacing:.05em}\ntable{border-collapse:collapse;width:100%;margin:8px 0 14px;font-size:12.5px}\nth,td{border:1px solid var(--line);padding:6px 9px;text-align:left;vertical-align:top}\nth{background:#161b22;font-weight:600;color:var(--mute);font-size:11.5px;text-transform:uppercase;letter-spacing:.04em}\ntr:hover td{background:#0f1318}\ncode{font-family:var(--mono);background:#161b22;padding:1px 5px;border-radius:3px;font-size:11.5px}\npre{background:#0d1117;border:1px solid var(--border);border-radius:6px;padding:12px;overflow-x:auto;font:11.5px/1.55 var(--mono);max-height:540px}\n.ok{color:var(--ok)} .warn{color:var(--warn)} .err{color:var(--err)} .mute{color:var(--mute)} .pick{color:var(--pick);font-weight:600} .gold{color:var(--gold)}\n.pill{display:inline-block;padding:1px 7px;border-radius:10px;font-size:10.5px;border:1px solid var(--line);background:#161b22;color:var(--mute)}\n.pill.g{color:var(--ok);border-color:#1e3a2a;background:#0d1f15}\n.pill.r{color:var(--err);border-color:#3a1d1d;background:#1f0e0e}\n.pill.y{color:var(--warn);border-color:#3a2c1d;background:#1f1810}\n.pill.b{color:var(--accent);border-color:#1d3050;background:#0e1a2b}\n.pill.v{color:var(--pick);border-color:#3a2848;background:#1a1124}\n.pill.gold{color:var(--gold);border-color:#3a2f1d;background:#1f1a0e}\n.tree{font-family:var(--mono);font-size:11.8px;line-height:1.65;padding:14px 18px;background:var(--panel);border:1px solid var(--border);border-radius:6px;overflow-x:auto}\n.tree ul{list-style:none;padding-left:18px;margin:0;border-left:1px dashed var(--line)}\n.tree>ul{padding-left:0;border-left:none}\n.tree li{position:relative;padding:1px 0 1px 14px}\n.tree li::before{content:\"\u2500\";position:absolute;left:0;color:var(--line)}\n.tree .root{color:var(--pick);font-weight:600}\n.tree .acc{color:var(--accent)}\n.tree .ai{color:var(--gold)}\n.tree small{color:var(--mute);margin-left:6px;font-size:10.5px}\n.grid2{display:grid;grid-template-columns:1fr 1fr;gap:14px}\n.callout{border-left:3px solid var(--accent);padding:10px 14px;background:rgba(88,166,255,.05);margin:12px 0;border-radius:4px}\n.callout.warn{border-left-color:var(--warn);background:rgba(240,136,62,.05)}\n.callout.err{border-left-color:var(--err);background:rgba(248,81,73,.05)}\n.callout.gold{border-left-color:var(--gold);background:rgba(227,179,65,.05)}\nhr.sep{border:0;border-top:1px solid var(--line);margin:30px 0}\n.col2{column-count:2;column-gap:24px}\n.col2 ul{margin:0;padding-left:18px}\n@media(max-width:1100px){.layout{grid-template-columns:1fr}nav{display:none}}\n\n\n\n\n\n\n\n\nFoundation Spec\nTL;DR\n\nI \u00b7 Foundation\n1 \u00b7 North-star principles\n2 \u00b7 Architecture\n3 \u00b7 Tenancy model\n4 \u00b7 Object hierarchy\n\nII \u00b7 Identity & Account\n5 \u00b7 Account plane\n6 \u00b7 Identity\n7 \u00b7 Network & security\n\nIII \u00b7 Data Plane\n8 \u00b7 Unity Catalog\n9 \u00b7 Tables & views\n10 \u00b7 Storage & volumes\n11 \u00b7 Functions, models, masks\n12 \u00b7 Sharing & marketplace\n13 \u00b7 Federation\n\nIV \u00b7 Compute\n14 \u00b7 Compute objects\n15 \u00b7 SQL warehouses\n16 \u00b7 Orchestration\n\nV \u00b7 Workspace\n17 \u00b7 Workspace assets\n18 \u00b7 Dashboards & queries\n19 \u00b7 Secrets\n\nVI \u00b7 AI / ML / Agents\n20 \u00b7 ML platform\n21 \u00b7 Model serving\n22 \u00b7 Vector & feature\n23 \u00b7 Agent platform\n24 \u00b7 MCP, tools, knowledge\n25 \u00b7 Guardrails & eval\n\nVII \u00b7 Permissions\n26 \u00b7 Privilege vocabulary\n27 \u00b7 Cascade rules\n28 \u00b7 Roles\n29 \u00b7 Resolver\n30 \u00b7 Anti-escalation\n\nVIII \u00b7 Cost\n31 \u00b7 Cost model\n32 \u00b7 SKU catalog\n33 \u00b7 Billing tables\n34 \u00b7 Budgets & quotas\n35 \u00b7 Pricing tiers\n\nIX \u00b7 Audit / Observability\n36 \u00b7 Audit events\n37 \u00b7 Lineage\n38 \u00b7 Traces (OTel)\n39 \u00b7 Metrics & alerts\n40 \u00b7 Retention\n\nX \u00b7 Appendix\n41 \u00b7 Full DDL\n42 \u00b7 Migration plan\n43 \u00b7 Anti-patterns\n44 \u00b7 Sources\n\n\n\n\n\n\nData & AI Platform \u00b7 Foundation Spec complete\n\nA Databricks-shaped end-to-end specification: every object, every attribute, every privilege, every cost dimension, every audit event, every retention rule. Consolidates 7 research agents (Databricks UC + cost model + Dify/LangGraph/n8n + audit/lineage/observability) into one foundation document. Implementation-state-agnostic \u2014 this is the target architecture, not a current snapshot.\n\n\n\n  \n\n10\nplanes\n  \n\n120+\nobject types\n  \n\n38\nprivilege verbs\n  \n\n15\ncost dimensions\n\n\n\n  \n\n7y\naudit retention (FedRAMP)\n  \n\n4-pass\nresolver\n  \n\nOTel\nnative trace shape\n  \n\nMCP\nfirst-class tool host\n\n\n\n\nTL;DR \u2014 the picture\n\n\n\nWhat this is. A foundation spec for a Databricks-shaped data platform that is also a first-class AI agent platform. Every securable in the system has a defined parent, attribute schema, owner, privilege set, cost dimension, and audit footprint. Adding a new feature = 1 row in securable_type + 1 row in securable_cascade + 1 row in rate_card. No retro migrations, no new ACL tables, no new audit tables.\n\nWhat it covers. 10 planes \u2014 Account \u00b7 Identity \u00b7 Network \u00b7 UC Data \u00b7 Sharing & Marketplace \u00b7 Compute \u00b7 Orchestration \u00b7 Workspace \u00b7 AI Agent \u00b7 Governance/Cost/Audit. 120+ object types. Full DDL appendix.\n\nHow it's organised. Hierarchy \u2192 object catalog (per plane, with attributes) \u2192 privileges \u2192 cost \u2192 audit \u2192 DDL. The hierarchy tree (\u00a74) is the single source of truth; everything else hangs off it.\n\n\n\n\n1 \u00b7 North-star principles\n\n\n\n#PrincipleWhy\n1Metadata-only platform. The platform never reads/writes user data. Compute (Spark, DuckDB, Trino, Iceberg engines) does that directly against object storage; the platform vends temporary credentials.Decouples scale of metadata from scale of data; no IO bottleneck through the control plane.\n2One graph, one resolver. Every securable lives in a single typed graph; one 4-pass resolver (forbid \u2192 admin \u2192 owner \u2192 permit) answers every authorization question.Eliminates drift between 6+ parallel ACL systems and 12+ admin-bypass codepaths that production platforms accumulate.\n3Owner = column, not grant. Each securable has an owner_id column; ownership is never represented as a grant row.Atomic transfer; can't lose ownership through grant deletion; matches Postgres semantics.\n4Cascade = data, not code. Parent \u2192 child privilege flow lives in securable_cascade rows; resolver JOINs at query time.New asset type ships without recompile; auditable matrix.\n5Default-deny, explicit-DENY wins. Order-independent (Cedar style). DENY at any ancestor blocks ALLOW at any descendant.Required by SOC2 / HIPAA. Avoids \"but the test passed locally\" bypass.\n6Multi-tenant via explicit account_id on every row + Postgres RLS.Belt-and-braces; one forgotten JOIN can't leak across tenants.\n7Append-only audit, hash-chained. Every state change emits one or more audit_event rows with prev_hash; daily Signed Tree Heads to WORM bucket.Forensic integrity for SOC2 / HIPAA / FedRAMP; tamper detection without trusting the operator.\n8OTel GenAI shape for traces. All agent / LLM / tool invocations recorded as OTel spans with gen_ai.* attributes from day 1.Avoids painful schema migration when LangSmith/LangFuse/Phoenix integrations land.\n9Cost is a first-class plane. Every billable resource has an SKU mapping; usage records JOIN rate_card for list cost, then committed_use + discount_policy for effective.Chargeback + budgets + invoices flow from the same ledger.\n10MCP is the canonical tool host. Built-in tools, OpenAPI tools, workflow-as-tool, and external MCP servers are polymorphic providers behind one tool row.Anthropic + OpenAI + Haystack + LangChain interop; future-proofs for tool ecosystem changes.\n11Versioning everywhere mutable. Every editable artifact (Agent, Workflow, PromptTemplate, ServingConfig) has an immutable *_version row + a default_version_id pointer + a deployment_slot for canary/staging/prod.Avoids Dify's \"edit-published-app-in-place\" footgun; enables eval gates on promotion.\n12Future assets are pre-reserved. Enum values + cascade rules + rate-card slots for unimplemented features are seeded at spec time.Cheap reservation, expensive retro-migration.\n\n\n\n\n2 \u00b7 Architecture \u2014 five planes\n\n\n\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502  CONTROL PLANE  (this spec)                                            \u2502\n\u2502  \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510   \u2502\n\u2502  \u2502 Account  \u2502 Identity \u2502   UC     \u2502 Compute  \u2502 Orchestr.\u2502 AI Agent \u2502   \u2502\n\u2502  \u2502 Workspace\u2502  RBAC    \u2502 Catalog  \u2502  Mgmt    \u2502  Engine  \u2502 Platform \u2502   \u2502\n\u2502  \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518   \u2502\n\u2502  \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510    \u2502\n\u2502  \u2502 Cost &   \u2502 Audit &  \u2502 Governance: ABAC tags, policies, masks   \u2502    \u2502\n\u2502  \u2502 Billing  \u2502 Lineage  \u2502  row-filters, retention, legal-hold      \u2502    \u2502\n\u2502  \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518    \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n                                   \u2502   vends creds, returns paths\n                                   \u25bc\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502  DATA PLANE                                                            \u2502\n\u2502  \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510     \u2502\n\u2502  \u2502 Spark / SQL\u2502 DuckDB   \u2502 Trino    \u2502 Iceberg-REST\u2502 Custom engine\u2502     \u2502\n\u2502  \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518     \u2502\n\u2502  reads/writes directly to: S3 / ADLS / GCS / Lakebase Postgres         \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n                                   \u2502   token usage, span data\n                                   \u25bc\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502  AGENT PLANE                                                           \u2502\n\u2502  \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510     \u2502\n\u2502  \u2502 Workflow   \u2502 Agent    \u2502 MCP host \u2502 Vector idx  \u2502 Eval runner  \u2502     \u2502\n\u2502  \u2502 runtime    \u2502 runtime  \u2502 proxy    \u2502  (pgvector) \u2502  (judge LLM) \u2502     \u2502\n\u2502  \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518     \u2502\n\u2502  emits: TraceSpan, MetricSample, BillableUsage(token), AuditEvent      \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\n\n\n\nPlaneOwnsTalks to\nControl planeMetadata, ACLs, vended credentials, audit ledger, billing ledgerPostgres for OLTP; system-table mirror for analytics\nData planeCustomer's data files in their bucket; Lakebase OLTPCloud storage APIs directly with vended STS tokens\nCompute planeSpark drivers, SQL warehouses, model-serving pods, agent runtimesCloud K8s/EC2; reports lifecycle to control plane\nAgent planeLLM gateway, tool exec, MCP host, eval runnerReads governance / audit / cost from control plane; emits OTel traces back\nEdge planeWeb console, REST API, JDBC/ODBC endpoints, public webhooksAuth via JWT/PAT/OAuth; rate-limited per workspace\n\n\n\n\n3 \u00b7 Tenancy model\n\nAccount-centric (Databricks shape). One Account is the billing entity, owns N Workspaces and 1+ Metastores, and federates principals from external IDPs. Cross-Account interactions go through Sharing/Marketplace, never direct grants.\n\n\n\nLevelObjectPurposeLifetime\nTier 0 (root)AccountBilling entity, top of identity tree, owns metastores + workspacespermanent until decommission\nTier 1WorkspaceCompute/UI/orchestration boundary; carries quotas, network bindings, IDP federationmutable; soft-delete with grace period\nTier 1MetastoreUC namespace root: catalogs, schemas, governance objects, sharing, secrets1 per region typically\nTier 2MetastoreAssignment(Workspace \u00d7 Metastore) \u2014 what data a workspace seesmutable\nTier 2WorkspaceCatalogBinding(Workspace \u00d7 Catalog, mode=R/RW) \u2014 finer-grained than metastore assignmentmutable\n\n\n\n\n4 \u00b7 Complete object hierarchy\n\nSingle source of truth. Every section below references nodes in this tree. Gold = AI agent plane (new). Blue = control plane.\n\n\n\n\n\n\nAccount\n  \n\n    \n\u21b3 Identity \u2014 User \u00b7 Group \u00b7 ServicePrincipal \u00b7 FederatedIdentity \u00b7 PAT \u00b7 OAuthApp \u00b7 EnterpriseApp \u00b7 IdentityFederationConfig \u00b7 ServiceProviderForOIDC\n    \n\u21b3 Account-only \u2014 Budget \u00b7 BudgetAlert \u00b7 CommittedUse \u00b7 DiscountPolicy \u00b7 RateCard \u00b7 Invoice \u00b7 CostAttributionPolicy \u00b7 BudgetPolicy \u00b7 IPAccessList(account) \u00b7 NetworkConfiguration \u00b7 PrivateAccessSettings \u00b7 NetworkConnectivityConfig \u00b7 CustomerManagedKey \u00b7 StorageConfiguration \u00b7 CredentialConfiguration \u00b7 LogDelivery \u00b7 AuditLogConfig\n      (account admin only)\n    \n\u21b3 Workspace\n      \n\n        \n\u21b3 Workspace settings \u2014 Token \u00b7 TokenManagement \u00b7 IPAccessList(ws) \u00b7 GitCredentials \u00b7 WorkspaceConf \u00b7 GlobalInitScript \u00b7 EnhancedSecurity\n        \n\u21b3 Compute \u2014 Cluster \u00b7 ClusterPolicy \u00b7 InstancePool \u00b7 InstanceProfile \u00b7 NodeType \u00b7 ClusterEvent \u00b7 Library \u00b7 Network v3\n        \n\u21b3 SQL \u2014 SQLWarehouse \u00b7 Query \u00b7 QuerySnippet \u00b7 Alert \u00b7 AlertDestination \u00b7 QueryHistory \u00b7 Subscription\n        \n\u21b3 Orchestration \u2014 Job \u00b7 JobRun \u00b7 JobTask \u00b7 TaskRun \u00b7 Pipeline (DLT) \u00b7 PipelineUpdate \u00b7 PipelineEvent \u00b7 Flow v3 \u00b7 FlowRun \u00b7 Trigger \u00b7 Webhook\n        \n\u21b3 Workspace assets \u2014 Notebook \u00b7 NotebookCell \u00b7 KernelSession \u00b7 File \u00b7 Directory \u00b7 Repo \u00b7 Dashboard \u00b7 DashboardDataset \u00b7 DashboardChat \u00b7 DashboardSchedule \u00b7 GenieSpace \u00b7 GenieMessage\n        \n\u21b3 ML platform \u2014 Experiment \u00b7 MLRun \u00b7 ModelEvaluation \u00b7 ModelMonitor \u00b7 ServingEndpoint \u00b7 ServedModel \u00b7 OnlineTable \u00b7 OnlineStore \u00b7 FeatureTable \u00b7 FeatureSpec \u00b7 VectorSearchEndpoint \u00b7 VectorIndex\n          \n\n\u21b3 InferenceLog\n        \n        \n\u21b3 AI Agent platform\n          \n\n            \n\u21b3 AIApp (Dify-style: chatbot/workflow/chatflow/agent/textgen)\n              \n\n                \n\u21b3 Agent \u2192 AgentVersion \u2192 DeploymentSlot\n                \n\u21b3 Workflow \u2192 WorkflowVersion \u2192 WorkflowNode \u2192 WorkflowRun \u2192 WorkflowNodeExecution\n                \n\u21b3 Conversation \u2192 Message \u2192 MessageAttachment \u00b7 Annotation\n                \n\u21b3 EndUser (SaaS user of the deployed app)\n                \n\u21b3 Trigger \u00b7 Webhook \u00b7 Variable \u00b7 SecretReference\n                \n\u21b3 Guardrail \u00b7 RateLimitPolicy \u00b7 CostBudget \u00b7 ApprovalPolicy \u00b7 DataResidencyPolicy \u00b7 PIIRedactor\n              \n            \n            \n\u21b3 MCPServer \u00b7 Tool \u00b7 ToolBinding \u00b7 ToolUsageQuota\n            \n\u21b3 KnowledgeBase \u2192 KBDocument \u2192 KBChunk \u00b7 MemoryStore\n            \n\u21b3 PromptTemplate \u2192 PromptVersion\n            \n\u21b3 ModelEndpoint (LLM provider alias) \u00b7 AIGateway\n            \n\u21b3 EvalDataset \u2192 EvalRun \u2192 EvalScore \u00b7 EvalGate\n          \n        \n        \n\u21b3 Apps \u2014 App (Lakehouse Apps) \u00b7 AppDeployment\n      \n    \n    \n\u21b3 Metastore\n      \n\n        \n\u21b3 UC objects\n          \n\n            \n\u21b3 Catalog \u2192 Schema\n              \n\n                \n\u21b3 Table \u00b7 StreamingTable \u00b7 View \u00b7 MaterializedView\n                \n\u21b3 Volume(managed/external) \u00b7 Function \u00b7 Procedure \u00b7 Model \u00b7 ModelVersion \u00b7 RegisteredModel\n                \n\u21b3 TableConstraint \u00b7 ColumnInfo \u00b7 ColumnMaskAttachment \u00b7 RowFilterAttachment\n              \n            \n          \n        \n        \n\u21b3 Storage & credentials \u2014 StorageCredential \u00b7 ExternalLocation \u00b7 Connection \u00b7 ServiceCredential \u00b7 CustomerManagedKey-binding\n        \n\u21b3 Federation \u2014 Connection \u2192 ForeignCatalog \u2192 ForeignSchema \u2192 ForeignTable\n        \n\u21b3 Sharing \u2014 Share \u00b7 ShareTable \u00b7 Provider \u00b7 Recipient \u00b7 CleanRoom \u00b7 CleanRoomAsset \u00b7 CleanRoomTask \u00b7 RecipientFederationPolicy\n        \n\u21b3 Marketplace \u2014 Listing \u00b7 Exchange \u00b7 ConsumerSubscription \u00b7 ConsumerInstallation \u00b7 ProviderProfile\n        \n\u21b3 Lakebase \u2014 DatabaseInstance \u00b7 DatabaseRole \u00b7 DatabaseSchema\n        \n\u21b3 Governance \u2014 GovernedTag \u00b7 TagAssignment \u00b7 TagPolicy \u00b7 ExternalMetadata \u00b7 DataAccessConfiguration \u00b7 LineageEntry\n        \n\u21b3 Secrets \u2014 SecretScope \u2192 Secret \u00b7 SecretAcl\n      \n    \n    \n\u21b3 MetastoreAssignment (Workspace \u00d7 Metastore)\n    \n\u21b3 WorkspaceCatalogBinding (Workspace \u00d7 Catalog, mode=R/RW)\n  \n\n\n\n\n\n\n5 \u00b7 Account plane\n\n\nAccount\n\n\nAttributeTypeNotes\naccount_idUUID PKimmutable\nnametextdisplay name\ncloudenum(AWS,AZURE,GCP)\u2014\nhome_regiontextbilling region\ncontract_typeenum(PAYG,ANNUAL,ENTERPRISE)governs SLA\ntierenum(STANDARD,PREMIUM,ENTERPRISE)feature gate (\u00a735)\nstatusenum(ACTIVE,SUSPENDED,DECOMMISSIONED)\u2014\ncreated_at, updated_attstz\u2014\nbilling_contact_emailtextinvoice destination\nsupport_tierenum(BUSINESS,ENTERPRISE,ENTERPRISE_PLUS)\u2014\n\n\nPrivileges: MANAGE_ACCOUNT (account_admin), BROWSE_ACCOUNT (account_user). Cascade: every workspace + metastore inherits. Audit: account.create / update / suspend / decommission.\n\n\nWorkspace\n\n\nAttributeTypeNotes\nworkspace_idUUID PK\u2014\naccount_idUUID FKtenancy\nname, deployment_nametextdeployment_name = subdomain\nregion, cloudtext, enum\u2014\nnetwork_idUUID FK?\u2192Networkcustomer-managed VPC\nstorage_config_idUUID FK?workspace storage bucket\ncredentials_idUUID FK?cross-account role\nprivate_access_settings_idUUID FK?PrivateLink\ncustomer_managed_key_idUUID FK?BYOK\npricing_tierenuminherited from account, overridable\ncompliance_security_profilejsonbHIPAA, FedRAMP-Mod toggles\nstatus, status_messageenum, textPROVISIONING / RUNNING / FAILED / CANCELLING\ncreated_at, deleted_attstz, tstz?soft delete\ntagsjsonbcost attribution\n\n\nPrivileges: MANAGE_WORKSPACE, USE_WORKSPACE, BROWSE_WORKSPACE. Children cascade BROWSE+MANAGE. Audit on every state change.\n\n\nOther account-only objects\n\n\nObjectKey attributesPrivileges\nNetworkConfigurationvpc_id, subnet_ids[], security_group_ids[], region, statusMANAGE\nPrivateAccessSettingsprivate_access_level, allowed_vpc_endpoint_ids[], public_access_enabledMANAGE\nNetworkConnectivityConfigegress nat config, allowed CIDRsMANAGE\nCustomerManagedKeykms_arn, use_cases[](MANAGED_DATA, STORAGE)MANAGE\nStorageConfigurationbucket_name, root_pathMANAGE\nCredentialConfigurationaws_credential_arn (cross-account role)MANAGE\nLogDeliveryconfig_name, log_type(BILLABLE_USAGE, AUDIT_LOGS), output_format, statusMANAGE\nAuditLogConfigverbose_audit_logs bool, log_delivery_idMANAGE\nIPAccessList(account)list_type(ALLOW,BLOCK), ip_addresses[]MANAGE\nOAuthPublishedApp / EnterpriseAppapp_id, redirect_uris[], scopes[]USE / MANAGE\n\n\n\n\n6 \u00b7 Identity plane\n\n\n\nObjectKey attributesCascade / privileges\nUseruser_id, account_id, external_id, user_name, display_name, emails[], active, entitlements[], roles[]self-mutable; admin-mutable. MANAGE_USER, MANAGE_PAT\nGroupgroup_id, account_id, display_name, members[], parent_groups[]nested groups, depth \u22643. MANAGE_GROUP\nServicePrincipalapplication_id, display_name, oauth_secrets[]MANAGE_SP\nFederatedIdentityissuer, subject, audience, attrs jsonbJIT-created; resolved at JWT validation\nPATtoken_id, owner_user_id, comment, lifetime_seconds, created_at, last_used_atowner-only; cascade from User\nIdentityFederationConfigidp_type(OIDC, SAML), issuer_url, jwks_uri, claim_mappings jsonbMANAGE\nGroupMembership(parent_id, member_id, account_id)edge-list; resolver expands transitively\n\n\n\n\nFederated identity contract. Inbound JWT validated against allow-listed issuer JWKS. (iss, sub, aud) tuple looked up in principal; missing \u2192 JIT-create row if issuer's auto_create flag is on. Token-exchange (RFC 8693) chains tracked in actor_session with depth \u2264 3 (rejected beyond).\n\n\n\n\n7 \u00b7 Network & security objects\n\nAll edge-policy objects. Read-only at the resolver level \u2014 i.e., they don't grant data access, they restrict who can authenticate at all.\n\n\n\nObjectScopeEffect\nIPAccessList(account)accountAllow/block list applied before workspace routing\nIPAccessList(workspace)workspacePer-workspace override\nPrivateAccessSettingsworkspaceFront-end / back-end PrivateLink\nEnhancedSecurityComplianceworkspaceHIPAA / FedRAMP / PCI / Compliance Security Profile\nNetworkConnectivityConfigaccountNAT, egress firewall\nCustomerManagedKey bindingworkspace + metastoreBYOK encryption; rotation tracked\nOAuthApp / EnterpriseAppaccountSSO / 3rd-party integration\n\n\n\n\n8 \u00b7 Unity Catalog \u2014 top of the data plane\n\n\nMetastore\n\n\nAttributeTypeNotes\nmetastore_idUUID PK\u2014\naccount_idUUID FK\u2014\nname, region, cloud\u20141 metastore per region\nstorage_roottexts3://bucket/path/ for managed tables\nstorage_root_credential_idUUID FK?StorageCredential FK\ndelta_sharing_scopeenum(INTERNAL_ONLY, INTERNAL_AND_EXTERNAL)\u2014\ndelta_sharing_recipient_token_lifetimeint (sec)\u2014\ndelta_sharing_organization_nametextshown to recipients\nowner_idUUID FK Principalmetastore admin\nprivilege_model_versiontext\"1.0\" current\ncreated_at, updated_attstz\u2014\n\n\n\nCatalog\n\n\nAttributeTypeNotes\ncatalog_id, nameUUID, text\u2014\nmetastore_idUUID FKparent\ncatalog_typeenum(MANAGED, FOREIGN, SHARE_CATALOG, SYSTEM_CATALOG, DELTASHARING_CATALOG)\u2014\nstorage_roottext?override metastore root\nprovider_name, share_nametext?for SHARE_CATALOG\nconnection_idUUID FK?for FOREIGN\noptionsjsonbconnector-specific\nisolation_modeenum(OPEN, ISOLATED)workspace binding mode\nowner_id, comment, properties (jsonb), tags (jsonb)\u2014\u2014\n\n\n\nSchema\n\n\nAttributeTypeNotes\nschema_id, name, catalog_idUUID, text, UUID\u2014\nstorage_roottext?override catalog\nstorage_locationtextresolved managed root\nowner_id, comment, properties, tags\u2014\u2014\nfull_nametext\"catalog.schema\" denormalised for fast lookup\n\n\n\n\n9 \u00b7 Tables & views\n\n\nCommon (Table / StreamingTable / View / MaterializedView)\n\nCREATE TABLE table_uc (\n  table_id              UUID PRIMARY KEY,\n  account_id            UUID NOT NULL,\n  metastore_id          UUID NOT NULL,\n  catalog_id            UUID NOT NULL,\n  schema_id             UUID NOT NULL,\n  name                  TEXT NOT NULL,\n  full_name             TEXT NOT NULL,\n  table_type            TEXT NOT NULL CHECK (table_type IN\n    ('MANAGED','EXTERNAL','VIEW','MATERIALIZED_VIEW','STREAMING_TABLE','FOREIGN','SHARED')),\n  data_source_format    TEXT,           -- DELTA / ICEBERG / PARQUET / ORC / CSV / JSON / JDBC\n  storage_location      TEXT,\n  view_definition       TEXT,           -- for VIEW/MV\n  view_dependencies     JSONB,\n  storage_credential_id UUID,\n  enable_predictive_optimization TEXT,  -- ENABLE/DISABLE/INHERIT\n  delta_runtime_properties_kvpairs JSONB,\n  -- Iceberg-specific\n  iceberg_table_format  TEXT,            -- v2/v3\n  -- Streaming\n  pipeline_id           UUID,            -- for ST: parent pipeline\n  refresh_status        TEXT,            -- for MV/ST\n  last_refresh_at       TIMESTAMPTZ,\n  -- Governance\n  owner_id              UUID NOT NULL,\n  comment               TEXT,\n  properties            JSONB,\n  tags                  JSONB,\n  -- Lifecycle\n  generation            BIGINT,\n  created_at            TIMESTAMPTZ NOT NULL,\n  updated_at            TIMESTAMPTZ NOT NULL,\n  deleted_at            TIMESTAMPTZ,\n  -- Children\n  -- columns: separate column_info table\n  -- constraints: separate table_constraint table\n  -- attachments: row_filter, column_mask\n);\n\n\nColumnInfo\n\nCREATE TABLE column_info (\n  table_id   UUID NOT NULL REFERENCES table_uc ON DELETE CASCADE,\n  position   INT NOT NULL,\n  name       TEXT NOT NULL,\n  type_name  TEXT NOT NULL,             -- canonical UC type\n  type_text  TEXT NOT NULL,             -- engine-rendered\n  type_json  JSONB,\n  type_precision INT, type_scale INT, type_interval_type TEXT,\n  nullable   BOOLEAN NOT NULL DEFAULT TRUE,\n  comment    TEXT,\n  partition_index INT,\n  mask_function_id UUID,                 -- column mask attachment\n  PRIMARY KEY (table_id, position)\n);\n\n\nTableConstraint\n\nCREATE TABLE table_constraint (\n  table_id    UUID NOT NULL REFERENCES table_uc ON DELETE CASCADE,\n  name        TEXT NOT NULL,\n  kind        TEXT NOT NULL CHECK (kind IN ('PRIMARY_KEY','FOREIGN_KEY','CHECK','NOT_NULL')),\n  columns     TEXT[] NOT NULL,\n  ref_table_id UUID,\n  ref_columns  TEXT[],\n  expression  TEXT,\n  rely        BOOLEAN NOT NULL DEFAULT FALSE,\n  PRIMARY KEY (table_id, name)\n);\n\n\nStreamingTable / MV refresh metadata\n\n-- refresh state per ST/MV\nCREATE TABLE table_refresh (\n  table_id          UUID PRIMARY KEY REFERENCES table_uc ON DELETE CASCADE,\n  schedule_cron     TEXT,\n  last_run_id       UUID,\n  last_run_status   TEXT,\n  next_run_at       TIMESTAMPTZ\n);\n\n\n\n10 \u00b7 Storage, volumes, credentials\n\n\n\nObjectPurposeKey attrs\nStorageCredentialcloud cred for managed/external storagename, aws_iam_role_arn / azure_managed_identity / gcp_sa_email, comment, isolation_mode, read_only bool, owner_id, used_for_managed_storage bool\nExternalLocation(path, credential) binding for external tables/volumesname, url, credential_id, fallback bool, encryption_details jsonb, browse_only bool, owner_id, comment\nVolumefile mount point under a schemavolume_id, schema_id, name, volume_type(MANAGED/EXTERNAL), storage_location, owner_id, comment\nConnectionfederation connection (Postgres/MySQL/Snowflake/BigQuery/Redshift/SF Open Catalog/etc.)name, connection_type, options jsonb (host/port/auth-ref), credential_id, read_only bool, properties jsonb\nServiceCredentialcred for invoking external compute/services (federation, AI gateway)name, service_type, auth_type, options jsonb, owner_id\nVolumeFilelogical row per file under Volume; mostly virtualvolume_id, path, size, content_type, modified_at\n\n\n\n\n11 \u00b7 Functions, models, masks\n\n\n\nObjectPurposeKey attrs\nFunctionSQL/Python UDF; also hosts row-filter and column-mask UDFsfunction_id, schema_id, name, input_params jsonb, return_type jsonb, routine_definition text, language(SQL,PYTHON,JAVA,SCALA), is_deterministic, is_null_call, sql_path, runtime_version, sql_data_access(NO_SQL,READS,MODIFIES), security_type(DEFINER,INVOKER), specific_name, comment, properties, tags, owner_id\nProcedureSQL stored procedure (different EXECUTE semantics)same shape as Function + procedure-only fields\nRegisteredModelMLflow model registry rootmodel_id, schema_id, name, owner_id, comment, tags\nModelVersionimmutable versionmodel_id, version int, source_uri, run_id, status(PENDING_REGISTRATION,READY,FAILED), run_link, model_signatures jsonb, owner\nRowFilterAttachment(table \u2192 filter_function) \u2014 applied at SELECT rewritetable_id PK, function_id, args text[]\nColumnMaskAttachment(table.col \u2192 mask_function) \u2014 applied at SELECT rewrite(table_id, column_name) PK, function_id, args text[]\nGovernedTagABAC tag definition (key + allowed values)tag_id, metastore_id, key, allowed_values text[], description, owner_id\nTagAssignment(securable, tag_key, tag_value)polymorphic FK\nTagPolicypredicate-style ABAC (\"tag PII denies SELECT for *\")policy_id, expression_cedar, applies_to_types text[], effect(ALLOW/DENY)\n\n\n\n\n12 \u00b7 Delta Sharing & Marketplace\n\n\n\nObjectKey attrs\nShareshare_id, metastore_id, name, owner_id, objects jsonb (refs to tables/views/notebooks/volumes/models), comment, created_at\nProviderprovider_id, name, authentication_type(TOKEN, OAUTH_CLIENT_CREDENTIALS, DATABRICKS), recipient_profile_str (for TOKEN), comment, owner_id, cloud, region\nRecipientrecipient_id, name, authentication_type, sharing_code (single-use), token_expiration_time, ip_access_list, owner_id, comment\nRecipientFederationPolicyrecipient_id, oidc_policy jsonb (issuer, subject, audience claim mapping)\nCleanRoomclean_room_id, name, comment, collaborators[] (workspace+region+org), output_catalog text, owner_id\nCleanRoomAssetclean_room_id, name, asset_type(TABLE,VIEW,VOLUME,NOTEBOOK,JAR,FOREIGN_TABLE), table_full_name, ownership_principal, added_at\nCleanRoomTaskclean_room_id, name, runtime, code_location, output_catalog, schedule_cron, status\nMarketplaceListinglisting_id, provider_id, summary jsonb (title, subtitle, description, categories[], regions[]), detail jsonb (price_model, support_link, terms), status(DRAFT,PUBLISHED,SUSPENDED)\nExchangeexchange_id, name, comment, filters jsonb\nConsumerSubscriptionlisting_id, subscriber_account_id, terms_accepted_at\nConsumerInstallationinstallation_id, listing_id, target_catalog_id, status, installed_at\n\n\n\n\n13 \u00b7 Lakehouse Federation\n\n\n\nObjectKey attrs\nConnectionsee \u00a710\nForeignCatalogcatalog_id (Catalog row with catalog_type=FOREIGN), connection_id, options jsonb, credential_id\nForeignSchemaFOREIGN catalog \u2192 schema cache; refreshed on access\nForeignTablecached metadata only; reads passthrough to source\nDatabaseInstance (Lakebase)instance_id, postgres_version, capacity_units, primary_endpoint, replicas[], parent_metastore_id, branch_of_instance_id?, owner_id, status\nDatabaseRole / DatabaseSchema(Lakebase) materialised UC rows for Postgres roles + schemas\nVendedCredential(audit-only) credential_id, requestor_principal, expires_at, used_at[]\nExternalMetadataexternal_metadata_id, kind(TABLEAU_DASHBOARD, SALESFORCE_REPORT, POWER_BI, \u2026), name, system_url, properties jsonb, owner_id, downstream_of full_names[]\n\n\n\n\n14 \u00b7 Compute objects\n\n\nCluster (full attribute set)\n\nCREATE TABLE cluster (\n  cluster_id           UUID PRIMARY KEY,\n  workspace_id         UUID NOT NULL,\n  cluster_name         TEXT NOT NULL,\n  state                TEXT NOT NULL,            -- PENDING, RUNNING, RESTARTING, RESIZING, TERMINATING, TERMINATED, ERROR\n  state_message        TEXT,\n  -- Sizing\n  spark_version        TEXT NOT NULL,            -- DBR version\n  node_type_id         TEXT NOT NULL,\n  driver_node_type_id  TEXT,\n  num_workers          INT,\n  autoscale_min        INT,\n  autoscale_max        INT,\n  -- Runtime\n  spark_conf           JSONB,\n  spark_env_vars       JSONB,\n  custom_tags          JSONB,\n  cluster_log_conf     JSONB,\n  init_scripts         JSONB,\n  enable_local_disk_encryption BOOLEAN,\n  data_security_mode   TEXT,                     -- SINGLE_USER, USER_ISOLATION, LEGACY_SINGLE_USER, NONE, USER_ISOLATION_NEXT\n  single_user_name     TEXT,\n  -- Billing\n  policy_id            UUID,                     -- compute policy\n  instance_pool_id     UUID,\n  driver_instance_pool_id UUID,\n  cluster_source       TEXT,                     -- API, JOB, UI\n  is_photon            BOOLEAN,\n  -- Cloud\n  aws_attributes       JSONB,                    -- spot_bid_price_percent, ebs_volume_*, instance_profile_arn\n  azure_attributes     JSONB,\n  gcp_attributes       JSONB,\n  -- Lifecycle\n  autotermination_minutes INT,\n  creator_user_name    TEXT,\n  start_time           TIMESTAMPTZ,\n  terminated_time      TIMESTAMPTZ,\n  last_activity_time   TIMESTAMPTZ,\n  termination_reason   JSONB\n);\n-- children: cluster_event, cluster_library, cluster_node_timeline (ts metric)\n\n\n\nObjectPurposeKey attrs\nClusterPolicyJSON policy document constraining cluster shapepolicy_id, name, definition jsonb, max_clusters_per_user, policy_family_id, created_at\nInstancePoolpre-warmed nodes for fast cluster startpool_id, name, node_type_id, min_idle_instances, max_capacity, idle_instance_autotermination_minutes, preloaded_spark_versions[], custom_tags\nInstanceProfile (AWS)cross-account IAM role for clustersarn, is_meta_instance_profile\nNodeTypesystem catalog of available SKUsnode_type_id, instance_type_id, memory_mb, num_cores, gpu_info jsonb\nClusterEventper-cluster timeline eventscluster_id, timestamp, type(CREATING, STARTING, \u2026), details jsonb\nLibrarycluster-attached libs (pypi, maven, jar, egg, whl)cluster_id, source(PYPI,MAVEN,JAR,EGG,WHL,CRAN), spec jsonb, status\nGlobalInitScriptaccount-level init scriptsscript_id, name, script (text), enabled, position, created_by\nNetwork (compute)VPC binding for clustersnetwork_id, vpc_id, subnet_ids[], security_group_ids[], status\n\n\n\n\n15 \u00b7 SQL warehouses\n\n\nObjectKey attrs\nSQLWarehousewarehouse_id, name, cluster_size(2X-Small..4X-Large), min_num_clusters, max_num_clusters, auto_stop_mins, channel(CHANNEL_NAME_CURRENT/PREVIEW), enable_photon, enable_serverless_compute, warehouse_type(CLASSIC,PRO), spot_instance_policy, tags jsonb, state, creator\nQueryquery_id, name, query_text, parent_path (folder), data_source_id (warehouse), user_id, options jsonb (parameter defs), tags, run_as(VIEWER, OWNER), schedule jsonb, created_at, updated_at, last_run_at\nQuerySnippetsnippet_id, name, content, trigger (text command), description, user_id\nAlertalert_id, name, query_id, condition jsonb (op, threshold, column), schedule, last_triggered_at, state(OK,TRIGGERED,UNKNOWN), notify_on_ok bool, custom_subject, custom_body, rearm_seconds, owner_user_id\nAlertDestinationdestination_id, type(EMAIL,SLACK,WEBHOOK,PAGERDUTY,MS_TEAMS), config jsonb (encrypted via secret_ref), name, owner\nQueryHistorysee \u00a738; system table\nSubscription(alert/dashboard subscriber) \u2014 recipient + cadence\n\n\n\n\n16 \u00b7 Orchestration \u2014 Jobs, Pipelines, Flows\n\n\nJob\n\nCREATE TABLE job (\n  job_id              UUID PRIMARY KEY,\n  workspace_id        UUID NOT NULL,\n  name                TEXT NOT NULL,\n  format              TEXT,                 -- SINGLE_TASK, MULTI_TASK\n  tasks               JSONB NOT NULL,       -- ordered DAG of task specs\n  job_clusters        JSONB,                -- shared cluster defs\n  email_notifications JSONB,\n  webhook_notifications JSONB,\n  notification_settings JSONB,\n  timeout_seconds     INT,\n  schedule            JSONB,                -- cron, timezone, pause_status\n  trigger             JSONB,                -- file_arrival, periodic\n  continuous          BOOLEAN,\n  max_concurrent_runs INT,\n  parameters          JSONB,                -- typed params with defaults\n  run_as              JSONB,                -- service principal or user\n  edit_mode           TEXT,                 -- UI_LOCKED, EDITABLE\n  deployment          JSONB,                -- bundle metadata\n  description         TEXT,\n  tags                JSONB,\n  budget_policy_id    UUID,                 -- for serverless tagging\n  performance_target  TEXT,                 -- STANDARD, PERFORMANCE_OPTIMIZED\n  queue               JSONB,                -- enabled bool\n  health              JSONB,                -- rules\n  owner_id            UUID NOT NULL,\n  created_at          TIMESTAMPTZ NOT NULL,\n  updated_at          TIMESTAMPTZ NOT NULL\n);\n\n\n\nObjectKey attrs\nJobRunrun_id, job_id, run_name, run_type(JOB_RUN, WORKFLOW_RUN, SUBMIT_RUN), trigger, state, life_cycle_state, result_state, start_time, end_time, duration_ms, queue_duration_ms, task_runs[], cluster_spec, params, attempt_number, original_attempt_run_id\nTaskRuntask_key, run_id, state, attempt_number, cluster_id, output_link, exit_code, error\nPipeline (DLT/Lakeflow)pipeline_id, name, edition(CORE,PRO,ADVANCED), continuous bool, development bool, photon bool, serverless bool, target catalog/schema, libraries[], notifications, configuration jsonb, clusters[], channel, expectations, root_path\nPipelineUpdateupdate_id, pipeline_id, creation_time, state(QUEUED, CREATED, WAITING_FOR_RESOURCES, INITIALIZING, RESETTING, SETTING_UP_TABLES, RUNNING, STOPPING, COMPLETED, FAILED, CANCELED), full_refresh, refresh_selection[], cause\nPipelineEventid, pipeline_id, update_id, event_type(create_update, flow_progress, dataset_quality, \u2026), timestamp, level, message, details jsonb\nFlow(generic orchestrator, distinct from Job/Pipeline) flow_id, name, dag jsonb, schedule, owner\nTrigger(generic) trigger_id, parent_id, kind(CRON,WEBHOOK,EVENT,FILE_ARRIVAL,MANUAL,CHAT), config, enabled, secret_ref\n\n\n\n\n17 \u00b7 Workspace assets\n\n\n\nObjectKey attrs\nNotebookobject_id, path, language(PYTHON,SCALA,SQL,R), source(JUPYTER,DATABRICKS,SOURCE), content_blob, size, created_at, modified_at, owner_id\nNotebookCell(notebook_id, position, cell_type(CODE,MARKDOWN,RAW), source, language_override, output_blob, last_run_at, last_run_status)\nKernelSessionsession_id, notebook_id, kernel_type(IPYTHON,SPARK,SQL), started_at, last_active_at, idle_seconds, allocated_cluster_id, owner_id\nFileobject_id, path, size, content_type, blob_uri, created_at, modified_at\nDirectoryobject_id, path\nReporepo_id, path, url, provider(GITHUB,GITLAB,BITBUCKET,AZURE_DEVOPS), branch, head_commit_id, sparse_checkout jsonb\nGitCredentialscredential_id, git_provider, git_username, personal_access_token (vended)\nWorkspaceConf(workspace-level settings k/v map)\nApp (Lakehouse Apps)app_id, name, source_code_path, default_source_code_path, app_status, compute_status, url, app_resources jsonb (sql_warehouse, secret_scope, model_endpoint, etc.), service_principal_client_id, owner\nAppDeploymentdeployment_id, app_id, source_code_path, mode(SNAPSHOT,AUTO_SYNC), status, deployment_artifacts\n\n\n\n\n18 \u00b7 Dashboards, Genie, Visualizations\n\n\n\nObjectKey attrs\nDashboard (Lakeview)dashboard_id, name, parent_path, serialized_dashboard (DSL/JSON), warehouse_id, lifecycle_state, created_at, updated_at, owner_id\nDashboardDataset(dashboard_id, name, query_text, parameters jsonb)\nDashboardSchedule(dashboard_id, cron, timezone, pause_status, subscriptions[])\nDashboardRefresh(dashboard_id, run_id, state, started_at, finished_at)\nGenieSpacespace_id, title, description, instructions, sample_questions[], catalog_id+schema_id (data scope), allowed_tables[], owner\nGenieMessagemessage_id, conversation_id, role, content jsonb, attachments[], created_at, sources[] (lineage)\nVisualization(query_id, type, options jsonb)\nSubscription(alert_id|dashboard_id, recipient, channel)\n\n\n\n\n19 \u00b7 Secrets\n\n\n\nObjectKey attrs\nSecretScopescope_id, name (unique within workspace), backend_type(DATABRICKS,AZURE_KEYVAULT), keyvault_metadata, initial_manage_principal\nSecret(scope_id, key, value (AES-256-GCM at rest, key from KEK), version, last_updated_timestamp)\nSecretAcl(scope_id, principal, permission(READ,WRITE,MANAGE)) \u2014 folded into unified grant table\n\n\n\n\n20 \u00b7 ML platform \u2014 experiments & registry\n\n\n\nObjectKey attrs\nExperiment (MLflow)experiment_id, name, artifact_location, lifecycle_stage, tags, owner\nMLRunrun_id, experiment_id, status, start_time, end_time, source_type, source_name, user, params jsonb, metrics jsonb (timeseries), tags, artifact_uri\nRegisteredModelsee \u00a711\nModelVersionsee \u00a711\nModelEvaluationeval_id, model_version_id, dataset_ref, metrics jsonb, status, started_at\nModelMonitormonitor_id, table_full_name, profile_type(TIMESERIES,SNAPSHOT,INFERENCE_LOG), schedule, baseline_table, slicing_exprs[], custom_metrics[], notifications\n\n\n\n\n21 \u00b7 Model Serving\n\n\n\nObjectKey attrs\nServingEndpointendpoint_id, name, config jsonb (served_entities[], traffic_config), state jsonb (ready, config_update), tags, route_optimized bool, ai_gateway jsonb, budget_policy_id\nServedEntity / ServedModelname, entity_name(model:catalog.schema.name OR foundation_model OR external_model), entity_version, scale_to_zero_enabled, workload_type(CPU/GPU_*), workload_size, environment_vars, instance_profile_arn, max_provisioned_throughput\nServingConfigUpdate(endpoint_id, version, config jsonb, state, started_at, finished_at)\nInferenceLog(endpoint_name, request_time, request_id, input_tokens, output_tokens, status, served_entity_id) \u2192 BillableUsage\nAIGateway(endpoint_id, usage_tracking_config, rate_limits[], inference_table_config, guardrails jsonb, fallback_config)\nRouteOptimizationendpoint-level low-latency routing config\n\n\n\n\n22 \u00b7 Vector search, feature, online\n\n\n\nObjectKey attrs\nVectorSearchEndpointendpoint_id, name, type(STANDARD,STORAGE_OPTIMIZED), state(PROVISIONING,ONLINE,OFFLINE), num_indexes\nVectorIndexindex_id, endpoint_id, name, primary_key, index_type(DELTA_SYNC,DIRECT_ACCESS), embedding_source_column, embedding_model_endpoint_name, schema jsonb, sync_status, source_table_full_name (for DELTA_SYNC), pipeline_type(TRIGGERED,CONTINUOUS)\nOnlineStorestore_id, name, capacity_units, status, parent_metastore\nOnlineTableonline_table_id, name (3-part), source_table_full_name, primary_keys[], spec jsonb (run_triggered/run_continuously, perform_full_copy_initially), status\nFeatureTablefeature_table_id, name, primary_keys[], timestamp_keys[], features jsonb (name, type, description), source_table_full_name\nFeatureSpecspec_id, schema_id, name, features[] (table_ref + name), function_specs[]\n\n\n\n\n23 \u00b7 AI Agent platform\n\nDistilled from Dify (App/Workflow/Conversation/KB), LangGraph (Thread/Checkpoint/Store), n8n (Trigger/Project/Credential), OpenAI Assistants (Run/RunStep), CrewAI (Crew/Memory), Anthropic MCP (Tool/Resource/Prompt). Modelled as first-class UC-grade securables.\n\n\nAIApp \u2014 top-level deployable\n\nCREATE TABLE ai_app (\n  app_id              UUID PRIMARY KEY,\n  workspace_id        UUID NOT NULL,\n  schema_id           UUID,                -- optional UC scoping\n  name                TEXT NOT NULL,\n  kind                TEXT NOT NULL,        -- CHATBOT, WORKFLOW, CHATFLOW, AGENT, TEXT_GEN\n  description         TEXT,\n  default_version_id  UUID,                 -- \u2192 ai_app_version\n  visibility          TEXT NOT NULL,        -- PRIVATE, INTERNAL, PUBLIC\n  channels            TEXT[],               -- WEB, API, SLACK, DISCORD, TEAMS, WHATSAPP, EMBED\n  budget_policy_id    UUID,\n  data_residency_policy_id UUID,\n  guardrail_ids       UUID[],\n  rate_limit_policy_id UUID,\n  approval_policy_id  UUID,\n  pii_redactor_id     UUID,\n  owner_id            UUID NOT NULL,\n  tags                JSONB,\n  created_at          TIMESTAMPTZ, updated_at TIMESTAMPTZ\n);\n-- versioned via ai_app_version (immutable manifest hash + parent pointer)\n\n\nAgent\n\nCREATE TABLE agent (\n  agent_id          UUID PRIMARY KEY,\n  app_id            UUID,                   -- nullable: agent can also live at schema scope\n  schema_id         UUID,\n  name              TEXT NOT NULL,\n  description       TEXT,                   -- shown to orchestrator (n8n / Crew / AutoGen distinguish from system_prompt)\n  default_version_id UUID,\n  owner_id          UUID NOT NULL,\n  tags              JSONB\n);\n\nCREATE TABLE agent_version (\n  version_id        UUID PRIMARY KEY,\n  agent_id          UUID NOT NULL REFERENCES agent ON DELETE CASCADE,\n  semver            TEXT NOT NULL,           -- 1.4.0\n  manifest_hash     BYTEA NOT NULL,          -- sha256(canonical_json(spec))\n  parent_version_id UUID,\n  status            TEXT NOT NULL,           -- DRAFT, STAGED, PUBLISHED, RETIRED\n  -- runtime spec\n  system_prompt_ref UUID,                    -- \u2192 prompt_version\n  model_endpoint_id UUID NOT NULL,\n  tools             UUID[],                  -- \u2192 tool ids (via tool_binding)\n  memory_config     JSONB,                   -- short_term, long_term, entity, episodic, semantic toggles + ttls\n  exit_conditions   JSONB,                   -- max_iters, target_state predicates\n  retries           JSONB,\n  reasoning_effort  TEXT,                    -- low/med/high (for thinking models)\n  output_schema     JSONB,                   -- structured output enforcement\n  guardrail_ids     UUID[],\n  created_by        UUID,\n  created_at        TIMESTAMPTZ\n);\n\nCREATE TABLE deployment_slot (\n  slot_id      UUID PRIMARY KEY,\n  app_id       UUID,\n  agent_id     UUID,\n  workflow_id  UUID,\n  slot_name    TEXT NOT NULL,                -- production, staging, canary, shadow\n  version_id   UUID NOT NULL,\n  traffic_pct  REAL NOT NULL DEFAULT 100,\n  promoted_at  TIMESTAMPTZ,\n  promoted_by  UUID,\n  rollback_of  UUID,\n  CHECK ((app_id IS NOT NULL)::int + (agent_id IS NOT NULL)::int + (workflow_id IS NOT NULL)::int = 1)\n);\n\n\nWorkflow + WorkflowNode + WorkflowRun\n\nCREATE TABLE workflow (\n  workflow_id        UUID PRIMARY KEY,\n  app_id             UUID, schema_id UUID,\n  name               TEXT NOT NULL,\n  default_version_id UUID,\n  trigger_refs       UUID[],\n  owner_id           UUID NOT NULL,\n  tags               JSONB\n);\n\nCREATE TABLE workflow_version (\n  version_id     UUID PRIMARY KEY,\n  workflow_id    UUID NOT NULL,\n  semver         TEXT NOT NULL,\n  graph_dsl      JSONB NOT NULL,           -- nodes + edges + state schema\n  schema_version TEXT NOT NULL,            -- \"lakeflow.workflow.v1\"\n  manifest_hash  BYTEA NOT NULL,\n  status         TEXT NOT NULL,\n  created_at     TIMESTAMPTZ\n);\n\nCREATE TABLE workflow_node (\n  node_id        UUID PRIMARY KEY,\n  workflow_version_id UUID NOT NULL,\n  position       INT NOT NULL,\n  node_type      TEXT NOT NULL,            -- START, END, ANSWER, LLM, CODE, KNOWLEDGE_RETRIEVAL,\n                                           -- QUESTION_CLASSIFIER, IF_ELSE, ITERATION, LOOP, HTTP,\n                                           -- TOOL, VARIABLE_AGGREGATOR, TEMPLATE_TRANSFORM,\n                                           -- PARAMETER_EXTRACTOR, LIST_OPERATOR, AGENT_SUB\n  config         JSONB NOT NULL,\n  retry_policy   JSONB\n);\n\nCREATE TABLE workflow_run (\n  run_id              UUID PRIMARY KEY,\n  workflow_version_id UUID NOT NULL,\n  thread_id           UUID,                 -- LangGraph-style thread\n  app_id              UUID,\n  status              TEXT NOT NULL,        -- QUEUED, RUNNING, INTERRUPTED, SUCCEEDED, FAILED, CANCELED\n  started_at          TIMESTAMPTZ NOT NULL,\n  finished_at         TIMESTAMPTZ,\n  trigger_ref         UUID,\n  inputs              JSONB,\n  outputs             JSONB,\n  error               JSONB,\n  triggered_by        UUID,\n  total_tokens        BIGINT,               -- denormalised from spans\n  total_cost_usd      NUMERIC(12,6)\n);\n\nCREATE TABLE workflow_node_execution (\n  exec_id        UUID PRIMARY KEY,\n  run_id         UUID NOT NULL REFERENCES workflow_run ON DELETE CASCADE,\n  node_id        UUID NOT NULL,\n  status         TEXT NOT NULL,\n  inputs         JSONB,\n  outputs        JSONB,\n  elapsed_ms     INT,\n  tokens_in      INT, tokens_out INT,\n  error          JSONB,\n  parent_exec_id UUID,\n  span_id        BYTEA                       -- \u2192 trace_span\n);\n\n-- LangGraph-style checkpoint chain for resumability\nCREATE TABLE checkpoint (\n  checkpoint_id UUID PRIMARY KEY,\n  thread_id     UUID NOT NULL,\n  parent_id     UUID,\n  state         JSONB NOT NULL,\n  next_nodes    TEXT[],\n  ts            TIMESTAMPTZ NOT NULL\n);\nCREATE INDEX checkpoint_thread ON checkpoint (thread_id, ts);\n\n\nConversation + Message\n\nCREATE TABLE conversation (\n  conversation_id UUID PRIMARY KEY,\n  app_id          UUID NOT NULL,\n  end_user_id     UUID,                       -- \u2192 end_user\n  thread_id       UUID,                       -- shared with workflow_run.thread_id when applicable\n  started_at      TIMESTAMPTZ NOT NULL,\n  last_message_at TIMESTAMPTZ,\n  summary         TEXT,\n  metadata        JSONB\n);\n\nCREATE TABLE message (\n  message_id    UUID PRIMARY KEY,\n  conversation_id UUID NOT NULL REFERENCES conversation ON DELETE CASCADE,\n  role          TEXT NOT NULL,                -- USER, ASSISTANT, SYSTEM, TOOL\n  content       JSONB NOT NULL,               -- multimodal: [{type:text|image|audio|file|tool_use|tool_result, \u2026}]\n  parent_id     UUID,\n  tool_calls    JSONB,\n  finish_reason TEXT,\n  -- usage\n  model_endpoint_id UUID,\n  tokens_in     INT,\n  tokens_out    INT,\n  cache_read_tokens INT,\n  reasoning_tokens INT,\n  cost_usd      NUMERIC(12,6),\n  latency_ms    INT,\n  -- audit / quality\n  span_id       BYTEA,                        -- \u2192 trace_span (root of message turn)\n  feedback      JSONB,                        -- \ud83d\udc4d/\ud83d\udc4e + reason\n  redaction_applied BOOLEAN NOT NULL DEFAULT FALSE,\n  created_at    TIMESTAMPTZ NOT NULL\n);\nCREATE INDEX message_conv_time ON message (conversation_id, created_at);\n\nCREATE TABLE message_attachment (\n  attachment_id UUID PRIMARY KEY,\n  message_id    UUID NOT NULL REFERENCES message ON DELETE CASCADE,\n  storage_uri   TEXT NOT NULL,\n  mime          TEXT, size BIGINT,\n  role          TEXT,                         -- INPUT, OUTPUT\n  scan_status   TEXT,                         -- PENDING, CLEAN, INFECTED, BLOCKED\n  scan_result   JSONB\n);\n\nCREATE TABLE annotation (\n  annotation_id UUID PRIMARY KEY,\n  target_kind   TEXT NOT NULL,                -- MESSAGE, CONVERSATION, TRACE_SPAN\n  target_id     UUID NOT NULL,\n  label         TEXT,\n  score         REAL,\n  rationale     TEXT,\n  author_id     UUID,\n  created_at    TIMESTAMPTZ\n);\n\n\nEndUser, Variable, SecretReference\n\n\nObjectKey attrs\nEndUserend_user_id, app_id, external_id, attributes jsonb (email, locale, plan), last_seen_at, ban_reason\nVariablevar_id, scope (WORKSPACE/APP/CONVERSATION/USER), key, value (jsonb), is_sensitive bool, ttl_seconds?\nSecretReferenceref_id, app_id, secret_store_uri, key, scope, allowed_consumers[] \u2014 never leaks the value\nTrigger / Webhooksee \u00a716 (shared shape)\n\n\n\nSpecifically reserved (not in v1 of most platforms but mandatory for production)\n\n\nObjectWhy first-classKey attrs\nGuardrailPrompt-injection / output-toxicity / PII / jailbreak detection bound pre+post callid, name, kind(PROMPT_INJECTION,OUTPUT_FILTER,CONTENT_CLASSIFIER,PII,TOXICITY), config jsonb, action(DENY,REDACT,QUARANTINE,LOG_ONLY), priority int\nRateLimitPolicyper app / agent / endpoint / end-userid, qps, rpm, tpm, max_concurrent_runs, scope\nCostBudget (agent-scope)monthly cap with hard-stop at app levelid, period, amount, currency, hard_stop bool, owners[]\nApprovalPolicyHITL gate on tool calls matching predicateid, predicate(cedar), approvers[], timeout, on_timeout(DENY/ALLOW)\nDataResidencyPolicyregion pinning of conv/message storageid, allowed_regions[], blocked_regions[], applies_to_kinds[]\nPIIRedactorpre-storage scrubber on Message/TraceSpan contentid, patterns jsonb, mode(REVERSIBLE,IRREVERSIBLE), key_ref\nToolUsageQuotaper (agent, tool) \u2014 prevents SerpAPI runaway(agent_id, tool_id, daily_limit, monthly_limit)\nMemoryStoretyped memory: short-term thread / long-term namespace / entity / episodic / semanticid, kind, namespace, owner_principal, ttl_seconds, retention_policy_id\nRetrievalCitation(message_id, kb_chunk_id, score) \u2014 RAG provenance\u2014\nEvalGaterequired-passing metric thresholds bound to (Agent/Workflow)Version \u2192 DeploymentSlot promotionid, criteria jsonb (metric, op, threshold), required_pass bool\n\n\n\n\n24 \u00b7 MCP, Tools, Knowledge Base\n\n\nTool unification \u2014 MCP is canonical\n\nCREATE TABLE tool (\n  tool_id     UUID PRIMARY KEY,\n  schema_id   UUID,                            -- UC-scoped\n  name        TEXT NOT NULL,\n  description TEXT,\n  input_schema  JSONB NOT NULL,                -- JSONSchema\n  output_schema JSONB,\n  provider_kind TEXT NOT NULL,                 -- BUILTIN, OPENAPI, MCP, WORKFLOW, FUNCTION\n  endpoint_ref  JSONB NOT NULL,                -- discriminated union\n  -- BUILTIN: { kind:'builtin', registry_id }\n  -- OPENAPI: { kind:'openapi', spec_url, base_url, auth_ref }\n  -- MCP    : { kind:'mcp', server_id, exposed_tool_name }\n  -- WORKFLOW: { kind:'workflow', workflow_version_id }\n  -- FUNCTION: { kind:'function', function_id }   -- UC SQL/Python UDF\n  cost_metadata JSONB,                         -- per-invocation cost hint\n  owner_id    UUID,\n  tags        JSONB\n);\n\nCREATE TABLE tool_binding (\n  binding_id UUID PRIMARY KEY,\n  parent_kind TEXT NOT NULL,                    -- AGENT, WORKFLOW, AIAPP\n  parent_id   UUID NOT NULL,\n  tool_id     UUID NOT NULL REFERENCES tool,\n  alias       TEXT,\n  allow_override BOOLEAN,\n  credential_ref UUID,                          -- per-binding override\n  enabled     BOOLEAN NOT NULL DEFAULT TRUE\n);\n\nCREATE TABLE mcp_server (\n  server_id   UUID PRIMARY KEY,\n  schema_id   UUID,\n  name        TEXT NOT NULL,\n  transport   TEXT NOT NULL,                    -- STDIO, HTTP, SSE, STREAMABLE_HTTP\n  endpoint    TEXT NOT NULL,\n  auth        JSONB,                            -- OAuth, API-key, mTLS\n  exposed_tools     JSONB,                      -- list discovered at connect\n  exposed_resources JSONB,                      -- MCP \"Resources\"\n  exposed_prompts   JSONB,                      -- MCP \"Prompts\"\n  server_instructions TEXT,                     -- \"Skills\" instructions\n  health_status TEXT,\n  last_handshake_at TIMESTAMPTZ,\n  owner_id    UUID\n);\n\n\nKnowledge Base / RAG\n\nCREATE TABLE knowledge_base (\n  kb_id              UUID PRIMARY KEY,\n  schema_id          UUID,\n  name               TEXT NOT NULL,\n  embedding_endpoint_id UUID NOT NULL,\n  retrieval_strategy TEXT NOT NULL,             -- VECTOR, FULLTEXT, HYBRID, KEYWORD\n  rerank_endpoint_id UUID,\n  indexing_strategy  TEXT NOT NULL,             -- HIGH_QUALITY, ECONOMY (Dify-style)\n  vector_index_id    UUID,                      -- \u2192 vector_index\n  chunk_size         INT, chunk_overlap INT,\n  chunk_strategy     TEXT,                      -- RECURSIVE, SENTENCE, SEMANTIC, MARKDOWN_HEADER\n  metadata_fields    JSONB,                     -- user-extensible filterable metadata schema\n  owner_id           UUID\n);\n\nCREATE TABLE kb_document (\n  doc_id        UUID PRIMARY KEY,\n  kb_id         UUID NOT NULL REFERENCES knowledge_base ON DELETE CASCADE,\n  source_uri    TEXT NOT NULL,                  -- s3://, https://, notion://, gdrive://\n  source_type   TEXT NOT NULL,                  -- FILE, WEB, NOTION, GDRIVE, CONFLUENCE, S3, JIRA, SLACK\n  mime          TEXT, size_bytes BIGINT,\n  status        TEXT NOT NULL,                  -- QUEUED, PARSING, EMBEDDING, READY, FAILED\n  metadata      JSONB,\n  uploader_id   UUID,\n  version       INT NOT NULL DEFAULT 1,\n  parent_doc_id UUID,                            -- for re-uploads\n  parsing_engine_used TEXT,                      -- builtin, unstructured, llm-parse\n  page_count    INT\n);\n\nCREATE TABLE kb_chunk (\n  chunk_id      UUID PRIMARY KEY,\n  doc_id        UUID NOT NULL REFERENCES kb_document ON DELETE CASCADE,\n  position      INT NOT NULL,\n  parent_chunk_id UUID,                          -- parent-child chunking\n  content       TEXT NOT NULL,\n  embedding_ref TEXT,                            -- vector index pk\n  metadata      JSONB,\n  score_floor   REAL,                            -- per-chunk min relevance threshold\n  hash          BYTEA NOT NULL                   -- dedup\n);\nCREATE INDEX kb_chunk_doc ON kb_chunk (doc_id, position);\n\n\nPrompts & Models\n\nCREATE TABLE prompt_template (\n  template_id     UUID PRIMARY KEY,\n  schema_id       UUID,\n  name            TEXT NOT NULL,\n  description     TEXT,\n  default_version_id UUID,\n  owner_id        UUID\n);\n\nCREATE TABLE prompt_version (\n  version_id           UUID PRIMARY KEY,\n  template_id          UUID NOT NULL REFERENCES prompt_template ON DELETE CASCADE,\n  body                 TEXT NOT NULL,             -- with {{vars}}\n  body_hash            BYTEA NOT NULL,            -- sha256\n  semver               TEXT NOT NULL,\n  input_schema         JSONB,\n  output_schema        JSONB,\n  model_compatibility  TEXT[],                    -- claude-*, gpt-*, llama-*\n  status               TEXT NOT NULL,\n  created_by           UUID, created_at TIMESTAMPTZ\n);\n\nCREATE TABLE model_endpoint (\n  endpoint_id     UUID PRIMARY KEY,\n  workspace_id    UUID,\n  alias           TEXT NOT NULL,\n  provider        TEXT NOT NULL,                  -- DATABRICKS, OPENAI, ANTHROPIC, BEDROCK, AZURE, GOOGLE, LOCAL\n  model_id        TEXT NOT NULL,                  -- claude-opus-4-7, gpt-4o, llama-3.3-70b\n  base_url        TEXT,\n  auth_ref        UUID,                           -- secret reference\n  modality        TEXT[],                         -- CHAT, EMBED, RERANK, STT, TTS, MODERATION, IMAGE\n  context_window  INT,\n  max_output_tokens INT,\n  default_params  JSONB,                          -- temperature, top_p, max_tokens\n  rate_limit_policy_id UUID,\n  cost_per_1m_in  NUMERIC(10,4),                  -- denormalised from rate_card for fast lookup\n  cost_per_1m_out NUMERIC(10,4),\n  capabilities    TEXT[]                          -- tool_use, json_mode, structured_output, reasoning, vision\n);\n\n\n\n25 \u00b7 Guardrails & Eval\n\n\nGuardrail (full schema)\n\nCREATE TABLE guardrail (\n  guardrail_id  UUID PRIMARY KEY,\n  workspace_id  UUID,\n  name          TEXT NOT NULL,\n  kind          TEXT NOT NULL,\n  -- Kinds:\n  --   PROMPT_INJECTION   - heuristic + judge LLM\n  --   OUTPUT_FILTER      - block patterns\n  --   CONTENT_CLASSIFIER - hate/violence/sexual/self-harm\n  --   PII                - regex + NER\n  --   TOXICITY           - judge LLM\n  --   JAILBREAK          - prompt-pattern catalog\n  --   FACTUALITY         - judge LLM (does answer match retrieved sources)\n  --   COST_RUNAWAY       - tokens/run threshold\n  --   TOOL_ALLOWLIST     - whitelist of tools\n  config        JSONB NOT NULL,\n  action        TEXT NOT NULL,         -- DENY, REDACT, QUARANTINE, LOG_ONLY, ESCALATE_HUMAN\n  priority      INT NOT NULL DEFAULT 100,\n  applies_to    TEXT NOT NULL,         -- INPUT, OUTPUT, BOTH\n  enabled       BOOLEAN NOT NULL DEFAULT TRUE,\n  owner_id      UUID\n);\n\n\nEval framework\n\nCREATE TABLE eval_dataset (\n  dataset_id   UUID PRIMARY KEY,\n  schema_id    UUID,\n  name         TEXT NOT NULL,\n  version      INT NOT NULL,\n  schema       JSONB NOT NULL,                  -- {input_field, expected_field, \u2026}\n  source       TEXT NOT NULL,                   -- GOLD, PRODUCTION_SAMPLED, SYNTHETIC\n  example_count INT NOT NULL,\n  owner_id     UUID, created_at TIMESTAMPTZ,\n  UNIQUE (schema_id, name, version)\n);\n\nCREATE TABLE eval_run (\n  run_id        UUID PRIMARY KEY,\n  dataset_id    UUID NOT NULL REFERENCES eval_dataset,\n  target_kind   TEXT NOT NULL,                   -- AGENT_VERSION, WORKFLOW_VERSION, PROMPT_VERSION\n  target_id     UUID NOT NULL,\n  judge_model_endpoint_id UUID,\n  judge_prompt_version_id UUID,\n  status        TEXT NOT NULL,                   -- RUNNING, SUCCEEDED, FAILED\n  started_at    TIMESTAMPTZ NOT NULL,\n  finished_at   TIMESTAMPTZ,\n  -- Aggregate metrics\n  pass_rate     REAL, mean_cost_usd NUMERIC(12,6),\n  p50_latency_ms INT, p95_latency_ms INT,\n  triggered_by  UUID\n);\n\nCREATE TABLE eval_score (\n  score_id        UUID PRIMARY KEY,\n  run_id          UUID NOT NULL REFERENCES eval_run ON DELETE CASCADE,\n  example_id      TEXT NOT NULL,\n  criterion       TEXT NOT NULL,                 -- correctness, faithfulness, relevance, toxicity,\n                                                 -- pii_leak, jailbreak_resistance, latency_p95, cost_per_run\n  score           REAL,\n  passed          BOOLEAN,\n  judge_rationale TEXT,\n  trace_id        UUID,                          -- \u2192 trace_span\n  human_override  BOOLEAN, reviewer_id UUID, reviewed_at TIMESTAMPTZ\n);\n\nCREATE TABLE eval_gate (\n  gate_id      UUID PRIMARY KEY,\n  target_kind  TEXT NOT NULL,\n  target_id    UUID NOT NULL,                    -- (Agent/Workflow)Version; required to pass before promotion\n  criteria     JSONB NOT NULL,                   -- [{metric,op,threshold}, \u2026]\n  required_pass BOOLEAN NOT NULL DEFAULT TRUE\n);\n\n\n\n26 \u00b7 Privilege vocabulary (38 verbs + macros)\n\n\nCategoryVerbs\nVisibilityBROWSE, BROWSE_METADATA\nRead/WriteSELECT, MODIFY, READ_VOLUME, WRITE_VOLUME, READ_FILES, WRITE_FILES\nExecuteEXECUTE, USE, USE_CATALOG, USE_SCHEMA, REFRESH\nFederation/SharingUSE_CONNECTION, USE_PROVIDER, USE_RECIPIENT, USE_SHARE, USE_MARKETPLACE_ASSETS\nExternal enginesEXTERNAL_USE_SCHEMA, EXTERNAL_USE_LOCATION\nTags / ABACASSIGN, CREATE_TAG, MANAGE_TAG\nCreateCREATE_CATALOG, CREATE_SCHEMA, CREATE_TABLE, CREATE_FUNCTION, CREATE_VOLUME, CREATE_MODEL, CREATE_MODEL_VERSION, CREATE_FOREIGN_CATALOG, CREATE_EXTERNAL_METADATA\nIdentityMANAGE_USER, MANAGE_GROUP, MANAGE_SP, MANAGE_PAT\nTop-levelMANAGE, MANAGE_ACCOUNT, MANAGE_WORKSPACE\nAI / AgentEXECUTE_AGENT, INVOKE_TOOL, USE_TOOL, USE_MCP, USE_KB, READ_TRACE, EXECUTE_EVAL, READ_EVAL, PROMOTE (slot promotion)\nCost / BillingVIEW_USAGE, MANAGE_BUDGET, MANAGE_RATE_CARD, MANAGE_DISCOUNT, VIEW_INVOICE, BILLING_ADMIN\nMacrosALL_PRIVILEGES (excludes EXTERNAL_USE_*, MANAGE_PAT, MANAGE_ACCOUNT)\n\n\n\nPermission-level \u2192 privilege mapping\n\n\nObjectLegacy permission levelUnified privilege\nClusterCAN_ATTACH_TO \u00b7 CAN_RESTART \u00b7 CAN_MANAGEUSE \u00b7 EXECUTE \u00b7 MANAGE\nJobCAN_VIEW \u00b7 CAN_MANAGE_RUN \u00b7 IS_OWNER \u00b7 CAN_MANAGEBROWSE \u00b7 EXECUTE \u00b7 (owner column) \u00b7 MANAGE\nPipeline (DLT)CAN_VIEW \u00b7 CAN_RUN \u00b7 CAN_MANAGE \u00b7 IS_OWNERBROWSE \u00b7 EXECUTE \u00b7 MANAGE \u00b7 (owner)\nNotebook/FileCAN_READ \u00b7 CAN_RUN \u00b7 CAN_EDIT \u00b7 CAN_MANAGEBROWSE \u00b7 EXECUTE \u00b7 MODIFY \u00b7 MANAGE\nFolder+ CAN_VIEW_METADATA+ BROWSE_METADATA\nServingEndpointCAN_QUERY \u00b7 CAN_MANAGEEXECUTE \u00b7 MANAGE\nSecretScopeREAD \u00b7 WRITE \u00b7 MANAGEBROWSE \u00b7 MODIFY \u00b7 MANAGE\nSQLWarehouseCAN_USE \u00b7 CAN_MANAGEUSE \u00b7 MANAGE\nAgentCAN_VIEW \u00b7 CAN_INVOKE \u00b7 CAN_EDIT \u00b7 CAN_MANAGE \u00b7 IS_OWNERBROWSE \u00b7 EXECUTE_AGENT \u00b7 MODIFY \u00b7 MANAGE \u00b7 (owner)\nMCPServerCAN_USE \u00b7 CAN_MANAGEUSE_MCP \u00b7 MANAGE\nKnowledgeBaseCAN_READ \u00b7 CAN_WRITE \u00b7 CAN_MANAGEUSE_KB \u00b7 MODIFY \u00b7 MANAGE\n\n\n\n\n27 \u00b7 Cascade rules \u2014 extract\n\nFull table is data-driven (securable_cascade); see DDL \u00a741. Highlights below.\n\n\n\nparent_typechild_typeprivileges flowing down\nAccountWorkspace \u00b7 MetastoreMANAGE \u00b7 BROWSE\nWorkspaceCluster \u00b7 Network \u00b7 Job \u00b7 Flow \u00b7 Pipeline \u00b7 Notebook \u00b7 Dashboard \u00b7 Directory \u00b7 ServingEndpoint \u00b7 SQLWarehouse \u00b7 AIApp \u00b7 MCPServerMANAGE \u00b7 BROWSE\nMetastoreCatalog \u00b7 StorageCredential \u00b7 ExternalLocation \u00b7 Connection \u00b7 Share \u00b7 Provider \u00b7 Recipient \u00b7 CleanRoom \u00b7 MarketplaceListing \u00b7 GovernedTag \u00b7 DatabaseInstance \u00b7 ExternalMetadata \u00b7 SecretScopeMANAGE \u00b7 BROWSE\nCatalogSchemaUSE_CATALOG \u00b7 BROWSE \u00b7 MANAGE\nSchemaTable \u00b7 StreamingTable \u00b7 View \u00b7 MV \u00b7 Volume \u00b7 Function \u00b7 Procedure \u00b7 Model \u00b7 VectorIndex \u00b7 FeatureTable \u00b7 OnlineTableUSE_SCHEMA \u00b7 BROWSE \u00b7 MANAGE (privileges per-type)\nJob \u00b7 Flow \u00b7 PipelineRun / UpdateBROWSE \u00b7 EXECUTE \u00b7 MANAGE (no SELECT \u2014 don't leak through logs)\nNotebook \u00b7 Dashboard \u00b7 AIApp \u00b7 Workflow \u00b7 Agentchildren (cell, dataset, version, conv, etc.)MANAGE \u00b7 BROWSE\nAIAppAgent \u00b7 Workflow \u00b7 Conversation \u00b7 TriggerMANAGE \u00b7 BROWSE\nAgentAgentVersion \u00b7 ConversationMANAGE \u00b7 BROWSE \u00b7 EXECUTE_AGENT\nConversationMessage \u00b7 MessageAttachment \u00b7 AnnotationBROWSE \u00b7 MODIFY (own)\nKnowledgeBaseKBDocument \u00b7 KBChunkUSE_KB \u00b7 MODIFY \u00b7 MANAGE\nMCPServer(exposed Tool refs)USE_MCP \u00b7 USE_TOOL\nSecretScopeSecretBROWSE \u00b7 SELECT \u00b7 MODIFY \u00b7 MANAGE\nUserPATMANAGE_PAT\n\n\n\n\n28 \u00b7 Built-in roles\n\n\nRoleScopePrivileges granted\naccount_adminAccountMANAGE_ACCOUNT (gates control plane only \u2014 does NOT auto-grant data-plane reads)\nbilling_adminAccountMANAGE_RATE_CARD, MANAGE_DISCOUNT, VIEW_INVOICE, MANAGE_BUDGET, VIEW_USAGE (all)\nbilling_readerAccount / WorkspaceVIEW_USAGE (filtered to own workspaces), VIEW_INVOICE\nworkspace_adminWorkspaceMANAGE_WORKSPACE + MANAGE on every securable in workspace\nmetastore_adminMetastoreMANAGE on Metastore (cascades to UC objects)\ncost_center_owner(custom group)MANAGE on Budgets matching their tag; VIEW_USAGE for their tag\nidentity_adminAccountMANAGE_USER, MANAGE_GROUP, MANAGE_SP\nauditorAccountBROWSE on audit_event, lineage_edge, trace_span (read-only); legal-hold permissions\nagent_developer(custom group)EXECUTE_AGENT, USE_TOOL, USE_KB, MODIFY on draft AgentVersions\ndata_engineer(custom group)USE_CATALOG, CREATE_SCHEMA, CREATE_TABLE, MODIFY in target schemas\n\n\n\n\n29 \u00b7 Resolver \u2014 single SQL CTE, 4 passes\n\nWITH RECURSIVE\neffective_principals AS (\n    SELECT $user_id::uuid AS id\n  UNION\n    SELECT pm.parent_id FROM principal_membership pm\n      JOIN effective_principals ep ON pm.member_id = ep.id\n),\nancestors AS (\n    SELECT $sec_id::uuid AS id, 0 AS depth\n  UNION ALL\n    SELECT s.parent_id, a.depth + 1\n      FROM securable s JOIN ancestors a ON s.id = a.id\n     WHERE s.parent_id IS NOT NULL\n)\nSELECT\n  -- Pass 1 \u2014 explicit DENY anywhere\n  EXISTS (SELECT 1 FROM grant g\n            JOIN ancestors a ON g.securable_id = a.id\n            JOIN effective_principals ep ON g.principal_id = ep.id\n           WHERE g.account_id=$acct AND g.privilege=$priv AND g.effect='DENY'\n             AND g.revoked_at IS NULL) AS forbid,\n  -- Pass 2 \u2014 control-plane admin (does NOT auto-cascade to data plane)\n  EXISTS (SELECT 1 FROM grant g\n            JOIN effective_principals ep ON g.principal_id = ep.id\n           WHERE g.account_id=$acct AND g.privilege='MANAGE_ACCOUNT'\n             AND g.effect='ALLOW' AND g.revoked_at IS NULL\n             AND $priv NOT IN ('SELECT','MODIFY','EXECUTE')) AS admin,\n  -- Pass 3 \u2014 owner column\n  EXISTS (SELECT 1 FROM securable s\n            JOIN effective_principals ep\n              ON ep.id IN (s.owner_id, s.pending_owner_id)\n           WHERE s.id=$sec_id) AS owner,\n  -- Pass 4 \u2014 explicit ALLOW with cascade-rule check\n  EXISTS (SELECT 1 FROM grant g\n            JOIN ancestors a ON g.securable_id=a.id\n            JOIN effective_principals ep ON g.principal_id=ep.id\n            JOIN securable child  ON child.id=$sec_id\n            JOIN securable parent ON parent.id=a.id\n       LEFT JOIN securable_cascade c\n              ON c.parent_type=parent.type AND c.child_type=child.type\n           WHERE g.account_id=$acct\n             AND g.privilege=$priv AND g.effect='ALLOW'\n             AND g.revoked_at IS NULL\n             AND g.valid_from<=now()\n             AND (g.expires_at IS NULL OR g.expires_at>now())\n             AND (a.depth=0 OR $priv = ANY(c.privileges))) AS permit;\n-- decision = NOT forbid AND (admin OR owner OR permit)\n-- if grant.condition_cedar IS NOT NULL \u2192 eval Cedar with (principal,resource,context)\n--   parse/eval error \u2192 DENY (fail-closed)\n\n\n\n30 \u00b7 Anti-escalation rules\n\n\n#RuleWhy\n1Grantor-holds-privilegeStop \"MANAGE on Catalog\" laundering into SELECT on every table\n2Two-phase ownership transfer (pending_owner_id, 7-day accept window)No owner-orphaning\n3DENY walks full ancestor chainCedar CVE-2024-25624 class\n4No self-grant of MANAGE_ACCOUNT (BEFORE INSERT trigger)Casbin CVE-2023-26485 class\n5Token-exchange chain depth \u2264 3OBO laundering\n6MANAGE_ACCOUNT does NOT auto-cascade to data plane (SELECT/MODIFY/EXECUTE)Account admins shouldn't silently see all data\n7ALL_PRIVILEGES excludes EXTERNAL_USE_*, MANAGE_PAT, MANAGE_ACCOUNTDatabricks-spec compliance\n8Grantor must hold both MANAGE on target AND the privilege being grantedPostgres semantics\n9Tool calls matching ApprovalPolicy predicate require HITL approvalHigh-blast-radius tool runaway\n10Cross-account principal cannot grant on intra-account securablesTenancy isolation\n\n\n\n\n31 \u00b7 Cost model architecture\n\n\nThree-layer pricing.\n1. RateCard \u2014 slowly changing rate per (sku, cloud, region, tier, time).\n2. BillableUsage \u2014 append-only ledger; 1 row per metering increment per resource. Computes list_cost = quantity \u00d7 unit_price.\n3. Effective \u2014 apply committed_use drawdown then discount_policy to produce effective_cost; aggregated into invoice at month-close.\nBudgets and alerts query the ledger; quotas enforce pre-flight; tags route attribution.\n\n\n\n\n32 \u00b7 SKU catalog (illustrative)\n\n\nFamilySKUUnitStandardPremiumEnterpriseNotes\nCompute (Classic)JOBS_COMPUTEDBU/hr0.100.150.20+Photon multiplier per node\nALL_PURPOSE_COMPUTEDBU/hr0.400.550.65interactive\nSQL_PRODBU/hr\u20140.550.65Premium+\nCompute (Serverless)JOBS_SERVERLESSDBU/hr\u20140.350.40bundled compute\nSQL_SERVERLESSDBU/hr\u20140.700.85\u2014\nNOTEBOOKS_SERVERLESSDBU/hr\u20140.400.50\u2014\nDLTDLT_COREDBU/hr0.200.200.20\u2014\nDLT_PRODBU/hr\u20140.250.25\u2014\nDLT_ADVANCEDDBU/hr\u2014\u20140.36data quality\nAIMODEL_SERVING_CPUDBU/hr\u20140.070.07concurrency \u00d7 uptime\nMODEL_SERVING_GPUDBU/hr\u20140.07-0.17\u2014GPU class dep\nVECTOR_SEARCH_VSUVSU/hr\u20140.350.35+ query units\nAI tokensFOUNDATION_MODEL_INPUTper 1M tokensper-model rate\u2014\nFOUNDATION_MODEL_OUTPUTper 1M tokens\u2014\u2014\nAGENT_EVAL_ROWper row\u20140.01-0.03\u2014judge LLM\nStorage / OLTPLAKEBASE_VCPUvCPU-hr\u2014per-class\u2014+ GB-month + IOPS\nONLINE_TABLE_GBGB-month\u2014per region\u2014+ requests\nVECTOR_INDEX_GBGB-month\u2014per class\u2014\u2014\nAppsAPPS_DBUDBU/hr\u20140.05\u2014compute-backed runtime\nNetworkEGRESS_GB / PRIVATELINK_GBGBcloud passthrough \u2014 not platform-priced\u2014\n\n\n\n\n33 \u00b7 Billing tables (system)\n\n\nRateCard\n\nCREATE TABLE rate_card (\n  rate_card_id     UUID PRIMARY KEY,\n  account_id       UUID NOT NULL,\n  sku_name         TEXT NOT NULL,\n  cloud            TEXT NOT NULL,\n  region           TEXT NOT NULL,\n  tier             TEXT NOT NULL,                 -- STANDARD/PREMIUM/ENTERPRISE\n  currency         TEXT NOT NULL DEFAULT 'USD',\n  unit             TEXT NOT NULL,                 -- DBU, TOKEN_IN, TOKEN_OUT, GB_HOUR, ROW, VSU_HOUR\n  unit_price       NUMERIC(20,10) NOT NULL,\n  photon_multiplier NUMERIC(8,4),\n  effective_from   TIMESTAMPTZ NOT NULL,\n  effective_to     TIMESTAMPTZ,                    -- NULL = current\n  source           TEXT,                            -- list, contract, custom\n  created_by       UUID, created_at TIMESTAMPTZ,\n  EXCLUDE USING gist (sku_name WITH =, cloud WITH =, region WITH =, tier WITH =,\n                      tstzrange(effective_from, effective_to) WITH &&)\n);\n\n\nBillableUsage\n\nCREATE TABLE billable_usage (\n  record_id          UUID PRIMARY KEY,             -- idempotency key\n  account_id         UUID NOT NULL,\n  workspace_id       UUID,\n  sku_name           TEXT NOT NULL,\n  cloud              TEXT NOT NULL,\n  usage_start_time   TIMESTAMPTZ NOT NULL,\n  usage_end_time     TIMESTAMPTZ NOT NULL,\n  usage_date         DATE NOT NULL,\n  usage_unit         TEXT NOT NULL,\n  usage_quantity     NUMERIC(38,18) NOT NULL,\n  usage_type         TEXT NOT NULL,                 -- COMPUTE_TIME, STORAGE_SPACE, NETWORK_BYTES,\n                                                    -- API_CALLS, TOKEN, GPU_TIME, ROW\n  custom_tags        JSONB NOT NULL DEFAULT '{}',\n  usage_metadata     JSONB NOT NULL DEFAULT '{}',   -- cluster_id, job_id, run_id, warehouse_id,\n                                                    -- endpoint_name, agent_id, conv_id, kb_id, \u2026\n  identity_metadata  JSONB,                         -- run_as, owned_by, created_by\n  record_type        TEXT NOT NULL DEFAULT 'ORIGINAL', -- ORIGINAL, RETRACTION, RESTATEMENT\n  ingestion_date     DATE NOT NULL DEFAULT CURRENT_DATE,\n  billing_origin     TEXT NOT NULL,                 -- JOBS, MODEL_SERVING, VECTOR_SEARCH, APPS,\n                                                    -- LAKEBASE, DLT, SQL, INTERACTIVE, NOTEBOOKS,\n                                                    -- AGENT_EVAL, AGENT_RUNTIME, KB_INGEST, \u2026\n  product_features   JSONB,                         -- is_serverless, is_photon, serving_type, \u2026\n  list_price         NUMERIC(20,10),                -- denormalised from rate_card\n  list_cost          NUMERIC(20,6) GENERATED ALWAYS AS (usage_quantity * list_price) STORED,\n  effective_cost     NUMERIC(20,6),                 -- after commit + discount, computed at invoice close\n  PRIMARY KEY (record_id, ingestion_date)\n) PARTITION BY RANGE (usage_date);\nCREATE INDEX bu_account_date ON billable_usage (account_id, usage_date);\nCREATE INDEX bu_workspace_date ON billable_usage (workspace_id, usage_date);\nCREATE INDEX bu_tags ON billable_usage USING GIN (custom_tags);\nCREATE INDEX bu_meta_jobid ON billable_usage ((usage_metadata->>'job_id'));\n\n\nCommittedUse, DiscountPolicy, Invoice, ChargebackReport\n\nCREATE TABLE committed_use (\n  commit_id        UUID PRIMARY KEY,\n  account_id       UUID NOT NULL,\n  term_months      INT NOT NULL,\n  total_amount     NUMERIC(20,2) NOT NULL,\n  currency         TEXT NOT NULL DEFAULT 'USD',\n  applies_to_skus  TEXT[],                           -- NULL = all\n  discount_pct     NUMERIC(5,2) NOT NULL,\n  start_date       DATE NOT NULL,\n  end_date         DATE NOT NULL,\n  drawdown         NUMERIC(20,2) NOT NULL DEFAULT 0,\n  status           TEXT NOT NULL DEFAULT 'ACTIVE'    -- ACTIVE, EXHAUSTED, EXPIRED\n);\n\nCREATE TABLE discount_policy (\n  discount_id      UUID PRIMARY KEY,\n  account_id       UUID NOT NULL,\n  name             TEXT NOT NULL,\n  kind             TEXT NOT NULL,                    -- PERCENT, FIXED, TIERED\n  config           JSONB NOT NULL,\n  applies_to       JSONB,                            -- {sku_in:[\u2026], tag_match:{\u2026}, workspace_in:[\u2026]}\n  start_date       DATE NOT NULL, end_date DATE,\n  stackable        BOOLEAN NOT NULL DEFAULT FALSE\n);\n\nCREATE TABLE invoice (\n  invoice_id       UUID PRIMARY KEY,\n  account_id       UUID NOT NULL,\n  period_start     DATE NOT NULL, period_end DATE NOT NULL,\n  currency         TEXT NOT NULL DEFAULT 'USD',\n  subtotal         NUMERIC(20,2),\n  commit_drawdown  NUMERIC(20,2),\n  discount_total   NUMERIC(20,2),\n  tax              NUMERIC(20,2),\n  total            NUMERIC(20,2),\n  status           TEXT NOT NULL,                     -- DRAFT, ISSUED, PAID, VOID\n  issued_at        TIMESTAMPTZ, due_at TIMESTAMPTZ,\n  line_items       JSONB                              -- per (sku \u00d7 workspace \u00d7 tag) rollup\n);\n\nCREATE TABLE chargeback_report (\n  report_id        UUID PRIMARY KEY,\n  account_id       UUID NOT NULL,\n  period_start     DATE NOT NULL, period_end DATE NOT NULL,\n  group_by         TEXT[] NOT NULL,                   -- ['tag:cost_center', 'workspace_id']\n  filter           JSONB,\n  generated_at     TIMESTAMPTZ NOT NULL,\n  rows             JSONB                              -- aggregated rollups\n);\n\n\n\n34 \u00b7 Budgets, BudgetPolicy, Quotas, CostTags\n\n\nCREATE TABLE budget (\n  budget_id        UUID PRIMARY KEY,\n  account_id       UUID NOT NULL,\n  name             TEXT NOT NULL,\n  period           TEXT NOT NULL,                     -- DAILY, MONTHLY, QUARTERLY, ANNUAL\n  amount           NUMERIC(20,2) NOT NULL,\n  currency         TEXT NOT NULL DEFAULT 'USD',\n  filter           JSONB NOT NULL DEFAULT '{}',       -- {workspace_in, tag_match, sku_in, run_as}\n  hard_stop        BOOLEAN NOT NULL DEFAULT FALSE,    -- when true, blocks new workloads\n  start_date       DATE NOT NULL, end_date DATE,\n  owner_id         UUID, created_at TIMESTAMPTZ\n);\n\nCREATE TABLE budget_alert (\n  alert_id         UUID PRIMARY KEY,\n  budget_id        UUID NOT NULL REFERENCES budget ON DELETE CASCADE,\n  threshold_pct    NUMERIC(5,2) NOT NULL,\n  kind             TEXT NOT NULL,                     -- ACTUAL, FORECAST\n  channel          TEXT NOT NULL,                     -- EMAIL, SLACK, WEBHOOK, PAGERDUTY\n  recipients       JSONB,\n  last_fired_at    TIMESTAMPTZ\n);\n\nCREATE TABLE budget_policy (                             -- serverless cost-tagging mandate\n  policy_id        UUID PRIMARY KEY,\n  account_id       UUID NOT NULL,\n  name             TEXT NOT NULL,\n  custom_tags      JSONB NOT NULL,                     -- stamped onto every workload\n  binding_workspace_ids UUID[] DEFAULT '{}',\n  enforcement      TEXT NOT NULL DEFAULT 'WARN',       -- OFF, WARN, BLOCK\n  created_by       UUID\n);\n\nCREATE TABLE compute_policy (                            -- pre-flight cluster shape constraints\n  policy_id        UUID PRIMARY KEY,\n  workspace_id     UUID NOT NULL,\n  name             TEXT NOT NULL,\n  definition       JSONB NOT NULL,                     -- per-field constraints\n  policy_family_id UUID,\n  max_clusters_per_user INT\n);\n\nCREATE TABLE quota (\n  quota_id         UUID PRIMARY KEY,\n  scope_kind       TEXT NOT NULL,                     -- ACCOUNT, WORKSPACE, USER\n  scope_id         UUID NOT NULL,\n  resource_type    TEXT NOT NULL,                     -- CLUSTER, JOB_RUN, PAT, ENDPOINT, AGENT_RUN, \u2026\n  hard_limit       INT NOT NULL,\n  soft_limit       INT,\n  reset_period     TEXT,                              -- NEVER, DAILY, MONTHLY\n  current_value    INT NOT NULL DEFAULT 0\n);\n\nCREATE TABLE cost_tag (                                  -- merged into BillableUsage.custom_tags\n  resource_kind    TEXT NOT NULL,\n  resource_id      UUID NOT NULL,\n  key              TEXT NOT NULL,\n  value            TEXT NOT NULL,\n  source           TEXT NOT NULL,                     -- WORKSPACE, RESOURCE, POLICY\n  PRIMARY KEY (resource_kind, resource_id, key, source)\n);\n-- Merge order: POLICY > RESOURCE > WORKSPACE\n\n\n\n35 \u00b7 Pricing tiers \u2014 feature gates\n\n\nFeatureStandardPremiumEnterprise\nWorkspaces, Jobs, Notebooks, UC core\u2713\u2713\u2713\nRow filters / column masks (ABAC)\u2014\u2713\u2713\nAudit logs (system.access.audit)\u2014\u2713\u2713\nIP access lists, customer-managed keys\u2014\u2713\u2713\nSCIM, token mgmt API\u2014\u2713\u2713\nCompliance: HIPAA / FedRAMP / PCI / Compliance Security Profile\u2014\u2014\u2713\nEnhanced security monitoring\u2014\u2014\u2713\nMarketplace listing (provider)\u2014\u2713\u2713\nAI Gateway, Agent Eval\u2014\u2713\u2713\nMulti-region replication\u2014\u2014\u2713\nPer-DBU multiplier1.0\u00d71.4\u00d7 typical1.6\u00d7 typical\n\n\n\n\n36 \u00b7 Audit events\n\n\nCREATE TABLE audit_event (\n  event_id          UUID PRIMARY KEY,                    -- UUIDv7\n  event_time        TIMESTAMPTZ NOT NULL,\n  event_date        DATE NOT NULL,\n  account_id        UUID NOT NULL,\n  workspace_id      UUID,                                -- NULL for ACCOUNT_LEVEL\n  audit_level       TEXT NOT NULL CHECK (audit_level IN ('WORKSPACE_LEVEL','ACCOUNT_LEVEL')),\n  service_name      TEXT NOT NULL,                       -- accounts, scim, unityCatalog, clusters,\n                                                         -- jobs, databrickssql, secrets, mlflow,\n                                                         -- notebook, workspace, vectorSearch,\n                                                         -- modelServing, ipAccessControl, genie,\n                                                         -- aiAgent, billing, retention, governance\n  action_name       TEXT NOT NULL,                       -- see catalogue below\n  request_id        TEXT NOT NULL,\n  source_ip         INET,\n  user_agent        TEXT,\n  session_id        TEXT,\n  user_email        TEXT,\n  user_subject      TEXT,\n  run_by            TEXT,\n  run_as            TEXT,\n  request_params    JSONB NOT NULL DEFAULT '{}',\n  response_status   INT,\n  response_error    TEXT,\n  response_result   JSONB,\n  -- forensic\n  before_state      JSONB,\n  after_state       JSONB,\n  rationale         TEXT,                                -- which resolver pass matched\n  prev_hash         BYTEA NOT NULL,\n  entry_hash        BYTEA NOT NULL UNIQUE,               -- sha256(event_id || time || canonical_json || prev_hash)\n  schema_version    TEXT NOT NULL DEFAULT '2.0'\n) PARTITION BY RANGE (event_date);\nCREATE INDEX ae_account_time ON audit_event (account_id, event_time DESC);\nCREATE INDEX ae_workspace_time ON audit_event (workspace_id, event_time DESC) WHERE workspace_id IS NOT NULL;\nCREATE INDEX ae_actor ON audit_event (account_id, user_email, event_time DESC);\nCREATE INDEX ae_service_action ON audit_event (account_id, service_name, action_name, event_time DESC);\nCREATE INDEX ae_params ON audit_event USING GIN (request_params jsonb_path_ops);\n\n-- Append-only enforcement\nREVOKE UPDATE, DELETE ON audit_event FROM PUBLIC;\nCREATE TRIGGER audit_no_modify BEFORE UPDATE OR DELETE ON audit_event\n  FOR EACH ROW EXECUTE FUNCTION raise_exception('audit_event is append-only');\n\n-- Hash-chain seal table (Signed Tree Heads, Sigstore-style)\nCREATE TABLE audit_seal (\n  seal_id          UUID PRIMARY KEY,\n  account_id       UUID NOT NULL,\n  sealed_at        TIMESTAMPTZ NOT NULL,\n  tree_size        BIGINT NOT NULL,\n  merkle_root      BYTEA NOT NULL,\n  signature        BYTEA NOT NULL,                       -- ed25519 over (tree_size || root || sealed_at)\n  external_anchor  TEXT                                  -- s3://worm-bucket/seals/\u2026\n);\n\n\nAction-name catalogue (representative subset)\n\n\nServiceActions\naccountslogin \u00b7 logout \u00b7 tokenLogin \u00b7 samlLogin \u00b7 oidcLogin \u00b7 mfaLogin \u00b7 createIpAccessList \u00b7 updateIpAccessList\nscimcreateUser \u00b7 updateUser \u00b7 deleteUser \u00b7 createGroup \u00b7 addPrincipalToGroup \u00b7 createServicePrincipal\nunityCatalogcreate/update/delete[Metastore|Catalog|Schema|Table|Volume|Function|Model|StorageCredential|ExternalLocation|Connection|GovernedTag] \u00b7 grantPermission \u00b7 revokePermission \u00b7 generateTemporaryTableCredential \u00b7 getTable\u26a1\nclusterscreate \u00b7 edit \u00b7 start \u00b7 restart \u00b7 delete \u00b7 attachNotebook \u00b7 changeAcl\njobscreate \u00b7 update \u00b7 delete \u00b7 runNow \u00b7 runStart \u00b7 runFailed \u00b7 runSucceeded \u00b7 cancel \u00b7 setAcl\ndatabrickssqlcreateEndpoint \u00b7 startEndpoint \u00b7 stopEndpoint \u00b7 executeAdHocQuery \u00b7 downloadQueryResult \u00b7 commandSubmit\u26a1 \u00b7 commandFinish\u26a1\nsecretscreateScope \u00b7 putSecret \u00b7 getSecret \u00b7 listSecrets \u00b7 putAcl\nvectorSearchcreateEndpoint \u00b7 createIndex \u00b7 queryIndex \u00b7 syncIndex\nmodelServingcreateServingEndpoint \u00b7 updateServingEndpoint \u00b7 inferenceRequest\u26a1\naiAgentcreateApp \u00b7 createAgent \u00b7 invokeAgent \u00b7 createTool \u00b7 invokeTool\u26a1 \u00b7 createConversation \u00b7 postMessage \u00b7 createKnowledgeBase \u00b7 ingestDocument \u00b7 runEval \u00b7 promoteVersion \u00b7 attachGuardrail \u00b7 approvalRequested \u00b7 approvalGranted \u00b7 approvalDenied\nbillingcreateBudget \u00b7 budgetExceed \u00b7 createInvoice \u00b7 drawdownCommit \u00b7 updateRateCard\nretentionupdatePolicy \u00b7 sweep \u00b7 legalHoldOn \u00b7 legalHoldOff\n\n\n\u26a1 = verbose-only; emitted when account-level \"verbose audit logs\" toggle is on.\n\n\n\n37 \u00b7 Lineage\n\n\nCREATE TABLE lineage_edge (\n  edge_id           UUID PRIMARY KEY,\n  event_time        TIMESTAMPTZ NOT NULL,\n  event_date        DATE NOT NULL,\n  metastore_id      UUID NOT NULL,\n  workspace_id      UUID,\n  statement_id      UUID,                                -- FK to query_history\n  entity_type       TEXT NOT NULL,                       -- NOTEBOOK, JOB, PIPELINE, DASHBOARD,\n                                                         -- DBSQL_QUERY, MODEL, AGENT_RUN, EXTERNAL\n  entity_id         TEXT,\n  entity_run_id     TEXT,\n  source_full_name  TEXT,\n  source_type       TEXT,                                -- TABLE, VIEW, PATH, MODEL, STREAM, KB_CHUNK\n  source_path       TEXT,\n  source_column     TEXT,                                -- NULL for table-grain\n  target_full_name  TEXT NOT NULL,\n  target_type       TEXT,\n  target_path       TEXT,\n  target_column     TEXT,\n  direct_access     BOOLEAN NOT NULL DEFAULT TRUE,\n  created_by        TEXT\n) PARTITION BY RANGE (event_date);\nCREATE INDEX le_target ON lineage_edge (target_full_name, event_time DESC);\nCREATE INDEX le_source ON lineage_edge (source_full_name, event_time DESC);\nCREATE UNIQUE INDEX le_dedup ON lineage_edge\n  (metastore_id, source_full_name, source_column, target_full_name, target_column, statement_id);\n\n-- RAG-specific extension: KB_CHUNK \u2192 MESSAGE citation\nCREATE TABLE retrieval_citation (\n  message_id   UUID NOT NULL REFERENCES message ON DELETE CASCADE,\n  kb_chunk_id  UUID NOT NULL REFERENCES kb_chunk,\n  score        REAL,\n  rank         INT,\n  PRIMARY KEY (message_id, kb_chunk_id)\n);\n\n\nDerivation: Spark logical plan introspection (QueryExecutionListener) for batch; Photon plan events for SQL; `direct_access=true` for raw S3 paths; OpenLineage HTTP ingest for external (Tableau/Salesforce/PowerBI). RAG citations emitted by KB retrieve nodes and joined to message_id.\n\n\n\n38 \u00b7 Traces (OTel GenAI shape)\n\n\nCREATE TABLE trace_span (\n  trace_id          UUID NOT NULL,                       -- UUIDv7\n  span_id           BYTEA NOT NULL,                      -- 8 bytes\n  parent_span_id    BYTEA,\n  account_id        UUID NOT NULL,\n  workspace_id      UUID,\n  service_name      TEXT NOT NULL,\n  span_name         TEXT NOT NULL,                       -- e.g. \"chat claude-opus-4-7\"\n  span_kind         TEXT NOT NULL,                       -- CLIENT, SERVER, INTERNAL, PRODUCER, CONSUMER\n  start_time        TIMESTAMPTZ NOT NULL,\n  end_time          TIMESTAMPTZ NOT NULL,\n  duration_ms       INT NOT NULL,\n  status_code       TEXT NOT NULL,                       -- OK, ERROR, UNSET\n  status_message    TEXT,\n  attributes        JSONB NOT NULL DEFAULT '{}',         -- gen_ai.* + custom\n  events            JSONB,                               -- prompt/completion (redactable)\n  links             JSONB,\n  resource          JSONB,                               -- service.name, deployment.environment\n  -- denormalised for fast filter\n  model             TEXT,\n  input_tokens      INT,\n  output_tokens     INT,\n  cache_read_tokens INT,\n  reasoning_tokens  INT,\n  cost_usd          NUMERIC(12,6),\n  PRIMARY KEY (trace_id, span_id)\n) PARTITION BY RANGE (start_time);\nCREATE INDEX ts_trace ON trace_span (trace_id);\nCREATE INDEX ts_account_time ON trace_span (account_id, start_time DESC);\nCREATE INDEX ts_model_time ON trace_span (model, start_time DESC) WHERE model IS NOT NULL;\nCREATE INDEX ts_attrs ON trace_span USING GIN (attributes jsonb_path_ops);\n\n\nOTel attribute catalogue (gen_ai.*)\n\n\nAttributeNotes\ngen_ai.systemdatabricks, anthropic, openai, bedrock, \u2026\ngen_ai.operation.namechat, text_completion, embeddings, invoke_agent, create_agent, execute_tool\ngen_ai.request.model / gen_ai.response.model\u2014\ngen_ai.request.temperature / top_p / max_tokens\u2014\ngen_ai.usage.input_tokens / output_tokens / cache_creation_input_tokens / cache_read_input_tokens / reasoning_tokensbilling source of truth\ngen_ai.response.id / finish_reasons\u2014\ngen_ai.tool.name / tool.call.id / tool.input / tool.output\u2014\ngen_ai.agent.id / agent.namespan hierarchy root\ngen_ai.prompt / completion (events)redactable per PIIRedactor policy\n\n\n\n\n39 \u00b7 Metrics & alerts\n\n\nCREATE TABLE metric_sample (\n  metric_name   TEXT NOT NULL,\n  ts            TIMESTAMPTZ NOT NULL,\n  account_id    UUID NOT NULL, workspace_id UUID,\n  resource_type TEXT NOT NULL,                           -- CLUSTER, WAREHOUSE, ENDPOINT, JOB, AGENT, KB\n  resource_id   TEXT NOT NULL,\n  value         DOUBLE PRECISION NOT NULL,\n  unit          TEXT,\n  labels        JSONB NOT NULL DEFAULT '{}',\n  PRIMARY KEY (metric_name, resource_id, ts)\n) PARTITION BY RANGE (ts);\n-- downsample tiers: raw 7d \u2192 5m 30d \u2192 1h 90d \u2192 1d 2y\n\nCREATE TABLE alert_rule (\n  rule_id          UUID PRIMARY KEY,\n  account_id       UUID NOT NULL, workspace_id UUID,\n  name             TEXT NOT NULL,\n  source           TEXT NOT NULL,                        -- AUDIT, METRIC, TRACE, EVAL, COST, LINEAGE\n  query            TEXT NOT NULL,                        -- SQL or PromQL\n  condition        JSONB NOT NULL,                       -- {op:'>', threshold:100, window:'5m'}\n  severity         TEXT NOT NULL,                        -- CRITICAL, HIGH, MEDIUM, LOW, INFO\n  destinations     UUID[] NOT NULL,\n  enabled          BOOLEAN NOT NULL DEFAULT TRUE,\n  cooldown_seconds INT NOT NULL DEFAULT 300,\n  owner_id         UUID, created_at TIMESTAMPTZ\n);\n\nCREATE TABLE alert_event (\n  event_id     UUID PRIMARY KEY,\n  rule_id      UUID NOT NULL REFERENCES alert_rule,\n  fired_at     TIMESTAMPTZ NOT NULL,\n  resolved_at  TIMESTAMPTZ,\n  state        TEXT NOT NULL,                            -- FIRING, RESOLVED, ACKED\n  context      JSONB NOT NULL,\n  fingerprint  TEXT NOT NULL,                            -- dedup\n  acked_by     UUID, acked_at TIMESTAMPTZ\n);\n\nCREATE TABLE notification_destination (\n  destination_id UUID PRIMARY KEY,\n  account_id     UUID NOT NULL, workspace_id UUID,\n  name           TEXT NOT NULL,\n  type           TEXT NOT NULL,                          -- EMAIL, SLACK, WEBHOOK, PAGERDUTY, MS_TEAMS, OPSGENIE\n  config         JSONB NOT NULL,                         -- type-specific\n  config_secret_ref TEXT,                                -- \u2192 secret store\n  verified       BOOLEAN NOT NULL DEFAULT FALSE\n);\n\n\n\n40 \u00b7 Retention & legal hold\n\nCREATE TABLE retention_policy (\n  policy_id        UUID PRIMARY KEY,\n  account_id       UUID NOT NULL,\n  table_name       TEXT NOT NULL,\n  hot_days         INT NOT NULL,\n  cold_days        INT NOT NULL,\n  cold_storage_uri TEXT,                                 -- s3://\u2026 (Object Lock COMPLIANCE)\n  legal_hold       BOOLEAN NOT NULL DEFAULT FALSE,\n  pii_redaction    BOOLEAN NOT NULL DEFAULT FALSE,\n  updated_by       UUID, updated_at TIMESTAMPTZ,\n  UNIQUE (account_id, table_name)\n);\n\n-- Default seed\nINSERT INTO retention_policy (table_name, hot_days, cold_days) VALUES\n  ('audit_event',          365,  2555),  -- 7y for FedRAMP / financial\n  ('data_access_event',     90,   365),\n  ('lineage_edge',         365,  1095),\n  ('trace_span',            30,    90),\n  ('metric_sample',          7,   730),\n  ('billable_usage',       730,  2555),  -- 7y financial\n  ('eval_run',             730,  1095),\n  ('alert_event',          365,   365),\n  ('message',              365,  1095),  -- agent conversations (HIPAA: 6y)\n  ('workflow_run',         365,   730),\n  ('ai_app_version',        -1,    -1),  -- never delete (immutable manifest)\n  ('agent_version',         -1,    -1);  -- never delete\n\n\n\n41 \u00b7 Full ACL DDL (paste-ready)\n\n-- Principal kinds\nCREATE TYPE principal_kind AS ENUM ('User','Group','ServicePrincipal','Recipient','FederatedIdentity');\n\nCREATE TABLE principal (\n  id                  UUID PRIMARY KEY DEFAULT gen_random_uuid(),\n  account_id          UUID NOT NULL,\n  kind                principal_kind NOT NULL,\n  name                TEXT NOT NULL,\n  external_id         TEXT,\n  attrs               JSONB NOT NULL DEFAULT '{}',         -- ABAC attributes\n  federated_issuer    TEXT, federated_subject TEXT, federated_audience TEXT,\n  created_at          TIMESTAMPTZ NOT NULL DEFAULT now(),\n  UNIQUE (account_id, kind, name)\n);\nCREATE UNIQUE INDEX p_federated ON principal (federated_issuer, federated_subject, federated_audience)\n  WHERE kind = 'FederatedIdentity';\n\nCREATE TABLE principal_membership (\n  parent_id  UUID NOT NULL REFERENCES principal ON DELETE CASCADE,\n  member_id  UUID NOT NULL REFERENCES principal ON DELETE CASCADE,\n  account_id UUID NOT NULL,\n  PRIMARY KEY (parent_id, member_id),\n  CHECK (parent_id != member_id)\n);\n-- Depth \u2264 3 enforced at write time\n\n-- Generic securable (typed graph)\nCREATE TABLE securable (\n  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),\n  account_id      UUID NOT NULL,\n  type            TEXT NOT NULL,                       -- Account, Workspace, Metastore, Catalog,\n                                                       -- Schema, Table, StreamingTable, View, MV,\n                                                       -- Volume, Function, Procedure, Model, Cluster,\n                                                       -- Network, MCPServer, Job, JobRun, Flow,\n                                                       -- Pipeline, Notebook, NotebookCell, KernelSession,\n                                                       -- File, Directory, Repo, Dashboard, GenieSpace,\n                                                       -- App, AIApp, Agent, AgentVersion, Workflow,\n                                                       -- WorkflowVersion, Conversation, Tool, Mcp\u2026,\n                                                       -- KnowledgeBase, KBDocument, KBChunk,\n                                                       -- PromptTemplate, PromptVersion, ModelEndpoint,\n                                                       -- Guardrail, RateLimitPolicy, ApprovalPolicy,\n                                                       -- Share, Provider, Recipient, CleanRoom,\n                                                       -- MarketplaceListing, Exchange, ExternalMetadata,\n                                                       -- DatabaseInstance, GovernedTag, TagPolicy,\n                                                       -- StorageCredential, ExternalLocation, Connection,\n                                                       -- ServiceCredential, SecretScope, \u2026\n  parent_id       UUID REFERENCES securable ON DELETE CASCADE,\n  name            TEXT NOT NULL,\n  full_name       TEXT NOT NULL,\n  owner_id        UUID NOT NULL REFERENCES principal,\n  pending_owner_id UUID REFERENCES principal,\n  pending_since   TIMESTAMPTZ,\n  metadata        JSONB NOT NULL DEFAULT '{}',\n  created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),\n  UNIQUE (account_id, type, full_name)\n);\nCREATE INDEX s_parent ON securable (parent_id);\nCREATE INDEX s_account_type ON securable (account_id, type);\n\n-- Cascade rules (data, not code)\nCREATE TABLE securable_cascade (\n  parent_type   TEXT NOT NULL,\n  child_type    TEXT NOT NULL,\n  privileges    TEXT[] NOT NULL,\n  PRIMARY KEY (parent_type, child_type)\n);\n\n-- Grants\nCREATE TABLE grant (\n  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),\n  account_id      UUID NOT NULL,\n  principal_id    UUID NOT NULL REFERENCES principal ON DELETE CASCADE,\n  securable_id    UUID NOT NULL REFERENCES securable ON DELETE CASCADE,\n  privilege       TEXT NOT NULL,\n  effect          TEXT NOT NULL DEFAULT 'ALLOW' CHECK (effect IN ('ALLOW','DENY')),\n  granted_by      UUID NOT NULL REFERENCES principal,\n  granted_at      TIMESTAMPTZ NOT NULL DEFAULT now(),\n  valid_from      TIMESTAMPTZ NOT NULL DEFAULT now(),\n  expires_at      TIMESTAMPTZ,\n  revoked_at      TIMESTAMPTZ, revoked_by UUID REFERENCES principal,\n  condition_cedar TEXT,\n  UNIQUE (account_id, principal_id, securable_id, privilege, effect)\n);\nCREATE INDEX g_live_allow ON grant (account_id, principal_id, securable_id)\n  WHERE effect='ALLOW' AND revoked_at IS NULL\n    AND valid_from<=now() AND (expires_at IS NULL OR expires_at>now());\nCREATE INDEX g_live_deny ON grant (account_id, principal_id, securable_id)\n  WHERE effect='DENY' AND revoked_at IS NULL;\n\n-- Row filter / column mask attachment (NOT grants)\nCREATE TABLE row_filter_attachment (\n  table_id    UUID PRIMARY KEY REFERENCES securable ON DELETE CASCADE,\n  function_id UUID NOT NULL REFERENCES securable,\n  args        TEXT[] NOT NULL DEFAULT '{}'\n);\nCREATE TABLE column_mask_attachment (\n  table_id    UUID NOT NULL REFERENCES securable ON DELETE CASCADE,\n  column_name TEXT NOT NULL,\n  function_id UUID NOT NULL REFERENCES securable,\n  args        TEXT[] NOT NULL DEFAULT '{}',\n  PRIMARY KEY (table_id, column_name)\n);\n\n-- Cache invalidation versions\nCREATE TABLE acl_version (\n  id INT PRIMARY KEY DEFAULT 1 CHECK (id=1),\n  grants_version BIGINT NOT NULL DEFAULT 0,\n  membership_version BIGINT NOT NULL DEFAULT 0,\n  ownership_version BIGINT NOT NULL DEFAULT 0,\n  cascade_version BIGINT NOT NULL DEFAULT 0\n);\nINSERT INTO acl_version DEFAULT VALUES;\n\n-- Multi-tenant RLS (defence in depth)\nALTER TABLE grant ENABLE ROW LEVEL SECURITY;\nCREATE POLICY grant_tenant_iso ON grant\n  USING (account_id = current_setting('app.account_id')::uuid);\nALTER TABLE securable ENABLE ROW LEVEL SECURITY;\nCREATE POLICY securable_tenant_iso ON securable\n  USING (account_id = current_setting('app.account_id')::uuid);\n\n-- Materialised \"what can this principal see\" \u2014 refreshed on grants_version bump\nCREATE MATERIALIZED VIEW effective_grants_mv AS\nWITH RECURSIVE eff_p AS (\n    SELECT p.id AS principal_id, p.id AS effective_id, p.account_id\n      FROM principal p WHERE p.kind IN ('User','ServicePrincipal','FederatedIdentity')\n  UNION\n    SELECT ep.principal_id, pm.parent_id, ep.account_id\n      FROM principal_membership pm\n      JOIN eff_p ep ON pm.member_id = ep.effective_id\n)\nSELECT g.account_id, ep.principal_id, g.securable_id, g.privilege, g.effect\nFROM grant g\nJOIN eff_p ep ON g.principal_id = ep.effective_id\nWHERE g.revoked_at IS NULL\n  AND g.valid_from <= now()\n  AND (g.expires_at IS NULL OR g.expires_at > now());\nCREATE UNIQUE INDEX eg_mv_pk\n  ON effective_grants_mv (account_id, principal_id, securable_id, privilege, effect);\n\n\n\n42 \u00b7 Rollout strategy\n\n\n\nPhaseStepRollback\n0Foundation tables (principal, securable, grant, audit_event, rate_card, billable_usage)DROP \u2014 no production traffic yet\n1UC objects with owner column + cascade rules; backfill grantsflip read flag\n2Workspace assets (notebook/file/dashboard/cluster/job/secret) folded into single grantflip flag\n3Account-level securables (Workspace, Metastore, Network, \u2026)\u2014\n4Audit + lineage tables; hash chain seal job\u2014\n5Cost: rate_card + billable_usage + budget/policy/quota; chargeback queries\u2014\n6AI agent plane: AIApp, Agent, Workflow, MCPServer, KB, Tool, ModelEndpoint, Guardrail, EvalGate, TraceSpan\u2014\n7Federation, Sharing, Marketplace, CleanRoom\u2014\n8Dual-write window (1 release): old ACL tables + grant; reads from newflip read flag\n9Drop legacy ACL tables (after 2 stable releases)irreversible\n10Postgres RLS on grant/securable; Cedar evaluator deployedflip flag\n11Materialised view list-pushdown + 500ms debounced refreshflip flag\n12OTel GenAI trace ingest from agent runtime; eval gate enforcement\u2014\n\n\n\n\n43 \u00b7 Anti-patterns (postmortem-derived)\n\n\n\n#Anti-patternMitigation\n1Hardcoded admin bypass scattered across handlersSingle resolver; admin = pass-2 only; no per-handler shortcut\n2Owner stored as grant rowOwner = column on securable; never deletable via REVOKE\n3Cascade hardcoded in codesecurable_cascade table; new asset = INSERT row\n4Group nesting via closure tableEdge list + recursive CTE, depth \u2264 3\n5Free-form JSON conditionsCedar grammar (total, decidable)\n6List endpoints fetch-then-filter (N+1)MV with predicate pushdown\n7Audit log without before/after + rationaleJSONB snapshots + matched-pass rationale + hash chain\n8account_id implicit in FK chainExplicit column on every row + RLS\n9MANAGE on parent auto-grants child SELECTGrantor-holds-privilege rule\n10Owner transfer single-phaseTwo-phase with pending_owner_id\n11Run-style children inherit SELECT (logs leak data)Cascade BROWSE+EXECUTE+MANAGE only\n12Cost tags computed only at invoice timeStamped at metering emission via budget_policy\n13Agent run cost computed from app-level counterSum from TraceSpan \u2014 sub-agent costs would otherwise be lost\n14Prompt edited in place (no version)prompt_version with content-addressed hash\n15Tool registered without input_schemaJSONSchema required at registration; reject otherwise\n16MCP server discovery cached foreverRe-handshake every N min; last_handshake_at tracked\n17Guardrail bypass via direct ModelEndpoint callAll chat traffic must go through Agent runtime that applies guardrails; ModelEndpoint USE requires no-bypass flag\n18Eval results not blocking promotioneval_gate wired into deployment_slot promotion API\n19Knowledge base re-index on every doc editContent-hash dedup on KB chunks; re-embed only if hash changes\n20Conversation retention indefiniteretention_policy.message defaults to 365d hot / 1095d cold; HIPAA bumps to 6y\n\n\n\n\n44 \u00b7 Sources\n\n\nDatabricks (canonical reference)\n\n\n\n\nDatabricks Workspace REST API\n\nDatabricks Account REST API\n\nUC securable objects\n\nUC privileges reference\n\nDatabricks pricing\n\nsystem.billing.usage\n\nsystem.access.audit\n\nsystem.access.{table,column}_lineage\n\nsystem.compute.*\n\nsystem.lakeflow.jobs\n\nBudget policies\n\nDLT/Lakeflow ACL\n\nMarketplace listings\n\nLakebase UC registration\n\nMosaic AI Gateway\n\n\n\nAI agent platforms\n\n\n\n\nDify \u00b7 key concepts\n\nDify \u00b7 GitHub\n\nLangGraph \u00b7 graph API\n\nn8n \u00b7 RBAC\n\nOpenAI \u00b7 Runs / Threads\n\nCrewAI \u00b7 crews\n\nAnthropic \u00b7 MCP\n\nMCP servers registry\n\nHaystack \u00b7 agents\n\nLlamaIndex \u00b7 Workflows\n\nFlowise\n\nCoze \u00b7 quickstart\n\n\n\nStandards & specs\n\n\n\n\nOTel GenAI spans\n\nOTel GenAI agent spans\n\nOpenLineage object model\n\nRFC 8693 \u00b7 OAuth Token Exchange\n\nCedar policy language\n\nSigstore Rekor (hash chain)\n\nOWASP ASVS V7 (Logging)\n\nDelta Sharing protocol\n\n\n\nPeer reference (cross-check)\n\n\n\n\nSnowflake QUERY_HISTORY\n\nSnowflake ACCESS_HISTORY\n\nBigQuery \u00b7 column masking\n\nAWS Lake Formation LF-TBAC\n\nApache Polaris RBAC\n\nAWS CloudTrail record\n\n\n\n\n\nFoundation spec \u00b7 47 securables active \u00b7 30+ reserved \u00b7 38 privileges \u00b7 15 cost dimensions \u00b7 OTel GenAI native \u00b7 Cedar conditions \u00b7 hash-chained audit \u00b7 MCP-first tool host \u00b7 4-pass resolver \u00b7 Postgres RLS\n\n\n\n\n\n", "creation_timestamp": "2026-05-10T03:58:34.000000Z"}, {"uuid": "ff01baf2-ab7d-4b9d-829f-98b496dc8852", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "86ecb4e1-bb32-44d5-9f39-8a4673af8385", "vulnerability": "CVE-2024-25625", "type": "published-proof-of-concept", "source": "https://github.com/pimcore/admin-ui-classic-bundle/security/advisories/GHSA-3qpq-6w89-f7mx", "content": "", "creation_timestamp": "2024-02-19T08:52:02.000000Z"}]}