Skip to content

Commit 624799e

Browse files
author
brian-brewer
committed
Added Use Cases, minor display fix to index
1 parent abf51cc commit 624799e

File tree

6 files changed

+442
-37
lines changed

6 files changed

+442
-37
lines changed

Manifesto.md

Lines changed: 35 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,18 @@
11
---
2-
title: DTE Manifesto
2+
title: Data Trust Engineering (DTE) Manifesto
33
---
44

5-
# Data Trust Engineering Manifesto
6-
75
## Mission
86

9-
Data Trust Engineering (DTE) unites data professionals to certify data systems by use case, risk, and value, replacing governance’s traditional approach with engineering-driven solutions. The manifesto defines the what and why—trusted, AI-ready systems—while our vendor-neutral community shapes the how, filling governance’s implementation gap. Join the #DTERevolution.
7+
Data Trust Engineering (DTE) unites data professionals to certify data systems by use case, risk, and value, replacing governance’s policy-driven or stewardship-driven approaches with engineering-driven solutions. The manifesto defines the what and why—trusted, AI-ready systems—while our vendor-neutral community shapes the how, filling governance’s implementation gap. Join the #DTERevolution.
108

119
## Rationale
1210

1311
The Data Trust Engineering (DTE) Manifesto emerged from the urgent need to address the persistent challenges of traditional data governance, which struggles with a 70-80% failure rate due to bureaucratic complexity and conflation of compliance with technical data management [Gartner, 2025]. Born from the collective insights of data professionals—engineers, scientists, architects, and analysts—DTE redefines data management as an engineering-driven discipline focused on certifying data systems by use case, risk, and value. Inspired by the agile principles of DevOps and the community-driven ethos of open-source movements, DTE moves beyond vendor-driven frameworks that prioritize billable hours over outcomes, evolving from governance’s SOX-era origins (2002). Instead, it empowers a vendor-neutral community to shape the *how* through collaborative tools like GitHub, while the manifesto defines the *what* (trusted, AI-ready systems) and *why* (to overcome governance’s inefficiencies and enable innovation in 2025’s 175-zettabyte AI age [Statista, 2025]).
1412

15-
DTE’s rationale is rooted in three core insights. First, governance’s one-size-fits-all approach struggles to address diverse needs—enterprises require compliance-adjacent trust, SMBs need lightweight solutions, and hyperscalers demand scalability. Second, the rise of AI, with 70% of models experiencing challenges due to data quality issues [MIT Technology Review, 2025], demands a new paradigm that prioritizes certification, observability, and semantic clarity (e.g., via knowledge graphs) over rigid policies. Third, compliance (e.g., GDPR, EU AI Act [2024]) is most effectively handled by legal and audit teams, freeing DTE to focus on technical trust and adaptability through tools like Apache Atlas and Great Expectations. By blending DataOps’ agility with strategic trust, DTE addresses the gap left by governance’s structural limitations, inviting all data professionals to join the #DTERevolution and build a trusted, scalable future.
13+
DTE’s rationale is rooted in three core insights. First, governance’s one-size-fits-all approach struggles to address diverse needs—enterprises require compliance-adjacent trust, SMBs need lightweight solutions, and hyperscalers demand scalability. Second, the rise of AI, with 70% of models experiencing challenges due to data quality issues [MIT Technology Review, 2025], demands a new paradigm that prioritizes certification, observability, and semantic clarity (e.g., via knowledge graphs) over rigid policies. Third, compliance (e.g., GDPR, EU AI Act [2024]) is most effectively handled by legal and audit teams, freeing DTE to focus on technical trust and adaptability through tools like Apache Atlas and Great Expectations.
14+
15+
By blending DataOps’ agility with strategic trust, DTE addresses the gap left by governance’s structural limitations. Unlike policy-driven, top-down approaches (e.g., DAMA frameworks), DTE cannot be corrupted into compliance theater. It is designed by engineers, for engineers, to deliver trusted systems that scale with AI realities. Join the #DTERevolution and help build a trusted, scalable future.
1616

1717
## The Evolution Beyond Data Governance
1818

@@ -42,25 +42,36 @@ Organizations can begin their DTE journey through these practical steps:
4242

4343
## Core Principles of DTE
4444

45-
1. **Trust**: Data must be accurate, secure, and accessible, earning confidence for enterprises, users, and AI systems in 2025.
46-
1. **Engineering Rigor**: DTE builds systems with software engineering precision, using automation, shift-left and first principles.
47-
1. **Adaptability Through Feedback**: TDD-inspired feedback loops enable rapid testing and pipeline optimization [Gartner, 2025].
48-
1. **Enablement**: DTE empowers data professionals to deliver value, fostering collaboration.
49-
1. **Cloud-Native**: DTE aligns with shared responsibility models (e.g., AWS, Azure).
50-
1. **Certification by Use Case, Risk, and Value**: DTE certifies datasets based on their use case, risk, and business value, prioritizing high-stakes scenarios (e.g., AI training, regulatory reporting) to meet EU AI Act (2024) standards for transparency, fairness, and oversight. Tools like Fairlearn and SDV support the improvement of AI systems experiencing quality challenges [MIT Technology Review, 2025].
51-
1. **Technical Debt Management**: DTE reframes data quality as technical debt, using dbt and Great Expectations for 20-30% debugging improvements [Gartner, 2025].
52-
1. **Community-Driven**: DTE evolves through GitHub collaboration, inviting contributions from thought leaders , data leaders and practitioners to contribute ideas, principles and technical content.
45+
1. **Trust**: Data must be accurate, secure, and accessible, earning confidence for enterprises, users, and AI systems in 2025.
46+
1. **Engineering Rigor**: DTE builds systems with software engineering precision, using automation, shift-left, and first principles.
47+
1. **Adaptability Through Feedback**: TDD-inspired feedback loops enable rapid testing and pipeline optimization [Gartner, 2025].
48+
1. **Enablement**: DTE empowers data professionals to deliver value, fostering collaboration.
49+
1. **Cloud-Native**: DTE aligns with shared responsibility models (e.g., AWS, Azure).
50+
1. **Certification by Use Case, Risk, and Value**: DTE certifies datasets based on their use case, risk, and business value. Certification provides **risk-tiered assurance for business outcomes**, not regulatory approval. Tools like Fairlearn and SDV support improving AI systems experiencing quality challenges [MIT Technology Review, 2025].
51+
1. **Technical Debt Management**: DTE reframes data quality as technical debt, using dbt and Great Expectations for 20–30% debugging improvements [Gartner, 2025].
52+
1. **Community-Driven**: DTE evolves through GitHub collaboration, inviting contributions from thought leaders, data leaders, and practitioners.
53+
1. **Not a Governance Framework**: DTE is not a data governance framework, policy, operating model, or governing body.
54+
1. **Not a Tool**: DTE is not a data catalog, metadata management platform, or governance tool.
55+
1. **Vendor Neutrality**: DTE must remain vendor-neutral. While open source is used to demonstrate use cases, commercial products—no matter how popular—introduce bias and are not the foundation.
56+
1. **Independence from Compliance**: DTE is not aligned with compliance regimes or regulatory checklists. Instead, it provides architectural and engineering principles that can be applied across contexts.
57+
1. **Not Policy-Driven or Stewardship-Driven**: DTE rejects top-down policy mandates and stewardship bureaucracies. Trust is achieved through engineering discipline and automation, not enforced checklists or roles.
58+
59+
60+
61+
5362

5463
## Why DTE Succeeds Where Traditional Approaches Struggle
5564

56-
- **Decouples Compliance**: Compliance is handled by legal/audit teams, with DTE supporting via certified datasets.
57-
- **Architected for Change**: Feedback loops ensure adaptability, minimizing technical debt [Gartner, 2025].
58-
- **Cloud-Ready**: DTE thrives in data mesh and hybrid systems.
59-
- **Efficiency-Focused**: DTE avoids the $1-2 billion in process overhead documented in traditional approaches [Industry Surveys, 2024-2025].
60-
- **AI-Enabled**: Tools like Fairlearn address AI challenges, improving system reliability [MIT Technology Review, 2025].
61-
- **Universal Appeal**: Enterprises, SMBs, and hyperscalers benefit from tailored certification.
62-
- **Certified Trust**: Certification ensures EU AI Act-compliant quality.
63-
- **Technical Debt Discipline**: Tools like dbt achieve significant cost improvements [Cloud Data Insights, 2025].
65+
- **Decouples Compliance**: Compliance is owned by legal/audit teams. DTE supports them with certified, auditable datasets.
66+
- **Architected for Change**: Feedback loops ensure adaptability, minimizing technical debt [Gartner, 2025].
67+
- **Cloud-Ready**: DTE thrives in data mesh and hybrid systems.
68+
- **Efficiency-Focused**: DTE avoids the $1–2B in process overhead documented in traditional approaches [Industry Surveys, 2024–2025].
69+
- **AI-Enabled**: Tools like Fairlearn address AI challenges, improving system reliability [MIT Technology Review, 2025].
70+
- **Universal Appeal**: Enterprises, SMBs, and hyperscalers benefit from tailored certification.
71+
- **Certified Trust (Risk-Tiered Assurance)**: Certification reflects engineering standards and business risk, producing artifacts that *support* (but never claim) compliance with regimes like the EU AI Act.
72+
- **Technical Debt Discipline**: Tools like dbt achieve significant cost improvements [Cloud Data Insights, 2025].
73+
- Beyond Policy & Stewardship: Unlike policy-driven or stewardship-driven models that stall execution, DTE embeds governance into engineering flow—delivering trust as a natural outcome of modernized systems.
74+
6475

6576
## Observability and Provenance Over Static Policy
6677

@@ -85,14 +96,14 @@ DTE reframes data quality issues as technical debt, using dbt and Great Expectat
8596
- **Security & Access**: IAM and encryption with provenance-driven controls.
8697
- **Scalability**: Architectures for data mesh and real-time analytics.
8798
- **Adaptability**: TDD and shift-left feedback loops.
88-
- **Certification**: Datasets meet EU AI Act standards with Fairlearn and SDV.
99+
- **Certification**: Datasets certified by use case, risk, and value. Fairlearn and SDV help create audit-ready evidence that *supports mapping* to frameworks, without asserting compliance.
89100
- **Technical Debt Management**: dbt and Deequ refactor pipelines.
90101
- **AI Governance**: Fairlearn, SHAP, and MLflow ensure fairness and explainability.
91102
- **Collaboration**: GitHub contributions drive DTE’s evolution.
92103

93104
## Case Studies: DTE in Action
94105

95-
- **Healthcare Provider**: Great Expectations reduced pipeline errors by 15%; Fairlearn and SHAP ensured EU AI Act compliance [MIT Technology Review, 2025].
106+
- **Healthcare Provider**: Great Expectations reduced pipeline errors by 15%; Fairlearn and SHAP provided evidence packs that supported the healthcare provider’s compliance team in demonstrating EU AI Act readiness. [MIT Technology Review, 2025].
96107
- **SMB Retail**: Apache Superset and dbt reduced stockouts by 18% and improved debugging efficiency by 25% [Gartner, 2025].
97108
- **Hyperscaler Media**: Apache Atlas reduced technical debt by 50%, generating $1.2 million in savings [Cloud Data Insights, 2025].
98109

@@ -105,7 +116,7 @@ DTE reframes data quality issues as technical debt, using dbt and Great Expectat
105116
|**Compliance Approach**|Conflates with management  |Indirectly supports via certification|
106117
|**Cloud Readiness**    |Limited (legacy-focused)   |Excellent (cloud-native)             |
107118
|**Team Empowerment**   |Process-focused            |High (engineer-driven)               |
108-
|**AI-Readiness**       |Limited adaptation         |Robust (EU AI Act-compliant)         |
119+
|**AI-Readiness**       |Limited adaptation         |Robust (risk-tiered assurance supporting EU AI Act readiness)         |
109120

110121

111122
## The Call to Action

README.md

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
Certify Data. Build Trust. Shape the Future.
1313
```
1414

15-
**Data Trust Engineering (DTE)** is a movement to replace traditional data governance’s 70-80% failure rate [Gartner, 2025] with engineering-driven trust, certification, and AI-readiness. Born from the *Data Trust Engineering Manifesto*, DTE certifies data systems by use case, risk, and value, blending DataOps’ agility (inspired by [dataopsmanifesto.org](https://dataopsmanifesto.org)) with community collaboration. Our first artifact, the **DTE Trust Dashboard**, visualizes AI governance metrics—more are coming! Join us at [datatrustmanifesto.org](https://datatrustmanifesto.org) and fuel the #DTERevolution.
15+
**Data Trust Engineering (DTE)** is a movement to replace traditional data governance’s 70-80% failure rate [Gartner, 2025] with engineering-driven trust, certification, and AI-readiness. Born from the *Data Trust Engineering Manifesto*, DTE certifies data systems by use case, risk, and value, blending DataOps’ agility (inspired by [datatrustmanifesto.org](https://datatrustmanifesto.org)) with community collaboration. Our first artifact, the **DTE Trust Dashboard**, visualizes AI governance metrics—more are coming! Join us at [datatrustmanifesto.org](https://datatrustmanifesto.org) and fuel the #DTERevolution.
1616

1717
## Why DTE?
1818

@@ -33,12 +33,10 @@ The **DTE Trust Dashboard** kicks off our open-source journey, visualizing AI go
3333
- **Compliance**: GDPR adherence metrics.
3434
- **Accuracy**: Model performance monitoring (MLflow-compatible).
3535

36-
**DataOps Features**:
37-
- Real-time refresh for agility (DataOps Principle #1).
38-
- Open-source for collaboration (DataOps Principle #4).
39-
- Lightweight, Superset-compatible visualizations (DataOps Principle #7).
4036

41-
Find it in `/tools/data-trust-dashboard/DTE_Trust_Dashboard.html`. More artifacts—like data quality patterns and lineage trackers—are on the way!
37+
Find it in `/tools/data-trust-dashboard/DTE_Trust_Dashboard.html`. More artifacts—like data quality patterns and trust evals—are on the way!
38+
39+
Explore [UseCases.md](UseCases.md) to see how to apply DTE principles with open-source code.
4240

4341
## Getting Started
4442

@@ -69,6 +67,7 @@ Find it in `/tools/data-trust-dashboard/DTE_Trust_Dashboard.html`. More artifact
6967
- `/community`: Contribution guidelines, code of conduct.
7068
- `/LICENSE.md`: MIT License.
7169

70+
7271
## Contributing
7372

7473
Help shape DTE’s future! Contribute:
@@ -87,7 +86,7 @@ Start with “good first issues” (e.g., typo fixes) or discuss ideas in GitHub
8786
## Why It Matters
8887

8988
DTE overcomes governance’s failures with:
90-
- **Certified Trust**: EU AI Act-compliant datasets.
89+
- **Certified Trust**: trust datasets.
9190
- **Engineering Excellence**: 20-30% debugging improvements [Gartner, 2025].
9291
- **Community Power**: Collaborative evolution via GitHub and Slack.
9392

@@ -108,7 +107,7 @@ DTE overcomes governance’s failures with:
108107

109108
DTE builds on the [DataOps Manifesto](https://dataopsmanifesto.org)’s agility and collaboration. Thanks to the DataOps community and contributors shaping DTE.
110109

111-
**Authors**: Brian Brewer | https://www.infolibcorp.com
110+
**Authors**: Brian Brewer | Founder | https://www.infolibcorp.com
112111

113112
#DTERevolution
114113

0 commit comments

Comments
 (0)