You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: Manifesto.md
+35-24Lines changed: 35 additions & 24 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,18 +1,18 @@
1
1
---
2
-
title: DTE Manifesto
2
+
title: Data Trust Engineering (DTE) Manifesto
3
3
---
4
4
5
-
# Data Trust Engineering Manifesto
6
-
7
5
## Mission
8
6
9
-
Data Trust Engineering (DTE) unites data professionals to certify data systems by use case, risk, and value, replacing governance’s traditional approach with engineering-driven solutions. The manifesto defines the what and why—trusted, AI-ready systems—while our vendor-neutral community shapes the how, filling governance’s implementation gap. Join the #DTERevolution.
7
+
Data Trust Engineering (DTE) unites data professionals to certify data systems by use case, risk, and value, replacing governance’s policy-driven or stewardship-driven approaches with engineering-driven solutions. The manifesto defines the what and why—trusted, AI-ready systems—while our vendor-neutral community shapes the how, filling governance’s implementation gap. Join the #DTERevolution.
10
8
11
9
## Rationale
12
10
13
11
The Data Trust Engineering (DTE) Manifesto emerged from the urgent need to address the persistent challenges of traditional data governance, which struggles with a 70-80% failure rate due to bureaucratic complexity and conflation of compliance with technical data management [Gartner, 2025]. Born from the collective insights of data professionals—engineers, scientists, architects, and analysts—DTE redefines data management as an engineering-driven discipline focused on certifying data systems by use case, risk, and value. Inspired by the agile principles of DevOps and the community-driven ethos of open-source movements, DTE moves beyond vendor-driven frameworks that prioritize billable hours over outcomes, evolving from governance’s SOX-era origins (2002). Instead, it empowers a vendor-neutral community to shape the *how* through collaborative tools like GitHub, while the manifesto defines the *what* (trusted, AI-ready systems) and *why* (to overcome governance’s inefficiencies and enable innovation in 2025’s 175-zettabyte AI age [Statista, 2025]).
14
12
15
-
DTE’s rationale is rooted in three core insights. First, governance’s one-size-fits-all approach struggles to address diverse needs—enterprises require compliance-adjacent trust, SMBs need lightweight solutions, and hyperscalers demand scalability. Second, the rise of AI, with 70% of models experiencing challenges due to data quality issues [MIT Technology Review, 2025], demands a new paradigm that prioritizes certification, observability, and semantic clarity (e.g., via knowledge graphs) over rigid policies. Third, compliance (e.g., GDPR, EU AI Act [2024]) is most effectively handled by legal and audit teams, freeing DTE to focus on technical trust and adaptability through tools like Apache Atlas and Great Expectations. By blending DataOps’ agility with strategic trust, DTE addresses the gap left by governance’s structural limitations, inviting all data professionals to join the #DTERevolution and build a trusted, scalable future.
13
+
DTE’s rationale is rooted in three core insights. First, governance’s one-size-fits-all approach struggles to address diverse needs—enterprises require compliance-adjacent trust, SMBs need lightweight solutions, and hyperscalers demand scalability. Second, the rise of AI, with 70% of models experiencing challenges due to data quality issues [MIT Technology Review, 2025], demands a new paradigm that prioritizes certification, observability, and semantic clarity (e.g., via knowledge graphs) over rigid policies. Third, compliance (e.g., GDPR, EU AI Act [2024]) is most effectively handled by legal and audit teams, freeing DTE to focus on technical trust and adaptability through tools like Apache Atlas and Great Expectations.
14
+
15
+
By blending DataOps’ agility with strategic trust, DTE addresses the gap left by governance’s structural limitations. Unlike policy-driven, top-down approaches (e.g., DAMA frameworks), DTE cannot be corrupted into compliance theater. It is designed by engineers, for engineers, to deliver trusted systems that scale with AI realities. Join the #DTERevolution and help build a trusted, scalable future.
16
16
17
17
## The Evolution Beyond Data Governance
18
18
@@ -42,25 +42,36 @@ Organizations can begin their DTE journey through these practical steps:
42
42
43
43
## Core Principles of DTE
44
44
45
-
1.**Trust**: Data must be accurate, secure, and accessible, earning confidence for enterprises, users, and AI systems in 2025.
46
-
1.**Engineering Rigor**: DTE builds systems with software engineering precision, using automation, shift-left and first principles.
47
-
1.**Adaptability Through Feedback**: TDD-inspired feedback loops enable rapid testing and pipeline optimization [Gartner, 2025].
48
-
1.**Enablement**: DTE empowers data professionals to deliver value, fostering collaboration.
49
-
1.**Cloud-Native**: DTE aligns with shared responsibility models (e.g., AWS, Azure).
50
-
1.**Certification by Use Case, Risk, and Value**: DTE certifies datasets based on their use case, risk, and business value, prioritizing high-stakes scenarios (e.g., AI training, regulatory reporting) to meet EU AI Act (2024) standards for transparency, fairness, and oversight. Tools like Fairlearn and SDV support the improvement of AI systems experiencing quality challenges [MIT Technology Review, 2025].
51
-
1.**Technical Debt Management**: DTE reframes data quality as technical debt, using dbt and Great Expectations for 20-30% debugging improvements [Gartner, 2025].
52
-
1.**Community-Driven**: DTE evolves through GitHub collaboration, inviting contributions from thought leaders , data leaders and practitioners to contribute ideas, principles and technical content.
45
+
1.**Trust**: Data must be accurate, secure, and accessible, earning confidence for enterprises, users, and AI systems in 2025.
46
+
1.**Engineering Rigor**: DTE builds systems with software engineering precision, using automation, shift-left, and first principles.
47
+
1.**Adaptability Through Feedback**: TDD-inspired feedback loops enable rapid testing and pipeline optimization [Gartner, 2025].
48
+
1.**Enablement**: DTE empowers data professionals to deliver value, fostering collaboration.
49
+
1.**Cloud-Native**: DTE aligns with shared responsibility models (e.g., AWS, Azure).
50
+
1.**Certification by Use Case, Risk, and Value**: DTE certifies datasets based on their use case, risk, and business value. Certification provides **risk-tiered assurance for business outcomes**, not regulatory approval. Tools like Fairlearn and SDV support improving AI systems experiencing quality challenges [MIT Technology Review, 2025].
51
+
1.**Technical Debt Management**: DTE reframes data quality as technical debt, using dbt and Great Expectations for 20–30% debugging improvements [Gartner, 2025].
52
+
1.**Community-Driven**: DTE evolves through GitHub collaboration, inviting contributions from thought leaders, data leaders, and practitioners.
53
+
1.**Not a Governance Framework**: DTE is not a data governance framework, policy, operating model, or governing body.
54
+
1.**Not a Tool**: DTE is not a data catalog, metadata management platform, or governance tool.
55
+
1.**Vendor Neutrality**: DTE must remain vendor-neutral. While open source is used to demonstrate use cases, commercial products—no matter how popular—introduce bias and are not the foundation.
56
+
1.**Independence from Compliance**: DTE is not aligned with compliance regimes or regulatory checklists. Instead, it provides architectural and engineering principles that can be applied across contexts.
57
+
1.**Not Policy-Driven or Stewardship-Driven**: DTE rejects top-down policy mandates and stewardship bureaucracies. Trust is achieved through engineering discipline and automation, not enforced checklists or roles.
58
+
59
+
60
+
61
+
53
62
54
63
## Why DTE Succeeds Where Traditional Approaches Struggle
55
64
56
-
-**Decouples Compliance**: Compliance is handled by legal/audit teams, with DTE supporting via certified datasets.
-**Cloud-Ready**: DTE thrives in data mesh and hybrid systems.
68
+
-**Efficiency-Focused**: DTE avoids the $1–2B in process overhead documented in traditional approaches [Industry Surveys, 2024–2025].
69
+
-**AI-Enabled**: Tools like Fairlearn address AI challenges, improving system reliability [MIT Technology Review, 2025].
70
+
-**Universal Appeal**: Enterprises, SMBs, and hyperscalers benefit from tailored certification.
71
+
-**Certified Trust (Risk-Tiered Assurance)**: Certification reflects engineering standards and business risk, producing artifacts that *support* (but never claim) compliance with regimes like the EU AI Act.
72
+
-**Technical Debt Discipline**: Tools like dbt achieve significant cost improvements [Cloud Data Insights, 2025].
73
+
- Beyond Policy & Stewardship: Unlike policy-driven or stewardship-driven models that stall execution, DTE embeds governance into engineering flow—delivering trust as a natural outcome of modernized systems.
74
+
64
75
65
76
## Observability and Provenance Over Static Policy
66
77
@@ -85,14 +96,14 @@ DTE reframes data quality issues as technical debt, using dbt and Great Expectat
85
96
-**Security & Access**: IAM and encryption with provenance-driven controls.
86
97
-**Scalability**: Architectures for data mesh and real-time analytics.
87
98
-**Adaptability**: TDD and shift-left feedback loops.
88
-
-**Certification**: Datasets meet EU AI Act standards with Fairlearn and SDV.
99
+
-**Certification**: Datasets certified by use case, risk, and value. Fairlearn and SDV help create audit-ready evidence that *supports mapping* to frameworks, without asserting compliance.
89
100
-**Technical Debt Management**: dbt and Deequ refactor pipelines.
90
101
-**AI Governance**: Fairlearn, SHAP, and MLflow ensure fairness and explainability.
-**Healthcare Provider**: Great Expectations reduced pipeline errors by 15%; Fairlearn and SHAP ensured EU AI Act compliance[MIT Technology Review, 2025].
106
+
-**Healthcare Provider**: Great Expectations reduced pipeline errors by 15%; Fairlearn and SHAP provided evidence packs that supported the healthcare provider’s compliance team in demonstrating EU AI Act readiness.[MIT Technology Review, 2025].
96
107
-**SMB Retail**: Apache Superset and dbt reduced stockouts by 18% and improved debugging efficiency by 25% [Gartner, 2025].
97
108
-**Hyperscaler Media**: Apache Atlas reduced technical debt by 50%, generating $1.2 million in savings [Cloud Data Insights, 2025].
98
109
@@ -105,7 +116,7 @@ DTE reframes data quality issues as technical debt, using dbt and Great Expectat
105
116
|**Compliance Approach**|Conflates with management |Indirectly supports via certification|
Copy file name to clipboardExpand all lines: README.md
+7-8Lines changed: 7 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,7 +12,7 @@
12
12
Certify Data. Build Trust. Shape the Future.
13
13
```
14
14
15
-
**Data Trust Engineering (DTE)** is a movement to replace traditional data governance’s 70-80% failure rate [Gartner, 2025] with engineering-driven trust, certification, and AI-readiness. Born from the *Data Trust Engineering Manifesto*, DTE certifies data systems by use case, risk, and value, blending DataOps’ agility (inspired by [dataopsmanifesto.org](https://dataopsmanifesto.org)) with community collaboration. Our first artifact, the **DTE Trust Dashboard**, visualizes AI governance metrics—more are coming! Join us at [datatrustmanifesto.org](https://datatrustmanifesto.org) and fuel the #DTERevolution.
15
+
**Data Trust Engineering (DTE)** is a movement to replace traditional data governance’s 70-80% failure rate [Gartner, 2025] with engineering-driven trust, certification, and AI-readiness. Born from the *Data Trust Engineering Manifesto*, DTE certifies data systems by use case, risk, and value, blending DataOps’ agility (inspired by [datatrustmanifesto.org](https://datatrustmanifesto.org)) with community collaboration. Our first artifact, the **DTE Trust Dashboard**, visualizes AI governance metrics—more are coming! Join us at [datatrustmanifesto.org](https://datatrustmanifesto.org) and fuel the #DTERevolution.
16
16
17
17
## Why DTE?
18
18
@@ -33,12 +33,10 @@ The **DTE Trust Dashboard** kicks off our open-source journey, visualizing AI go
33
33
-**Compliance**: GDPR adherence metrics.
34
34
-**Accuracy**: Model performance monitoring (MLflow-compatible).
35
35
36
-
**DataOps Features**:
37
-
- Real-time refresh for agility (DataOps Principle #1).
38
-
- Open-source for collaboration (DataOps Principle #4).
DTE builds on the [DataOps Manifesto](https://dataopsmanifesto.org)’s agility and collaboration. Thanks to the DataOps community and contributors shaping DTE.
110
109
111
-
**Authors**: Brian Brewer | https://www.infolibcorp.com
110
+
**Authors**: Brian Brewer | Founder | https://www.infolibcorp.com
0 commit comments