Skip to content

Commit de32389

Browse files
tmosleyIIItmosleyIII
authored
update docs (#24)
* update docs * PR fixes * updated docker command
1 parent a908646 commit de32389

File tree

6 files changed

+659
-39
lines changed

6 files changed

+659
-39
lines changed

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,3 +64,4 @@ flowct
6464
docs/cdp-vs-benthos-comparison.md
6565
docs/future-inspired-enhancements.md
6666
docs/future-best-practices-for-cdp.md
67+
docs/SETUP_GITHUB_PAGES.md

‎README.md‎

Lines changed: 152 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,153 @@
11
# CDP Pipeline Workflow
22

3-
A data pipeline for processing Stellar blockchain data, with support for payment and account creation operations(WIP). Many of the consumers and processors are experimental and may not perform as they should.
3+
A modular data pipeline for processing Stellar blockchain data at scale. Process ledgers, transactions, operations, and contract events from multiple sources with flexible output destinations.
4+
5+
**Quick Start:** Visit [withobsrvr.github.io/cdp-pipeline-workflow](https://withobsrvr.github.io/cdp-pipeline-workflow) for installation instructions.
46

57
## Features
68

7-
- Processes Stellar blockchain data from multiple sources:
9+
- **Multiple Data Sources:**
810
- Amazon S3
9-
- Google Cloud Storage (with OAuth or Service Account)
11+
- Google Cloud Storage (OAuth or Service Account)
1012
- Local filesystem
11-
- Transforms operations into standardized formats
12-
- Supports both payment and create account operations(WIP)
13-
- Outputs to multiple destinations (MongoDB, ZeroMQ, PostgreSQL, DuckDB)
14-
- Processes account data and stores it in PostgreSQL or DuckDB
13+
- Stellar RPC endpoints
14+
- Captive Core (direct connection)
15+
16+
- **Flexible Processing:**
17+
- Payment operations
18+
- Account creation and management
19+
- Contract invocations and events
20+
- DEX trades and market data
21+
- Validator analytics
22+
23+
- **Output Destinations:**
24+
- PostgreSQL, MongoDB, ClickHouse, DuckDB
25+
- Redis (caching and orderbooks)
26+
- Google Cloud Storage, Excel
27+
- ZeroMQ, WebSocket streaming
1528

16-
## Prerequisites
29+
## Installation
1730

18-
- Go 1.22 or later
19-
- Access to one of:
20-
- Amazon S3
21-
- Google Cloud Storage
22-
- Local filesystem
23-
- MongoDB instance
24-
- ZeroMQ (optional)
31+
### Option 1: Docker (Recommended)
32+
33+
Pull the image:
34+
```bash
35+
docker pull withobsrvr/obsrvr-flow-pipeline:latest
36+
```
37+
38+
**Basic usage (local files):**
39+
```bash
40+
docker run --rm \
41+
-v $(pwd)/config:/app/config \
42+
withobsrvr/obsrvr-flow-pipeline:latest \
43+
/app/cdp-pipeline-workflow -config /app/config/pipeline.yaml
44+
```
45+
46+
**With Google Cloud Storage:**
47+
```bash
48+
# First authenticate: gcloud auth application-default login
49+
50+
docker run --rm --network host \
51+
-v "$HOME/.config/gcloud/application_default_credentials.json":/.config/gcp/credentials.json:ro \
52+
-e GOOGLE_APPLICATION_CREDENTIALS=/.config/gcp/credentials.json \
53+
-v $(pwd)/config:/app/config \
54+
withobsrvr/obsrvr-flow-pipeline:latest \
55+
/app/cdp-pipeline-workflow -config /app/config/pipeline.yaml
56+
```
57+
58+
**With AWS S3:**
59+
```bash
60+
docker run --rm --network host \
61+
-e AWS_ACCESS_KEY_ID=your_access_key \
62+
-e AWS_SECRET_ACCESS_KEY=your_secret_key \
63+
-e AWS_REGION=us-east-1 \
64+
-v $(pwd)/config:/app/config \
65+
withobsrvr/obsrvr-flow-pipeline:latest \
66+
/app/cdp-pipeline-workflow -config /app/config/pipeline.yaml
67+
```
68+
69+
### Option 2: Build from Source
70+
71+
**Prerequisites:**
72+
- Go 1.24 or later
73+
- GCC (for CGO support)
74+
- System libraries: libzmq3-dev, libczmq-dev, libsodium-dev
75+
76+
**Install dependencies:**
77+
78+
```bash
79+
# Ubuntu/Debian
80+
sudo apt-get install -y libzmq3-dev libczmq-dev libsodium-dev build-essential
81+
82+
# macOS
83+
brew install zeromq czmq libsodium
84+
```
85+
86+
**Build:**
87+
88+
```bash
89+
git clone https://github.com/withObsrvr/cdp-pipeline-workflow.git
90+
cd cdp-pipeline-workflow
91+
CGO_ENABLED=1 go build -o cdp-pipeline-workflow
92+
./cdp-pipeline-workflow -config config.yaml
93+
```
94+
95+
## Quick Start
96+
97+
Create a `config.yaml` file:
98+
99+
```yaml
100+
pipeline:
101+
name: MyFirstPipeline
102+
source:
103+
type: BufferedStorageSourceAdapter
104+
config:
105+
bucket_name: "stellar-ledgers"
106+
network: "mainnet"
107+
start_ledger: 1000
108+
end_ledger: 2000
109+
num_workers: 4
110+
processors:
111+
- type: LedgerMetaDecoder
112+
consumers:
113+
- type: SaveToPostgreSQL
114+
config:
115+
host: "localhost"
116+
port: 5432
117+
database: "stellar"
118+
username: "postgres"
119+
password: "password"
120+
```
121+
122+
Run the pipeline:
123+
124+
```bash
125+
# Docker
126+
docker run --rm \
127+
-v $(pwd)/config:/app/config \
128+
withobsrvr/obsrvr-flow-pipeline:latest \
129+
/app/cdp-pipeline-workflow -config /app/config/config.yaml
130+
131+
# Or from source
132+
./cdp-pipeline-workflow -config config.yaml
133+
```
134+
135+
## Environment Variables (for Source Builds)
136+
137+
When building from source, you'll need to set these environment variables:
138+
139+
**AWS S3:**
140+
```bash
141+
export AWS_ACCESS_KEY_ID=your_access_key
142+
export AWS_SECRET_ACCESS_KEY=your_secret_key
143+
export AWS_REGION=us-east-1
144+
```
145+
146+
**Google Cloud Storage:**
147+
```bash
148+
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
149+
# Or authenticate with: gcloud auth application-default login
150+
```
25151

26152
## Configuration
27153

@@ -206,36 +332,23 @@ pipelines:
206332
db_path: "data/stellar_accounts.duckdb"
207333
```
208334

209-
## Installation
210-
211-
```bash
212-
go get github.com/withObsrvr/cdp-pipeline-workflow
213-
```
214-
215-
## Usage
335+
## Additional Resources
216336

217-
For GCS authentication:
218-
```bash
219-
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
220-
./cdp-pipeline-workflow -config /path/to/config.yaml
221-
```
337+
- **Documentation:** [Full configuration guide](https://github.com/withObsrvr/cdp-pipeline-workflow/blob/main/README.md)
338+
- **Examples:** Check the `config/` directory for sample configurations
339+
- **Issues:** [Report bugs or request features](https://github.com/withObsrvr/cdp-pipeline-workflow/issues)
340+
- **Contributing:** Pull requests are welcome!
222341

223-
For S3 authentication:
224-
```bash
225-
export AWS_ACCESS_KEY_ID=your_access_key
226-
export AWS_SECRET_ACCESS_KEY=your_secret_key
227-
./cdp-pipeline-workflow -config /path/to/config.yaml
228-
```
342+
## GCS OAuth Authentication (Advanced)
229343

230-
For GCS OAuth authentication, you'll need to:
344+
For OAuth-based Google Cloud Storage access:
231345

232346
1. Create a Google Cloud OAuth 2.0 Client ID:
233-
- Go to Google Cloud Console -> APIs & Services -> Credentials
347+
- Go to Google Cloud Console → APIs & Services → Credentials
234348
- Create a new OAuth 2.0 Client ID
235349
- Download the client configuration file
236350

237-
2. Get an OAuth token using the Google OAuth 2.0 Playground:
238-
- Visit https://developers.google.com/oauthplayground/
351+
2. Get an OAuth token using the [Google OAuth 2.0 Playground](https://developers.google.com/oauthplayground/):
239352
- Configure OAuth 2.0 with your client ID and secret
240353
- Select and authorize the "Google Cloud Storage API v1"
241354
- Exchange authorization code for tokens
@@ -249,6 +362,6 @@ pipeline:
249362
type: GCSBufferedStorageSourceAdapter
250363
config:
251364
access_token: "your-access-token"
252-
```
365+
```
253366

254-
Note: OAuth tokens are temporary and will expire. For production use, consider using service account authentication instead.
367+
**Note:** OAuth tokens are temporary and will expire. For production use, consider using service account authentication instead (via `GOOGLE_APPLICATION_CREDENTIALS`).

‎docs/.nojekyll‎

Whitespace-only changes.

‎docs/README.md‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# CDP Pipeline Workflow - Documentation
2+
3+
This directory contains the GitHub Pages landing page for CDP Pipeline Workflow.
4+
5+
## Contents
6+
7+
- **index.html** - Main installation landing page (rustup.rs style)
8+
- **_config.yml** - GitHub Pages configuration
9+
- **.nojekyll** - Prevents Jekyll processing (we use custom HTML)
10+
- **SETUP_GITHUB_PAGES.md** - Guide for enabling GitHub Pages
11+
12+
## Live Site
13+
14+
Once GitHub Pages is enabled, the landing page will be available at:
15+
16+
**https://withobsrvr.github.io/cdp-pipeline-workflow/**
17+
18+
## Local Development
19+
20+
To preview the landing page locally:
21+
22+
```bash
23+
# Simple HTTP server (Python)
24+
cd docs/
25+
python3 -m http.server 8000
26+
27+
# Or use Node.js
28+
npx serve
29+
30+
# Or use PHP
31+
php -S localhost:8000
32+
```
33+
34+
Then visit: http://localhost:8000
35+
36+
## Features
37+
38+
The landing page provides:
39+
40+
- **Two Installation Options:**
41+
- Docker (recommended)
42+
- Build from source
43+
44+
- **Copy-to-Clipboard:** One-click command copying
45+
- **Responsive Design:** Works on mobile and desktop
46+
- **Minimal Dependencies:** Pure HTML/CSS/JS (no frameworks)
47+
48+
## Updating
49+
50+
To update the landing page:
51+
52+
1. Edit `index.html`
53+
2. Test locally (optional)
54+
3. Commit and push:
55+
```bash
56+
git add docs/index.html
57+
git commit -m "Update landing page"
58+
git push
59+
```
60+
61+
GitHub Pages will automatically rebuild in 1-3 minutes.
62+
63+
## Customization
64+
65+
### Colors
66+
67+
The gradient background uses these colors:
68+
- Primary: `#667eea` (purple)
69+
- Secondary: `#764ba2` (darker purple)
70+
71+
To change colors, edit the CSS in `index.html`:
72+
73+
```css
74+
body {
75+
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
76+
}
77+
```
78+
79+
### Tabs
80+
81+
To add/remove installation tabs, edit the HTML structure:
82+
83+
```html
84+
<div class="tabs">
85+
<div class="tab active" onclick="switchTab('docker')">Docker</div>
86+
<!-- Add more tabs here -->
87+
</div>
88+
89+
<div id="docker" class="tab-content active">
90+
<!-- Tab content here -->
91+
</div>
92+
```
93+
94+
### Links
95+
96+
Update links in the footer section:
97+
98+
```html
99+
<a href="https://github.com/withObsrvr/cdp-pipeline-workflow" class="link-button">
100+
View on GitHub
101+
</a>
102+
```
103+
104+
## Browser Compatibility
105+
106+
Tested and working on:
107+
- Chrome/Edge (latest)
108+
- Firefox (latest)
109+
- Safari (latest)
110+
- Mobile browsers (iOS Safari, Chrome Mobile)
111+
112+
## License
113+
114+
Same as the main project license.

‎docs/_config.yml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# GitHub Pages Configuration
2+
# This file configures GitHub Pages for the CDP Pipeline Workflow documentation
3+
4+
title: CDP Pipeline Workflow
5+
description: Process Stellar blockchain data at scale
6+
baseurl: "/cdp-pipeline-workflow"
7+
url: "https://withobsrvr.github.io"
8+
9+
# Theme (optional - can use default)
10+
# theme: minima
11+
12+
# Plugins
13+
plugins:
14+
- jekyll-feed
15+
- jekyll-seo-tag
16+
17+
# Build settings
18+
markdown: kramdown
19+
20+
# Exclude from processing
21+
exclude:
22+
- README.md

0 commit comments

Comments
 (0)