Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 39 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,8 @@
| Cloud-dependent AI tools | **Run AI locally** with Ollama/LM Studio — no API keys, no data leaves your machine |
| Web-only dashboards | **Native desktop app** (Tauri) for macOS, Windows, and Linux + installable PWA with offline map support |
| Flat 2D maps | **3D WebGL globe** with deck.gl rendering and 36+ toggleable data layers |
| Siloed financial data | **Finance variant** with 92 stock exchanges, 19 financial centers, 13 central banks, and Gulf FDI tracking |
| Undocumented, fragile APIs | **Proto-first API contracts** — 17 typed services with auto-generated clients, servers, and OpenAPI docs |
| Siloed financial data | **Finance variant** with 92 stock exchanges, 19 financial centers, 13 central banks, BIS data, WTO trade policy, and Gulf FDI tracking |
| Undocumented, fragile APIs | **Proto-first API contracts** — 19 typed services with auto-generated clients, servers, and OpenAPI docs |

---

Expand Down Expand Up @@ -166,8 +166,10 @@ All four variants run from a single codebase — switch between them with one cl
- 92 global stock exchanges — mega (NYSE, NASDAQ, Shanghai, Euronext, Tokyo), major (Hong Kong, London, NSE/BSE, Toronto, Korea, Saudi Tadawul), and emerging markets — with market caps and trading hours
- 19 financial centers — ranked by Global Financial Centres Index (New York #1 through offshore centers: Cayman Islands, Luxembourg, Bermuda, Channel Islands)
- 13 central banks — Federal Reserve, ECB, BoJ, BoE, PBoC, SNB, RBA, BoC, RBI, BoK, BCB, SAMA, plus supranational institutions (BIS, IMF)
- BIS central bank data — policy rates across major economies, real effective exchange rates (REER), and credit-to-GDP ratios sourced from the Bank for International Settlements
- 10 commodity hubs — exchanges (CME Group, ICE, LME, SHFE, DCE, TOCOM, DGCX, MCX) and physical hubs (Rotterdam, Houston)
- Gulf FDI investment layer — 64 Saudi/UAE foreign direct investments plotted globally, color-coded by status (operational, under-construction, announced), sized by investment amount
- WTO trade policy intelligence — active trade restrictions, tariff trends, bilateral trade flows, and SPS/TBT barriers sourced from the World Trade Organization

</details>

Expand Down Expand Up @@ -227,7 +229,7 @@ All four variants run from a single codebase — switch between them with one cl
- Prediction market integration (Polymarket) with 3-tier JA3 bypass (browser-direct → Tauri native TLS → cloud proxy)
- Service status monitoring (cloud providers, AI services)
- Shareable map state via URL parameters (view, zoom, coordinates, time range, active layers)
- Data freshness monitoring across 14 data sources with explicit intelligence gap reporting
- Data freshness monitoring across 16 data sources with explicit intelligence gap reporting
- Per-feed circuit breakers with 5-minute cooldowns to prevent cascading failures
- Browser-side ML worker (Transformers.js) for NER and sentiment analysis without server dependency
- **Cmd+K command palette** — fuzzy search across 20+ result types (news, countries, hotspots, markets, bases, cables, datacenters, nuclear facilities, and more), plus layer toggle commands, layer presets (e.g., `layers:finance`), and instant country brief navigation for all ISO countries
Expand Down Expand Up @@ -542,7 +544,7 @@ Detected spikes are auto-summarized via Groq (rate-limited to 5 summaries/hour)

The entire API surface is defined in Protocol Buffer (`.proto`) files using [sebuf](https://github.com/SebastienMelki/sebuf) HTTP annotations. Code generation produces TypeScript clients, server handler stubs, and OpenAPI 3.1.0 documentation from a single source of truth — eliminating request/response schema drift between frontend and backend.

**17 service domains** cover every data vertical:
**19 service domains** cover every data vertical:

| Domain | RPCs |
| ---------------- | ------------------------------------------------ |
Expand All @@ -551,7 +553,7 @@ The entire API surface is defined in Protocol Buffer (`.proto`) files using [seb
| `conflict` | ACLED events, UCDP events, humanitarian summaries|
| `cyber` | Cyber threat IOCs |
| `displacement` | Population displacement, exposure data |
| `economic` | Energy prices, FRED series, macro signals, World Bank |
| `economic` | Energy prices, FRED series, macro signals, World Bank, BIS policy rates, exchange rates, credit-to-GDP |
| `infrastructure` | Internet outages, service statuses, temporal baselines |
| `intelligence` | Event classification, country briefs, risk scores|
| `maritime` | Vessel snapshots, navigational warnings |
Expand All @@ -561,6 +563,7 @@ The entire API surface is defined in Protocol Buffer (`.proto`) files using [seb
| `prediction` | Prediction markets |
| `research` | arXiv papers, HackerNews, tech events |
| `seismology` | Earthquakes |
| `trade` | WTO trade restrictions, tariff trends, trade flows, trade barriers |
| `unrest` | Protest/unrest events |
| `wildfire` | Fire detections |

Expand Down Expand Up @@ -806,7 +809,7 @@ Activity spikes at individual locations boost the aggregate score (+10 per spike

### Data Freshness & Intelligence Gaps

A singleton tracker monitors 22 data sources (GDELT, RSS, AIS, military flights, earthquakes, weather, outages, ACLED, Polymarket, economic indicators, NASA FIRMS, cyber threat feeds, trending keywords, oil/energy, population exposure, and more) with status categorization: fresh (<15 min), stale (1h), very_stale (6h), no_data, error, disabled. It explicitly reports **intelligence gaps** — what analysts can't see — preventing false confidence when critical data sources are down or degraded.
A singleton tracker monitors 24 data sources (GDELT, RSS, AIS, military flights, earthquakes, weather, outages, ACLED, Polymarket, economic indicators, NASA FIRMS, cyber threat feeds, trending keywords, oil/energy, population exposure, BIS central bank data, WTO trade policy, and more) with status categorization: fresh (<15 min), stale (1h), very_stale (6h), no_data, error, disabled. It explicitly reports **intelligence gaps** — what analysts can't see — preventing false confidence when critical data sources are down or degraded.

### Prediction Markets as Leading Indicators

Expand Down Expand Up @@ -890,6 +893,31 @@ The Oil & Energy panel tracks four key indicators from the U.S. Energy Informati

Trend detection flags week-over-week changes exceeding ±0.5% as rising or falling, with flat readings within the threshold shown as stable. Results are cached client-side for 30 minutes. The panel provides energy market context for geopolitical analysis — price spikes often correlate with supply disruptions in monitored conflict zones and chokepoint closures.

### BIS Central Bank Data

The Economic panel integrates data from the Bank for International Settlements (BIS), the central bank of central banks, providing three complementary datasets:

| Dataset | Description | Use Case |
| --- | --- | --- |
| **Policy Rates** | Current central bank policy rates across major economies | Monetary policy stance comparison — tight vs. accommodative |
| **Real Effective Exchange Rates** | Trade-weighted currency indices adjusted for inflation (REER) | Currency competitiveness — rising REER = strengthening, falling = weakening |
| **Credit-to-GDP** | Total credit to the non-financial sector as percentage of GDP | Credit bubble detection — high ratios signal overleveraged economies |

Data is fetched through three dedicated BIS RPCs (`GetBisPolicyRates`, `GetBisExchangeRates`, `GetBisCredit`) in the `economic/v1` proto service. Each dataset uses independent circuit breakers with 30-minute cache TTLs. The panel renders policy rates as a sorted table with spark bars, exchange rates with directional trend indicators, and credit-to-GDP as a ranked list. BIS data freshness is tracked in the intelligence gap system — staleness or failures surface as explicit warnings rather than silent gaps.

### WTO Trade Policy Intelligence

The Trade Policy panel provides real-time visibility into global trade restrictions, tariffs, and barriers — critical for tracking economic warfare, sanctions impact, and supply chain disruption risk. Four data views are available:

| Tab | Data Source | Content |
| --- | --- | --- |
| **Restrictions** | WTO trade monitoring | Active trade restrictions with imposing/affected countries, product categories, and enforcement dates |
| **Tariffs** | WTO tariff database | Tariff rate trends between country pairs (e.g., US↔China) with historical datapoints |
| **Flows** | WTO trade statistics | Bilateral trade flow volumes with year-over-year change indicators |
| **Barriers** | WTO SPS/TBT notifications | Sanitary, phytosanitary, and technical barriers to trade with status tracking |

The `trade/v1` proto service defines four RPCs, each with its own circuit breaker (30-minute cache TTL) and `upstreamUnavailable` signaling for graceful degradation when WTO endpoints are temporarily unreachable. The panel is available on FULL and FINANCE variants. Trade policy data feeds into the data freshness tracker as `wto_trade`, with intelligence gap warnings when the WTO feed goes stale.

### BTC ETF Flow Estimation

Ten spot Bitcoin ETFs are tracked via Yahoo Finance's 5-day chart API (IBIT, FBTC, ARKB, BITB, GBTC, HODL, BRRR, EZBC, BTCO, BTCW). Since ETF flow data requires expensive terminal subscriptions, the system estimates flow direction from publicly available signals:
Expand All @@ -911,7 +939,7 @@ A single codebase produces three specialized dashboards, each with distinct feed
| **Domain** | worldmonitor.app | tech.worldmonitor.app | finance.worldmonitor.app |
| **Focus** | Geopolitics, military, conflicts | AI/ML, startups, cybersecurity | Markets, trading, central banks |
| **RSS Feeds** | ~25 categories (politics, MENA, Africa, think tanks) | ~20 categories (AI, VC blogs, startups, GitHub) | ~18 categories (forex, bonds, commodities, IPOs) |
| **Panels** | 44 (strategic posture, CII, cascade) | 31 (AI labs, unicorns, accelerators) | 30 (forex, bonds, derivatives, institutional) |
| **Panels** | 45 (strategic posture, CII, cascade, trade policy) | 31 (AI labs, unicorns, accelerators) | 31 (forex, bonds, derivatives, trade policy) |
| **Unique Map Layers** | Military bases, nuclear facilities, hotspots | Tech HQs, cloud regions, startup hubs | Stock exchanges, central banks, Gulf investments |
| **Desktop App** | World Monitor.app / .exe / .AppImage | Tech Monitor.app / .exe / .AppImage | Finance Monitor.app / .exe / .AppImage |

Expand Down Expand Up @@ -962,11 +990,13 @@ World Monitor uses 60+ Vercel Edge Functions as a lightweight API layer, split i

- **RSS Proxy** — domain-allowlisted proxy for 100+ feeds, preventing CORS issues and hiding origin servers. Feeds from domains that block Vercel IPs are automatically routed through the Railway relay.
- **AI Pipeline** — Groq and OpenRouter edge functions with Redis deduplication, so identical headlines across concurrent users only trigger one LLM call. The classify-event endpoint pauses its queue on 500 errors to avoid wasting API quota.
- **Data Adapters** — GDELT, ACLED, OpenSky, USGS, NASA FIRMS, FRED, Yahoo Finance, CoinGecko, mempool.space, and others each have dedicated edge functions that normalize responses into consistent schemas
- **Data Adapters** — GDELT, ACLED, OpenSky, USGS, NASA FIRMS, FRED, Yahoo Finance, CoinGecko, mempool.space, BIS, WTO, and others each have dedicated edge functions that normalize responses into consistent schemas
- **Market Intelligence** — macro signals, ETF flows, and stablecoin monitors compute derived analytics server-side (VWAP, SMA, peg deviation, flow estimates) and cache results in Redis
- **Temporal Baseline** — Welford's algorithm state is persisted in Redis across requests, building statistical baselines without a traditional database
- **Custom Scrapers** — sources without RSS feeds (FwdStart, GitHub Trending, tech events) are scraped and transformed into RSS-compatible formats
- **Finance Geo Data** — stock exchanges (92), financial centers (19), central banks (13), and commodity hubs (10) are served as static typed datasets with market caps, GFCI rankings, trading hours, and commodity specializations
- **BIS Integration** — policy rates, real effective exchange rates, and credit-to-GDP ratios from the Bank for International Settlements, cached with 30-minute TTL
- **WTO Trade Policy** — trade restrictions, tariff trends, bilateral trade flows, and SPS/TBT barriers from the World Trade Organization

All edge functions include circuit breaker logic and return cached stale data when upstream APIs are unavailable, ensuring the dashboard never shows blank panels.

Expand All @@ -980,7 +1010,7 @@ All three variants run on three platforms that work together:
┌─────────────────────────────────────┐
│ Vercel (Edge) │
│ 60+ edge functions · static SPA │
│ Proto gateway (17 typed services) │
│ Proto gateway (19 typed services) │
│ CORS allowlist · Redis cache │
│ AI pipeline · market analytics │
│ CDN caching (s-maxage) · PWA host │
Expand Down
13 changes: 9 additions & 4 deletions server/worldmonitor/trade/v1/_shared.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,12 @@ import { CHROME_UA } from '../../../_shared/constants';
/** WTO Timeseries API base URL. */
export const WTO_API_BASE = 'https://api.wto.org/timeseries/v1';

/** Merchandise exports (total). */
/** Merchandise exports (total) — annual. */
export const ITS_MTV_AX = 'ITS_MTV_AX';
/** Merchandise imports (total). */
/** Merchandise imports (total) — annual. */
export const ITS_MTV_AM = 'ITS_MTV_AM';
/** Applied tariff — HS simple average. */
export const HS_M_0010 = 'HS_M_0010';
/** Simple average MFN applied tariff — all products. */
export const TP_A_0010 = 'TP_A_0010';

/**
* WTO member numeric codes → human-readable names.
Expand All @@ -42,6 +42,9 @@ export const WTO_MEMBER_CODES: Record<string, string> = {
/**
* Fetch JSON from the WTO Timeseries API.
* Returns parsed JSON on success, or null if the API key is missing or the request fails.
*
* IMPORTANT: The WTO API does NOT support comma-separated indicator codes in the `i` param.
* Each indicator must be queried separately.
*/
export async function wtoFetch(
path: string,
Expand All @@ -66,6 +69,8 @@ export async function wtoFetch(
signal: AbortSignal.timeout(15000),
});

// 204 = No Content (valid query, no matching data)
if (res.status === 204) return { Dataset: [] };
if (!res.ok) return null;
return await res.json();
} catch {
Expand Down
25 changes: 12 additions & 13 deletions server/worldmonitor/trade/v1/get-tariff-trends.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
/**
* RPC: getTariffTrends -- WTO applied tariff trend data
* Fetches HS simple average applied tariff rates over time.
* Fetches MFN simple average applied tariff rates over time.
*
* NOTE: Tariff indicators (TP_A_*) do NOT have a partner dimension.
* The `partnerCountry` request field is accepted but not sent to the API.
*/

declare const process: { env: Record<string, string | undefined> };
Expand All @@ -13,7 +16,7 @@ import type {
} from '../../../../src/generated/server/worldmonitor/trade/v1/service_server';

import { getCachedJson, setCachedJson } from '../../../_shared/redis';
import { wtoFetch, WTO_MEMBER_CODES, HS_M_0010 } from './_shared';
import { wtoFetch, WTO_MEMBER_CODES, TP_A_0010 } from './_shared';

const REDIS_CACHE_TTL = 21600; // 6h

Expand All @@ -37,38 +40,34 @@ function toDataPoint(row: any, reporter: string, partner: string): TariffDataPoi
reportingCountry:
WTO_MEMBER_CODES[reporter] ?? String(row.ReportingEconomy ?? row.reportingEconomy ?? reporter),
partnerCountry:
WTO_MEMBER_CODES[partner] ?? String(row.PartnerEconomy ?? row.partnerEconomy ?? partner),
productSector: String(row.ProductOrSector ?? row.productOrSector ?? 'Total'),
WTO_MEMBER_CODES[partner] ?? String(row.PartnerEconomy ?? row.partnerEconomy ?? (partner || 'World')),
productSector: String(row.ProductOrSector ?? row.productOrSector ?? 'All products'),
year,
tariffRate: Math.round(tariffRate * 100) / 100,
boundRate: parseFloat(row.BoundRate ?? row.boundRate ?? '0') || 0,
indicatorCode: String(row.IndicatorCode ?? row.indicatorCode ?? HS_M_0010),
indicatorCode: String(row.IndicatorCode ?? row.indicatorCode ?? TP_A_0010),
};
}

async function fetchTariffTrends(
reporter: string,
partner: string,
productSector: string,
_productSector: string,
years: number,
): Promise<{ datapoints: TariffDataPoint[]; ok: boolean }> {
const currentYear = new Date().getFullYear();
const startYear = currentYear - years;

// Tariff indicators do NOT support the partner (p) parameter.
const params: Record<string, string> = {
i: HS_M_0010,
i: TP_A_0010,
r: reporter,
p: partner || '000',
ps: `${startYear}-${currentYear}`,
fmt: 'json',
mode: 'full',
max: '500',
};

if (productSector) {
params.pc = productSector;
}

const data = await wtoFetch('/data', params);
if (!data) return { datapoints: [], ok: false };

Expand All @@ -92,7 +91,7 @@ export async function getTariffTrends(
const productSector = isValidCode(req.productSector) ? req.productSector : '';
const years = Math.max(1, Math.min(req.years > 0 ? req.years : 10, 30));

const cacheKey = `trade:tariffs:v1:${reporter}:${partner}:${productSector || 'all'}:${years}`;
const cacheKey = `trade:tariffs:v1:${reporter}:${productSector || 'all'}:${years}`;
const cached = (await getCachedJson(cacheKey)) as GetTariffTrendsResponse | null;
if (cached?.datapoints?.length) return cached;

Expand Down
Loading