Claude/fetch stock gold data ak t ea #215
Open
+2,607
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Demonstrates how to use vnstock library to fetch: - Historical stock prices (OHLCV) from KBS/VCI/TCBS sources - SJC gold prices by date - BTMC gold prices (current) - Company overview information - VCB exchange rates https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
Automated script that runs daily after market close (15:30) to collect: - All listed stock symbols (HOSE, HNX, UPCOM) - Price board data (batch) for all stocks - OHLCV history for each stock - SJC and BTMC gold prices - VCB exchange rates Supports: date selection, backfill, cron install/remove, VCI/KBS sources. Data saved to data/YYYY-MM-DD/ with latest symlink. https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
- Scheduled cron at 15:30 Vietnam time (08:30 UTC), Mon-Fri - Manual trigger via workflow_dispatch with date/skip_ohlcv options - Auto commits collected data back to repo - Also uploads data as artifacts (90-day retention) - Updated .gitignore to allow CSV files in data/ directory https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
- Increase batch size 15 -> 50 symbols per batch - Reduce delays: batch 3.5s -> 0.5s, request 0.3s -> 0.12s - Enable full OHLCV collection by default (skip_ohlcv default: false) - Add API key registration step in workflow - Reduce workflow timeout 60 -> 30 min (faster with Golden tier) https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
New workflow "Backfill Historical Data" with manual trigger to fetch: - OHLCV history for all stocks (or specific symbols) over a date range - SJC gold price history day by day - Data saved to data/history/ohlcv_all.csv and sjc_gold_history.csv Supports custom start/end dates, symbol filtering, VCI/KBS sources. https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
New workflow "Collect Market Indices" fetches daily data for: - VNINDEX, HNXINDEX, UPCOMINDEX, VN30, HNX30 Includes technical indicators (SMA, RSI, MACD, Bollinger Bands) and generates visualization charts: - Overview comparison (normalized %) - Per-index detailed charts (Price + MA + RSI + MACD) - Volume comparison across exchanges Also adds standalone indicators.py module for TA calculations. https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
Expanded from 5 to 18 indices: - Main: VNINDEX, HNXINDEX, UPCOMINDEX, VN30, HNX30 - Size: VN100, VNMIDCAP, VNSML - Sectors: VNFIN, VNREAL, VNIT, VNHEAL, VNENE, VNCONS, VNMAT, VNCOND - Investment: VNDIAMOND, VNFINSELECT Added KBS _INDEX_MAPPING for sector/size/investment indices. Charts now grouped: overview_main, overview_sectors, overview_size, overview_invest, overview_all + individual TA charts per index. https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
KBS only supports 6 main indices (VNINDEX, HNXINDEX, UPCOMINDEX, VN30, HNX30, VN100). For the 12 remaining indices (VNFIN, VNREAL, VNIT, etc.), call VCI's chart API directly with requests, bypassing vnstock's validation that rejects unsupported symbols. Also fixes symbol names: VNMIDCAP -> VNMID (correct symbol per HOSE). https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
- Auto-detect timestamp format (seconds/milliseconds/string) - Log VCI response status, structure, and timestamp samples for debugging - Handle error responses gracefully (dict responses, non-200 status) - Move VN100 from KBS to VCI-direct (KBS only returns 1 row for VN100) - Drop NaT rows from parsed timestamps https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
VCI chart API returns empty data for all sector indices (VNFIN, VNREAL, etc.) - confirmed by debugging. Switched to SSI FiinTrade REST API (fiin-market.ssi.com.vn/MarketInDepth/GetIndexSeries) which was the data source used by vnstock v0.x for all 25+ Vietnamese market indices. Strategy: - KBS: 5 main indices (VNINDEX, HNXINDEX, UPCOMINDEX, VN30, HNX30) with full OHLCV data - SSI FiinTrade: 13 sector/size/investment indices (VNFIN, VNREAL, VN100, VNMID, VNSML, etc.) with close + volume data SSI API provides indexValue (close) and totalMatchVolume. Open/High/Low are set equal to close since the API doesn't provide them - sufficient for comparison charts, SMA, RSI, MACD analysis. https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
Replace SSI FiinTrade API (which required auth headers and only returned close+volume) with VNDirect dchart API (no auth, full OHLCV data in TradingView UDF format). This is the same backend used by vnstock-data's Data Explorer, confirmed to support all 13 non-KBS indices including sector (VNFIN, VNREAL, etc.), size (VN100, VNMID, VNSML), and investment (VNDIAMOND, VNFINSELECT) indices. https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
pd.to_numeric() returns numpy.ndarray which doesn't have .fillna(). Use np.nan_to_num() instead to handle NaN values in volume data. https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
KBS data was inaccurate for indices (e.g. VNINDEX showing 1760 instead of 1755 on Feb 6). Remove KBS dependency entirely and use VNDirect dchart API as the single data source for all 18 indices. VNDirect dchart API provides full OHLCV data via TradingView UDF format, no auth needed, and supports all indices including main, sector, size, and investment indices. Also added debug logging showing last row OHLCV values for each index to make it easy to verify data accuracy. https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
Add get_top_movers() function to daily_collector.py that: - Fetches KBS price_board data for all stocks (flat format with symbol, percent_change, listed_shares, close_price) - Computes market cap = close_price * listed_shares - Filters top 500 stocks by market cap - Picks top 30 gainers and top 30 losers by percent_change - Saves as top_gainers.csv and top_losers.csv Output columns: symbol, close_price, percent_change, total_trades, total_value - ranked by percent change. https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
Add scripts/db_cache.py with SQLite-based caching: - index_ohlcv table: caches index OHLCV data (18 indices) - stock_ohlcv table: caches stock OHLCV data (~1600 stocks) - price_board table: reserved for future use collect_indices.py integration: - Check cache before calling VNDirect API - Only fetch dates not in cache (+ re-fetch last 3 days for freshness) - Log cache hit stats (e.g. "252 phien (API: 3 moi)") daily_collector.py integration: - Skip stocks already cached for the target date - Log cache vs API breakdown (e.g. "cache: 1500, API: 100") - Dramatically reduces API calls on re-runs Database stored at data/cache.db (auto-created on first run). https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
- Refactor: fetch KBS price_board once, derive multiple datasets - Add market breadth (advancing/declining/unchanged per exchange) - Add foreign flow (buy/sell/net volume + value per exchange) - Add top 20 foreign net buyers/sellers - Update collect-indices.yml commit message for all 18 indices - Daily collector now 8 steps (was 6) New output files: market_breadth.csv, foreign_flow.csv, foreign_top_buy.csv, foreign_top_sell.csv https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
- compute_index_impact(): approximate each HOSE stock's impact on VNINDEX using market_cap * price_change - Output: index_impact_positive.csv (top 20 kéo tăng), index_impact_negative.csv (top 20 kéo giảm) - Daily collector now 9 steps (was 8) - Completes all 3 dashboard data gaps: market breadth, foreign flow, index impact https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
- Xóa daily-collector.yml và collect-indices.yml - Tạo dashboard.yml chạy cả 2 script tuần tự: [1/2] daily_collector.py (cổ phiếu, vàng, tỷ giá, breadth, foreign, impact) [2/2] collect_indices.py (18 chỉ số + charts) - Cron: 08:30 UTC (15:30 VN), thứ 2-6 - 1 commit duy nhất cho tất cả data - Timeout 45 phút (đủ cho cả 2 script) https://claude.ai/code/session_01YYqYGm6Pyi53jiC59fNoCf
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.