- chore(deps): update all non-major dependencies (
1a1af87
)
- fix(deps): update all non-major dependencies (
cd0d706
)
- fix(deps): update all non-major dependencies (
c6c347d
)
-
fix(spider): update pattern for get_product_quantity (
34660e0
) -
fix(spider): update coldstorage spider due to change in site html structure (
6509cac
) -
fix(spider): update hopshop spider due to change in site html structure (
42f1c30
)
- fix(deps): update all non-major dependencies (
350bf81
)
- fix(deps): update all non-major dependencies (
a7e1318
)
- fix(deps): update all non-major dependencies (
4612a86
)
- fix(deps): update all non-major dependencies (
f6085c6
)
- chore: update pre-commit hooks versions (
d7b5b12
)
- fix(spider): use localized cookies when sending requests (
a1c50cb
)
- fix(deps): update all non-major dependencies (
ea35f0b
)
- chore(deps): update dependency flake8 to v7 (
1ccc5de
)
- fix(dockerfile): revert ARG changes (
0764e61
)
- Merge pull request #162 from ngshiheng/renovate/flake8-7.x
chore(deps): update dependency flake8 to v7 (aa872d2
)
-
chore: bump python version to ^3.10 (
775a62f
) -
chore(deps): update dependency mypy to v1 (
79c3ee4
) -
chore(deps): update abatilo/actions-poetry action to v3 (
7b1d851
)
- fix(deps): update all non-major dependencies (
093e5a4
)
- Merge pull request #158 from ngshiheng/renovate/all-minor-patch
fix(deps): update all non-major dependencies (f61a214
)
- Merge pull request #145 from ngshiheng/renovate/mypy-1.x
chore(deps): update dependency mypy to v1 (479f864
)
- Merge pull request #161 from ngshiheng/renovate/abatilo-actions-poetry-3.x
chore(deps): update abatilo/actions-poetry action to v3 (ed8f6bb
)
- chore(deps): update actions/setup-python action to v5 (
12bde47
)
- fix(spider): update craftbeersg due to site html layout change (
b9eaf08
)
- Merge pull request #160 from ngshiheng/fix-craftbeersg-spider
fix(spider): update craftbeersg due to site html layout change (8328121
)
- Merge pull request #159 from ngshiheng/renovate/actions-setup-python-5.x
chore(deps): update actions/setup-python action to v5 (9634979
)
- chore(deps): update dependency pylint to v3 (
071cbd3
)
- Merge pull request #157 from ngshiheng/renovate/all-minor-patch
fix(deps): update all non-major dependencies (ad03b2f
)
- Merge pull request #156 from ngshiheng/renovate/pylint-3.x
chore(deps): update dependency pylint to v3 (6cea93f
)
- chore(deps): update actions/checkout action to v4 (
47c3127
)
- fix(deps): update all non-major dependencies (
063d96c
)
- Merge pull request #154 from ngshiheng/renovate/actions-checkout-4.x
chore(deps): update actions/checkout action to v4 (169a7ee
)
- fix(deps): update all non-major dependencies (
7c26430
)
- fix(deps): update all non-major dependencies (
b9d8fa5
)
- fix(deps): update all non-major dependencies (
f031116
)
- fix(deps): update all non-major dependencies (
9d2c966
)
- fix(deps): update all non-major dependencies (
a7115e1
)
- fix(deps): update all non-major dependencies (
30bbed8
)
- fix: update domain name (
633cc70
)
- chore(deps): update all non-major dependencies (
672a325
)
- fix(deps): update all non-major dependencies (
9cb0f35
)
- chore(deps): update dependency pre-commit to v3 (
c299328
)
-
ci: update poeetry version to 1.3.2 and remove paths (
8ecd762
) -
ci: update python-semantic-releas to master (
706b9b1
)
- style: update log colors (
2f31d89
)
- Merge pull request #140 from ngshiheng/renovate/pre-commit-3.x
chore(deps): update dependency pre-commit to v3 (4a3ebf9
)
- chore: remove docker inspect running state check (
368175a
)
- fix(spider): handle page not found error and invalid price (
f40ca3b
)
-
chore(docker): bump POETRY_VERSION (
8d292a9
) -
chore(renovate): group all non major updates (
fc92cd9
) -
chore: update pre-commit isort (
0f60365
)
-
fix(spider): handle cass where product_info does not have abv (
7b4b722
) -
fix(spider): skip craftbeersg item without quantity (
6d1161c
)
- refactor: update docstrings, logs, and other misc. things (
a3c24c3
)
- fix(deps): update dependency sentry-sdk to v1.14.0 (
07bc1a5
)
-
chore(deps): update dependency pre-commit to v2.21.0 (
a6337aa
) -
chore(deps): update dependency mypy to v0.991 (
967dd30
) -
chore(deps): update dependency pylint to v2.15.10 (
d2a220a
)
- fix(deps): update dependency ipython to v8.9.0 (
a4efa4e
)
- fix(deps): update dependency alembic to v1.9.2 (
7fad6b4
)
- Merge pull request #137 from ngshiheng/renovate/alembic-1.x
fix(deps): update dependency alembic to v1.9.2 (08c60a5
)
- feat: add craftbeersg spider (
f9badd0
)
- style: update comments and spacing (
6dfb862
)
- Merge pull request #142 from ngshiheng/add-craftbeersg-spider
feat: add craftbeersg spider (7f7d0ed
)
-
chore(deps): pin dependency mypy to 0.910 (
ad0e004
) -
chore: remove setuptools (
9f891ce
) -
chore: update Makefile to add new target to setup postgres in Docker (
7ab3e6f
) -
chore(deps): update dependency autopep8 to v2.0.1 (
37ca00d
)
-
docs: move some content to DEVELOPMENT.md (
e1bd085
) -
docs: move sections into different files (
e267f1d
)
- fix(deps): update dependency sqlalchemy to v1.4.46 (
c31a78f
)
- Merge pull request #134 from ngshiheng/renovate/sqlalchemy-1.x
fix(deps): update dependency sqlalchemy to v1.4.46 (af58e15
)
- Merge pull request #131 from ngshiheng/renovate/pin-dependencies
chore(deps): pin dependency mypy to 0.910 (4ecc2ef
)
-
BREAKING CHANGE: thirsty site html layout change (
5804ddd
) -
BREAKING CHANGE: removed giant and shopee spiders (
2b687b7
)
- fix(spider): increase download delay for shopee (
46d154a
)
- fix(spider): handle captcha challenge (
2bc54e7
)
-
chore(deps): update dependency flake8 to v6 (
b515441
) -
chore(deps): update dependency autopep8 to v2 (
418728d
)
- fix(spider): api change (
ca66838
)
- Merge pull request #130 from ngshiheng/renovate/flake8-6.x
chore(deps): update dependency flake8 to v6 (b375e9d
)
- Merge pull request #123 from ngshiheng/renovate/autopep8-2.x
chore(deps): update dependency autopep8 to v2 (9b03570
)
- fix(deps): update dependency sentry-sdk to v1.11.1 (
237bc08
)
- fix(deps): update dependency scrapy to v2.7.1 (
5f708e6
)
-
chore(deps): update dependency pylint to v2.15.8 (
17ea8f0
) -
chore(deps): update dependency autopep8 to v1.7.0 (
1436a21
)
- fix(deps): update dependency ipython to v8.7.0 (
4ee24b4
)
- fix: poetry tries to remove setuptools and then complains it can't find it (
88b2a81
)
- chore: run poetry update (
157619e
)
- ci: remove heroku deploy (
01fd51e
)
- fix(locators): beerforce spider theme change (
cafeba6
)
- Merge pull request #126 from ngshiheng/ci-remove-heroku
ci: remove heroku deploy (eb05522
)
- fix(deps): update dependency colorlog to v6.7.0 (
a9fb005
)
- Merge pull request #120 from ngshiheng/renovate/colorlog-6.x
fix(deps): update dependency colorlog to v6.7.0 (0e5757a
)
- Merge pull request #116 from ngshiheng/renovate/pin-dependencies
fix(deps): pin dependencies (62773e1
)
- ci: downgrade poetry-version to 1.1.13 (
0b9af92
)
- fix: install setuptool (
9e94e5d
)
- Merge pull request #125 from ngshiheng/fix-setuptool-error
chore: add setuptools (169ab08
)
- Merge pull request #124 from ngshiheng/fix-setuptool-error
fix: install setuptool (9e5b4de
)
-
ci: add workflow_dispatch to cd (
d4574aa
) -
ci: remove cache poetry (
8d65fb6
) -
ci: cache poetry dependencies (
18772a1
) -
ci: upgrade poetry version (
14b17d8
)
- fix(spider): handle case where product is returned as null from fairprice api (
700980b
)
- feat: add item deduplication pipeline (
424a3ce
)
- fix(spider): skip merchandise page for beerforce (
1036068
)
- refactor(spider): lazada spider to always use proxy (
a20f815
)
- feat: add proxy middleware (
fa0b3ec
)
- feat: use scrapeops for job monitoring (
113444e
)
- refactor: makefile to be self-documenting (
abfbbdd
)
- Merge pull request #115 from ngshiheng/scrapeops-scrapy-integration
feat: scrapeops scrapy integration (96ff70c
)
- docs: add section to delete products via scrapy shell (
9bd7750
)
- feat: add FiltersPipeline to filter unwanted products (
5ffbdcc
)
- fix(spider): lazada spider url param path (
7f84e11
)
- fix(spider): shopee spider regression due to api requirement change (
7e0fa59
)
-
fix(spider): remove rate limiting flag & fetch beer style from filterItems (
5831825
) -
fix: missing trailing comma (
a2e4f30
)
- fix: downgrade to python3.9 (
e8949d1
)
- fix(deps): update dependency sqlalchemy to v1.4.42 (
07cad9b
)
- chore: run poetry install (
6891d1f
)
- fix(deps): update dependency psycopg2-binary to v2.9.5 (
d869198
)
- fix(deps): update dependency sentry-sdk to v1.9.9 (
677826f
)
-
chore: improve Makefile (
7134b23
) -
chore(deps): update abatilo/actions-poetry action to v2.1.6 (
ea1d7d6
) -
chore(deps): update dependency flake8 to v5.0.4 (
fc36d43
)
- fix(deps): update dependency scrapy to v2.6.3 (
935ba9c
)
- chore(spider): update mainstream beer brand list (
e35dc7f
)
- fix(spider): update lazada spider endpoint (
077cd48
)
- Merge pull request #108 from ngshiheng/update-lazada-spider
fix(spider): update lazada spider endpoint (47bd287
)
-
chore(renovate): update schedule to monthly (
117ce16
) -
chore(deps): update dependency flake8 to v5 (
088b91c
)
- fix(spider): skip items if brand is None for giant.sg spider (
29d3368
)
- Merge pull request #107 from ngshiheng/renovate/flake8-5.x
chore(deps): update dependency flake8 to v5 (fb76fc9
)
- fix(deps): update dependency sentry-sdk to v1.9.0 (
daf2f21
)
- chore: add FUNDING.yml (
f86fdb5
)
- fix(deps): update dependency scrapy to v2.6.2 (
b3271e7
)
- chore(deps): update dependency pylint to v2.14.5 (
5c896de
)
- fix(deps): update dependency sentry-sdk to v1.8.0 (
dd3716c
)
- fix(deps): update dependency sentry-sdk to v1.7.2 (
fed5a2f
)
- fix(deps): update dependency sentry-sdk to v1.7.1 (
212864d
)
- fix(deps): update dependency alembic to v1.8.1 (
bfe5b90
)
-
chore(deps): update dependency pre-commit to v2.20.0 (
f1c0885
) -
chore(deps): update abatilo/actions-poetry action to v2.1.5 (
683174d
) -
chore(deps): update dependency pylint to v2.14.4 (
5ab3169
)
- fix(deps): update dependency sentry-sdk to v1.7.0 (
0ed1c1d
)
- refactor(spider): move greatbeerexperiment xpaths to locators/ (
230b7bc
)
- style: update docstring (
fc1ecfe
)
- test(parser): add test_parse_name (
c875fa2
)
- fix(deps): update dependency sqlalchemy to v1.4.39 (
4abd32c
)
-
chore(deps): update dependency pylint to v2.14.3 (
413a1b5
) -
chore(deps): update dependency pylint to v2.14.2 (
c82845e
) -
chore: update pre-commit versions (
0c443a5
) -
chore(deps): update actions/setup-python action to v4 (
3247f60
) -
chore(deps): update dependency pylint to v2.14.1 (
eb1a40e
) -
chore: update style & flake8 exclude list (
606db23
)
- ci: only run cd for changes in *.py files and poetry.lock (
9f0389c
)
-
docs: add features (
3685743
) -
docs: add badges (
dc5304e
) -
docs: move example sql queries to .sql files (
7636768
)
- fix(deps): update dependency sentry-sdk to v1.6.0 (
973d0aa
)
- Merge pull request #91 from ngshiheng/renovate/actions-setup-python-4.x
chore(deps): update actions/setup-python action to v4 (148eb8c
)
- chore(spider): update contract (
e00cc80
)
- fix(spider): skip personalised items for troublebrewing spider (
38aea1f
)
- refactor(locator): update imports (
e1bf084
)
- fix: missing trailing comma which causes unintended concat (
e89fead
)
-
chore: run poetry update (
17e910f
) -
chore: add Makefile (
b05e944
) -
chore(deps): update dependency pylint to v2.14.0 (
2fe78de
)
- feat: skip non-beer items (
45bab87
)
- fix(deps): update dependency alembic to v1.8.0 (
a7d2bd2
)
- chore: ignore python-semantic-release (
9394acb
)
- fix(deps): update dependency sqlalchemy to v1.4.37 (
5c6d125
)
-
chore: update autpmergeType to pr (
af33b9a
) -
chore(deps): update relekang/python-semantic-release action to v7.29.0 (
87c4340
)
-
ci: revert relekang/python-semantic-release action to v7.28.1 (
783648c
) -
ci: update release job to use relekang/python-semantic-release@master (
0c9e037
)
- fix(deps): update dependency ipython to v8.4.0 (
e62ffe8
)
-
chore: bump POETRY_VERSION to 1.1.13 (
112feec
) -
chore(deps): update dependency pylint to v2.13.9 (
736d6dd
) -
chore: run poetry update (
7a8636c
) -
chore(deps): update dependency pre-commit to v2.19.0 (
0e4c062
) -
chore(deps): update dependency pylint to v2.13.8 (
4cefa4d
) -
chore: move bash scripts to folder (
2a748ab
)
-
ci: pin relekang/python-semantic-release@v7.28.1 due to #442 (
302a357
) -
ci: avoid running the same test job twice (
81e483b
) -
ci: make release and deploy require test (
d2575d0
)
- feat: skip more mainstream beer brands (
b9039c5
)
-
fix(deps): update dependency sentry-sdk to v1.5.12 (
ccb340c
) -
fix(deps): update dependency ipython to v8.3.0 (
a9dc049
) -
fix(deps): update dependency sentry-sdk to v1.5.11 (
75c9ab6
)
-
refactor(spider): add typing for parse* methods (
76391b7
) -
refactor(spider): move hopshop spider XPath to use locator class (
675b1ac
)
-
style(ci): reformat yml (
ab85ee5
) -
style(spider): remove redundant xpath (
b86233f
) -
style(spider): group locators according to items.py (
c4c3869
)
- fix(spider): invalid xpath expression error (
b3676b4
)
- feat(command): allow prune.sh to specify stale_days (
8a0b4bb
)
- fix(command): move optparse to argparse (
b0175bb
)
- chore: update vscode settings (
de31764
)
- docs: add sentry details (
c66f5b0
)
- style: ordering and line break (
1dd31af
)
- Merge pull request #85 from ngshiheng/improve-filter-and-log
feat: improve filter and log (3ef1269
)
- chore: update branch name to main (
7c6143a
)
- fix(deps): update dependency sqlalchemy to v1.4.36 (
3cfb9c6
)
- chore: update LICENSE year (
4c0e66e
)
- refactor: move coldstorage spider XPath to use locator class (
0276702
)
- Merge pull request #84 from ngshiheng/add-giant-spider
feat: add giant.sg spider (17364d7
)
- chore(deps): update dependency pylint to v2.13.7 (
b980c46
)
- docs: add disclaimer (
db7aede
)
- fix: api query param currentPage to be 0-indexed (
b127e13
)
- chore(deps): update dependency pre-commit to v2.18.1 (
222c417
)
- fix(deps): update dependency sentry-sdk to v1.5.10 (
2c55e7b
)
- fix(deps): update dependency sentry-sdk to v1.5.9 (
0d2c520
)
- chore(deps): update dependency pylint to v2.13.5 (
58c9498
)
- fix(deps): update dependency sqlalchemy to v1.4.35 (
8ec5708
)
- chore(deps): update dependency pylint to v2.13.4 (
9db05f6
)
- fix(deps): update dependency scrapy to v2.6.1 (
7c4c677
)
- fix(deps): update dependency ipython to v8.2.0 (
1214d35
)
- fix(deps): update dependency ipython to v8.1.1 (
14ff8b6
)
- fix(deps): update dependency sqlalchemy to v1.4.32 (
6265f7b
)
- fix(deps): update dependency sentry-sdk to v1.5.8 (
fe7237e
)
- chore: run poetry lock (
2afe045
)
- docs: update semantic commit url (
6f2976e
)
- fix(deps): update dependency alembic to v1.7.7 (
85f326c
)
- style: update order and formatting (
cdf6f45
)
-
chore: update renovate schedule to daily (
b979bc0
) -
chore(deps): update actions/checkout action to v3 (
e63a0c2
) -
chore: run poetry update (
931107a
) -
chore(deps): update actions/checkout action to v3 (
0aa3b7a
) -
chore(deps): update actions/setup-python action to v3 (
bae2973
) -
chore: retain stale data for 60 days instead of 14 days (
aab0211
) -
chore(deps): update dependency pre-commit to v2.17.0 (
d03b33a
) -
chore: remove needs (
afa71ab
) -
chore: update branches to production (
6ed3e07
) -
chore: only release on pushes for main branch (
3d3667b
) -
chore: update script to prune last 14 days of data (
446babb
) -
chore: update timezone and schedule (
4d5295d
) -
chore: update timezone and schedule (
8657393
) -
chore(deps): update dependency autopep8 to v1.6.0 (
e2795c8
) -
chore(deps): update dependency flake8 to v4 (
aa8e5c3
) -
chore: add schedule for typical non-office hours (
957bc9b
)
-
ci: bump poetry version & add python 3.10 to matrix (
f8a84ad
) -
ci: use relekang/python-semantic-release@master (
6b46022
)
- docs: remove github page (
3b2bf5a
)
- feat: use semanticCommits (
17abcdb
)
-
fix(deps): update dependency sqlalchemy to v1.4.31 (
4e7a24d
) -
fix(deps): update dependency sentry-sdk to v1.5.4 (
d2420a3
) -
fix(deps): update dependency ipython to v8.0.1 (
4b2757d
) -
fix: html layout change (
3becc92
) -
fix(deps): update dependency colorlog to v6 (
c843e60
) -
fix(deps): update dependency ipython to v8 (
8551978
)
-
refactor: make lazada spider use scraper api proxy (
3a55501
) -
refactor: log request headers & ip_address (
8afa195
) -
refactor: add download delay (
7af13ce
) -
refactor: add ROBOTSTXT_OBEY (
0ea189a
) -
refactor: rework redmart spider to use lazada api; exclude more mainstream beer brands (
b67611c
) -
refactor: remove craftbeergsg spider (
4334018
) -
refactor: use XPATH with page object model (POM) (
327f6d2
)
- style: add line break (
fc70651
)
- Merge pull request #83 from ngshiheng/renovate/actions-checkout-3.x
chore(deps): update actions/checkout action to v3 (e094e95
)
- Merge pull request #82 from ngshiheng/update-ci
ci: add python 3.10 into matrix & bump poetry version (706b4fd
)
- Merge pull request #81 from ngshiheng/rework-redmart-spider
refactor: rework redmart spider to use lazada api; exclude more mainstream beer brands (3931504
)
- Merge pull request #79 from ngshiheng/renovate/actions-checkout-3.x
chore(deps): update actions/checkout action to v3 (7f9b11a
)
- Merge pull request #78 from ngshiheng/renovate/actions-setup-python-3.x
chore(deps): update actions/setup-python action to v3 (ea6d15a
)
-
Update CNAME (
7f69548
) -
Merge branch 'production' of github.com:ngshiheng/burplist into production (
205c733
) -
Merge pull request #74 from ngshiheng/fix-beerforce-spider
fix: html layout change (3ec61eb
)
- Merge pull request #71 from ngshiheng/renovate/colorlog-6.x
fix(deps): update dependency colorlog to v6 (55eddb3
)
- Merge pull request #68 from ngshiheng/renovate/autopep8-1.x
chore(deps): update dependency autopep8 to v1.6.0 (c32192d
)
- Merge pull request #70 from ngshiheng/renovate/flake8-4.x
chore(deps): update dependency flake8 to v4 (dab1fb5
)
- Merge pull request #72 from ngshiheng/renovate/ipython-8.x
fix(deps): update dependency ipython to v8 (0926ff4
)
-
chore: create PR for major update (
d3c0b49
) -
chore: change config file to json5 (
dea6b5b
) -
chore(deps): update python docker tag to v3.10 (
ae7d810
) -
chore: add automerge for non-major update (
33535e3
) -
chore: use pin rangeStrategy (
a3a3b50
) -
chore(deps): update abatilo/actions-poetry action to v2.1.4 (
6384089
) -
chore(deps): add renovate.json (
098e6f1
) -
chore: run poetry update to update lxml (
d50f0e4
)
-
fix: remove baseBranches (
0ff705e
) -
fix(deps): pin dependencies (
d7c295c
) -
fix: spider API requires new query param (
a9894cf
)
- Merge pull request #73 from ngshiheng/add-renovate-bot
chore: create PR for major update (87da684
)
- Merge pull request #67 from ngshiheng/add-renovate-bot
chore: add automerge for non-major update (209f28d
)
- Merge pull request #66 from ngshiheng/renovate/pin-dependencies
fix(deps): pin dependencies (dc1b921
)
- Merge pull request #65 from ngshiheng/add-renovate-bot
chore: use pin rangeStrategy (b075b19
)
- Merge pull request #62 from ngshiheng/renovate/abatilo-actions-poetry-2.x
chore(deps): update abatilo/actions-poetry action to v2.1.4 (ed96e30
)
- Merge pull request #61 from ngshiheng/renovate/configure
Configure Renovate (9dec6ca
)
- Merge pull request #59 from ngshiheng/fix-fairprice-spider
fix: spider API requires new query param (142f5c5
)
- chore: add unittest to pre-commit hook (
c3299b4
)
-
docs: add adsense code (
b8afcbb
) -
docs: update default view to gallery (
96f3c14
) -
docs: update README with more detailed information (
47a6ca7
) -
docs: update favicon and add Google Analytics tag (
b4931a2
)
-
fix: missing text() (
205fb28
) -
fix: wrong import (
d459be2
) -
fix: inconsistent API response where sometimes it returns quantity as a (
e86814e
)
-
refactor: use contains (
1512397
) -
refactor: use XPATH with page object model (POM) (
07a45e3
) -
refactor: add docker volume for pdb and move command to Dockerfile (
1b61db4
)
- style: minor nitpicks (
081d144
)
- test: add basic test cases for parse_quantity (
c2ec5ca
)
-
Update CNAME (
503d439
) -
Merge pull request #58 from ngshiheng/fix-spiders
fix: inconsistent API response where sometimes it returns quantity as a (1ae6410
)
- Merge pull request #57 from ngshiheng/refactor-docker
Refactor docker (20278b9
)
-
docs: add favicon (
eb857ce
) -
docs: add html title and meta description (
6323aa8
) -
docs: add index.html (
16fa26d
)
-
fix: bug where all products have the same image_url (
b4677e4
) -
fix: filter products with the name containing 'gift' (
4b1230b
)
- refactor: Make stale_days >= 7 instead of > 0 to prevent accidental prune (
b70b0e5
)
-
Merge branch 'production' of github.com:ngshiheng/burplist into production (
d8020d4
) -
Create CNAME (
32cc9c0
) -
Merge pull request #56 from ngshiheng/ignore-gift-cards
Filter products with the name containing 'gift' (11c5b5e
)
- build: Add docker-compose support (
5d39ece
)
- chore: run poetry update (
610db6d
)
- docs: add PR template (
22273a0
)
- Merge pull request #55 from ngshiheng/update-site-spider-layout-change
fix: site html layout change (8018c49
)
- Merge pull request #54 from ngshiheng/dockerize-app
build:Add docker-compose support (1f467f5
)
- Merge pull request #53 from ngshiheng/dockerize-app
Dockerize app (605722e
)
-
Add multi-stage build Dockerfile (
3b77124
) -
Run poetry update (
a63d762
) -
Raise exception if spider doesn't find any existing product (
f1fde08
) -
Fix xpath and query param (
8ae68d2
) -
Update README (
6fdf610
) -
Update README and add MIT license (
ef7d1c9
) -
Minor refactoring (
2169d1b
) -
Clean up docs and minor style changes with poetry update (
c6460ed
) -
Clean up docs and minor style changes (
aa25e66
) -
Handle case where description only has 2 attributes instead of 3 (
e598809
) -
Move SENTRY_DSN to environment var (
cdd3545
) -
Packages update (
0a764f6
) -
Handle case where description is not deliminated by | (
1e15d1d
) -
Packages update (
b1181d5
) -
Add doc description for quantize_price (
5563352
)
- Merge pull request #52 from ngshiheng/update-currency-type
feat: Update price to numeric type (20856ab
)
-
Remove str conversion at spider level and compare prices in Decimal (
b7a54a2
) -
Update price column to be numeric (
37fe87b
) -
Make alembic autogenerate detects column type change (
1013e9d
) -
Regular packages update (
7983fb7
) -
Update lock file (
bf2bf22
) -
Merge pull request #51 from ngshiheng/auto-add-semantic-release
ci: Add auto semantic release (67dd155
)
-
Add release job to GA workflow (
38e431b
) -
Update python version (
a3c3f87
) -
Packages update (
e5fd6db
) -
Add auto semantic release (
25540ca
) -
Update product description xpath to be more flexible (
fc6de51
) -
Remove unused setup (
24b9fc7
) -
Add typing (
fb8869d
) -
Refactor spiders to use ProductLoader (
698a824
) -
Use scoped session (
0324a5e
) -
Add test_prevent_remove_stale_products_from_deleting_all (
7d1817d
) -
Add proxy unit test (
d644dab
) -
Packages update (
f2aeb5c
) -
Merge pull request #50 from ngshiheng/scrape-image-urls
Add image_url to product table (d9fa49a
)
-
Parse image_url for troublebrewing (
ac3991b
) -
Parse image_url for thirsty (
4463292
) -
Parse image_url for shopee (
a519a6b
) -
Parse image_url for redmart (
3312648
) -
Parse image_url for ishopchangi (
cb1c59e
) -
Parse image_url for hopshop (
25ba411
) -
Parse image_url for greatbeerexperiment (
4280638
) -
Parse image_url for fairprice (
6f24070
) -
Parse image_url for craftbeersg (
91de518
) -
Parse image_url for coldstorage (
b1e9a03
) -
Parse image_url for beerforce (
346a9c7
) -
Parse image_url for alcoholdelivery (
12eb2e5
) -
Parse image_url for alcohaul (
412bbd1
) -
Add image_url to product table (
f73edc0
) -
Update SQL query for price list (
a3cad3e
) -
Update github action (
f8a3812
) -
Merge pull request #49 from ngshiheng/add-mypy
Add mypy and type annotations (2274e28
)
-
Add mypy and type annotations (
93b043d
) -
Packages update (
4f4caa3
) -
Update craftbeersg spider to perform nested crawl (
406cfc7
) -
Fix incorrect price error (
d4f653b
) -
Add temporary hotfix to craftbeersg due to page change (
609d4e1
) -
Remove check json (
53aa265
) -
Run poetry lock (
210b964
) -
Update python version for lock file (
bbe2afc
) -
Update python version in GitHub action (
5e10f98
) -
Specify python version (
f0ec8bf
) -
Update packages and troublebrewing spider (
cf4124c
) -
Fix bug where default raw_name is not returned because remove_spaces may be whitespace (
68147b3
) -
Update documentation (
3d5451e
) -
Add str.strip to Item MapCompose (
5454dd1
) -
Return raw_name when parse_name returns an empty str (
80923d6
) -
Update spider specific utility functions to be staticmethods (
e5dc360
) -
Avoid use of generic exceptions (
964517d
) -
Update shopee spider (
9048da6
) -
Enable auto-migration upon Heroku release (
f8310e0
) -
Temporary remove Procfile to run alembic stamp head first (
9d688c9
) -
Set sqlalchemy.url with DATABASE_CONNECTION_STRING (
bb642df
) -
Merge pull request #47 from ngshiheng/add-alembic-migration
Add alembic (07f0b14
)
-
Add Procfile to automate migration (
6a4c7f8
) -
Create zero state migration for existing database (
600340d
) -
Initialize alembic (
53bbfdb
) -
Add alembic (
75359f2
) -
Update install instruction (
02a755c
) -
Temporarily remove scrapy check (
661b52f
) -
Ignore DOWNLOADER_MIDDLEWARES on CI/CD (
909ad6c
) -
Fix spider contract issue scrapy/scrapy#3222 (
6fb6b35
) -
Specify an exact Python version (
6ac17ee
) -
Merge pull request #46 from ngshiheng/migrate-pipenv-to-poetry
Migrate from pipenv to poetry (3d3e40a
)
-
Update README (
b977ebe
) -
Update pythonPath (
d579b12
) -
Migrate from pipenv to poetry (
f49e31f
) -
Minor code refactoring based on pylint (
36ff0a4
) -
Polish scrapy contracts (
6777f76
) -
Polish scrapy contracts (
18e7252
) -
Add more contracts for existing spiders (
cf076ce
) -
Add contracts for existing spiders (
c161387
) -
Update precommit hooks to include scrapy check (
b7ccfbf
) -
Add precommit hooks (
6c1ee10
) -
Handle case where listItems do not exist (
b13397b
) -
Fix where could be returned as or by the API (
fbb3363
) -
Fix bug where parsers always return None (
0e3b1b9
) -
Remove the use of commit method when using session context manager (
befcd78
) -
Use session with context manager (
52ed2cb
) -
Rename error to exception (
1454e2c
) -
Move db_connect and create_table to database.utils (
3893115
) -
Fix bug where exception is not raised when caught (
3bb56d1
) -
Update Pipfile (
7545038
) -
Add util functions to get distinct beer brands and styles (
450634f
) -
Add bash script to prune stale products (
70f3029
) -
Add scrapy prune custom management command (
c87552e
) -
Perform some minor code refactoring and update packages (
12af5e5
) -
Move middlewares into folder (
1cd0b74
) -
Move models.py to database folder (
acd5860
) -
Filter out merchandise (
49e0cdc
) -
Add greatbeerexperiment spider (
61ca0c5
) -
Fix minor bug (
b74b1ee
) -
Sanitize POPULAR_BRANDS list (
89d7e06
) -
Parse brand by product name (
14ca2cc
) -
Skip out of stock products for alcoholdelivery (
4124c5c
) -
Fix bug where parse_quantity is called twice in alcohaul (
15fab94
) -
Add alcohaul spider (
84940ad
) -
Add a function to delete stale products (
59842e3
) -
Refactor quantity parser (
2532bde
) -
Merge pull request #45 from ngshiheng/add-redmart-spider
Revive redmart spider (333d7f1
)
-
Ignore mainstream beer brands (
63f1330
) -
Update name parser (
1a87793
) -
Revive redmart spider (
7cdceb9
) -
Merge pull request #44 from ngshiheng/add-shopee-spider
Revive shopee spider (f846ac6
)
-
Add to-do (
d42445f
) -
Revive shopee spider (
fe312fd
) -
Merge pull request #41 from ngshiheng/major-refactoring
Major refactoring (ef21f40
)
-
🧹 Clean up codebase and add some notes (
ea76c70
) -
Update hopshop spider with new database model (
f9100ac
) -
Update craftbeersg spider with new database model (
b92ad57
) -
Update troublebrewing spider (
06a3286
) -
Update ishopchangi spider (
bd7e95d
) -
Add style parser (
889806e
) -
Update SQL example (
2c23f80
) -
Remove lazada and shopee spiders for now (
d916005
) -
Make thirsty spider start crawling from 'browse by beer style page' in order to get beer style (
50117ea
) -
Use follow_all (
ecaf325
) -
Update thirsty spider with the latest database model (
d74ab1d
) -
Update ExistingProductPricePipeline to update product info (
808f797
) -
Update beerforce spider (
71807e7
) -
Add fallback when some ABV are displayed only as int (
b36c89f
) -
Make column nullable (
bbcd3b5
) -
Remove old tests and brewerkz spider (
66543e0
) -
Update to support new database models (
98a54ed
) -
Add more parsers (
f689c3d
) -
Update database models (
00b0da6
) -
Remove remdart spider and scrape only craft beer data for some sites (
d8025c5
) -
Update vendor to platform (
7ddcb43
) -
Update regex (
ff5ebd0
) -
Merge pull request #40 from ngshiheng/add-shopee-spider
Add shopee spider (926a7f6
)
-
Add shopee spider (
62b3f97
) -
Update reference link (
a7f0c18
) -
Remove duplicated prices before commiting to DB (
cfdefdf
) -
Refactor ishopchangi to use regex groups (
a480fc0
) -
Refactor brewerkz spider to use get_product_name_quantity (
d0e7f76
) -
Use regex groups (
1cea872
) -
Add comments about additional product information that could be scraped (
bb919d0
) -
Make sure product url always starts with https (
e4b1d1a
) -
Merge pull request #38 from ngshiheng/add-lazada-spider
Add lazada spider (24c8489
)
-
Remove headers because it is automatically populated (
0cc1bca
) -
Disable HTTPCACHE for redmart due to failing retries (
a210462
) -
Fix case where using custom_settings completely left out global DOWNLOADER_MIDDLEWARES (
dc62336
) -
Make use of productInfo from redmart's API (
dcef1cc
) -
Add DelayedRequestsMiddleware to add delay in between retries (
1be9a51
) -
Add get_retry_request to retry requests (
2a0bfad
) -
Refactoring (
9720886
) -
Update redmart spider to scrape from a single URL first (
8a30fd9
) -
Add redmart spider (
79bf5a7
) -
Add long DOWNLOAD_DELAY for lazada spider (
963763b
) -
Use middleware to automatically rotate User-Agent in request headers (
33609de
) -
Add scrapy-fake-useragent (
e5c6f6d
) -
Make lazada spider use rotating proxy (
c011dc2
) -
Add lazada spider (
9b37aee
) -
Add useful SQL queries (
a6b5653
) -
Merge pull request #36 from ngshiheng/add-ishopchangi-spider
Add ishopchangi spider (02358c0
)
-
Add test for ishopchangi and fix trailing comma name (
441e7e5
) -
Add ishopchangi spider (
7fbdc12
) -
Add index to product name (
e4328c9
) -
Merge pull request #33 from ngshiheng/add-troublebrewing-spider
Add troublebrewing spider (78e8e73
)
-
Temporarily disable statsmailer (
1fd4a46
) -
Add pylint and fix pylint warnings (
dc42aab
) -
Dependencies update (
42d0226
) -
Add troublebrewing spider (
7d6f426
) -
Update README (
6bdaa3b
) -
Add Sentry warning whenever and have different length (
1ca19a3
) -
Merge pull request #32 from ngshiheng/add-hopshop-spider
Add hopshop spider (14b16a8
)
-
Add hopshop spider (
e0ca36c
) -
Merge pull request #30 from ngshiheng/fix-last-price-bug
Fix CardinalityViolation bug with Product.last_price (76c0e6f
)
-
Update settings to only send email in production (
67e4610
) -
Fix CardinalityViolation bug with Product.last_price (
069c17c
) -
Merge pull request #29 from ngshiheng/add-email-notification
Use StatsMailer extension to send Scrapy info (76c9d0c
)
-
Use StatsMailer extension to send Scrapy info (
2ef73f9
) -
Update README instructions (
9692be9
) -
Uncomment scrapy commands (
73c0240
) -
Add task.sh for Heroku Scheduler to run spiders (
b70f22a
) -
Set ROBOTSTXT_OBEY False for thirsty spider (
acabfae
) -
Fix bug where we constantly run into UniqueViolation because fairprice API returns duplicated item (
f983ec4
) -
Merge pull request #27 from ngshiheng/update-database-model
Update database model (066122c
)
-
Fix brewerkz spider bug where it attemps to scrape duplicated data on side panel (
a56adee
) -
Add UniqueConstraint for Product quantity and url (
dd7dd11
) -
Merge pull request #23 from ngshiheng/bug-fixes
Fix issue where duplicated products are created when there is a change in price (66d36ed
)
-
Fix issue where duplicated products are created when there is a change in price (
b9aa520
) -
Prettier README (
720b321
) -
Add gif to README (
0c4219a
) -
Read database name from environment variable (
eae84c7
) -
Merge pull request #13 from ngshiheng/zyte-deployment
Update Sentry integration and code refactoring (6622534
)
-
Remove sample output (
825ae4e
) -
Add additional test case for trailing 's' (
58b9a7b
) -
Add basic brewerkz spider (
7d494d6
) -
Use bulk_insert_mappings for even faster performance (
8901ac3
) -
Allow Sentry transaction and use SqlalchemyIntegration (
0c3095f
) -
Update README (
a81ae4a
) -
Update documentations (
69ee0b5
) -
Merge pull request #11 from ngshiheng/code-refactoring
Code refactoring and improvements (04d7bab
)
-
Use bulk operation to insert new products and prices to DB (
67c0e6f
) -
Ignore .scrapy/ generated by httpcache (
2808722
) -
Enable AUTOTHROTTLE_ENABLED and HTTPCACHE_ENABLED (
5d90aab
) -
Fix timeout error issue where SQLAlchemy hits QueuePool limit due to unclosed session (
a7f8f9f
) -
Add column_property to query for last_price and set Sentry environment (
5f5e96c
) -
Add ipython (
597a0da
) -
Merge pull request #10 from ngshiheng/color-logging
Color logging (b931c2e
)
-
Fix bug where discounted product price is not extracted (
ee178d5
) -
Fix bug where condition check should be 'is True' not 'is None' (
b1c44a9
) -
Update sample output of all spiders (
21d8eef
) -
Add unit tests (
3bda2e1
) -
Set Sentry to use sentry-sdk (
16a5f15
) -
Add colourized log (
07db31a
) -
Merge pull request #8 from ngshiheng/add-sentry
Add sentry and price parser (17853b1
)
-
Add NewProductPricePipeline and DuplicatePricePipeline (
c4808b8
) -
Update model and add pipeline to add item into database (
53a63ec
) -
Add SQLAlchemy and some basic pipelines (
bfcfbda
) -
Use price_parser and fix coldstorage bug where price element has different class name (
3f9c3b0
) -
Add Sentry (
62980c9
) -
Merge pull request #7 from ngshiheng/add-pipelines
Update coldstorage spider to scrape next pages (e0388f2
)
-
Update coldstorage spider to scrape next pages (
77fa0a0
) -
Merge pull request #3 from ngshiheng/add-beerforce-spider
Add basic spider to scrape from beerforce.sg (f345ba8
)
-
Clear flake8 errors (
80bc7ba
) -
Update alcoholdelivery sample output (
bd3af22
) -
Scrape product quantity from thirsty (
720503a
) -
Scrape product quantity from fairprice (
95e5a59
) -
Sanitize craftbeersg beer name and get product quantity (
c0e98c9
) -
Remove Unit Enum, change unit to quantity (
f81765e
) -
Move example output folder (
8da40ee
) -
Fix bug where beerforce spider did not recursively follow the link to next page (
c398c72
) -
Update scrapy ItemLoader to parse price correctly (
4e227ce
) -
Filter out kegs (
73b89ca
) -
Add basic spider for alcoholdelivery.com (
f148bb3
) -
Add basic spider to scrape from beerforce.sg (
e4e45aa
) -
Merge pull request #2 from ngshiheng/scaper-api-proxy
Use Scaper API as a proxy management measure (c9f6d3a
)
-
Use Scaper API as a proxy management measure (
482cf70
) -
Merge pull request #1 from ngshiheng/add-craftbeersg-spider
Add basic spiders for all main sites (5f153c1
)
-
Add flake8 and scrapy check to github action (
434dd2a
) -
Add basic spider to scrape from thirsty.com (
2265749
) -
Scrape fairprice using their API (
f4a142b
) -
Add basic spider to scrape from fairprice.com (
ab9b996
) -
Add scrapy-splash for selecting dynamically loaded content (
b9fe2da
) -
Add basic spider to scrape from coldstorage.com (
c816152
) -
Add basic spider to scrape from craftbeersg.com (
6aabec3
) -
Initialize scrapy project (
eade821
) -
Initialize project (
42e44e5
)