Skip to content

Latest commit

 

History

History
129 lines (94 loc) · 4.73 KB

README.md

File metadata and controls

129 lines (94 loc) · 4.73 KB
Raw

About

Python library to call the APIs that the store itself calls.
To get good parsed results from those API calls see parser.py.

Python Version from PEP 621 TOML Formatter github commits

Usage

Prerequisites 
sudo apt-get install git
python -m ensurepip --upgrade

git clone https://github.com/allendema/rewe_dl && cd rewe_dl
pip install requirements.txt
pip install -e rewe_dl

Optional apprise can be used to send send notifications to more than 100 services/apps.

See Config options!

Examples

You can:

  • save product infos to JSON
  • save product infos to JSONL
  • save product infos to SQL
  • save product images
  • save product calories or other details
  • monitor your basket with your fav products
  • compare prices and get notified when prices change
  • save products to SQL for further analysis
  • analyse the output data the way you like, for example: inflation analysis.
  • and whatever you want.
examples/discounted_to_json.py

rewe_dl/rewe_dl/examples/discounted_to_json.py

Lines 26 to 37 in 7948de8

def main():
my_store = STORE(store_id="8534540")
discounted_products = my_store.get_discounted_products()
all_products = Parser().parse_search_results_products(discounted_products)
todays_date = datetime.today().strftime("%Y-%m-%d")
this_file = Path(__file__).stem
options = {"directory": DATA_FOLDER, "filename": f"{this_file}-{todays_date}.json", "mode": "json"}
MetadataPP(all_products, options=options).run()
examples/discounted_to_sql.py

rewe_dl/rewe_dl/examples/discounted_to_sql.py

Lines 23 to 35 in 7948de8

def main():
my_store = STORE(store_id="8534540")
discounted_products = my_store.get_discounted_products()
all_products = Parser().parse_search_results_products(discounted_products)
this_file = Path(__file__).stem
todays_date = datetime.today().strftime("%Y-%m-%d")
file_name = f"{this_file}-{todays_date}.sqlite3"
SqlPP.save_to_sql(all_products, file_name)
examples/my_basket.py

rewe_dl/rewe_dl/examples/my_basket.py

Lines 23 to 37 in 7948de8

my_basket = [
"https://www.rewe.de/produkte/oro-di-parma-tomatenmark-mit-paprika-scharf-200g/265601",
"https://www.rewe.de/produkte/tuc-cracker-original-100g/215147",
"https://www.rewe.de/produkte/kerrygold-cheddar-scheiben-herzhaft-150g/2602784",
"https://www.rewe.de/produkte/gouda-jung-80g/2621809",
"https://www.rewe.de/produkte/maretti-bruschette-chips-tomato-olives-und-oregano-150g/2632965",
"https://www.rewe.de/produkte/rauch-eistee-pfirsisch-0-5l/1091511",
"https://www.rewe.de/produkte/schaer-choco-chip-cookies-glutenfrei-200g/1344325",
"https://www.rewe.de/produkte/bauer-joghurt-vanille-250g/2559285",
"https://www.rewe.de/produkte/baerenmarke-eiskaffee-cappuccino-1l/8008368",
]
offers = Cli(store_id=MY_STORE_ID).from_links(my_basket)
SqlPP.save_to_sql(offers, file_name="my_basket.sqlite3")
examples/newest_products_to_sql.py

rewe_dl/rewe_dl/examples/newest_products_to_sql.py

Lines 19 to 26 in 7948de8

def main():
search_results = STORE().get_new_products()
all_products = Parser().parse_search_results_products(search_results)
this_file = Path(__file__).stem
SqlPP.save_to_sql(all_products, file_name=this_file + ".sqlite3")
examples/notify_when_price.py

rewe_dl/rewe_dl/examples/notify_when_price.py

Lines 54 to 66 in 7948de8

def main():
my_basket_cookies = [
"https://www.rewe.de/produkte/schaer-choco-chip-cookies-glutenfrei-200g/1344325",
"https://www.rewe.de/produkte/schaer-haferkeks-avena-glutenfrei-130g/2731396",
]
hikes_watchlist = [
"https://www.rewe.de/produkte/durstloescher-multivitamin-0-5l/8831846",
"https://www.rewe.de/produkte/durstloescher-eistee-pfirsich-geschmack-0-5l/951490",
]
compare(my_basket_cookies, type="below", limit_price=2.3)
compare(hikes_watchlist, type="above", limit_price=0.65)
examples/raw_responses_to_json.py

rewe_dl/rewe_dl/examples/raw_responses_to_json.py

Lines 26 to 43 in 7948de8

def main():
my_store = STORE(store_id="8534540")
query = "ja"
todays_date = datetime.today().strftime("%Y-%m-%d")
this_file = Path(__file__).stem
paginated = list(my_store.search(query, max_page=1))
for page, response in enumerate(paginated, 1):
options = {
"directory": DATA_FOLDER,
"filename": f"{this_file}-{query}-{page:02}-{todays_date}.json",
"mode": "json",
"ident": 4,
}
MetadataPP(response, options=options).run()

See rewe_dl/examples folder for some more starting examples to embedd it into your own project.

Config

Getting the cookies

Depending on store location, some stores will have different prices.

To not use the default store, go to https://shop.rewe.de
and select your pickup store.

Right click -> Inspect source -> Web Storage -> Cookies
and copy the value from marketsCookie into the config.json file.

Notifications/Webhooks

If you don't want to install apprise, you can use matrix.org or telegram.org.

  • Matrix: Fill the matrix dict in rewe_dl/config.json file.
  • Telegram: Fill the telegram dict in rewe_dl/config.json file.
  • Apprise (Optional): The config file in ~/.config/apprise (without any extension) will be used.
    Configure your wanted notifications/webhooks options based on the wiki.

Changes

API endpoints or data structure might change! If you have any issues report them or open a pull request.

This project will be a part of another project which uses multiple stores using an REST API with Fastapi.

Development

Development

If you want to modify something:

  • pip install requirements-dev.txt
  • The pyproject.toml file is used for linting/formatting with ruff.
  • Make your changes.
  • If you add tests, run them with python3 ./scripts/run_tests.py.
  • Run ruff check --fix .
  • Run ruff format .
  • Create a Pull Request.

Credits

Inspired from gallery-dl, the code structure is similiar and some is code straight reused.
Where applicable - docstrings point to original code creators.