Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Web scraper woolworths #129

Merged
merged 30 commits into from
Sep 27, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
86ea1c4
created scraper file for woolworths
Krygsmancode Sep 21, 2023
257713d
added scaper code - needs adjustment
Krygsmancode Sep 21, 2023
6176555
added way to get sitemap[ links
Krygsmancode Sep 21, 2023
11011cb
fixed scrapiong navigation
Krygsmancode Sep 21, 2023
7b08ebb
added extraction of weight
Krygsmancode Sep 21, 2023
432da16
adjusted interval time
Krygsmancode Sep 21, 2023
91bfdf1
naming change
Krygsmancode Sep 21, 2023
b180124
removed the android folder
Krygsmancode Sep 21, 2023
effcb4c
added debugging logs to backend
Krygsmancode Sep 22, 2023
5ff6a04
changed the product controller
Krygsmancode Sep 22, 2023
917ec56
changes rexex for foodmodel name
Krygsmancode Sep 22, 2023
47b0b90
removed type from tovititmodel declaration
Krygsmancode Sep 22, 2023
f706c6a
ovberided boierlplate code
Krygsmancode Sep 22, 2023
214aa5b
commented out checkers scraper
Krygsmancode Sep 22, 2023
7b92286
changed linkservice for woolworths
Krygsmancode Sep 22, 2023
b61aa11
imoplementde webscraping service for woolworthg
Krygsmancode Sep 22, 2023
f19161a
woolworthwebscraperservice
Krygsmancode Sep 22, 2023
5c1cf75
added my own model
Krygsmancode Sep 24, 2023
bd21ab9
added visitlink model
Krygsmancode Sep 24, 2023
7b81123
fixed scraper again
Krygsmancode Sep 24, 2023
b9a4249
deletedwool
Krygsmancode Sep 24, 2023
245b70e
deleted
Krygsmancode Sep 24, 2023
bbbcdce
removed stopping of scraper fuction
Krygsmancode Sep 24, 2023
5e0eeb3
prep for pull
Krygsmancode Sep 24, 2023
3cb8cb5
pull prep
Krygsmancode Sep 24, 2023
d07f9fa
prep
Krygsmancode Sep 24, 2023
abc7e92
added checkrs
Krygsmancode Sep 24, 2023
91d0786
prep finished
Krygsmancode Sep 24, 2023
e818750
Merge branch 'development' into webscraper-woolworths
Krygsmancode Sep 24, 2023
9e608e9
👷 new cicd
SkulderLock Sep 27, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 51 additions & 41 deletions .github/workflows/buildAndTest.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,3 @@
# This workflow will do a clean installation of node dependencies, cache/restore them, build the source code and run tests across different versions of node
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-nodejs

name: Build&Test Node.js

on:
Expand All @@ -15,52 +12,65 @@ on:

jobs:
build:

runs-on: ubuntu-latest

services:
mongodb:
image: mongo:latest
ports:
- 27017:27017

strategy:
matrix:
node-version: [ 16.x ]
node-version: [16.x]
# See supported Node.js release schedule at https://nodejs.org/en/about/releases/

steps:
- uses: actions/checkout@v3

- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}
cache: 'npm'
cache-dependency-path: package-lock.json
- run: npm ci
- run: npm run build --if-present
- uses: actions/checkout@v3

- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}
cache: "npm"
cache-dependency-path: package-lock.json

- run: npm ci
- run: npm run build --if-present

- name: Test with Angular
run: npm run test:dev

- name: Set up JDK 17
uses: actions/setup-java@v2
with:
distribution: "adopt"
java-version: "17"

- name: Grant execute permission for gradlew
run: chmod +x gradlew
working-directory: backend

- name: Test with Angular
run: npm run test:dev
# Add this step to wait for MongoDB to be ready
- name: Wait for MongoDB
run: |
while ! nc -z localhost 27017; do
sleep 1
done

- name: Set up JDK 17
uses: actions/setup-java@v2
with:
distribution: 'adopt'
java-version: '17'
- name: Test with Gradle
run: ./gradlew test
working-directory: backend
env:
JWT_SECRET: ${{ secrets.JWT_SECRET }}
DB_PASSWORD: ${{ secrets.DB_PASSWORD }}
DB_USERNAME: ${{ secrets.DB_USERNAME }}
DB_URI: ${{ secrets.DB_URI }}

- name: Grant execute permission for gradlew
run: chmod +x gradlew
working-directory: backend
- name: Download artifacts
uses: actions/download-artifact@v2

- name: Test with Gradle
run: ./gradlew test
working-directory: backend
env:
JWT_SECRET: ${{ secrets.JWT_SECRET }}
DB_PASSWORD: ${{ secrets.DB_PASSWORD }}
DB_USERNAME: ${{ secrets.DB_USERNAME }}
DB_URI: ${{ secrets.DB_URI }}

- name: Download artifacts
uses: actions/download-artifact@v2

- name: Upload coverage reports to Codecov
uses: codecov/codecov-action@v3
with:
token: ${{ secrets.CODECOV_TOKEN }}
- name: Upload coverage reports to Codecov
uses: codecov/codecov-action@v3
with:
token: ${{ secrets.CODECOV_TOKEN }}
101 changes: 0 additions & 101 deletions android/.gitignore

This file was deleted.

2 changes: 0 additions & 2 deletions android/app/.gitignore

This file was deleted.

54 changes: 0 additions & 54 deletions android/app/build.gradle

This file was deleted.

24 changes: 0 additions & 24 deletions android/app/capacitor.build.gradle

This file was deleted.

21 changes: 0 additions & 21 deletions android/app/proguard-rules.pro

This file was deleted.

This file was deleted.

49 changes: 0 additions & 49 deletions android/app/src/main/AndroidManifest.xml

This file was deleted.

Loading
Loading