Skip to content

Commit

Permalink
Merge branch 'master' into feat/readme_selenium_chrome
Browse files Browse the repository at this point in the history
  • Loading branch information
HonzaTuron authored Jul 17, 2023
2 parents 9971426 + ad53df9 commit e9cc093
Show file tree
Hide file tree
Showing 146 changed files with 17,924 additions and 10,595 deletions.
33 changes: 0 additions & 33 deletions .dependabot/config.yml

This file was deleted.

1 change: 1 addition & 0 deletions .eslintignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
templates
6 changes: 5 additions & 1 deletion .eslintrc.json
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
{
"extends": ["@apify"]
"extends": ["@apify"],
"plugins": ["jest"],
"rules": {
"no-console": "off"
}
}
36 changes: 36 additions & 0 deletions .github/workflows/build_archives.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: Build template archives

on:
push:
paths-ignore:
- 'dist/**'
branches:
- master

jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout repo
uses: actions/checkout@v3
with:
token: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}

- name: Setup Node.js
uses: actions/setup-node@v3
with:
node-version-file: .nvmrc

- name: Install dependencies
run: npm install

- name: Build template archives
run: npm run build

- name: Commit the updated template archives
run: |
git config user.name 'GitHub Actions'
git config user.email 'github-actions[bot]@users.noreply.github.com'
git add dist/*
git diff-index --quiet HEAD || git commit -m 'chore: Update template archives [skip ci]' || true
git push
34 changes: 0 additions & 34 deletions .github/workflows/buildarchives.yml

This file was deleted.

133 changes: 133 additions & 0 deletions .github/workflows/lint_and_test.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
name: Test Templates

on:
push:
paths-ignore:
- 'dist/**'

jobs:
lint_and_test:
name: Lint and test (without templates)
runs-on: ubuntu-latest

steps:
- name: Checkout repo
uses: actions/checkout@v3

- name: Setup Node.js
uses: actions/setup-node@v3
with:
node-version-file: .nvmrc
cache: 'npm'

- name: Get Node.js version
id: get-node-version
run: echo "version=$(node --version | tr -d 'v' | cut -d '.' -f 1)" >> "$GITHUB_OUTPUT"

- name: Cache node_modules
uses: actions/cache@v3
with:
path: node_modules
key: ${{ runner.os }}-node-${{ steps.get-node-version.outputs.version }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-node-${{ steps.get-node-version.outputs.version }}-
- name: Install dependencies
run: npm install

- name: Lint
run: npm run lint

- name: Test building templates
run: npm run build

- name: Test template manifest
run: npm run test-without-templates

test-node-templates:
name: Test Node.js templates
strategy:
matrix:
os: [ubuntu-latest, windows-latest]
node-version: [ 16, 18 ]
runs-on: ${{ matrix.os }}

steps:
- name: Checkout repo
uses: actions/checkout@v3

- name: Setup Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}
cache: 'npm'

- name: Cache node_modules
uses: actions/cache@v3
with:
path: node_modules
key: ${{ runner.os }}-node-${{ matrix.node-version }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-node-${{ matrix.node-version }}
- name: Install dependencies
run: |
npm install
npm install -g apify-cli
./node_modules/.bin/playwright install chromium --with-deps
- name: Test Node.js templates
run: npm run test-node-templates
env:
APIFY_TOKEN: ${{ secrets.APIFY_TEST_USER_API_TOKEN }}

test-python-templates:
name: Test Python templates
strategy:
matrix:
os: [ubuntu-latest, windows-latest]
python-version: [ '3.8', '3.9', '3.10', '3.11' ]
runs-on: ${{ matrix.os }}

steps:
- name: Checkout repo
uses: actions/checkout@v3

- name: Setup Node.js
uses: actions/setup-node@v3
with:
node-version-file: .nvmrc
cache: 'npm'

- name: Get Node.js version
id: get-node-version
run: echo "version=$(node --version | tr -d 'v' | cut -d '.' -f 1)" >> "$GITHUB_OUTPUT"

- name: Cache node_modules
uses: actions/cache@v3
with:
path: node_modules
key: ${{ runner.os }}-node-${{ steps.get-node-version.outputs.version }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-node-${{ steps.get-node-version.outputs.version }}-
- name: Install dependencies
run: |
npm install
npm install -g apify-cli
./node_modules/.bin/playwright install chromium --with-deps
- name: Setup Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}

- name: Setup Chromedriver
uses: nanasess/setup-chromedriver@v2
with:
chromedriver-version: '113.0.5672.63'

- name: Test Python templates
run: npm run test-python-templates
env:
APIFY_TOKEN: ${{ secrets.APIFY_TEST_USER_API_TOKEN }}
27 changes: 0 additions & 27 deletions .github/workflows/nodejs.yml

This file was deleted.

File renamed without changes.
1 change: 1 addition & 0 deletions .nvmrc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
16
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,13 @@ You can find more code examples in the

### Project boilerplate

If you're already familiar with actors, you can use the following templates to bootstrap new projects using an empty project template or Crawlee templates:
If you're already familiar with Actors, you can use the following templates to bootstrap new projects using an empty project templates or Crawlee templates:

- [Empty project](./templates/project_empty/) - Template with very little boilerplate code.
#### Empty projects
- [Empty JavaScript project](./templates/js-empty/) - Start a new web scraping project quickly and easily in JavaScript (Node.js) with our empty project template. It provides a basic structure for the Actor with [Apify SDK](https://docs.apify.com/sdk/js/) and allows you to easily add your own functionality.


#### Crawlee projects
- [CheerioCrawler](./templates/project_cheerio_crawler_js/) ([TypeScript version](./templates/project_cheerio_crawler_ts/)) - Standard and up to date template for developing with Crawlee's CheerioCrawler.
- [PlaywrightCrawler](./templates/project_playwright_crawler_js/) ([TypeScript version](./templates/project_playwright_crawler_ts/)) - Standard and up to date template for developing with Crawlee's PlaywrightCrawler.
- [PuppeteerCrawler](./templates/project_cheerio_crawler_js/) ([TypeScript version](./templates/project_puppeteer_crawler_ts/)) - Standard and up to date template for developing with Crawlee's PuppeteerCrawler.
Expand Down
Binary file not shown.
Binary file modified dist/templates/js-crawlee-cheerio.zip
Binary file not shown.
Binary file modified dist/templates/js-crawlee-playwright-chrome.zip
Binary file not shown.
Binary file modified dist/templates/js-crawlee-puppeteer-chrome.zip
Binary file not shown.
Binary file modified dist/templates/js-cypress.zip
Binary file not shown.
Binary file modified dist/templates/js-empty.zip
Binary file not shown.
Binary file modified dist/templates/js-start.zip
Binary file not shown.
Binary file modified dist/templates/python-beautifulsoup.zip
Binary file not shown.
Binary file added dist/templates/python-empty.zip
Binary file not shown.
Binary file modified dist/templates/python-playwright.zip
Binary file not shown.
Binary file modified dist/templates/python-scrapy.zip
Binary file not shown.
Binary file modified dist/templates/python-selenium.zip
Binary file not shown.
Binary file modified dist/templates/python-start.zip
Binary file not shown.
Binary file modified dist/templates/ts-crawlee-cheerio.zip
Binary file not shown.
Binary file modified dist/templates/ts-crawlee-playwright-chrome.zip
Binary file not shown.
Binary file modified dist/templates/ts-crawlee-puppeteer-chrome.zip
Binary file not shown.
Binary file added dist/templates/ts-empty.zip
Binary file not shown.
Binary file modified dist/templates/ts-playwright-test-runner.zip
Binary file not shown.
Binary file modified dist/templates/ts-start.zip
Binary file not shown.
7 changes: 3 additions & 4 deletions jest.config.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
module.exports = {
testRunner: 'jest-circus/runner',
verbose: false,
testEnvironment: 'node',
testPathIgnorePatterns: ['/templates/ts-playwright-test-runner/tests/'],
roots: ['test'],
forceExit: true,
testTimeout: 60000,
};
Loading

0 comments on commit e9cc093

Please sign in to comment.