Skip to content

Commit

Permalink
Merge pull request #9 from nadar/ignore
Browse files Browse the repository at this point in the history
fix crawl ignore, trim link value
  • Loading branch information
nadar authored Dec 17, 2020
2 parents 76e089e + dbfbffc commit e05164d
Show file tree
Hide file tree
Showing 7 changed files with 552 additions and 164 deletions.
1 change: 1 addition & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ jobs:
php-version: ${{ matrix.php }}
extensions: apc, curl, dom, intl, mbstring, mcrypt
ini-values: date.timezone='UTC'
coverage: xdebug

## install composer
- name: Install dependencies
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
All notable changes to this project will be documented in this file. This project adheres to [Semantic Versioning](http://semver.org/).
In order to read more about upgrading and BC breaks have a look at the [UPGRADE Document](UPGRADE.md).

## 1.2.1 (17. December 2020)

+ [#9](https://github.com/nadar/crawler/pull/9) Fix issue where `CRAWL_IGNORE` tag had no effect. Trim the value of a links.

## 1.2.0 (14. November 2020)

+ [#7](https://github.com/nadar/crawler/pull/7/files) By default, response content which is bigger then 5MB won't be passed to Parsers. In order to turn off this behavior use `'maxSize' => false` or increase the limit `'maxSize' => 15000000` (which is 15MB for example). The value must be provided in Bytes. The main goal is to ensure that the PDF Parser won't run into very large memory consumption. This restriction won't stop the Crawler from downloading the URL (whether its large the the maxSize definition or not), but preventing memory leaks when the Parsers start to interact with the response content.
Expand Down
2 changes: 1 addition & 1 deletion composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,6 @@
"smalot/pdfparser": "^0.16.1"
},
"suggest": {
"smalot/pdfparser" : "In order to use the PdfParser this library is required."
"smalot/pdfparser" : "In order to use the PdfParser the smalot/pdfparser library is required."
}
}
Loading

0 comments on commit e05164d

Please sign in to comment.