Improvements for captcha detection (WIP) #65

jifeon · 2015-12-09T22:21:49Z

mousedown and mouseup action
wait for visible, pattern, page actions
back action Ability to parse deep row information from sub-page of item #52
provide collection action - for cases when rules are not defined before actions are performed Return new parsing scope from action #64
postActions support
concurrent cases actions support - when we don't know what to do exactly
performance improved for wait action
actions results rule type - for cases when we don't need parse
possibility to use blank scopes for rules, for virtual rules which can get results from actions for example
ability to get partial results On any error we should provide parsed results #54

Refs #57
🐗

wait for visible, pattern, page actions back action #52 provide collection action - for cases when rules are not defined before actions are performed #64 postActions support concurrent cases actions support - when we don't know what to do exactly performance improved for wait action actions results rule type - for cases when we don't need parse possibility to use blank scopes for rules, for virtual rules which can get results from actions for example Refs #57 🐗

jifeon · 2015-12-09T22:22:40Z

lib/PhantomEnvironment.js

@@ -457,7 +496,8 @@ PhantomEnvironment.prototype = _.create(Environment.prototype, /**@lends Phantom
            var args = [].slice.call(arguments);
            args = args.slice(3);
            this._page.onResourceRequested(callback, /* @covignore */ function (requestData) {
-                debug('Resource requested %s', requestData.url);
+                // todo: decide, remove or leave
+                // debug('Resource requested %s', requestData.url);


it's really annoying, what do you think @maZahaca

Yep. We can cut this. That was made only for testing work which is related to loading resources.

maZahaca · 2015-12-10T08:11:56Z

lib/Environment.js

@@ -3,6 +3,8 @@ var vow = require('vow'),

 function Environment(options) {
    debug('Initializing...');
+
+    this._errbacks = [];


What is that?

errback is callback for error )) In this particular case it's errbacks for evaluateJs fucntions. Sometimes we need to know that something going wrong. For example: if we use wait and on first iteration code fails, we should stop waiting. We add errback before wait and remove it when wait is completed.

maZahaca · 2015-12-10T08:12:37Z

Am I right, that this PR includes #61 ?

maZahaca · 2015-12-10T08:14:53Z

lib/Actions.js

-        EXIST: 'exist'
+        EXIST: 'exist',
+        BACK: 'back',
+        PROVIDE_COLLECTION: 'provideCollection'


What is it for?

Sometimes we do not know what are rules for parsing before actions are performed. Also we have conditional actions and last action can return collection to parse, this collection will be attached to the rule which actions are performed for.

jifeon · 2015-12-10T22:18:09Z

Am I right, that this PR includes #61 ?

yep

use action results in next action "show more" pagination :boar:

waitForQuery action transforms for all type of rules :boar:

🐗

open action :boar:

🐗

PhantomEnvironment._page memory leak fixed :boar:

…-goose-babe # Conflicts: # lib/PhantomEnvironment.js

🐗

…-goose-babe # Conflicts: # lib/PhantomEnvironment.js

🐗

🪿

pause action wait reverted :boar:

…-goose-babe

🐗

some screenshots disabled :boar:

🐗

possible leak removed phantom 2.1.1 :boar:

…action.rules * Improved using rule.type = array for string results * Added decodeURI action

Improved provideRules

browser env injection :boar:

post transformations :boar:

rules from actions fixed :boar:

inject for rules received from actions :boar:

possibility to return scope from actions together with collection etc. :boar:

🪿

jifeon reviewed Dec 9, 2015
View reviewed changes

jifeon changed the title ~~Improvements for captcha detection~~ Improvements for captcha detection (WIP) Dec 9, 2015

maZahaca reviewed Dec 10, 2015
View reviewed changes

maZahaca mentioned this pull request Dec 10, 2015

Feature use actions result #61

Closed

maZahaca reviewed Dec 10, 2015
View reviewed changes

jifeon and others added 5 commits December 17, 2015 19:38

snapshot actions

ba6bd7c

use action results in next action "show more" pagination :boar:

getters and setters

8854d7a

waitForQuery action transforms for all type of rules :boar:

catching errors for rules

c3a8202

🐗

added new version

9893626

killing phantom harder

a685dbe

🐗

jifeon force-pushed the feature-goose-babe branch from 7cf7df4 to a685dbe Compare December 24, 2015 20:21

jifeon added 3 commits December 25, 2015 01:50

smarter error handling for actions

51e504d

open action :boar:

force tearDown phantom on fail

70c9e8c

🐗

trim transformation

ee0f85e

🐗

jifeon and others added 21 commits January 15, 2016 02:08

PhantomEnvironment refactored

56edc09

PhantomEnvironment._page memory leak fixed :boar:

Merge remote-tracking branch 'origin/feature-goose-babe' into feature…

5bf99e3

…-goose-babe # Conflicts: # lib/PhantomEnvironment.js

fixed silent errors

0619a94

memory leaks fixed

c5b822e

🐗

Merge remote-tracking branch 'origin/feature-goose-babe' into feature…

79e49ef

…-goose-babe # Conflicts: # lib/PhantomEnvironment.js

googse babe alpha 5

6ac33f7

🐗

Added rule.child -> child index in the selected childNodes

9f966d0

🪿

graceful shutdown

1f7beb1

pause action wait reverted :boar:

Merge remote-tracking branch 'origin/feature-goose-babe' into feature…

ddfba68

…-goose-babe

alpha 7

5fb5d1c

🐗

Cleaning DOM after scroll parsing

731c33e

Fixed phantom resource timeout

58ff81e

alpha 10

0493f60

some screenshots disabled :boar:

alpha 11

68092fe

some screenshots disabled :boar:

kill phantom harder

664e5cf

🐗

alpha 13

1ffee08

possible leak removed phantom 2.1.1 :boar:

* Fixed 'provideCollection' -> 'provideRules' & action.collection -> …

c23593f

…action.rules * Improved using rule.type = array for string results * Added decodeURI action

Fixed couple of bugs

515b97e

Merge pull request #67 from redco/feature-optimized-action-provide-rules

d01af67

Improved provideRules

new version

475ab5e

alpha 15

fc3476c

browser env injection :boar:

jifeon force-pushed the feature-goose-babe branch from 6dcf779 to fc3476c Compare January 31, 2016 21:13

jifeon and others added 7 commits February 9, 2016 03:24

alpha 16

236c3d5

post transformations :boar:

Added rule.virtual and rule.set

260cf9e

alpha 18

94aba50

rules from actions fixed :boar:

alpha 19

9196ab2

rules from actions fixed :boar:

alpha 20

684fc9c

inject for rules received from actions :boar:

alpha 21

bebdbcc

possibility to return scope from actions together with collection etc. :boar:

Fixed position error behavior

46a99f9

🪿

maZahaca closed this Mar 18, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improvements for captcha detection (WIP) #65

Improvements for captcha detection (WIP) #65

jifeon commented Dec 9, 2015

jifeon Dec 9, 2015

maZahaca Dec 10, 2015

maZahaca Dec 10, 2015

jifeon Dec 10, 2015

maZahaca commented Dec 10, 2015

maZahaca Dec 10, 2015

jifeon Dec 10, 2015

jifeon commented Dec 10, 2015

Improvements for captcha detection (WIP) #65

Improvements for captcha detection (WIP) #65

Conversation

jifeon commented Dec 9, 2015

jifeon Dec 9, 2015

Choose a reason for hiding this comment

maZahaca Dec 10, 2015

Choose a reason for hiding this comment

maZahaca Dec 10, 2015

Choose a reason for hiding this comment

jifeon Dec 10, 2015

Choose a reason for hiding this comment

maZahaca commented Dec 10, 2015

maZahaca Dec 10, 2015

Choose a reason for hiding this comment

jifeon Dec 10, 2015

Choose a reason for hiding this comment

jifeon commented Dec 10, 2015