http://www.diamondproducers.com multiple 200 pages marked as 404

[mkantautas]

http://www.diamondproducers.com multiple 200 pages marked as 404 .

Is it a bug or an unusual way of blocking crawlers ?

[stevenvachon]

Does the same occur with the v0.8.0 branch?

[hakudev]

This is not a bug.

This website doesn't seem to accept the request method 'HEAD', which is the default value as specified in: https://github.com/stevenvachon/broken-link-checker#optionsrequestmethod

Solution:

let options = {
    /* [...] */
    requestMethod: 'GET',
};

[beaulac]

Agreed, this is not a bug, but can we turn this into a feature request?

It would be nice to allow specifying which error codes should cause a HEAD to be retried with a GET.

The proposed workaround (requestMethod: 'GET') works, but is undesirable in the case where only a small portion of links handle HEAD incorrectly (w/ non-405 error codes). HEAD is much faster in many other cases -- GET should ideally only be used to retry when encountering an error, to keep things speedy.

By default, this list of error codes would just be 405, as it is now, and we could maintain backwards compatibility by making the retry405Head option modify this list of error codes.

[mkantautas]

Upon Forking this repo and mergin this PR can say that this does fix the issue. Great work! #tested

[merav2110]

Can I control this from the command line ?

[beaulac]

@merav2110 Not yet, unfortunately. It shouldn't be too hard to implement yourself if you need it; fork my fix branch and see #115 for inspiration.