Dealing with good crawlers

[jl-678]

Hi,

I am running the latest version and see the dashboard overflying with traffic from "good crawlers." I see 50,000 hits from Microsoft and AWS crawlers This skews the dashboards. What is the best way to manage this? I know that there are many new parameters in v1, but I cannot figure out how to use them most effectively to address something like this.

Ideally, I would want to configure Krawl to limit traffic served to "good crawlers" and maximize the wastage of bad ones. Is there a way to do that with the new parameters?

As an aside, the table on the repo, lists all the parameters, but I think that we could user further clarification about what exactly they mean and why someone would change them. Maybe even some examples. I would be willing to help with something like that, but I am not sure where to start in understanding them.

TIA!

[carnivuth]

I am running the latest version and see the dashboard overflying with traffic from "good crawlers." I see 50,000 hits from Microsoft and AWS crawlers This skews the dashboards. What is the best way to manage this?

we are working on a feature to export other ips classification other than malicious ones, maybe that can be used to setup fw rules to limit traffic, something like

curl https://your-krawl-istance/api/something?type=good-krawlers | while read ip; do
## custom code to write fw rules to limit traffic from ips 
done

This could work on a network level, while krawl ban feature is limited to requests that gets to the krawl server (@BlessedRebuS correct me if i'm wrong on this one)

As an aside, the table on the repo, lists all the parameters, but I think that we could user further clarification about what exactly they mean and why someone would change them. Maybe even some examples. I would be willing to help with something like that, but I am not sure where to start in understanding them.

Reading the tables with fresh eyes i noticed what you are saying, for example

variable	description
KRAWL_SERVER_HEADER	HTTP Server header for deception

Can be written as:

variable	description
KRAWL_SERVER_HEADER	HTTP Server header that krawl send to clients to mimic known vulnerable web servers, set this to trick attackers that relies on this header to choose targets

Plese tell me if i'm missing something

[jl-678]

Thank you. The suggestion that we block good crawlers via firewall rules and thus only allow bad traffic to reach Krawl would work. It feels a bit complex, but it would accomplish the desired outcome.

Yes on your table update. A bit more detail like that will help users determine what the parameters mean and when/if to change them.

TY!

[BlessedRebuS]

@jl-678 In the 1.0.0 release we added

KRAWL_MAX_PAGES_LIMIT
KRAWL_BAN_DURATION_SECONDS

That should block external IPs after KRAWL_MAX_PAGES_LIMIT requests for KRAWL_BAN_DURATION_SECONDS seconds
Could you try to set those parameters and increase their values?

[jl-678]

@BlessedRebuS Thank you. So if I read this correctly, these settings would impact every user, correct? So a bad crawler and a good crawler would both be banned for KRAWL_BAN_DURATION_SECONDS after hitting KRAWL_MAX_PAGES_LIMIT. Is that right?

It would be cool if you could treat good crawlers and bad crawlers separately to minimize good crawler resources and maximize bad crawler wastage.

[BlessedRebuS]

@jl-678 Yes, that’s correct.

As @carnivuth suggested, we could implement this feature by reusing the same logic we apply to attackers and rely on a banlist to differentiate between good and bad crawlers.

However, I believe we should still define a threshold to classify a crawler as "good". A crawler with only 1–2 hits could be a false positive and might later turn out to be a malicious attacker so we must consider that

[Lore09]

Quick update on the matter, currently we did some minor upgrades on the UI part of the visualization now the recent suspicious activity tab only shows actual suspicious activity without the good crawlers noise.
You can find the updated version on the public demo of Krawl https://demo.krawlme.com/das_dashboard#overview