[FEATURE REQUEST] Threshold flag to remove false positives that sink scan time

[NotoriousRebel]

Is your feature request related to a problem? Please describe.
Sometimes certain hosts will say every single word in your wordlist is matching when it could be due to a WAF, how website is configured, or something else.

Describe the solution you'd like
Adding a --threshold flag or some flag under the same vein, for example --threshold 250 if a url in your scan says 250 words within your wordlist are valid, it will stop scanning that host and move on to the next.

Describe alternatives you've considered
You can run feroxbuster with a much smaller wordlist and if it matches on everything there you remove it, could also write a trivial python script using asyncio and aiohttp to perform the brute forcing to see if it matches on everything.

Additional context
Been running into this for months, and finally decided to create an issue. Some hosts just want to match on everything within your wordlist ¯_(ツ)_/¯

P.S Feroxbuster is dope!

[epi052]

That's interesting. I'd like to know what's causing the server to respond that way, as there may be a better solution than just bailing. Do you have any public targets where we can test?

The actual implementation could leverage the machinery used to do the auto-tune/bail policy stuff

https://epi052.github.io/feroxbuster-docs/docs/examples/auto-tune/#auto-tune-or-auto-bail-from-scans

[godylockz]

@epi052 I think I know what he is referring to. No matter what your fuzzing, it always ends up either a 302 or 200 due to the implementation of the server. So the option is to use number of words, response length, etc. Gobuster does the following and always tests a directory that doesn't exist first, and if it gets a 200, it knows that something must be wrong.

Error: the server returns a status code that matches the provided options for non existing urls. http://ipaddress1/9b9353c0-3de2-4df5-abd7-0f618e4d70ab => 200. To force processing of Wildcard responses, specify the ‘–wildcard’ switch

Let me know if i got that right @NotoriousRebel

[epi052]

@godylockz ferox does this as well in heuristics.rs. we go a step further and detect the requested url being reflected in the response (i.e. response size changes, depending on the request itself).

For situations where there's a csrf token or something less predictable in a custom 404 or similar, we have the --filter-similar-to option, which does some performant hashing to filter responses above a certain similarity threshold.

On top of those, there's also --dont-scan, which takes in regex and prevents requests from even making it off the client machine. Along with --auto-tune and --auto-bail linked above, which enforce policies based on server reported errors.

Do any of these fit/handle the scenario reported?

https://github.com/epi052/feroxbuster/blob/main/src/heuristics.rs#L83

https://epi052.github.io/feroxbuster-docs/docs/examples/filter-similar/

https://epi052.github.io/feroxbuster-docs/docs/examples/deny-list/

[NotoriousRebel]

Apologies for the late response, I unfortunately do not have an example url I can provide. However, typically when I run into this it's usually because no matter what I give for example: https://example.com/FUZZ will return the same page no matter what. I know I could filter it out easily with it's content length but that would mean a request would still be made for each word in a wordlist. I usually run feroxbuster with --auto-tune, I have not used --auto-bail much before so would need to research that a bit more.

In regards to if any of these fit/handle the scenario reported?
Let's take a look

Option 1: Forewarning my knowledge in rust is flimsy to say the least but I will try, Hmmmmm I always run feroxbuster in silent mode so I'm not sure if I would ever see these messages print in the first place and know if a wildcard was detected. I will have to try running feroxbuster again and see if it's detecting certain urls as wildcards. I want to say it's not though as when I look at the json results it seems to continue visiting urls that always return a 200 no matter what word in the wordlist

Option 2: Unfotunately I do not see --filter-similar-to being feasible as I don't see how it can scale easily as if I have httpx results for let's just say 700 subdomains & ips I would have to first determine which urls will always give the same response and then do the --filter-similar-to for each of those which I don't see scaling easily without writing code to do that.

Option 3: I can see this being useful if the urls instead of always returning a 200 are just redirecting to a subdirectory which using the deny list would be very powerful. However, I don't see how exactly this alleviates the aforementioned root problem as well as scaling concerns.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[epi052]

Going to pin this for now. Can you please update the ticket when you have a target that exhibits the behavior AND is a public target (i.e. HTB/THM/VulnHub, open invite bug bounty, etc...).

I understand the concerns with scaling, and agree, if you're running in a huge pipeline / wide swath of targets, the fine-grained controls aren't going to be much use.

[mzpqnxow]

Until there's a native feature to solve this case, I like the idea of a workaround that uses two passes, with this general process/logic, similar to what @NotoriousRebel described

For each host:

1. Run with a short time limit and/or a small/preliminary wordlist
2. Process the results to determine if the server is behaving in a way that causes positives for most/all requests
3. If the results indicate the server seems to be sane, run the second (real) pass with the full wordlist, and with a longer time limit (or no time limit)

Currently, I cop-out of this problem when running at large scale by simply setting an arbitrary run-limit for each session. This has a bunch of obvious downsides, but it at least ensures a site doesn't get endlessly hammered

Looking forward to seeing what you can come up with

[epi052]

@ anyone who still cares enough to read this more than a year later... 😁

i updated how ferox handles wildcard responses to be more general and also utilize the similarity filters automatically as well. I haven't officially released 2.8.0, but it's ready and can be tested by downloading a pre-built binary from here or building yourself from the main branch.

The update isn't a threshold flag, but it should more accurately detect when servers are doing things like responding to everything (and filter those responses)

https://github.com/epi052/feroxbuster/actions/runs/4292981464