Add subwords capability to ffuf_shortnames #2237
Open
+145
−20
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## dev #2237 +/- ##
=====================================
- Coverage 93% 93% -0%
=====================================
Files 378 378
Lines 29363 29443 +80
=====================================
+ Hits 27155 27227 +72
- Misses 2208 2216 +8 ☔ View full report in Codecov by Sentry. |
bbot/modules/ffuf_shortnames.py
Outdated
|
|
||
| from bbot.modules.deadly.ffuf import ffuf | ||
|
|
||
|
|
||
| class ffuf_shortnames(ffuf): | ||
| watched_events = ["URL_HINT"] | ||
| produced_events = ["URL_UNVERIFIED"] | ||
| deps_pip = ["numpy"] | ||
| deps_pip = ["numpy", "nltk"] |
There was a problem hiding this comment.
Is there an advantage to using nltk over the builtin subword helper?
There was a problem hiding this comment.
mainly just the size of the wordlist, which is massive, and should be well maintained being part of nltk
Also the functionality of that helper doesn't quite match the use, that finds all the subwords (as a list), whereas this is just checking for prefixes
bbot/modules/ffuf_shortnames.py
Outdated
| self.debug("NLTK words data already present") | ||
| except LookupError: | ||
| self.debug("NLTK words data not found, downloading") | ||
| nltk.download("words", download_dir=self.nltk_dir, quiet=True) |
There was a problem hiding this comment.
self.helpers.wordlist("https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/packages/corpora/words.zip")
TheTechromancer
approved these changes
Feb 12, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Also adds ignore_case option to ffuf (useful for IIS where case doesn't matter)
Subwords uses python nltk (natural language toolkit) to try and find smaller words at the beginning of shortnames. If it does, it sends the remainder off to the predictor. This works well because web developers have a habit of making lots of "VerbAction" type two-word file names.