urlscan&tld to generate domain_regex.list

[cjbd]

hello, i found linux&python tool urlscan, it searches all url in text file, can this tool to scan & generate domain_regex.list? not sure if it's thorough enough

e.g.

$ urlscan -n /path/to/chromium/src/android_webview/browser/aw_browser_context.cc
crbug.com
http://crbug.com/898555)
https://security.googleblog.com/2017/09/chromes-plan-to-distrust-symantec.html
http://crbug.com/921750)

[Eloston]

It depends on how the tool works. Domain substitution is pretty simple; it just searches for strings anywhere in the file that match the given patterns. Anything more complex might miss some strings.

[cjbd]

@Eloston , thanks, i meant if this can be used to generate domain_regex.list, i'm trying to do domain substitution for chromiumos too
that project probably has more domain names to substitute, i want to check all domain names in source code

[cjbd]

i scanned both chromium and chromiumos source code using urlscan and another python package tld, with exception of non-text files and text files under test folder, as some test file would fail urlscan, i assume those test files won't go to the final product, but i might be wrong, anyway, if needed, i can include those too and only skip files can't be handled by urlscan

i'm attaching the url & tld list from my scan: tld.zip, url.zip

[Eloston]

Those lists look pretty long and are hitting a lot of fake/test domains. We should try filtering out test files, as those are only used for unit/integration tests.