Parallelize amass enumeration

[drtychai]

amass creates a huge bottleneck at the end of subdomain enumeration. One way to speed this up is to separate it's enumeration over multiple amass DBs and concat the resulting JSONs.

This can be achieved by explicitly disabling all but a few sources in the config.ini and increasing -max-dns-queries.

All amass data sources disabled:

output_directory = /amass<i>
maximum_dns_queries = 1000

[disabled_data_sources]
data_source = AlienVault
data_source = ArchiveIt
data_source = ArchiveToday
data_source = Arquivo
data_source = Ask
data_source = Baidu
data_source = BinaryEdge
data_source = Bing
data_source = BufferOver
data_source = Censys
data_source = CertSpotter
data_source = CIRCL
data_source = CommonCrawl
data_source = Crtsh
data_source = DNSDB
data_source = DNSDumpster
data_source = DNSTable
data_source = Dogpile
data_source = Entrust
data_source = Exalead
data_source = GitHub
data_source = Google
data_source = GoogleCT
data_source = HackerOne
data_source = HackerTarget
data_source = IPToASN
data_source = IPv4Info
data_source = LoCArchive
data_source = Mnemonic
data_source = Netcraft
data_source = NetworksDB
data_source = OpenUKArchive
data_source = PassiveTotal
data_source = Pastebin
data_source = PTRArchive
data_source = RADb
data_source = Riddler
data_source = Robtex
data_source = SiteDossier
data_source = SecurityTrails
data_source = ShadowServer
data_source = Shodan
data_source = Spyse
data_source = Sublist3rAPI
data_source = TeamCymru
data_source = ThreatCrowd
data_source = Twitter
data_source = UKGovArchive
data_source = Umbrella
data_source = URLScan
data_source = ViewDNS
data_source = VirusTotal
data_source = Wayback
data_source = WhoisXML
data_source = Yahoo

Since amass locks it's DB during execution, we need to provide different output directories for each execution of it. The initial idea is to just incremented the output_directory in the config.

[drtychai]

brute forcing functionality can also be separated (with all sources disabled)

[drtychai]

Swapped positions to run amass alongside all other subdomain tools while massdns is now used to filter out bad results from subdomain tools. Speed and accuracy show non-trivial improvement.

Remaining lag from amass is coming from rate limiting from it's sources - implementing a timeout here may be good