I wrote an article taking PentestGPT for a spin and gave my thoughts on it.

[redactedaccount]

https://mantisek.com/taking-pentestgpt-for-a-spin

I thought it fair to include this, as I offer quite a few criticisms.

[GreyDGL]

Thanks for the feedback!
I'm glad that you mentioned all the issues that LLM-based solutions, including PentestGPT, may have against penetration testing tasks. The point I totally agree with is that there's no chance for an LLM-based solution to replace penetration testers.

For the three key points you mentioned:

LLMs have a serious problem with holding context, and since LLMs dont actually evaluate any data, this is a problem that will persist even with greater memory and other fixes.

I agree. That's why people are using embedding together with LLMs to address this, and I'm doing some tests on that. I won't say it's promising, but memory issues might be resolved in near future. You may also want to try the 32K token GPT-4, which works amazing in some complex tasks.

Without control over the model, users will have to send sensitive data to OpenAI (or whichever LLM) which is a serious issue for penetration tests where confidentiality is critical.

Agree. Unfortunately, even fine-tuned open-source LLMs cannot outperform GPT-4. I did some testing on GLM-130B but it didn't work well. This can be a long-lasting issue until some solid open-source models come out.

Trying to put an LLM on rails by forcing it into a structured conversation, using multiple sessions as its 'memory' is always going to come at odds with an LLM's inherent 'fuzziness'

I still hold the opposite opinion and believe that this can be a solid solution for more complex tasks.
If you consider PentestGPT (or any other potential LLM tools) as a supporting tool to pentesters, then users should always discuss with the tool for the next step. For example, when PentestGPT gives a complete exploit chain in HTB forest, the user can use the discuss function and tell it like hey this is not correct. I don't think the complete exploit chain works out. Can you give some recommendations for the first step?. Nevertheless, PentestGPT is still mostly non-sense to experienced pentester:( Many times it just point out the direction that is obviously not correct.
If you consider PentestGPT as a helper for new testers with zero security backgrounds, then they can always try the whole exploit chain (even though not correct) and discuss with the tool like The exploit did not work out; the output from xxxx is xxx. What should I do?.

I do believe that with the improvement of LLMs, those issues can be solved one day. Let's see the performance of Google Sec-PaLM.

Lastly, thanks so much for the feedback and the criticisms. I'll do more testing and try to make it better:)

[redactedaccount]

Just picked up a job wherein my duties are trying to solve this very problem. I'll be in touch, maybe we can chat later.

[GreyDGL]

Sure. Happy to chat.