A laundry list of issues

[colin-stubbs]

1. BBOT prefers the last key in api_keys rather than the explicit value from api_key. e.g. this affects watchdog and agent behaviour where they should only ever have a single API key configured under api_key.

2. Case transform and sensitivity issues between server/watchdog/agent when uppercase UUID's have been used, e.g. as generated by uuidgen on MacOS.

e.g

agent-1     | [ERROR] Error making POST request -> http://server:8807/v1/scans/agents/?name=6664c9774a25&description=Auto+generated+container+agent: 401 {"detail":"Invalid API key \"1c629bcd-17db-4f88-aa61-ada9a3bfda6f\" not in {'1C629BCD-17DB-4F88-AA61-ADA9A3BFDA6F', 'ECAC6D64-342A-47F7-A790-6103C4033966', '350A73CC-FFBB-45E4-BFE2-5F3130091BAB'}"}

3. Agent's first scan always fails, agent exits without warning or error during python module setup. Second and subsequent scans are fine.

4. All containers are running code as root, they should run processes as bbot and have limited ability to write files outside the home directory of the user they are running as.

NOTE: server and watchdog run fine as a non-root user, but the agent container probably should run as root, although agent may be able to use sudo automatically so that may be an option?

5. Some server API responses do not seem to include Content-Type or Content-Length headers even when returning a JSON body.

6. If a bbot server preset also includes a HTTP module configuration it overrides delivery to bbot server, and X-API-Key will be sent to the overridden URL.

7a. Processes currently create a debug.log as ~/.bbot_server/debug.log.gz ... except it's plain text and not actually a gzip compressed file at all.

# gunzip -c .bbot_server/debug.log.gz

gzip: .bbot_server/debug.log.gz: not in gzip format
# file .bbot_server/debug.log.gz
.bbot_server/debug.log.gz: ASCII text
# cat .bbot_server/debug.log.gz
[09:22:49] [INFO] [serverctl.py:45] First run detected. Adding a new API key...
[09:22:49] [INFO] [apikeyctl.py:34] New API key added. Please restart the server for the new key to be recognized:
[09:22:49] [INFO] [apikeyctl.py:35]     - API KEY: 030db6a1-23da-4b1f-b181-3a0e481dd086
[09:22:49] [INFO] [redis.py:99] Subscribed to bbot:stream:events
[09:22:49] [INFO] [redis.py:99] Subscribed to bbot:stream:assets
# 

7b. Watchdog and Agent should never generate a key, they should error if they have not been correctly configured with one, e.g. via config file or via environment variable.

8. Many API endpoints currently do not work and instead except as a HTTP 500 response,

root@bbot-server-agent:/app# curl -v -H 'x-api-key: %{API_KEY}%' http://bbot-server:8807/v1/assets/emails/evilcorp.com
*   Trying 172.31.255.6:8807...
* Connected to bbot-server (172.31.255.6) port 8807 (#0)
> GET /v1/assets/emails/evilcorp.com HTTP/1.1
> Host: bbot-server:8807
> User-Agent: curl/7.88.1
> Accept: */*
> x-api-key: %{API_KEY}%
> 
< HTTP/1.1 500 Internal Server Error
< date: Wed, 25 Jun 2025 04:26:38 GMT
< server: uvicorn
< content-length: 2277
< content-type: text/plain; charset=utf-8
< 
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/bbot_server/applets/emails.py", line 17, in get_emails
    matching_assets = await self.root.assets.get_assets(host=domain)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: AssetsApplet.get_assets() got an unexpected keyword argument 'host'
* Connection #0 to host bbot-server left intact
root@bbot-server-agent:/app# 

9. At present when an agent starts with a valid agent.yml config file it does connect to bbot-server but it does nothing to validate that the server knows anything about it. If the registration is missing the agent will sit connected to the server via websocket but it will never see any scan events issued to it by the watchdog because the watchdog does not in fact know that the agent exists. What should happen is that the agent should check the server knows about it and if not request and complete registration.

10. X-API-Key based "authentication" actually responds with a list of valid API keys when an incorrect key is utilised...

{
  "detail": "Invalid API key \"ecac6d64-342a-47f7-a790-6103c4033966\" not in {'b10f15c9-beb5-4cb2-8bf8-59b6e13281a3', 'e345f022-871e-4811-898b-ca8d6ff94fef', '14806145-a58f-48c2-8bb2-70cd7696c3c7'}"
}

[colin-stubbs]

Decided to write down a list of issues I've come across, I'll try to sort these out alongside my containerisation improvements.

[TheTechromancer]

@colin-stubbs thanks for keeping track of these. In BBOT server's current state it makes sense to keep them as a checklist instead of individual issues. I plan to return to devving after next week.

As for the missing Content-Length, this is probably for streaming endpoints. I use this approach for a few endpoints that tend to return huge amounts of data, I.e. thousands or millions of entries. It lets you lazy load the data at your leisure.

[colin-stubbs]

@TheTechromancer possibly, I found some were fine others were not... didn't really get time to look deeper into it. I'll throw Cursor at it later tonight and tell it to work it out :-P

Also I'll add this... this is not really an authentication mechanism at this point though I get why it's probably like this for the time being!

[TheTechromancer]

@colin-stubbs working through these here:

• Misc fixes #68

A few are fixed and a few we may need specific steps to reproduce

[colin-stubbs]

Awesome, I'll sync up with dev... I made a lot of progress then got lost in the tests and life happened again :-(