[WIP] feat: UE3 LAN query support

[RattleSN4K3]

This will add support for querying LAN servers for UE3 based games as issued by #631. But before merging this pull request, I like to support respectively work on another game, like Toxikk and/or Renegade X.

This request has also various changes which might be be revised/checked. This pull request should be compatbile to all other protocols, but as the core is changed, it's a slightly critical change.

Additional related issues: #629 #628

---

By default the LAN query will provide the same structure as a (master/online) server does. However as the query provider is different with a LAN query the data is limited and will not provide any player data.

There is an additional settings resolver in patterns/ue3pattern.js which currently holds the structure defined by UT3 and allows parsing the data in a more structured way and also providing being able to customize any given payload/pattern by a simple JSON passed as parameter (WIP).

A setup for a query looks like this (for now) + some options for configuring the output:

const gamedig = new GameDig({
  listenUdpPort: 14001
})

gamedig.query({
  type: 'unrealtournament3lan',
  address: '192.168.56.56'

  //port: 14002
  // outputAsArray: true, // instead of a single JSON item, this will output an array

  // packetVersion: 5,
  // lanGameUniqueId: '0x4D5707DB',
  // lanPacketPlatformMask: 1,

  // NOT IMPLEMENTED ANYMORE 
  // subsystem: 'lan'
  // usePattern: true // whether to use the new pattern logic by resolving the settings from a response into known properties
  // includeFlat: false, // [Pattern] if the root object should have all properties included
  // includeAll: false, // [Pattern] if the root object should have all properties included (in combiniation with includeLegacy=false and the property might be removed)
  // includeLegacy: true, // [Pattern] if the root object should include legacy/gamedig properties
  // includeNull: false, // [Pattern] whether the property should be included even if a mapped value cannot be resolved
})

An example response with the options usePattern: true and outputAsArray: true:

Warning

Outdated json response. See here for an updated https://gist.github.com/RattleSN4K3/55958bfc0639c4ce2347f5298cd70c6a

[
  {
    "name": "Player",
    "version": "",
    "maxplayers": 8,
    "numplayers": 0,
    "players": [],
    "bots": [],
    "queryPort": 14001,
    "NumOpenPublicConnections": 8,
    "NumOpenPrivateConnections": 0,
    "NumPublicConnections": 8,
    "NumPrivateConnections": 0,
    "bShouldAdvertise": true,
    "bIsLanMatch": true,
    "bUsesStats": true,
    "bAllowJoinInProgress": true,
    "bAllowInvites": true,
    "bUsesPresence": true,
    "bAllowJoinViaPresence": true,
    "bUsesArbitration": false,
    "bAntiCheatProtected": true,
    "OwningPlayerName": "Player",
    "OwningPlayerId": "0000000000000000",
    "[server]": {
      "BotSkill": "Experienced",
      "PureServer": "Yes",
      "LockedServer": "No",
      "AllowKeyboard": "No",
      "IsFullServer": "No",
      "IsEmptyServer": 1,
      "IsDedicated": "Yes",
      "SteamID": 0,
      "ServerDescription": ""
    },
    "[game]": {
      "MapName": "Custom",
      "VsBots": "2:1",
      "Campaign": "No",
      "ForceRespawn": "No",
      "GameMode": "Deathmatch",
      "NumBots": -1,
      "GoalScore": 3,
      "TimeLimit": 2,
      "NumBotsIA": 6,
      "OfficialMutators": [
        "Low Gravity"
      ],
      "SteamVAC": 0,
      "CustomMapName": "DM-1on1-Pure-Pro2",
      "CustomGameMode": "UTGame.UTDeathmatch",
      "CustomMutators": "EasyDodge",
      "CustomMutClasses": "\u001cEasyDodgeMutator\u001c"
    },
    "connect": "192.168.56.56:7777",
    "ping": 23
  }
]

The UnrealEngine3 protocol will serialize all provided data. Most of that data is also provided with a master/online query.

Note

Some changes are removed due to a rebase of this PR.

Changes:

---

Code

• Protocols are now imported and re-exported in protocols/index.js for better maintainability
• Subsystem protocols are searched by appending it to the given type with some common delimiters ($, _, : or nothing) which should be compliant to naming rule
• Some changes regarding the call of core:run to provide overloading
• Some changes regarding providing/overloading options

Query/Core

• User-Options are prioritized over any default/game/protocol/subsystem options
• New method core:updateOptionsDefaults(..), passed with final options to core/protocol to handle given options before the protocol runs
• Added core:getOptionsDefaults(outOptions) to provide default options per protocol
• Added core:getOptionsOverrides(outOptions) to provide overriding options per protocol
• New method core:prepareRun() to run before core:run(state)
• New method core:finishRun() to run after core:run(state)
• New method core:createState(), for custom handling of creating initial state
• New method core:finishState(), for custom handling of finishing the state
• New method core:populateState(), for populate the state with queried data (moved from core:run()
• Method setupOptions(..) to handle building up options list (should not be overloaded)

New Core protocol for LAN-protocols

• Defaults address to 255.255.255.255
• Defaults givenPortOnly to true
• Enables broadcast mode based on resolved/given address

UnrealEngine3/LAN protocol:

Provides basic functionality of parsing UE3 LAN packets send over UDP broadcast

• LAN query data does not contain player data
• UE3 UDP packet definition is compatible to most UE3 based games, some newer engines support Steam which adds an additional 4-byte value (see packetVersion)
• Games can be added by providing packetVersion and gameUniqueId (+ port, see config for [IpDrv.OnlineGameInterfaceImpl] and property LanAnnouncePort)
• Some games are using the same LAN port and unique game id (such as UDK SDK or Toxikk), these games need to be differentiated on consuming client)

UT3 LAN protocol:

• Added subsystem "LAN", which overrides the query port to be 14001, and sets specific query options for a proper LAN (broadcast) query
• Sets packetVersion to 5, and gameUniqueId to 0x4D5707DB

Reader:

• Added method to read 8-byte double values (double())
• Added peek parameter for reader:remaining(..) to check if the reader can read the additional data

[mmorrisontx]

-1 for putting non-standard query output into the main object. all of that should be in raw, or else it becomes part of the promised standard which is impossible to maintain cross-game.
-1 for subsystem. It may as well just be a different game ID if it's a different query protocol. That's probably up to the current maintainers though.
-1 for doing discovery within a game protocol. Gamedig has always been about querying servers that you know about, and discovery was left out of scope. If discovery is needed, I'd argue that it should be an independent method call, totally separate from the existing query api.

[RattleSN4K3]

Thanks for your feedback.

-1 for putting non-standard query output into the main object. all of that should be in raw, or else it becomes part of the promised standard which is impossible to maintain cross-game.

That's applicable in a sec. Good to know.

-1 for subsystem. It may as well just be a different game ID if it's a different query protocol. That's probably up to the current maintainers though.

If the game-entry (in games.js) corresponds to a client, isn't it the same game ID then, or am I missing something?

By looking at Minecraft how it is set up, each entry would require a different client to connect to a given game ID (same with GTA 5 what I can see). Whereas with UT3, you are still bound to the game, same client, same connection method even when having the subsystem.

Both response data (for unrealtournament3 and such unrealtournament3lan game ID) would have the same output (except players, or bots). What's the ideal approach I can follow?

-1 for doing discovery within a game protocol. Gamedig has always been about querying servers that you know about, and discovery was left out of scope. If discovery is needed, I'd argue that it should be an independent method call, totally separate from the existing query api.

In various tests the LAN servers for UT3 didn't respond to unicast query. That was/is the goal of this feature.

Currently, my testing server running on localhost responds to a unicast using the same subsystem with the option address: '192.168.56.56' But since the response wouldn't be parsed properly with the existing protocol(s), the custom handling would be required. Thus the "subsystem" protocol (although it can be any protocol).

Adding support for discovery was just a small change (7b16301). The only critical change in that would be the non-standard response by receiving 1:n server responses.

---

I likely split the settings-resolver part from this pull request since that can be an additional one. How I can continue changing some bits of the PR?

[Hornochs]

Having two game entries for the same game, like unrealtournament3 and unrealtournament3lan has pro and cons.

The pro of having two GameIDs is that it’s more visible for the user that there are two different approaches for the game. Which means when user setup gamedig and seeing the gamelist they don‘t have to look deep into the documentation or notes.

Con is that, at least for me, it would be more „clean“ to handle. Having one gameID and a sub parameter makes more sense. On different games are multiple parameters necessary as well, like nadeo with Userlogin.

[CosminPerRam]

Same opinion as @mmorrisontx, but to extend some stuff:

If the game-entry (in games.js) corresponds to a client, isn't it the same game ID then, or am I missing something?

Hope I got your question right, #608 would benefit from exactly what you describe but introducing something like subsystem would be a breaking change (can definitely be considered for the next major version, also considering the plans for rcon support, which would generate more GIDs).

-1 for doing discovery within a game protocol.

Strongly agreed, #484 is similar to this, and it would be something separated from the query api.

[...] The only critical change in that would be the non-standard response by receiving 1:n server responses.

For now at least, this could be written in the protocol implementation (or let that be a base protocol).
If multiple games needing this appear, we could then move it over.

[CosminPerRam]

I appreciate a lot the addition of docs comments, but they should come in a separate PR to this one.

[RattleSN4K3]

I appreciate a lot the addition of docs comments, but they should come in a separate PR to this one.

Right. I was just using that logic specifically and just added the JSDoc comment.

[RattleSN4K3]

I've rebased the pull request's source branch as I reduced the complexity, based on the given feedback

• fully removed subsystem
• added unique/separate game ID
• removed unrelated code changes
• backup is here backup/ue3-lan_1

I reply to some of your messages. The current codebase for this PR has changed a bit so you guys may need to review the changes. It is still not fully tested with additional games LAN query.

tl;dr The current PR will result into such queries:
https://gist.github.com/RattleSN4K3/55958bfc0639c4ce2347f5298cd70c6a

---

mmorrisontx: Gamedig has always been about querying servers that you know about

(1) I thought about that a bit. There is some smaller issue with that regarding providing the data. Even by knowing the server's IP (on a LAN environment), the single query can lead to (receiving multiple or) just a single random respone since you query the IP using the "LanAnnouncePort" which is shared across any instance on the same system. (from the same game).

A LAN setup where a single machine would host several instances of the same game (if supported) could be the norm.

CosminPerRam: subsystem would be a breaking change

It was opt-in. Out of interest, could you elaborate what could be considered breaking? All existing protocol wouldn't be affected.

That subsystem logic could be seen as a set of properties, specified on a game level, to be ready to use. For convience it was auto-resolving the subsystem (when set) to a given protocol, which could be seen as breaking.

CosminPerRam: Strongly agreed, #484 is similar to this, and it would be something separated from the query api.

I see what you guys mean by splitting any Discovery API from Query, but having a setup like mentioned in (1), the query would be incomplete since you cannot specifiy a single system on such LAN setup. That would require various filter options (like only responding to given ports). All game instance (of UT3, and various UE3 games) use the same "query" port.

Since the single query call and response for the LAN protocol is the same with having gamedig being set as broadcast, I would guess query would fit for getting data from LAN servers.
Without setBroadcast the socket will throw an error when using a broadcast address.

When using the unicast address for a network address/machine which may have/run multiple servers, you won't receive additional ones, only one random (although it seems to be the first one bound to the port).

So the goal of this PR i basically fixing the query of single IP:Port on a local network. The discovery of instances on another network address could be seen as bonus 😁

---

I had breaking changes in mind that's the reason of enabledBroadcast required to be truthy. Only the Core LAN protocol(s) would set this (based on detecting a broadcast IP). So it would never change how the udpSocket acts for any existing protocol (or setup, since it is a new field to be set).

But at the end I might not see the broad system to actualy judge how breaking a change could be. 🙃

[Hornochs]

(1) I thought about that a bit. There is some smaller issue with that regarding providing the data. Even by knowing the server's IP (on a LAN environment), the single query can lead to (receiving multiple or) just a single random respone since you query the IP using the "LanAnnouncePort" which is shared across any instance on the same system. (from the same game).

And that is one of the reason that there is the need of a check if the received Package is from the server you query. Querying ingame uses the same system, the gameclient sends the broadcast to the beacon Port. Imagine the following scenario:

• GameDig gets started to query a server
• Just in that time a player sends an broadcast request

Of course the server should sends two times, but it could be an timing issue and GameDig will receive a Answer Package from a different gameserver, or even of a different game if the unique LANID is different 😐

[RattleSN4K3]

And that is one of the reason that there is the need of a check if the received Package is from the server you query.

The initiated query/lookup by a game or GameDig will only receive-and-process the response(s) meant for each specific client instance (for UE3). That's what the nonce is for which is not static but generated for each client (UE3 engine wide and in this PR).

node-gamedig/protocols/unrealengine3lan.js

Lines 32 to 33 in 2ce6690

	// generate unique client id
	this.clientNonce = unrealengine3.generateNonce(8)

node-gamedig/protocols/unrealengine3lan.js

Lines 283 to 287 in 2ce6690

	// is response from same client?
	const nonceRaw = reader.part(8)
	const nonceHex = nonceRaw.toString('hex')
	const clientNonceHex = Buffer.from(this.clientNonce).toString('hex')
	bIsValid = (nonceHex === clientNonceHex)

Receiving responses from different games than expecting (due to the games having wrongly used game ids) isn't a problem of Gamedig, the broadcast or the current UDP broadcast implementation of this PR.

[CosminPerRam]

Out of interest, could you elaborate what could be considered breaking? All existing protocol wouldn't be affected.

It isn't a breaking change in the sense that the existing protocols wouldn't be affected, but rather than how all the games are currently being handled by some users, for example Homepage selects their game solely by the GID, introducing this in v5 would leave some entries in being unreachable by them (although this could be solved by also adding GIDs for these?).