Replay

[vjeux]

Now that there's a leaderboard, would be awesome to also upload a replay file so we can see how they got that record. Replay should have both the inputs and the car position. Then we can replay the engine on-top of the inputs and see that it actually matches to verify.

[wuharvey]

Would also be cool to show time differentials on checkpoints like in Gran Turismo!

On the main topic, wonder if we should just save the input timestamps and just "replay" through the game logic.

[Gusted]

@njm222 we had discussed this a bit in another way to validate server-sides the input.

1 Problem I see here is that the data will be stored somewhere in a array, which can become either 1 pretty memory hungry if someone just wants to test out the map. 60x such stats will be pushed to an array, also should account for users that experience webGL Lag due to driver issues etc.

[vjeux]

should account for users that experience webGL Lag due to driver issues etc.

For this, you probably want to decouple the rendering from the engine. Ideally we should be able to run the physics algorithm without webgl involvement.

[lortschi]

What about capturing to video file, kind like this somehow: https://github.com/jbaicoianu/threecap You could set up a game position of the car where it would start.

[Gusted]

For this, you probably want to decouple the rendering from the engine. Ideally we should be able to run the physics algorithm without webgl involvement.

Oh, I meant with WebGL lag that Render engines sometimes just jitter(firefox + nvidia + linux?) sometimes, because 🤷🏽. And as nature of javascript it will be blocked on that exact GL call and thus make a couple of frames to be the same stats which is due to physics not possible and looks like cheating.

What about capturing to video file, kind like this somehow: https://github.com/jbaicoianu/threecap You could set up a game position of the car where it would start.

Storing raw data is verifiable by anyone to check. Making a video will be 1 not so great for performance, even if it's in a another thread. Because we don't know which GPU supports which encoding of a format. And if it's an integrated GPU, it won't like you that it has to encode something + handle WebGL. Also storing videos isn't great on space size 😄.

[lortschi]

But if the video replay just for ssr enabled and not provided to the local open source, then could be on the other hand performing well 😜 It could be stored as a gif in few 100 kb in less quality.

[Gusted]

But if the video replay just for ssr enabled and not provided to the local open source

Huh? You want the video to be rendered server side?

then could be on the other hand performing well stuck_out_tongue_winking_eye It could be stored as a gif in few 100 kb in less quality.

If you want the server to render it you still need the collect raw data and still mitigate and account for lag etc. This whole render a video is not necessary for a replay system in my eyes. A lot if not all replay systems I know(ex. trackmanio, Mario kart TV) will fetch the raw input data and feed that the current physics system and let them figure out how to display it etc.

[njm222]

I think the only way this is feasible is to save the user's raw input in an array.
Since we have the raw input we can validate & replay. (I think it's the most flexible way)
This still leaves the issue with lag & random jitters... a few ways to combat this:

1. save device info (replicate the same environment when validating)
2. save timestamps of lag & jitters (to help with false positives)
3. send start and end timestamps (for tampered inputs)
4. save vehicle position every x seconds (to keep in sync [I think GTA does this])

Notes:

• Timestamps should be THREE.Clock?
• The validation can be pretty basic in the beginning and we can build on it.
  (Ideally if we have enough data we can label them and use a ML model to help detect cheating)

Regarding Gusted's concern:

1 Problem I see here is that the data will be stored somewhere in a array, which can become either 1 pretty memory hungry

Perhaps we just save one 'time' per user? So if you go faster than your previous time we overwrite it? (not the best solution.. just throwing it out there)

Also compressing the inputs, especially repeated ones, for example: for most of the run the user will be holding down up instead of capturing all the inputs we just capture when keydown and keyup for each key... should reduce size by quite a bit

[Gusted]

Also compressing the inputs, especially repeated ones, for example: for most of the run the user will be holding down up instead of capturing all the inputs we just capture when keydown and keyup for each key... should reduce size by quite a bit

So saving delta values? Should be okay. I will be looking into some kind of prototype, especially to combat the jitter etc. While not shaving of valuable frames.

Timestamps should be THREE.Clock?

If we somehow can ensure that every x seconds or ms we have some kind of raw input, we can assume the times when to play so no needs for timestamps.

[njm222]

Not sure what you mean by delta values.. I mean the array we save will look something like this (sorted by timestampDown):

[
  {
    key: string, // action is probably better here so it works for mobile as well
    timestampDown: THREE.Clock.getElapsedTime(),
    timestampUp: THREE.Clock.getElapsedTime()
  },
  ...
]

If we somehow can ensure that every x seconds or ms we have some kind of raw input, we can assume the times when to play so no needs for timestamps.

I don't like this way because it tends to feel laggy and can miss input.. (listening for input > expecting input)

I don't see the issue with using timestamps, but maybe I'm missing something?

[Gusted]

Not sure what you mean by delta values.

Only store a property if it differ from the previous state.

I don't see the issue with using timestamps, but maybe I'm missing something?

It's that we're talking about a replay system so we need every input that the user uses. So using timestamps and guesses what the user inputs might be wrong because it registerd a up when the code needed it but not a up when the "collector" checked it.

[lortschi]

Would you like to save the complete race user control inputs or just specific areas from track? If you compare with other racing games you can see they just captures specific track sections and uses other cameras with different positions to have the fun level higher, eq. Mario Cart: https://youtu.be/AGJcFEIMDwU

[Gusted]

Would you like to save the complete race user control inputs

That's pretty much my idea, still. As looking into recording/getting video capture will open up other problems.

If you compare with other racing games you can see they just captures specific track sections and uses other cameras with different positions to have the fun level higher,

I'm not sure as I cannot find any information online. But they probably just capture the raw input and send them to anyone who want to see the replay and feed that data to the physics and then let them control the replay with different camera's etc. As doing that well gaming is a no-go in performance as I pointed in:

Because we don't know which GPU supports which encoding of a format. And if it's an integrated GPU, it won't like you that it has to encode something + handle WebGL.

[lortschi]

yes that‘s really curious why online couldn‘t be found any inputs regarding replay uses 🤔

[Gusted]

As looking into recording/getting video capture will open up other problems.

To clarify: No problem to render such thing afterwards. But while in-game just collect raw input to get it performant as-is.

[njm222]

I don't see the issue with using timestamps, but maybe I'm missing something?

It's that we're talking about a replay system so we need every input that the user uses. So using timestamps and guesses what the user inputs might be wrong because it registerd a up when the code needed it but not a up when the "collector" checked it.

I think you may have misunderstood, there is no 'guesses'. I am saying we save every input with a timestamp like the example array I posted before.

• It is saved on each user action not with a 'collector'.
• After the run that array is fed to a 'validator' that replays the actions.
• When replaying it may be out of sync slightly.. that's where the vehicle position comes in. Every x seconds we will need to sync the replayed vehicle with the saved points (IFF it is within some deviation)
• The validator will be doing something like this:

let i = 0

useFrame(({clock}) => {
 if (clock.getElapsedTime > savedActions[i].timstampDown) {
  expectingUpActions.push(savedActions[i])
  // do DOWN savedActions[i].action
  i++
 }
 expectingUpActions.forEach(upAction => {
  if (clock.getElapseTime > upAction.timestampUp) {
   // remove upAction from expectingUpActions
   // do UP savedActions[i].action 
  }
 }
 // check currVehicle position against savedVehicle position
 // if outside deviation => run is invalid
 // if goal is reached => run is valid
})

[Gusted]

@njm222 I understand you now. Seems more do-able.