Testing Client Server and Client Only implementations

[alex-dukhno]

This issue serves a place where we can discuss how we are going to implement driver part to test frames flow through the implementations:

• client-server
• client-only

[alex-dukhno]

Possible testing flow for Client-Server

[alex-dukhno]

Possible flow for testing Client-Only

[whyoleg]

I have first proposal for client testing

Main problem is that we need to somehow say RSocket client what to do, using some driver. Well then, we need some general, language agnostic way to do it. I think, that we need to control which high-level commands can client do.

Commands

• System commands
  • CaseStart - tell driver, that need to run test case (contains name of case, and any other meta info)
  • CaseEnd - tell driver, that test case finished (contains status: success, failed; can contain timestamp, or other meta info)
  • TestResult - tell driver that testing is finished (contains full tests info, stats (all/ignored/succeeded/failed tests amount), can contain other info)
• Connection commands
  • SetupConnection - tell driver how to setup connection (contains mime types, payload, optional resume token and other info)
  • ResumeConnection - tell driver that connection need to be resumed (contains info needed to resume)
  • CloseConnection - tell driver to close connection (TODO seems needed, but not sure, can be used just to close/cleanup connection)
• Request commands (stream initiation)
  • MetadataPush - tell driver, that client need to push metadata (contains metadata - to check it then in tck case) - not increment streamId
  • FireAndForget - tell driver, that client need to do fire and forget (contains data, metadata - to check it in tck case) - increment streamId
  • RequestResponse - tell driver, that client need to get response (contains data, metadata - to check it in tck case) - increment streamId
  • RequestStream - tell driver, that client need to initiate stream (contains data, metadata - to check it in tck case) - increment streamId
  • RequestChannel - tell driver, that client need to initiate channel (contains data, metadata - to check it in tck case) - increment streamId
  • Payload - tell driver, that client need to continue sending payload in channel (contains streamId of channel, data, metadata, flags - to handle it in tck case)
  • RequestN - tell driver, that client need to request more elements in stream or channel (contains streamId of channel or stream, amount of elements to request - to use it in tck case)
• Request finish commands (stream completion)
  • CancelStream - tell driver, that stream(RR, RS, RC) need to be canceled (contains streamId of stream to cancel)
  • FailStream - tell driver, that stream(RR, RS, RC) need to be closed with error (contains streamId of stream to cancel)

For the first POC I propose that we will have some tck-server with 2 routes:

• client/commands - route to connect tck-server and client-driver - f.e. WebSockets
• client/rsocket - route for connecting RSocket client to run test cases - can be TCP, WS, Aeron or any other transport, depending on tck-server configuration
  So we will have simple single client server. After TCK launched, tck-server will be started and user will need to start client-driver.

Client driver

Driver will connect to commands route and listen and execute commands in loop. Simple single test case will contains following list of commands:

1. CaseStart - start case handling (more o less driver specific)
2. SetupConnection - create RSocket client, init some connection with rsocket route
3. RequestResponse - do some request through RSocket client
4. CloseConnection - [TODO] if needed close connection by driver
5. CaseEnd - finish case, do smth in driver if needed
6. TestResult - print/save results, do anything

TCK Case

Code of Kotlin test case for such example will look similar to (numbers in comments linked to numbers in client driver commands list):

//[1] start of case, command will be sent automatically 
sendCommand(SetupConnection(resume = false, lease=false ...)) //[2] send command to setup some connection
val setup: SetupFrame = receiveFrame().asSetup() //receive setup frame
setup expect SetupFrame(resume = false, lease = false ...) // check that setup is as expected, all flags and other properties valid
sendCommand(RequestResponse(data = "test")) //[3] send command to do request
val requestResponseFrame: RequestResponseFrame = receiveFrame().asRequestResponse() //receiver sent frame
requestResponseFrame expect RequestResponseFrame(data = "test") //check validity of frame
//[5] case finished, command will be sent automatically
//[6] if no more cases needed to run, command with test results will be sent automatically

P.S.

• Commands can be just json/protobuf(or of other format) serialized payloads which can be parsed by driver.
• Also, need to support reverse sending commands from client-driver, f.e. to send back data from response and then check it in test case (it will be not so hard)
• Similar approach can be applied for testing RSocket server using server-driver and tck-client
• Configuration of tck-server can be created using json/yaml or other DSL, it will contain:
  • transport: ws, tcp, aeron
  • cases to run or ignore using some pattern
  • can be extended if needed

[alex-dukhno]

I have first proposal for client testing

Could you please clarify for what kind of RSocket implementation client-server or client-only or both?

I like the list of commands - that will help to write spec cases.
However, I have one concern: overall, the high level problem, at least as I see it, is that there is no way that we can test as dummy as possible RSocket implementation. Those who are going to implement RSocket protocol in any programming language should build some sort of software that will mimic application code to send/receive packages to/from RSocket TCK. And what I'd like to see is the solution that will require the least of effort to build that additional software.
Or we have to come up with idea how RSocket implementations could be tested without additional software.

[whyoleg]

To be honest Im not sure, that I understand client-server case, can you explain it in more details?
Workflow as I proposed is for testing RSocket client, so I think it's client-only

for the second part, I will answer a little later

[whyoleg]

Im not sure, that it's possible to test protocol implementation without some driver/ additional software?.
So, I think, that if we use such arch as I propose, those who are implementing RSocket protocol will be needed to write really not so much code:

1. Serialization/Deserialization of json/proto - can be easily done in any language
2. Connect to some WS connection - same
3. Handling commands - just a simple loop

If to use some pseudo language, command handling will be look like this:

//assume we have already written/generated commands as some classes/structures

//we need to connect to WS
ws = wsconnect("localhost:PORT/client/commands")
streams = Map STREAM_IDENTIFICATOR -> STREAM
rsocketClient = null
while(true) {
 command = deserialize(ws.receive())
 when(command) {
  is CaseStart -> print("start case")
  is CaseEnd -> print("print some stats from command data"); streams.clean()
  is TestResult -> exit

  is SetupConnection -> rsocketClient = RSocket.client(/*get data from setup and create client*/)
  
  is RequestResponse -> streams.put(/* get STREAM_IDENTIFICATOR from command payload*/, rsocketClient.requestResponse(/*get payload from command/*))

  is CancelStream -> streams.get(/*get STREAM_IDENTIFICATOR from command payload/*).cancel()
  is RequestN -> streams.get(/*get STREAM_IDENTIFICATOR from command payload/*).request(/* get request number from payload*/)
 
 }
}

It's simplified a lot, but I think you understand an idea, and that it's really needed to do a little to implement such driver - it can be even one file of code if to not count commands serialization