Compilation update #46
Conversation
|
|
||
| // Asking LLM to generate test. Here, we have a loop to make feedback cycle for LLm in case of wrong responses. | ||
| while (!generatedTestsArePassing) { | ||
| log.info("New iterations of requests") | ||
|
|
||
| if (processStopped(project, indicator)) return | ||
|
|
||
| if (requestsCount >= maxRequests) { | ||
| if (isLastIteration(requestsCount) && project.service<Workspace>().testGenerationData.compilableTestCases.isEmpty()) { |
There was a problem hiding this comment.
What about the tests that are generated in the last request? In this case, the last request which is saved in line 196, will be ignored.
There was a problem hiding this comment.
Fixed loop:
- checking for the last iteration
- request with the updated prompt
- if some error → update the prompt && continue
- else → save result from the request in this iteration
There was a problem hiding this comment.
If the maximum number of requests are set to 1, the loop will show invalidGrazieResult without sending any request. This is because at the beginning of the loop, the number of requests will be increased to 1 and isLastIteration checks whether the number of requests is greater equal to the maximum number of requests (here 1). I think to solve this issue, the isLastItteration method should be changed to:
private fun isLastIteration(requestsCount: Int) = requestsCount > maxRequests
|
|
||
| // check is .class file exists | ||
| if (File(classFilePath).exists()) { | ||
| project.service<Workspace>().testGenerationData.compilableTestCases.add(testCases[index]) |
There was a problem hiding this comment.
When you send feedback, LLM will print the whole test file again. So, it is possible that we have some duplicate in compilable tests
There was a problem hiding this comment.
Also, we only add test cases here. We need to check if they are still compilable with the final test suite body of the test file (with imports and mocking-related annotations).
There was a problem hiding this comment.
I checked it, because each test case is in a separate file. I created them here: https://github.com/JetBrains-Research/TestGenie/pull/46/files/0a67016bc734ba460ee0d79ba8e836b35ae04ade#diff-87db339e5713928f5d39e84379b8955a6a684be4aef50c12171fc981e823f2c6R236
And I avoid duplicates, because this is a set of test cases 🙂
Description of changes made
Saving correct test cases during the incorrect common result