Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stability testing with ADE repo #238

Closed
PauliusKu opened this issue Sep 30, 2024 · 6 comments · Fixed by #292
Closed

Stability testing with ADE repo #238

PauliusKu opened this issue Sep 30, 2024 · 6 comments · Fixed by #292
Assignees
Labels
enhancement New feature or request

Comments

@PauliusKu
Copy link
Collaborator

PauliusKu commented Sep 30, 2024

task for code text change testing with ADE repo.

Current goal is to reach less than 1%

@PauliusKu PauliusKu self-assigned this Sep 30, 2024
@PauliusKu PauliusKu added the enhancement New feature or request label Sep 30, 2024
@PauliusKu
Copy link
Collaborator Author

328 failing tests so far (4396 passing)

@PauliusKu
Copy link
Collaborator Author

266 with current develop

@PauliusKu
Copy link
Collaborator Author

PauliusKu commented Nov 26, 2024

174 with tree-sitter 0044 (3.68% failed)

@PauliusKu
Copy link
Collaborator Author

PauliusKu commented Nov 26, 2024

After making this change #267 the number of errors was reduced to 91 (1.93%)

@PauliusKu
Copy link
Collaborator Author

82 with tree-sitter 0045

@PauliusKu
Copy link
Collaborator Author

PauliusKu commented Jan 7, 2025

TODO list:

  1. Improve framework to allow easier debugging (write failing test results to files)
  2. Go through all existing issues and analyze them. Categorize by issue's source: Tree-Sitter parser error, Tree-Sitter parsing incorrect, Formatter bug, else ...

TODO part 2:

  1. In addition to before & after files, create a simple list (like ignore file) for failing test, so we could use it as input in known issues file (see part 2).
  2. Add known issues file, so that framework might not show some known failures
  3. Add full project relevant file paths to the file names used while testing, because in a code base there might be multiple files with same name but different location.
  4. Improve the testing framework and attach it to GitHub pipeline as sort of regression tests.

@darkeris345 darkeris345 linked a pull request Jan 15, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants