Skip to content

Add Google Test suite for regression testing baseline.#276

Open
fdcastel wants to merge 2 commits intoFirebirdSQL:masterfrom
fdcastel:add-tests-from-v4-pr
Open

Add Google Test suite for regression testing baseline.#276
fdcastel wants to merge 2 commits intoFirebirdSQL:masterfrom
fdcastel:add-tests-from-v4-pr

Conversation

@fdcastel
Copy link
Member

Summary

Adds a comprehensive Google Test suite (375 tests across 38 test suites) that exercises the Firebird ODBC driver through the standard ODBC API via the Driver Manager. This establishes a regression testing baseline on the current, unmodified driver — per the plan discussed in PR #275.

No driver source code is modified by this PR.

Motivation

Before making any code changes (bugfixes, build system, CI), we need a testing baseline that documents what the current driver can do and where improvements are needed. This test suite:

  1. Verifies existing functionality — 230 tests pass, confirming data types, parameter binding, cursors, catalog functions, etc. all work correctly.
  2. Documents known gaps — 145 tests are marked GTEST_SKIP() with explanations of what improvement they require (e.g., "Requires Phase 7 (OC-1): SQLCopyDesc null records crash fix").
  3. Provides a regression gate — future PRs can remove individual GTEST_SKIP() markers as fixes are merged, turning them into pass/fail tests.

What's included

Infrastructure

File Description
tests/CMakeLists.txt Standalone CMake project; fetches Google Test v1.14.0 via FetchContent
tests/test_main.cpp GTest entry point
tests/test_helpers.h OdbcConnectedTest fixture, TempTable RAII guard, error helpers
tests/README.md Build/run instructions, test categories, connection string format

Test files (30 files)

Category A — Tests that pass on vanilla master (16 files)

File Description
test_data_types.cpp All SQL data types: INTEGER, VARCHAR, NUMERIC, DATE, TIME, TIMESTAMP, etc.
test_result_conversions.cpp SQLGetData type conversions
test_param_conversions.cpp SQLBindParameter type conversions
test_prepare.cpp SQLPrepare / SQLExecute lifecycle
test_cursors.cpp Cursor behavior across statements
test_cursor_commit.cpp Cursor behavior across transactions
test_cursor_name.cpp SQLSetCursorName / SQLGetCursorName
test_data_at_execution.cpp SQL_DATA_AT_EXEC / SQLPutData
test_array_binding.cpp Column-wise + row-wise parameter arrays
test_bindcol.cpp Dynamic unbind/rebind mid-fetch
test_descrec.cpp SQLGetDescRec for all column types
test_blob.cpp BLOB read/write (small, large, NULL)
test_multi_statement.cpp Multiple statement handles on one connection
test_stmthandles.cpp Stress test: 100+ simultaneous statement handles
test_wchar.cpp SQL_C_WCHAR bind/fetch and truncation
test_escape_sequences.cpp ODBC escape sequence passthrough

Category B — Mixed pass/skip (6 files)

These files contain tests that pass and tests that are GTEST_SKIP()'d because they depend on driver improvements not yet merged.

File Pass Skip Skip reason
test_descriptor.cpp 6 7 SQLCopyDesc crash, SetDescCount allocation
test_connect_options.cpp 6 30 CONNECTION_TIMEOUT, ASYNC_ENABLE, QUERY_TIMEOUT, RESET_CONNECTION
test_errors.cpp 11 7 SQL_DIAG_ROW_COUNT, string truncation indicator
test_catalogfunctions.cpp 26 3 SQLGetTypeInfo ordering, GUID searchability
test_server_version.cpp 4 2 FB4+ type info detection
test_scrollable_cursor.cpp 5 4 Scrollable cursor edge cases

Category C — All tests SKIP'd (8 files)

These files test features that don't exist on vanilla master. Every test is GTEST_SKIP()'d with an explanation.

File Tests Feature required
test_null_handles.cpp 65 Crash prevention: NULL handle validation
test_savepoint.cpp 4 Savepoint transaction isolation
test_conn_settings.cpp 3 ConnSettings DSN attribute
test_odbc38_compliance.cpp 12 ODBC 3.8 (SQL_OV_ODBC3_80, RESET_CONNECTION)
test_guid_and_binary.cpp 14 SQL_GUID type mapping, FB4+ types
test_odbc_string.cpp 26 OdbcString class (guarded with __has_include)
test_phase7_crusher_fixes.cpp Not compiled (documentation only, tests duplicated in Cat B files)
test_phase11_typeinfo_timeout_pool.cpp Not compiled (documentation only, tests duplicated in Cat B files)

How to build and run

Prerequisites

  • The Firebird ODBC driver must be pre-built (from Builds/MsVc2022.win/OdbcFb.sln) and registered as an ODBC data source
  • A Firebird 3.0+ database must be accessible
  • CMake ≥ 3.14

Build

cmake -B build-tests -S tests
cmake --build build-tests --config Release

Run

rem Windows
set FIREBIRD_ODBC_CONNECTION=Driver={Firebird ODBC Driver};Dbname=localhost:C:\path\to\test.fdb;Uid=SYSDBA;Pwd=masterkey;
ctest --test-dir build-tests --output-on-failure -C Release

Expected output

[==========] 375 tests from 38 test suites ran.
[  PASSED  ] 230 tests.
[  SKIPPED ] 145 tests.
[  FAILED  ] 0 tests.

Without FIREBIRD_ODBC_CONNECTION set, all 375 tests skip gracefully.

Design decisions

  1. Standalone CMake project — Since there's no root CMakeLists.txt yet (that's a future PR), the test suite is self-contained with its own project() declaration. It will integrate via add_subdirectory(tests) when a root CMake build is added.

  2. Tests via Driver Manager, not direct DLL calls — Tests link against odbc32/odbccp32 (the Windows ODBC Driver Manager) and connect using a standard connection string. This tests the driver as applications actually use it. The one exception is test_null_handles.cpp (which loads the DLL directly to test entry-point validation) — all its tests are SKIP'd.

  3. GTEST_SKIP() preserves test code — Rather than deleting tests that don't pass on vanilla, each is wrapped with GTEST_SKIP() << "reason". The test code remains intact so that future PRs can simply remove the SKIP line to activate the test.

  4. No driver code changes — This PR adds only the tests/ directory. No existing source files are modified.

Checklist

  • Tests build with CMake 3.14+ / MSVC 2022
  • 230 tests pass against vanilla Firebird ODBC driver
  • 145 tests skip with clear explanations
  • 0 test failures
  • Graceful SKIP when no database connection is available
  • README with build and run instructions

@fdcastel
Copy link
Member Author

Pushed fixes for four issues identified in @irodushka’s review.

@fdcastel
Copy link
Member Author

@irodushka I’d love to continue working on the tasks from #275, but I need this one as a foundation for the rest.

When you have a chance, please take a look. 😉

@irodushka
Copy link
Contributor

irodushka commented Feb 23, 2026 via email

@fdcastel
Copy link
Member Author

P.s: I’m not sure I understood you correctly, but are you modifying the code to make the tests pass?

If so, please do not do that. At the very least, wait until we start working on the first third step of The Plan™️.

Also, remember that any additional changes you introduce now will only make it more difficult for me to merge this back into the existing code in the other branch (and, in there, all tests are passing).

For now, the goal is only to add the test suite -- not to make all the tests pass, yet.

After the 2nd and 3rd steps we will have a better infrastructure to make these changes and fixes, all automated in GitHub actions.

@irodushka
Copy link
Contributor

You don't catch - the question is not whether the test was passed or not. All the test suite is crashing on segmentation fault))
I see only 2 ways: 1) fix the bug OR 2) comment out the test(s) that causes the crash.

@irodushka
Copy link
Contributor

Hi @fdcastel

Well... I've found a buggy legacy code in OdbcStatement.cpp

-                       bindOffsetIndColumnWiseBinding = ( bindOffsetPtrTmp + rowNumberParamArray ) * sizeof ( SQLINTEGER );
+                       bindOffsetIndColumnWiseBinding = ( bindOffsetPtrTmp + rowNumberParamArray ) * sizeof ( SQLLEN );

and I see the correspoinding fix in your main PR.

But that means we cannot use the test suite on the master branch (because it crashes) without merging your changes in OdbcStatement.cpp. And I took a look on these changes (+886 -233) and you know, I'm not ready (even if it might be merged without all the rest staff, and it's impossible).

And when I switched of the crashing tests, the suite hanged up on some next test..

I think it makes sense to merge the tests if they work (they may pass or fail, no care), but if they crash? or hangup?.. no sense.

So, what can we do.

  1. Exclude the test cases that cause crashes/coredumps or hangs and cannot be fixed in the master branch in a simple way (not as above)
  2. Fix the master branch if it could be done in a simple way (as above)

Expected result: we can run a set of tests, some tests pass, some don't, but overall everything works, without freezing or crashing.

What do you think about it?

@fdcastel
Copy link
Member Author

fdcastel commented Feb 24, 2026

You don't catch - the question is not whether the test was passed or not. All the test suite is crashing on segmentation fault)) I see only 2 ways: 1) fix the bug OR 2) comment out the test(s) that causes the crash.

I understand your point now.

What’s puzzling is that the crash doesn’t happen on my system or in GitHub Actions. That suggests there may be differences in your development environment.

I recently ran into similar issues while trying to build https://github.com/asfernandes/fb-cpp. It became so problematic that I ended up creating a full set of installation scripts (https://github.com/fdcastel/cpp-dev) to replicate the GitHub Actions environment locally.

Rather than spending time debugging the crash specifically on your machine, I think the better approach would be for me to submit the second and third PRs (which introduce the new build system and GitHub automation). That would give you a clean, reproducible baseline to compare against your local setup and help identify what’s different.

The second PR would build on this one, and the third would build on the second. This way, you can review everything incrementally without having to merge all changes at once.

@irodushka
Copy link
Contributor

Hi @fdcastel

Sorry, I was dealing with some urgent family matters last week and I'm still busy with them and the backlog of things to do.
I will return to you and to Mr. Firebird ASAP.

Regards

@fdcastel
Copy link
Member Author

fdcastel commented Mar 2, 2026

Sure, man! Totally understand!

Take your time. But, of course, as soon as you unlock this, I’ll jump right into the next PRs 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants