Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update catch2 tags assigned to object store tests #6788

Merged
merged 5 commits into from
Jul 14, 2023

Conversation

michael-wb
Copy link
Contributor

@michael-wb michael-wb commented Jul 14, 2023

What, How & Why?

To help identify the tests that currently use the baas server, the [baas] tag was added previously, but there were some tests that were missed. Also, the [local] tag was removed, since it was a bit ambiguous and not needed, since using ~[baas] will run those tests that do not depend on baas.
Other tags that were added were [sync], for those that use the sync component and either [pbs] or [flx] to label the type of sync that is performed between the client and the (local or baas) server.

The list of available tags can be viewed using the ./realm-object-store-tests --list-tags command.

☑️ ToDos

  • [ ] 📝 Changelog update
  • [ ] 🚦 Tests (or not relevant)
  • [ ] C-API, if public C++ API changed.

@danieltabacaru
Copy link
Collaborator

The list of available tags can be viewed using the ./realm-object-store-tests --list-tags command.

We run our tests through ctest, so I am not sure you can use those tags at all. You can expose them in ctest too. This is how I used to do it in CMakeLists, but it does not work anymore:

include(Catch)
catch_discover_tests(ObjectStoreTests)

@michael-wb
Copy link
Contributor Author

michael-wb commented Jul 14, 2023

The list of available tags can be viewed using the ./realm-object-store-tests --list-tags command.

We run our tests through ctest, so I am not sure you can use those tags at all. You can expose them in ctest too. This is how I used to do it in CMakeLists, but it does not work anymore:

include(Catch)
catch_discover_tests(ObjectStoreTests)

It turns out that we can use the ctest LABELS to break up the object-store-tests into baas vs local runs using the Catch2 tags:

FILE: tools/cmake/Utilities.cmake

macro(add_labeled_test _name _label _target _args)
    if(NOT APPLE)
        add_test(NAME ${_name} COMMAND ${_target} ${_args})
    else()
        # When given a target name, add_test() is supposed to automatically
        # determine the path to the executable. However, this is very broken on
        # Apple platforms. ctest doesn't perform macro expansion, so the path
        # is left with a `$(EFFECTIVE_PLATFORM_NAME)` in it. The generator
        # expression `$<TARGET_FILE:target>` is also just wrong on macOS, as it
        # uses a path suitable for an iOS bundle rather than a macOS bundle. As
        # a result, we have to construct the path manually.
        add_test(NAME ${_name} COMMAND $<TARGET_FILE_NAME:${_target}>.app/Contents/MacOS/$<TARGET_FILE_NAME:${_target}> ${_args})
    endif()
    set_tests_properties(${_name} PROPERTIES LABELS "${_label}")
endmacro()

And the implementation:
FILE: test/object-store/CMakeLists.txt

#add_bundled_test(ObjectStoreTests)
add_labeled_test(ObjectStoreTests-local LOCAL ObjectStoreTests "~[baas]")
add_labeled_test(ObjectStoreTests-baas BAAS ObjectStoreTests "[baas]")

When the test runs, it prints a summary for each label:

Label Time Summary:
BAAS         = 1169.52 sec*proc (1 test)
LOCAL        =  44.33 sec*proc (1 test)

To run just one of the tests using the label name, you would use ctest -VV -L <label-name>
For example:

$ ctest -L LOCAL -VV
Test project /Users/michael.wilkersonbarker/Dev/realm-core4/debug
    Start 4: ObjectStoreTests-local
. . .
1/1 Test #4: ObjectStoreTests-local ...........   Passed   38.31 sec

100% tests passed, 0 tests failed out of 1

Label Time Summary:
LOCAL    =  38.31 sec*proc (1 test)

Total Test time (real) =  38.35 sec

Copy link
Collaborator

@danieltabacaru danieltabacaru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work. Do you know if it's also possible to just run a specific test?

@michael-wb michael-wb merged commit 6225a93 into master Jul 14, 2023
2 checks passed
@michael-wb michael-wb deleted the mwb/update-objstore-test-tags branch July 14, 2023 18:19
@michael-wb michael-wb linked an issue Jul 17, 2023 that may be closed by this pull request
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 21, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update Catch2 tags for object store tests
3 participants