From 8340d6963ae942d27b1e9607dd368acfdc1e4060 Mon Sep 17 00:00:00 2001 From: Max-Julian Jakobitsch Date: Thu, 4 Jul 2024 17:23:48 +0200 Subject: [PATCH 1/4] docs: update readme add info on flutter patch, headless mode, reports, config --- README.md | 111 +++++++++++++++--- .../flutter/get-running-async-tasks.diff | 38 ++++++ 2 files changed, 135 insertions(+), 14 deletions(-) create mode 100644 doc/patches/flutter/get-running-async-tasks.diff diff --git a/README.md b/README.md index c77e9f42..c7690307 100644 --- a/README.md +++ b/README.md @@ -189,9 +189,34 @@ Flaky is flaky, and we are aware of it. It is neither failed nor success, so you ### CI / headless mode -This tool can be run without GUI and only produce log data and exit code, making it suitable to be run in a CI. If you want to examine the details with GUI, just open the generated artifact in the GUI using `Load Report` button. +[`packages/convenient_test_manager_dart`](packages/convenient_test_manager_dart) runs without GUI +and only produce log data and exit code, making it suitable to be run in a CI. +Simply use `convenient_test_manager_dart` instead of `convenient_test_manager`. +See [Run the `manager`](#run-the-manager) and [Getting started](#getting-started). + +```sh +# in one shell +convenient_test_manager_dart +# in another shell, run your worker app +flutter run integration_test/main_test.dart --host-vmservice-port 9753 --disable-service-auth-codes --dart-define "CONVENIENT_TEST_APP_CODE_DIR=$PWD" +``` + +### Reports + +If you want to examine the details with GUI, just open the generated artifact in the GUI using `Load Report` button. + +By default the manager will save reports to a temporary directory. Typically: + +- macOS: `$HOME/Library/Caches/ConvenientTest` +- Linux: `/tmp/ConvenientTest` +- Windows: `%LocalAppData%\Temp\ConvenientTest` + +Use `--report-save-path /some/path` (or `--dart-define "CONVENIENT_MANAGER_REPORT_SAVE_PATH=/some/path"`) to change this. -// TODO: Add more doc about how to use it (informal doc here: https://github.com/fzyzcjy/flutter_convenient_test/issues/240) +#### Format + +The reports are exported using [protobuf](https://protobuf.dev/). +See the schema at [`convenient_test.proto`](packages/convenient_test_common_dart/proto/convenient_test.proto). ### Run single test/group @@ -233,32 +258,84 @@ Following are about the toggles: ## Tutorial: Run examples 1. Clone this repository and enter the `packages/convenient_test/example` folder. -2. Run the example app (e.g. using iOS simulator) via `flutter run /path/to/flutter_convenient_test/packages/convenient_test/example/integration_test/main_test.dart --host-vmservice-port 9753 --disable-service-auth-codes --dart-define CONVENIENT_TEST_APP_CODE_DIR=/path/to/this/app`. Can also be run via VSCode or Android Studio with similar commands. +2. Run the example app (e.g. using iOS simulator) via: + ```sh + flutter run integration_test/main_test.dart --host-vmservice-port 9753 --disable-service-auth-codes --dart-define "CONVENIENT_TEST_APP_CODE_DIR=$PWD" + ``` + Can also be run via VSCode or Android Studio with similar commands. 3. Run the GUI located in `packages/convenient_test_manager`. (See details in section below) 4. Enjoy the GUI! ## Getting started -1. In `pubspec.yaml`, add `convenient_test: ^1.0.0` to the `dependencies` section, and `convenient_test_dev: ^1.0.0` to the `dev_dependencies` section. As normal, we need to `flutter pub get`. -2. Create `integration_test/main_test.dart` file in your app. Fill it like `void main() => convenientTestMain(MyConvenientTestSlot(), () { ... the normal test code you write });`. See [the example package](https://github.com/fzyzcjy/flutter_convenient_test/blob/master/packages/convenient_test/example/integration_test/main_test.dart) for demonstration. -3. Run your app (e.g. using iOS simulator) via `flutter run /path/to/your/app/integration_test/main_test.dart --host-vmservice-port 9753 --disable-service-auth-codes --dart-define CONVENIENT_TEST_APP_CODE_DIR=/path/to/this/app`. Can also be run via VSCode or Android Studio with similar commands. +1. Navigate to the flutter home directory and apply [this patch](doc/patches/flutter/get-running-async-tasks.diff): + `cd $FLUTTER_HOME && curl https://raw.githubusercontent.com/fzyzcjy/flutter_convenient_test/master/doc/patches/flutter/get-running-async-tasks.diff | git apply` + - while this is not strictly necessary, it is recommended to improve reliability when running multiple async tasks. see [#337](https://github.com/fzyzcjy/flutter_convenient_test/issues/337) for more information +2. Navigate to your project dir and run `flutter pub add convenient_test dev:convenient_test_dev` + - or manually update your `pubspec.yaml` to include `convenient_test: ^1.0.0` in the `dependencies` section, and `convenient_test_dev: ^1.0.0` in the `dev_dependencies` section. As normal, we need to `flutter pub get`. +3. Create `integration_test/main_test.dart` file in your app. Fill it like `void main() => convenientTestMain(MyConvenientTestSlot(), () { ... the normal test code you write });`. See the example [`main_test.dart`](packages/convenient_test/example/integration_test/main_test.dart) for demonstration. +4. Run your app (e.g. using iOS simulator) via `flutter run integration_test/main_test.dart --host-vmservice-port 9753 --disable-service-auth-codes --dart-define "CONVENIENT_TEST_APP_CODE_DIR=$PWD"`. Can also be run via VSCode or Android Studio with similar commands. - for use with the Android emulator, add `--dart-define CONVENIENT_TEST_MANAGER_HOST=10.0.2.2` -4. Run the GUI located in `packages/convenient_test_manager`. (See details in section below) -5. Enjoy the GUI! +5. Run the Convenient Test Manager (See details in section below) ### Run the `manager` +- [`packages/convenient_test_manager`](packages/convenient_test_manager) is a flutter GUI app that allows running and inspecting individual tests. + Use this for development and debugging. +- [`packages/convenient_test_manager_dart`](packages/convenient_test_manager_dart) is a dart CLI app that runs all tests and terminates with a corresponding exit code. + Use this for CI and automated testing. See [CI / headless mode](#ci--headless-mode) + #### Method 1: Download precompiled binary -There are precompiled binaries for each commit, at the "artifacts" section in [the CI page](https://github.com/fzyzcjy/flutter_convenient_test/actions/workflows/ci.yaml). For example, download `manager_macos` artifact, unzip it, and open it. +See [Releases](https://github.com/fzyzcjy/flutter_convenient_test/releases) for binaries. + +Additionally, there are precompiled binaries for each commit, at the "artifacts" section in [the CI page](https://github.com/fzyzcjy/flutter_convenient_test/actions/workflows/ci.yaml) +(download only available for logged in GitHub users). +For example, go to the most recent _CI_ run, download `manager_gui_macos` artifact, unzip it, and open it. + +#### Method 2: `flutter run` / `flutter build` + +Check the requirements for developing flutter desktop apps for +[macOS](https://docs.flutter.dev/platform-integration/macos/install-macos/install-macos-from-ios), +[Linux](https://docs.flutter.dev/platform-integration/linux/install-linux/install-linux-from-android), +and [Windows](https://docs.flutter.dev/platform-integration/windows/install-windows/install-windows-from-android). + +```sh +git clone https://github.com/fzyzcjy/flutter_convenient_test.git +cd flutter_convenient_test/packages/convenient_test_manager -Currently only MacOS and Linux has CI for this, and I am willing to accept a PR for Windows. +# compile executable. platform = linux | macos | windows +flutter build +# or just run directly +flutter run -d +``` + +##### For CLI manager: + +```sh +cd flutter_convenient_test/packages/convenient_test_manager + +dart compile exe bin/convenient_test_manager_dart.dart +# or just run directly +dart run +``` + +#### Method 3: `dart pub global` (only CLI manager) -#### Method 2: `flutter run` / `flutter profile` / `flutter build` +Set up once (re-run to update): -`packages/convenient_test_manager` is nothing but a *normal* Flutter Windows/MacOS/Linux app, so run it following the Flutter official doc. +```sh +dart pub global activate --source git https://github.com/fzyzcjy/flutter_convenient_test.git --git-path packages/convenient_test_manager_dart +``` + +> [!TIP] +> Add `--git-ref v1.x.x` to `dart pub global activate` to check out a specific [release](https://github.com/fzyzcjy/flutter_convenient_test/releases). + +Run using: -Surely, it can also be run via `flutter profile` mode to speed up. Or use `flutter build` and the standard approaches to generate a release version binary and use it. +```sh +dart pub global run convenient_test_manager_dart +``` ## Miscellaneous @@ -271,7 +348,13 @@ There are a few ways to configure the manager: 3. From command line arguments (when run via `convenient_test_manager_dart` command line) 4. Change configurations via GUI (e.g. switches at right-top) -For all options that are configurable and all environment variable names, please see [`consts.dart`](packages/convenient_test_common_dart/lib/src/consts.dart). (By doing so you can always see the up-to-date information and there is no possiblility to see outdated doc.) +For all options that are configurable and all environment variable names, please see + +- [`consts.dart`](packages/convenient_test_common_dart/lib/src/consts.dart) +- [`static_config.dart`](packages/convenient_test_dev/lib/src/support/static_config.dart) +- [`global_config_store.dart`](packages/convenient_test_manager_dart/lib/stores/global_config_store.dart) only manager related options + +(By doing so you can always see the up-to-date information and there is no possibility to see outdated doc.) If you are trying to use this package with Android Virtual Emulator (AVD), you may need to set `kConvenientTestManagerHost` to `10.0.2.2` (add `--dart-define CONVENIENT_TEST_MANAGER_HOST=10.0.2.2`) because of its [network topology](https://stackoverflow.com/questions/9808560/why-do-we-use-10-0-2-2-to-connect-to-local-web-server-instead-of-using-computer). Please see [issue #253](https://github.com/fzyzcjy/flutter_convenient_test/issues/253) for more details. diff --git a/doc/patches/flutter/get-running-async-tasks.diff b/doc/patches/flutter/get-running-async-tasks.diff new file mode 100644 index 00000000..b4c8648a --- /dev/null +++ b/doc/patches/flutter/get-running-async-tasks.diff @@ -0,0 +1,38 @@ +Enable re-entrance for `WidgetTester.runAsyncEnhanced` +(`ExtWidgetTesterPump` in `flutter_convenient_test/packages/convenient_test_dev/lib/src/functions/widget_tester.dart`) + +see also https://github.com/fzyzcjy/flutter_convenient_test/issues/337 + +this diff is based on flutter v3.22.2 + +diff --git a/packages/flutter_test/lib/src/binding.dart b/packages/flutter_test/lib/src/binding.dart +index 95a80d2..be8ba34 100644 +--- a/packages/flutter_test/lib/src/binding.dart ++++ b/packages/flutter_test/lib/src/binding.dart +@@ -414,6 +414,8 @@ abstract class TestWidgetsFlutterBinding extends BindingBase + /// [LiveTestWidgetsFlutterBinding] (tests running using `flutter run`). + test_package.Timeout get defaultTestTimeout; + ++ bool get runningAsyncTasks; // NOTE MODIFIED add ++ + /// The current time. + /// + /// In the automated test environment (`flutter test`), this is a fake clock +@@ -1230,6 +1232,8 @@ class AutomatedTestWidgetsFlutterBinding extends TestWidgetsFlutterBinding { + FakeAsync? _currentFakeAsync; // set in runTest; cleared in postTest + Completer? _pendingAsyncTasks; + ++ bool get runningAsyncTasks => _pendingAsyncTasks != null; // NOTE MODIFIED add ++ + @override + Clock get clock { + assert(inTest); +@@ -1722,6 +1726,8 @@ class LiveTestWidgetsFlutterBinding extends TestWidgetsFlutterBinding { + bool _viewNeedsPaint = false; + bool _runningAsyncTasks = false; + ++ bool get runningAsyncTasks => _runningAsyncTasks; // NOTE MODIFIED add ++ + /// The strategy for [pump]ing and requesting new frames. + /// + /// The policy decides whether [pump] (with a duration) pumps a single frame From adfbdb2e2eef9f9bfe9cc190b2660b10a810925e Mon Sep 17 00:00:00 2001 From: Max Jakobitsch Date: Sat, 6 Jul 2024 01:51:17 +0200 Subject: [PATCH 2/4] Update README.md Co-authored-by: fzyzcjy <5236035+fzyzcjy@users.noreply.github.com> --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c7690307..58b40ec8 100644 --- a/README.md +++ b/README.md @@ -268,7 +268,7 @@ Following are about the toggles: ## Getting started -1. Navigate to the flutter home directory and apply [this patch](doc/patches/flutter/get-running-async-tasks.diff): +1. (Optional but recommended) Navigate to the flutter home directory and apply [this patch](doc/patches/flutter/get-running-async-tasks.diff): `cd $FLUTTER_HOME && curl https://raw.githubusercontent.com/fzyzcjy/flutter_convenient_test/master/doc/patches/flutter/get-running-async-tasks.diff | git apply` - while this is not strictly necessary, it is recommended to improve reliability when running multiple async tasks. see [#337](https://github.com/fzyzcjy/flutter_convenient_test/issues/337) for more information 2. Navigate to your project dir and run `flutter pub add convenient_test dev:convenient_test_dev` From 29d970aeced6a5d4fabc8677946add9a053b09fb Mon Sep 17 00:00:00 2001 From: Max Jakobitsch Date: Sat, 6 Jul 2024 01:51:39 +0200 Subject: [PATCH 3/4] Update README.md Co-authored-by: fzyzcjy <5236035+fzyzcjy@users.noreply.github.com> --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 58b40ec8..1c30379d 100644 --- a/README.md +++ b/README.md @@ -270,7 +270,7 @@ Following are about the toggles: 1. (Optional but recommended) Navigate to the flutter home directory and apply [this patch](doc/patches/flutter/get-running-async-tasks.diff): `cd $FLUTTER_HOME && curl https://raw.githubusercontent.com/fzyzcjy/flutter_convenient_test/master/doc/patches/flutter/get-running-async-tasks.diff | git apply` - - while this is not strictly necessary, it is recommended to improve reliability when running multiple async tasks. see [#337](https://github.com/fzyzcjy/flutter_convenient_test/issues/337) for more information + - While this is not strictly necessary, it is recommended to improve scenarios when running multiple async tasks. see [#337](https://github.com/fzyzcjy/flutter_convenient_test/issues/337) for more information 2. Navigate to your project dir and run `flutter pub add convenient_test dev:convenient_test_dev` - or manually update your `pubspec.yaml` to include `convenient_test: ^1.0.0` in the `dependencies` section, and `convenient_test_dev: ^1.0.0` in the `dev_dependencies` section. As normal, we need to `flutter pub get`. 3. Create `integration_test/main_test.dart` file in your app. Fill it like `void main() => convenientTestMain(MyConvenientTestSlot(), () { ... the normal test code you write });`. See the example [`main_test.dart`](packages/convenient_test/example/integration_test/main_test.dart) for demonstration. From e3872c82b0b5f1368d46a5dbd0f2f4ea5b9a0935 Mon Sep 17 00:00:00 2001 From: Max Jakobitsch Date: Sat, 6 Jul 2024 21:43:57 +0200 Subject: [PATCH 4/4] docs: add comment that managers are just normal flutter/dart projects --- README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 1c30379d..d26c60ba 100644 --- a/README.md +++ b/README.md @@ -295,10 +295,13 @@ For example, go to the most recent _CI_ run, download `manager_gui_macos` artifa #### Method 2: `flutter run` / `flutter build` +The GUI and CLI managers are just flutter and dart projects as well, respectively. +So they can be run and built in the same way one would with any other flutter or dart project: + Check the requirements for developing flutter desktop apps for [macOS](https://docs.flutter.dev/platform-integration/macos/install-macos/install-macos-from-ios), [Linux](https://docs.flutter.dev/platform-integration/linux/install-linux/install-linux-from-android), -and [Windows](https://docs.flutter.dev/platform-integration/windows/install-windows/install-windows-from-android). +and [Windows](https://docs.flutter.dev/platform-integration/windows/install-windows/install-windows-from-android). Then: ```sh git clone https://github.com/fzyzcjy/flutter_convenient_test.git @@ -310,7 +313,7 @@ flutter build flutter run -d ``` -##### For CLI manager: +For CLI manager: ```sh cd flutter_convenient_test/packages/convenient_test_manager