Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update docs #11

Merged
merged 2 commits into from
Dec 2, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions Docs/XCUITests.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@

# XCUITests

After the tests finish execution, the essential thing that we need is the simulator needs to stay alive.
The problem with `XCUITests` is that after testing finish execution, it quit the simulator.

# Things I've tried:

1. Trying to find a way to preserve the running program after tests finish running

- I can't find any way to preserve the running program.

2. At the end of a test, run shell script to generate `memgraph` before the running program quit

For this idea, I put a breakpoint before the test finish execution. Then we can custom that breakpoint to execute a shell script command.
However, when running on CI, we will execute test using script, not from Xcode. So, using breakpoint to execute shell script will not work for CI

=> Only work on Xcode

3. From Xcode13, Apple provide `-enablePerformanceTestsDiagnostics` to generate memgraph after a test finish executing.

```bash
xcodebuild test -project MemoryLeaksCheck.xcodeproj \
-scheme LeaksCheckerUITests \
-destination platform=iOS,name="Tuan iPhone" \
-enablePerformanceTestsDiagnostics YES
```

> Note: In the scheme configuration, open `Options` under `Test`, unselect "Delete if test succeeds" for Attachments.

<img src=../resources/xcuitests.png width=800/>

However, **this only works for real device, not for simulator.**

Based on [Apple docs](https://developer.apple.com/documentation/xcode-release-notes/xcode-13-release-notes)

> xcodebuild has a new option -enablePerformanceTestsDiagnostics YES that collects diagnostics for Performance XCTests. The option collects a ktrace file for non-XCTMemoryMetrics, and a series of memory graphs for XCTMemoryMetrics. xcodebuild attaches diagnostics to the generated xcresult bundle. **Note that memory graph collection isn’t available in simulated devices. (64495534)**

=> Only work on local with real device, doesn't work on CI

# Conclusion

XCUITests is not appropriate for this approach. (for now)
30 changes: 29 additions & 1 deletion LeaksCheckerUITests/LeaksCheckerUITests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,9 @@ final class LeaksCheckerUITests: XCTestCase {
super.tearDown()
}

func testExample() throws {
/// This is a solution to generate memgraph using breakpoint.
/// => this solution only work when you running test using Xcode.
func testExample_usingBreakpointToGenrateMemgraph() throws {

// UI tests must launch the application that they test.
app = XCUIApplication(bundleIdentifier: "Hoang-Anh-Tuan.MemoryLeaksCheck")
Expand All @@ -41,5 +43,31 @@ final class LeaksCheckerUITests: XCTestCase {
/// - Parameters: The parameters is passed in the shell script is the program name, which is our app name. For this project, the app name is *MemoryLeaksCheck*.
debugPrint("Start checking for leaks... 🔎")
}

/// This test will generate memgraph via command line. However, it only works on physical device, not simulator.
/// For more info, please read README.
func testExample() throws {
let app = XCUIApplication()
let options = XCTMeasureOptions()

measure(
metrics: [XCTMemoryMetric(application: app)],
options: options
) {
app.launch()
startMeasuring()

app.staticTexts["Abandoned Memory Example"].tap()
app.buttons["Scenarios"].tap()

app.staticTexts["Leaks Memory Example"].tap()

let simulateLogoutThenLoginActionButton = app.buttons["Simulate Logout then Login Action"]
simulateLogoutThenLoginActionButton.tap()
simulateLogoutThenLoginActionButton.tap()
simulateLogoutThenLoginActionButton.tap()
simulateLogoutThenLoginActionButton.tap()
}
}

}
11 changes: 7 additions & 4 deletions MemoryLeaksCheck.xcodeproj/project.pbxproj
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,7 @@
};
4CBC59372AC3299B00D83CA5 = {
CreatedOnToolsVersion = 14.2;
TestTargetID = 4C28E3C92ABEDFFB0004394E;
};
};
};
Expand Down Expand Up @@ -382,7 +383,7 @@
ASSETCATALOG_COMPILER_GLOBAL_ACCENT_COLOR_NAME = AccentColor;
CODE_SIGN_STYLE = Automatic;
CURRENT_PROJECT_VERSION = 1;
DEVELOPMENT_TEAM = RCQA85T6C2;
DEVELOPMENT_TEAM = 98D3824HFQ;
GENERATE_INFOPLIST_FILE = YES;
INFOPLIST_FILE = MemoryLeaksCheck/Info.plist;
INFOPLIST_KEY_UIApplicationSupportsIndirectInputEvents = YES;
Expand Down Expand Up @@ -414,7 +415,7 @@
ASSETCATALOG_COMPILER_GLOBAL_ACCENT_COLOR_NAME = AccentColor;
CODE_SIGN_STYLE = Automatic;
CURRENT_PROJECT_VERSION = 1;
DEVELOPMENT_TEAM = RCQA85T6C2;
DEVELOPMENT_TEAM = 98D3824HFQ;
GENERATE_INFOPLIST_FILE = YES;
INFOPLIST_FILE = MemoryLeaksCheck/Info.plist;
INFOPLIST_KEY_UIApplicationSupportsIndirectInputEvents = YES;
Expand Down Expand Up @@ -444,7 +445,7 @@
buildSettings = {
CODE_SIGN_STYLE = Automatic;
CURRENT_PROJECT_VERSION = 1;
DEVELOPMENT_TEAM = RCQA85T6C2;
DEVELOPMENT_TEAM = 98D3824HFQ;
GENERATE_INFOPLIST_FILE = YES;
IPHONEOS_DEPLOYMENT_TARGET = 14.0;
MARKETING_VERSION = 1.0;
Expand All @@ -457,6 +458,7 @@
SWIFT_EMIT_LOC_STRINGS = NO;
SWIFT_VERSION = 5.0;
TARGETED_DEVICE_FAMILY = 1;
TEST_TARGET_NAME = MemoryLeaksCheck;
};
name = Debug;
};
Expand All @@ -465,7 +467,7 @@
buildSettings = {
CODE_SIGN_STYLE = Automatic;
CURRENT_PROJECT_VERSION = 1;
DEVELOPMENT_TEAM = RCQA85T6C2;
DEVELOPMENT_TEAM = 98D3824HFQ;
GENERATE_INFOPLIST_FILE = YES;
IPHONEOS_DEPLOYMENT_TARGET = 14.0;
MARKETING_VERSION = 1.0;
Expand All @@ -478,6 +480,7 @@
SWIFT_EMIT_LOC_STRINGS = NO;
SWIFT_VERSION = 5.0;
TARGETED_DEVICE_FAMILY = 1;
TEST_TARGET_NAME = MemoryLeaksCheck;
};
name = Release;
};
Expand Down
Binary file not shown.
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,16 @@
BreakpointExtensionID = "Xcode.Breakpoint.FileBreakpoint">
<BreakpointContent
uuid = "36593BB7-BE9C-457C-8CE7-A83C591BB7A4"
shouldBeEnabled = "Yes"
shouldBeEnabled = "No"
nameForDebugger = "CheckingForLeaks"
ignoreCount = "0"
continueAfterRunningActions = "No"
filePath = "LeaksCheckerUITests/LeaksCheckerUITests.swift"
startingColumnNumber = "9223372036854775807"
endingColumnNumber = "9223372036854775807"
startingLineNumber = "42"
endingLineNumber = "42"
landmarkName = "testExample()"
startingLineNumber = "44"
endingLineNumber = "44"
landmarkName = "testExample_usingBreakpointToGenrateMemgraph()"
landmarkType = "7">
<Actions>
<BreakpointActionProxy
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@
buildConfiguration = "Debug"
selectedDebuggerIdentifier = "Xcode.DebuggerFoundation.Debugger.LLDB"
selectedLauncherIdentifier = "Xcode.DebuggerFoundation.Launcher.LLDB"
shouldUseLaunchSchemeArgsEnv = "YES">
shouldUseLaunchSchemeArgsEnv = "YES"
userAttachmentLifetime = "keepAlways">
<AdditionalOptions>
<AdditionalOption
key = "MallocStackLogging"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>SchemeUserState</key>
<dict>
<key>LeaksCheckerUITests.xcscheme_^#shared#^_</key>
<dict>
<key>orderHint</key>
<integer>0</integer>
</dict>
<key>MemoryLeaksCheck.xcscheme_^#shared#^_</key>
<dict>
<key>orderHint</key>
<integer>1</integer>
</dict>
<key>SnapKitPlayground (Playground) 1.xcscheme</key>
<dict>
<key>isShown</key>
<false/>
<key>orderHint</key>
<integer>3</integer>
</dict>
<key>SnapKitPlayground (Playground) 2.xcscheme</key>
<dict>
<key>isShown</key>
<false/>
<key>orderHint</key>
<integer>4</integer>
</dict>
<key>SnapKitPlayground (Playground).xcscheme</key>
<dict>
<key>isShown</key>
<false/>
<key>orderHint</key>
<integer>2</integer>
</dict>
</dict>
</dict>
</plist>
10 changes: 3 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,10 @@ Learn more about `Maestro` [here](https://maestro.mobile.dev/)
leaksdetector -processName $YOUR_APP_NAME -e $SUPPORTED_TESTING_FRAMEWORKS -d $PATH_TO_DANGER_FILE
```

## Current support testing frameworks
## Current testing frameworks

- [Maestro](https://maestro.mobile.dev/) ✅
- [XCUITest](https://developer.apple.com/documentation/xctest) (In progress ⚙️)
- [XCUITest](https://developer.apple.com/documentation/xctest) (XCUITest is not supported. Read more [here](./Docs/XCUITests.md)) ❌

## How it works

Expand All @@ -40,11 +40,7 @@ Find more about `leaks` tool and `memgraph` [here](https://developer.apple.com/v
## Why I used Maestro?

1. I need a testing tool which doesn't kill the program after the testing finished execution. And Maestro support that. Also Maestro is very easy to integrate & use.
2. I've tried to used XCUItest, which is really promissing. Based on this [WWDC video](https://developer.apple.com/videos/play/wwdc2021/10180/) from Apple, XCUITest even *allows us to capture the stacktrace where leaks occur & generate a memgraph*. However, I've tried to follow the video but Xcode didn't generate any memgraph.

=> I'm working on this.

I've posted some questions on [swiftforum](https://forums.swift.org/t/xctest-doesnt-generate-memgraph-file-after-ui-test-finish-execution/67982) and [developer.apple](https://developer.apple.com/forums/thread/738659). If you have any ideas, feel free to reply on the threads
2. XCUITest can not preserve running program after test execution. Read more at [here](./Docs/XCUITests.md)

## How to support your testing frameworks

Expand Down
Binary file added resources/xcuitests.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading