Skip to content

Commit

Permalink
feat: Add support for native screen recording (#330)
Browse files Browse the repository at this point in the history
  • Loading branch information
mykola-mokhnach authored Feb 7, 2025
1 parent e00a5a6 commit d2f5b25
Show file tree
Hide file tree
Showing 28 changed files with 1,363 additions and 62 deletions.
22 changes: 0 additions & 22 deletions .eslintrc.json

This file was deleted.

2 changes: 2 additions & 0 deletions .npmrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
save-exact=true
package-lock=false
83 changes: 81 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -586,7 +586,7 @@ Retrieves a screenshot of each display available to macOS.

Name | Type | Required | Description | Example
--- | --- | --- | --- | ---
displayId | number | no | Display identifier to take a screenshot for. If not provided then all display screenshots are going to be returned. If no matches were found then an error is thrown. | 1
displayId | number | no | Display identifier to take a screenshot for. If not provided then all display screenshots are going to be returned. If no matches were found then an error is thrown. Use the `system_profiler -json SPDisplaysDataType` Terminal command to list IDs of connected displays or the [macos: listDisplays](#macos-listdisplays) API. | 1

#### Returns

Expand All @@ -595,7 +595,7 @@ A list of dictionaries where each item has the following keys:
- `isMain`: Whether this display is the main one
- `payload`: The actual PNG screenshot data encoded to base64 string

### mobile: deepLink
### macos: deepLink

Opens the given URL with the default or the given application.
Xcode must be at version 14.3+.
Expand All @@ -607,6 +607,85 @@ Name | Type | Required | Description | Example
url | string | yes | The URL to be opened. This parameter is manadatory. | https://apple.com, myscheme:yolo
bundleId | string | no | The bundle identifier of an application to open the given url with. If not provided then the default application for the given url scheme is going to be used. | com.myapp.yolo

### macos: startNativeScreenRecording

Initiates a new native screen recording session via XCTest.
If the screen recording is already running then this call results in noop.
A screen recording is running until a testing session is finished.
If a recording has never been stopped explicitly during a test session
then it would be stopped automatically upon the test session termination,
and leftover videos would be deleted as well.
Xcode must be at version 15+.

#### Arguments

Name | Type | Required | Description | Example
--- | --- | --- | --- | ---
fps | number | no | Frame Per Second setting for the resulting screen recording. 24 by default. Higher FPS values may significantly increase the size of the resulting video. | 60
codec | number | no | Possible codec value, where `0` means H264 (the default setting), `1` means HEVC | 1
displayId | number | no | Valid display identifier to record the video from. Main display ID is assumed by default. Use the `system_profiler -json SPDisplaysDataType` Terminal command to list IDs of connected displays or the [macos: listDisplays](#macos-listdisplays) API. | 1

#### Returns

The information about the asynchronously running video recording, which includes the following items:

Name | Type | Description | Example
--- | --- | --- | ---
fps | number | Frame Per Second value | 24
codec | number | Codec value, where `0` means H264 (the default setting), `1` means HEVC | 1
displayId | number | Display identifier used to record this video for. | 1
uuid | string | Unique video identifier. It is also used by XCTest to store the video on the file system. Look for `$HOME/Library/Daemon Containers/<testmanager_id>/Data/Attachments/<uuid>` to find the appropriate video file. Add the `.mp4` extension to it to make it openable by video players.
startedAt | number | Unix timestamp of the video startup moment | 123456789

### macos: getNativeScreenRecordingInfo

Fetches the information of the currently running native video recording.
Xcode must be at version 15+.

#### Returns

Either `null` if no native video recording is currently active or the same map that [macos: startNativeScreenRecording](#macos-startnativescreenrecording) returns.

### macos: stopNativeScreenRecording

Stops native screen recording previously started by
[macos: startNativeScreenRecording](#macos-startnativescreenrecording)
and returns the video payload or uploads it to a remote location,
depending on the provided arguments.
The actual video file is removed from the local file system after the video payload is
successfully consumed.
If no screen recording has been started before then this API throws an exception.
Xcode must be at version 15+.

#### Arguments

Name | Type | Required | Description | Example
--- | --- | --- | --- | ---
remotePath | string | no | The path to the remote location, where the resulting video should be uploaded. The following protocols are supported: http/https, ftp. Null or empty string value (the default setting) means the content of resulting file should be encoded as Base64 and passed as the endpoint response value. An exception will be thrown if the generated media file is too big to fit into the available process memory. | https://myserver.com/upload/video.mp4
user | string | no | The name of the user for the remote authentication. | myname
pass | string | no | The password for the remote authentication. | mypassword
method | string | no | The http multipart upload method name. The 'PUT' one is used by default. | POST
headers | map | no | Additional headers mapping for multipart http(s) uploads | `{"header": "value"}`
fileFieldName | string | no | The name of the form field, where the file content BLOB should be stored for http(s) uploads. `file` by default | payload
formFields | Map or `Array<Pair>` | no | Additional form fields for multipart http(s) uploads | `{"field1": "value1", "field2": "value2"}` or `[["field1", "value1"], ["field2", "value2"]]`

#### Returns

Base64-encoded content of the recorded media file if `remotePath` parameter is falsy or an empty string.

### macos: listDisplays

Fetches information about available displays.

#### Returns

A map where keys are display identifiers represented as strings and values are display infos containing the following items:

Name | Type | Description | Example
--- | --- | --- | ---
id | number | Display identifier | 12345
isMain | boolean | Is `true` if the display is configured as a main system display | false


## Application Under Test Concept

Expand Down
57 changes: 57 additions & 0 deletions WebDriverAgentMac/IntegrationTests/AMVideoRecordingTests.m
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* See the NOTICE file distributed with this work for additional
* information regarding copyright ownership.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

#import <XCTest/XCTest.h>

#import "AMIntegrationTestCase.h"
#import "AMVideoRecorder.h"
#import "FBScreenRecordingRequest.h"
#import "FBTestMacros.h"
#import "FBScreenRecordingContainer.h"
#import "FBScreenRecordingPromise.h"


@interface AMVideoRecordingTests : AMIntegrationTestCase
@end

@implementation AMVideoRecordingTests

- (void)setUp
{
[super setUp];
static dispatch_once_t onceToken;
dispatch_once(&onceToken, ^{
[self launchApplication];
});
}

- (void)testVideoRecording
{
AMVideoRecorder *recorder = AMVideoRecorder.sharedInstance;
FBScreenRecordingRequest *request = [[FBScreenRecordingRequest alloc] initWithFps:24
codec:0
displayID:nil];
NSError *error;
FBScreenRecordingPromise *promise = [recorder startScreenRecordingWithRequest:request
error:&error];
XCTAssertNotNil(promise);
XCTAssertNil(error);
FBWaitExact(5);
XCTAssertTrue([recorder stopScreenRecordingWithUUID:promise.identifier error:&error]);
XCTAssertNil(error);
}

@end
27 changes: 27 additions & 0 deletions WebDriverAgentMac/WebDriverAgentLib/Commands/AMVideoCommands.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* See the NOTICE file distributed with this work for additional
* information regarding copyright ownership.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

#import <Foundation/Foundation.h>

#import <WebDriverAgentLib/FBCommandHandler.h>

NS_ASSUME_NONNULL_BEGIN

@interface AMVideoCommands : NSObject <FBCommandHandler>

@end

NS_ASSUME_NONNULL_END
120 changes: 120 additions & 0 deletions WebDriverAgentMac/WebDriverAgentLib/Commands/AMVideoCommands.m
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* See the NOTICE file distributed with this work for additional
* information regarding copyright ownership.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

#import "AMVideoCommands.h"

#import "FBRouteRequest.h"
#import "FBScreenRecordingContainer.h"
#import "FBScreenRecordingPromise.h"
#import "FBScreenRecordingRequest.h"
#import "AMScreenUtils.h"
#import "FBSession.h"
#import "AMVideoRecorder.h"
#import "FBErrorBuilder.h"

const NSUInteger DEFAULT_FPS = 24;
const NSUInteger DEFAULT_CODEC = 0;

@implementation AMVideoCommands

+ (NSArray *)routes
{
return
@[
[[FBRoute POST:@"/wda/video/start"] respondWithTarget:self action:@selector(handleStartVideoRecording:)],
[[FBRoute POST:@"/wda/video/stop"] respondWithTarget:self action:@selector(handleStopVideoRecording:)],
[[FBRoute GET:@"/wda/video"] respondWithTarget:self action:@selector(handleGetVideoRecording:)],

[[FBRoute POST:@"/wda/video/start"].withoutSession respondWithTarget:self action:@selector(handleStartVideoRecording:)],
[[FBRoute POST:@"/wda/video/stop"].withoutSession respondWithTarget:self action:@selector(handleStopVideoRecording:)],
[[FBRoute GET:@"/wda/video"].withoutSession respondWithTarget:self action:@selector(handleGetVideoRecording:)],
];
}

+ (id<FBResponsePayload>)handleStartVideoRecording:(FBRouteRequest *)request
{
FBScreenRecordingPromise *activeScreenRecording = FBScreenRecordingContainer.sharedInstance.screenRecordingPromise;
if (nil != activeScreenRecording) {
return FBResponseWithObject([FBScreenRecordingContainer.sharedInstance toDictionary] ?: [NSNull null]);
}

NSNumber *fps = (NSNumber *)request.arguments[@"fps"] ?: @(DEFAULT_FPS);
NSNumber *codec = (NSNumber *)request.arguments[@"codec"] ?: @(DEFAULT_CODEC);
NSNumber *displayID = (NSNumber *)request.arguments[@"displayId"];
if (nil != displayID) {
NSError *error;
if (![self verifyDisplayWithID:displayID.longLongValue error:&error]) {
return FBResponseWithStatus([FBCommandStatus invalidArgumentErrorWithMessage:error.description
traceback:nil]);
}
}
FBScreenRecordingRequest *recordingRequest = [[FBScreenRecordingRequest alloc] initWithFps:fps.integerValue
codec:codec.longLongValue
displayID:displayID];
NSError *error;
FBScreenRecordingPromise* promise = [AMVideoRecorder.sharedInstance startScreenRecordingWithRequest:recordingRequest
error:&error];
if (nil == promise) {
[FBScreenRecordingContainer.sharedInstance reset];
return FBResponseWithUnknownError(error);
}
[FBScreenRecordingContainer.sharedInstance storeScreenRecordingPromise:promise
fps:fps.integerValue
codec:codec.longLongValue
displayID:displayID];
return FBResponseWithObject([FBScreenRecordingContainer.sharedInstance toDictionary]);
}

+ (id<FBResponsePayload>)handleStopVideoRecording:(FBRouteRequest *)request
{
FBScreenRecordingPromise *activeScreenRecording = FBScreenRecordingContainer.sharedInstance.screenRecordingPromise;
if (nil == activeScreenRecording) {
return FBResponseWithOK();
}

NSUUID *recordingId = activeScreenRecording.identifier;
NSDictionary *response = [FBScreenRecordingContainer.sharedInstance toDictionary];
NSError *error;
if (![AMVideoRecorder.sharedInstance stopScreenRecordingWithUUID:recordingId error:&error]) {
[FBScreenRecordingContainer.sharedInstance reset];
return FBResponseWithUnknownError(error);
}
[FBScreenRecordingContainer.sharedInstance reset];
return FBResponseWithObject(response);
}

+ (id<FBResponsePayload>)handleGetVideoRecording:(FBRouteRequest *)request
{
return FBResponseWithObject([FBScreenRecordingContainer.sharedInstance toDictionary] ?: [NSNull null]);
}

+ (BOOL)verifyDisplayWithID:(long long)displayID error:(NSError **)error
{
NSMutableArray* availableIds = [NSMutableArray array];
for (XCUIScreen *screen in XCUIScreen.screens) {
long long currentDisplayId = AMFetchScreenId(screen);
if (displayID == currentDisplayId) {
return YES;
}
[availableIds addObject:@(currentDisplayId)];
}
return [[[FBErrorBuilder builder]
withDescriptionFormat:@"The provided display identifier %lld is not known. Only the following values are allowed: %@",
displayID, [availableIds componentsJoinedByString:@","]]
buildError:error];
}

@end
17 changes: 17 additions & 0 deletions WebDriverAgentMac/WebDriverAgentLib/Commands/FBDebugCommands.m
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@

#import "FBDebugCommands.h"

#import "AMScreenUtils.h"
#import "FBRouteRequest.h"
#import "FBSession.h"
#import "XCUIApplication+AMSource.h"
Expand All @@ -23,6 +24,9 @@ + (NSArray *)routes
@[
[[FBRoute GET:@"/source"] respondWithTarget:self action:@selector(handleGetSourceCommand:)],
[[FBRoute GET:@"/source"].withoutSession respondWithTarget:self action:@selector(handleGetSourceCommand:)],

[[FBRoute GET:@"/wda/displays/list"] respondWithTarget:self action:@selector(handleListDisplays:)],
[[FBRoute GET:@"/wda/displays/list"].withoutSession respondWithTarget:self action:@selector(handleListDisplays:)],
];
}

Expand Down Expand Up @@ -53,4 +57,17 @@ + (NSArray *)routes
return FBResponseWithObject(result);
}

+ (id<FBResponsePayload>)handleListDisplays:(FBRouteRequest *)request
{
NSArray<AMScreenProperties *> *screenInfos = AMListScreens();
NSMutableDictionary <NSString *, NSDictionary<NSString *, id> *> *result = [NSMutableDictionary new];
for (AMScreenProperties *screenInfo in screenInfos) {
result[[NSString stringWithFormat:@"%lld", screenInfo.identifier]] = @{
@"id": @(screenInfo.identifier),
@"isMain": @(screenInfo.isMain),
};
}
return FBResponseWithObject(result.copy);
}

@end
Loading

0 comments on commit d2f5b25

Please sign in to comment.