-
Notifications
You must be signed in to change notification settings - Fork 26
Add utility to collect headers from google/mediapipe #10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
b625bf4
adds cmd to pull header files from google/mediapipe
craiglabenz 29fd288
polish and missing parts from git surgery
craiglabenz ff3cedb
More comments and touch ups
craiglabenz a6f609e
Apply suggestions from code review
Piinks 4c81a41
moves build command into `tool/` directory
craiglabenz ac1c59d
complete build_cmd -> builder rename
craiglabenz d827567
Update readme
craiglabenz 271c244
Added licenses
craiglabenz b794c46
Adds DownloadModelCommand
craiglabenz File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| // Used to normalize the paths of commands. | ||
| // The contents of this file do not matter. |
Binary file not shown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| # https://dart.dev/guides/libraries/private-files | ||
| # Created by `dart pub` | ||
| .dart_tool/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| ## 1.0.0 | ||
|
|
||
| - Initial version with headers command to sync C headers from `google/mediapipe`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| # Flutter MediaPipe builder | ||
|
|
||
| Helper utility which performs build or CI/CD step operations necessary to develop, release, and use `flutter-mediapipe`. | ||
|
|
||
| ### Usage: | ||
|
|
||
| Usage depends on which task you need to accomplish. All supported workflows are described below. | ||
|
|
||
| #### Header aggregation | ||
|
|
||
| Header files across all tasks in `google/flutter-mediapipe` have to stay in sync with their origin, `google/mediapipe`. To | ||
| resync these files, check out both repositories on the same machine (ideally next to each other on the file system) and run: | ||
|
|
||
| ```sh | ||
| $ dart tool/builder/bin/main.dart headers | ||
| ``` | ||
|
|
||
| -- | ||
|
|
||
| Check back in the future for any additional development workflows this command may support. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| include: package:lints/recommended.yaml | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| // Copyright 2014 The Flutter Authors. All rights reserved. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As this is a new file the year should be 2023. Unless it's copied from somewhere?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is the standard Flutter authors license. |
||
| // Use of this source code is governed by a BSD-style license that can be | ||
| // found in the LICENSE file. | ||
|
|
||
| import 'dart:io' as io; | ||
| import 'package:args/command_runner.dart'; | ||
| import 'package:builder/download_model.dart'; | ||
| import 'package:builder/sync_headers.dart'; | ||
| import 'package:logging/logging.dart'; | ||
|
|
||
| final runner = CommandRunner( | ||
| 'build', | ||
| 'Performs build operations for google/flutter-mediapipe that ' | ||
| 'depend on contents in this repository', | ||
| ) | ||
| ..addCommand(SyncHeadersCommand()) | ||
| ..addCommand(DownloadModelCommand()); | ||
|
|
||
| void main(List<String> arguments) { | ||
| Logger.root.onRecord.listen((LogRecord record) { | ||
| io.stdout | ||
| .writeln('${record.level.name}: ${record.time}: ${record.message}'); | ||
| }); | ||
| runner.run(arguments); | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,144 @@ | ||
| // Copyright 2014 The Flutter Authors. All rights reserved. | ||
| // Use of this source code is governed by a BSD-style license that can be | ||
| // found in the LICENSE file. | ||
|
|
||
| import 'dart:io' as io; | ||
| import 'package:args/command_runner.dart'; | ||
| import 'package:builder/repo_finder.dart'; | ||
| import 'package:http/http.dart' as http; | ||
| import 'package:logging/logging.dart'; | ||
| import 'package:path/path.dart' as path; | ||
|
|
||
| final _log = Logger('DownloadModelCommand'); | ||
|
|
||
| enum Models { | ||
| textclassification, | ||
| languagedetection, | ||
| } | ||
|
|
||
| class DownloadModelCommand extends Command with RepoFinderMixin { | ||
| @override | ||
| String description = 'Downloads a given MediaPipe model and places it in ' | ||
| 'the designated location.'; | ||
| @override | ||
| String name = 'model'; | ||
|
|
||
| DownloadModelCommand() { | ||
| argParser | ||
| ..addOption( | ||
| 'model', | ||
| abbr: 'm', | ||
| allowed: [ | ||
| // Values will be added to this as the repository gets more | ||
| // integration tests that require new models. | ||
| Models.textclassification.name, | ||
| Models.languagedetection.name, | ||
| ], | ||
| help: 'The desired model to download. Use this option if you want the ' | ||
| 'standard model for a given task. Using this option also removes any ' | ||
| 'need to use the `destination` option, as a value here implies a ' | ||
| 'destination. However, you still can specify a destination to ' | ||
| 'override the default location where the model is placed.\n' | ||
| '\n' | ||
| 'Note: Either this or `custommodel` must be used. If both are ' | ||
| 'supplied, `model` is used.', | ||
| ) | ||
| ..addOption( | ||
| 'custommodel', | ||
| abbr: 'c', | ||
| help: 'The desired model to download. Use this option if you want to ' | ||
| 'specify a specific and nonstandard model. Using this option means ' | ||
| 'you *must* use the `destination` option.\n' | ||
| '\n' | ||
| 'Note: Either this or `model` must be used. If both are supplied, ' | ||
| '`model` is used.', | ||
| ) | ||
| ..addOption( | ||
| 'destination', | ||
| abbr: 'd', | ||
| help: | ||
| 'The location to place the downloaded model. This value is required ' | ||
| 'if you use the `custommodel` option, but optional if you use the ' | ||
| '`model` option.', | ||
| ); | ||
| } | ||
|
|
||
| static final Map<String, String> _standardModelSources = { | ||
| Models.textclassification.name: | ||
| 'https://storage.googleapis.com/mediapipe-models/text_classifier/bert_classifier/float32/1/bert_classifier.tflite', | ||
| Models.languagedetection.name: | ||
| 'https://storage.googleapis.com/mediapipe-models/language_detector/language_detector/float32/1/language_detector.tflite', | ||
| }; | ||
|
|
||
| static final Map<String, String> _standardModelDestinations = { | ||
| Models.textclassification.name: | ||
| 'packages/mediapipe-task-text/example/assets/', | ||
| Models.languagedetection.name: | ||
| 'packages/mediapipe-task-text/example/assets/', | ||
| }; | ||
|
|
||
| @override | ||
| Future<void> run() async { | ||
| final io.Directory flutterMediaPipeDirectory = findFlutterMediaPipeRoot(); | ||
|
|
||
| late final String modelSource; | ||
| late final String modelDestination; | ||
|
|
||
| if (argResults!['model'] != null) { | ||
| modelSource = _standardModelSources[argResults!['model']]!; | ||
| modelDestination = (_isArgProvided(argResults!['destination'])) | ||
| ? argResults!['destination'] | ||
| : _standardModelDestinations[argResults!['model']]!; | ||
| } else { | ||
| if (argResults!['custommodel'] == null) { | ||
| throw Exception( | ||
| 'You must use either the `model` or `custommodel` option.', | ||
| ); | ||
| } | ||
| if (argResults!['destination'] == null) { | ||
| throw Exception( | ||
| 'If you do not use the `model` option, then you must supply a ' | ||
| '`destination`, as a "standard" destination cannot be used.', | ||
| ); | ||
| } | ||
| modelSource = argResults!['custommodel']; | ||
| modelDestination = argResults!['destination']; | ||
| } | ||
|
|
||
| io.File destinationFile = io.File( | ||
| path.joinAll([ | ||
| flutterMediaPipeDirectory.absolute.path, | ||
| modelDestination, | ||
| modelSource.split('/').last, | ||
| ]), | ||
| ); | ||
| ensureFolders(destinationFile); | ||
| await downloadModel(modelSource, destinationFile); | ||
| } | ||
|
|
||
| Future<void> downloadModel( | ||
| String modelSource, | ||
| io.File destinationFile, | ||
| ) async { | ||
| _log.info('Downloading $modelSource'); | ||
|
|
||
| // TODO(craiglabenz): Convert to StreamedResponse | ||
| final response = await http.get(Uri.parse(modelSource)); | ||
|
|
||
| if (response.statusCode != 200) { | ||
| throw Exception('${response.statusCode} ${response.reasonPhrase} :: ' | ||
| '$modelSource'); | ||
| } | ||
|
|
||
| if (!(await destinationFile.exists())) { | ||
| _log.fine('Creating file at ${destinationFile.absolute.path}'); | ||
| await destinationFile.create(); | ||
| } | ||
|
|
||
| _log.fine('Downloaded ${response.contentLength} bytes'); | ||
| _log.info('Saving to ${destinationFile.absolute.path}'); | ||
| await destinationFile.writeAsBytes(response.bodyBytes); | ||
| } | ||
| } | ||
|
|
||
| bool _isArgProvided(String? val) => val != null && val != ''; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,120 @@ | ||
| // Copyright 2014 The Flutter Authors. All rights reserved. | ||
| // Use of this source code is governed by a BSD-style license that can be | ||
| // found in the LICENSE file. | ||
|
|
||
| import 'dart:io' as io; | ||
| import 'package:args/args.dart'; | ||
| import 'package:args/command_runner.dart'; | ||
| import 'package:path/path.dart' as path; | ||
| import 'package:io/ansi.dart'; | ||
|
|
||
| /// Mixin to help [Command] subclasses locate both `google/mediapipe` and | ||
| /// the root of `google/flutter-mediapipe` (this repository). | ||
| /// | ||
| /// The primary methods are [findFlutterMediaPipeRoot] and [findMediaPipeRoot]. | ||
| /// | ||
| /// By default, the root for `google/flutter-mediapipe` is determined by the | ||
| /// firest ancestor directory which contains a `.flutter-mediapipe-root` file | ||
| /// (whose contents are irrelevant), and the root of `google/mediapipe` is | ||
| /// expected to be a sibling of that. However, the `--source` flag can overwrite | ||
| /// this expectation and specify an absolute path where to find `google/mediapipe`. | ||
| /// | ||
| /// Note that it is not possible to override the method of locating the root of | ||
| /// `google/flutter-mediapipe`. | ||
| mixin RepoFinderMixin on Command { | ||
| /// Name of the file which, when found, indicates the root of this repository. | ||
| static String sentinelFileName = '.flutter-mediapipe-root'; | ||
|
|
||
| void addSourceOption(ArgParser argParser) { | ||
| argParser.addOption( | ||
| 'source', | ||
| abbr: 's', | ||
| help: 'The location of google/mediapipe. Defaults to being ' | ||
| 'adjacent to google/flutter-mediapipe.', | ||
| ); | ||
| } | ||
|
|
||
| /// Looks upward for the root of the `google/mediapipe` repository. This assumes | ||
| /// the `dart build` command is executed from within said repository. If it is | ||
| /// not executed from within, then this searching algorithm will reach the root | ||
| /// of the file system, log the error, and exit. | ||
| io.Directory findFlutterMediaPipeRoot() { | ||
| final placesChecked = <io.Directory>[]; | ||
| io.Directory dir = io.Directory(path.current); | ||
| while (true) { | ||
| if (_isFlutterMediaPipeRoot(dir)) { | ||
| return dir; | ||
| } | ||
| placesChecked.add(dir); | ||
| dir = dir.parent; | ||
| if (dir.parent.path == dir.path) { | ||
| io.stderr.writeln( | ||
| wrapWith( | ||
| 'Failed to find google/flutter-mediapipe root directory. ' | ||
| 'Did you execute this command from within the repository?\n' | ||
| 'Looked in:', | ||
| [red], | ||
| ), | ||
| ); | ||
| io.stderr.writeln( | ||
| wrapWith( | ||
| placesChecked | ||
| .map<String>((dir) => ' - ${dir.absolute.path}') | ||
| .toList() | ||
| .join('\n'), | ||
| [red], | ||
| ), | ||
| ); | ||
| io.exit(1); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| /// Finds the `google/mediapipe` checkout where artifacts built in this | ||
| /// repository should be sourced. By default, this command assumes the two | ||
| /// repositories are siblings on the file system, but the `--source` flag | ||
| /// allows for this assumption to be overridden. | ||
| io.Directory findMediaPipeRoot( | ||
| io.Directory flutterMediaPipeDir, | ||
| String? source, | ||
| ) { | ||
| final mediaPipeDirectory = io.Directory( | ||
| source ?? | ||
| path.joinAll([flutterMediaPipeDir.parent.absolute.path, 'mediapipe']), | ||
| ); | ||
|
|
||
| if (!mediaPipeDirectory.existsSync()) { | ||
| io.stderr.writeln( | ||
| 'Could not find ${mediaPipeDirectory.absolute.path}. ' | ||
| 'Folder does not exist.', | ||
| ); | ||
| io.exit(1); | ||
| } | ||
| return mediaPipeDirectory; | ||
| } | ||
|
|
||
| /// Looks for the sentinel file of this repository's root directory. This allows | ||
| /// the `dart build` command to be run from various locations within the | ||
| /// `google/mediapipe` repository and still correctly set paths for all of its | ||
| /// operations. | ||
| bool _isFlutterMediaPipeRoot(io.Directory dir) { | ||
| return io.File( | ||
| path.joinAll( | ||
| [dir.absolute.path, sentinelFileName], | ||
| ), | ||
| ).existsSync(); | ||
| } | ||
|
|
||
| /// Builds any missing folders between the file and the root of the repository | ||
| void ensureFolders(io.File file) { | ||
| io.Directory parent = file.parent; | ||
| List<io.Directory> dirsToCreate = []; | ||
| while (!parent.existsSync()) { | ||
| dirsToCreate.add(parent); | ||
| parent = parent.parent; | ||
| } | ||
| for (io.Directory dir in dirsToCreate.reversed) { | ||
| dir.createSync(); | ||
| } | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.