Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update doc, and code formatting #202

Merged
merged 1 commit into from
Oct 3, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,6 @@ The content is converted to CMAF format with both a DASH and HLS manifest to sup
* Marks migrated assets and provides HTML summary on analyze
* Support for statically encrypting the content while packaging.

## Open Issues
* Direct migration from an Azure Storage account without using the AMS API is not supported but will be in a future version of this tool.

# Types of Migration
The tool supports various types of migration depending on the asset format and the command line options.
* For non-streamable assets, It can simply upload the files to the new storage account.
Expand Down Expand Up @@ -245,7 +242,9 @@ Below are some tip(s) about error(s) that may occurs, (more will be added later

So in this case, if you look the the output container path, you will find a __migrate blob that is locked via lease which you'll need to break the lease of prior to another run that writes to this location. So if you know that another tool is not working on this output path, then you will need to break the lease the '__migrate' blob. This can be done manually using the azure portal or azure storage explorer. Another approach is to run the same AMSMigrate 'asset' command that writes to this output path but append '--break-output-lease' flag which will automatically break the lease of the __migrate blob for you.

# Post-migration
Following AMS retirement, you can proceed with asset migration from a storage account using the "storage" command. For additional information, please see [storageCommand.md](doc/storageCommand.md) document.
# Post AMS shutdown migration
After AMS shutdown, you can not migrate from your AMS account any longer as it doesn't exist anymore. If you have any unmigrated contents, then the tool provides an alternative (but somewhat limited in functionality, e.g. no decryption support) way for you to migrate directly from the storage container where your asset is located. However, we recommed that you finish your migration prior to AMS shutdown.

To migrate from directly storage account, you need to identify the storage container yourself and then you can follow the instruction in the [direct storage migration](doc/storageCommand.md) document.


4 changes: 2 additions & 2 deletions ams/AssetAnalyzer.cs
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ public override async Task MigrateAsync(CancellationToken cancellationToken)
{
var watch = Stopwatch.StartNew();
_logger.LogInformation("Begin analysis of items for account: {name}", _analysisOptions.AccountName);
var (isAMSAcc, account) = await IsAMSAccountAsync(_analysisOptions.AccountName, cancellationToken);
var (isAMSAcc, account) = await IsAMSAccountAsync(_analysisOptions.AccountName, cancellationToken);
var reportGenerator = new ReportGenerator(_globalOptions.HtmlReportFile, _globalOptions.JsonReportFile, _logger);
reportGenerator.WriteHeader();
var statistics = new AssetStats();
Expand Down Expand Up @@ -236,7 +236,7 @@ await MigrateInParallel(assets, filteredList, async (asset, cancellationToken) =
await progress;
_logger.LogDebug("Finished analysis of assets for account: {name}. Time taken {elapsed}", _analysisOptions.AccountName, watch.Elapsed);
}

WriteSummary(statistics, assetTypes);
WriteDetails(assetTypes);

Expand Down
34 changes: 17 additions & 17 deletions ams/BaseMigrator.cs
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ public BaseMigrator(

public abstract Task MigrateAsync(CancellationToken cancellationToken);



protected async Task MigrateInParallel<T>(
IAsyncEnumerable<T> values,
Expand All @@ -61,25 +61,25 @@ protected async Task MigrateInParallel<T>(
await Parallel.ForEachAsync(values, options, processItem);
}
}
protected async Task<(bool, MediaServicesAccountResource?)> IsAMSAccountAsync(string accountName, CancellationToken cancellationToken)
{
MediaServicesAccountResource? amsAccount = null;

try
{
amsAccount = await _resourceProvider.GetMediaAccountAsync(accountName, cancellationToken);
}
catch (Exception ex)
protected async Task<(bool, MediaServicesAccountResource?)> IsAMSAccountAsync(string accountName, CancellationToken cancellationToken)
{
if (ex is OutOfMemoryException) throw; // It is a fatal error.
MediaServicesAccountResource? amsAccount = null;

// For any other exception, swallow the exception, treat it as not-AMS account,
// The caller then has a chance to treat it as storage account and try it again,
// if it is still failed, the caller will throw exception appropriately.
}
try
{
amsAccount = await _resourceProvider.GetMediaAccountAsync(accountName, cancellationToken);
}
catch (Exception ex)
{
if (ex is OutOfMemoryException) throw; // It is a fatal error.

return (amsAccount != null, amsAccount);
}
// For any other exception, swallow the exception, treat it as not-AMS account,
// The caller then has a chance to treat it as storage account and try it again,
// if it is still failed, the caller will throw exception appropriately.
}

return (amsAccount != null, amsAccount);
}

protected async Task<double> GetStorageBlobMetricAsync(ResourceIdentifier accountId, CancellationToken cancellationToken)
{
Expand Down
2 changes: 1 addition & 1 deletion ams/CleanupCommand.cs
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ public CleanupCommand(GlobalOptions globalOptions,

public override async Task MigrateAsync(CancellationToken cancellationToken)
{

var (isAMSAcc, account) = await IsAMSAccountAsync(_options.AccountName, cancellationToken);
if (!isAMSAcc || account == null)
{
Expand Down
10 changes: 5 additions & 5 deletions ams/ResetCommand.cs
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ private async Task<bool> MigrateTaskAsync(BlobContainerClient container, Cancell
if (properties?.Metadata != null && properties.Metadata.Count == 0)
{
_logger.LogInformation("Container '{container}' does not have metadata.", container.Name);

}
else
{ // Clear container metadata
Expand All @@ -113,7 +113,7 @@ private async Task<bool> MigrateTaskAsync(BlobContainerClient container, Cancell
// It is a container for the DMT generated content, don't reset it.
isDmtGeneratedContainer = true;
_logger.LogInformation("The container '{container}' is created by migration tool, don't reset this one.", container.Name);

}
if (!isDmtGeneratedContainer && !string.IsNullOrEmpty(assetType))
{
Expand All @@ -130,7 +130,7 @@ private async Task<bool> MigrateTaskAsync(BlobContainerClient container, Cancell
else
{
_logger.LogInformation("Metadata in Container '{container}' does not exist or was not deleted.", container.Name);

}

}
Expand All @@ -139,10 +139,10 @@ private async Task<bool> MigrateTaskAsync(BlobContainerClient container, Cancell
catch (Exception ex)
{
_logger.LogError("An unexpected error occurred: {message}", ex.Message);

}
}
return isReseted;
return isReseted;
}
}
}
34 changes: 21 additions & 13 deletions doc/storageCommand.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,28 @@
As the default behavior, the program will initially search for an AMS account; if none is located, it will then proceed to seek a storage account for the migration process.
# Post AMS shutdown migration

# avaiable commands for migrating assets from Azure blob storage:
To migrate from storage account, you'll need to use 'storage' command instead of 'assets' command. An example of the command line is

1. storage command:
AMSMigrate.exe storage -s <subscription> -g <Resource group for the storage account to be migrated> -n <the name of storage account to be migrated> -o <output storage account uri>
```
AMSMigrate.exe storage -s <subscription> -g <Resource group for the storage account to be migrated> -n <the name of storage account to be migrated> -o <output storage account uri>
```

AES encryption is also supported with storage command,

3. reset command:
AMSMigrate.exe reset -s <subscription> -g <Resource group for the storage account to be migrated> -n <the name of storage account to be migrated> -o <output storage account uri>
```
AMSMigrate.exe storage -s <subscription> -g <resource group of media service> -n <the name of storage account to be migrated> -o <output storage account uri> -e --key-vault-uri https://<your azure key vault name>.vault.azure.net
```

4. analyze command:

AMSMigrate.exe analyze -s <subscription> -g <Resource group for the storage account to be migrated> -n <the name of storage account to be migrated>
The 'reset' and 'analyze' command works as well with storage account,

5. AES Encryption:
e.g.

AMSMigrate.exe assets -s <subscription> -g <resource group of media service> -n <the name of storage account to be migrated> -o <output storage account uri> -e --key-vault-uri https://<your azure key vault name>.vault.azure.net
```
AMSMigrate.exe reset -s <subscription> -g <Resource group for the storage account to be migrated> -n <the name of storage account to be migrated>
```
and

```
AMSMigrate.exe analyze -s <subscription> -g <Resource group for the storage account to be migrated> -n <the name of storage account to be migrated>
```

The 'cleanup' command only works with AMS account, so if you need to do 'cleanup' on a storage container, you'll have to do it manually yourself using Azure storage API / CLI.