When you are migrating to Azure DevOps you MUST provide an PAT

[nkofctg]

While running 15.2, I was able to run migrations from many sources to many destinations. After upgrading to 16.0.1, all migration attempts end after a few seconds, with the error stating that the PersonalAccessToken is missing from my Target configuration. However, my configuration does include a PAT.

I have attempted this with several 15.2 versioned config files, as well as the above, which was taken from a newly initialized config file (with relevant configs added, including of course my PAT). I have also attempted to change to Prompt authentication, and tried adding / removing the NetworkCredentials block.

"Target": {
        "EndpointType": "TfsTeamProjectEndpoint",
        "Collection": "https://dev.azure.com/ContosoInc/",
        "Project": "Company Project",
        "Authentication": {
          "AuthenticationMode": "AccessToken",
          "AccessToken": "thisiswhereatokengoesanditalmostkindalookslikethis"
        },
        "ReflectedWorkItemIdField": "Custom.ReflectedWorkItemId",
        "AllowCrossProjectLinking": false,
        "LanguageMaps": {
          "AreaPath": "Area",
          "IterationPath": "Iteration"
        },

[08:26:13 INF] [16.0.1] Migration Context Start: TfsWorkItemMigrationProcessor
[08:26:13 ERR] [16.0.1] When you are migrating to Azure DevOps you MUST provide an PAT so that we can call the REST API for certain actions. For example we would be unable to deal with a Work item Type change. 
System.InvalidOperationException: Missing PersonalAccessToken from Target

[nkdagility-actions-bot[bot]]

Hi there,

ACTION REQUIRED

In order for us to help you will need to provide both the log, and the configration file that you are using.

Generate and Attach Log

Steps:

1. In the Config file, change LogLevel to Debug.
2. Identify a work item ID that exhibits this issue and create a query that only selects this work item.
3. Delete any existing copy of that work item from the target.
4. Run the migration.
5. Comment here with the full Log attached as well as the full (redacted) configration file that you are using.

Your assistance is greatly appreciated!

Thanks, Azure DevOps Migration Tools team

[MrHinsh]

Can you provide the full configuration?

Things to check:

1. Ensure the endpoint is listed under "MigrationTools:Endpoints".
2. Verify that the processor references the correct endpoint.

---

Note on the new format:

The new configuration format supports multiple endpoints. We transitioned from a grouped set of lists to a single set of named entries. This change allows overrides via the command line and environment variables.

The validation error you're encountering adds an extra layer to ensure a Personal Access Token (PAT) is used when connecting to Azure DevOps (ADO).

---

PAT Token Validation Logic:

private void ValidatePatTokenRequirement()
{
    string collUrl = Target.Options.Collection.ToString();
    if (collUrl.Contains("dev.azure.com") || collUrl.Contains(".visualstudio.com"))
    {
        var token = Target.Options.Authentication.AccessToken;
        
        // Check if the token is missing
        if (token.IsNullOrEmpty())
        {
            var ex = new InvalidOperationException("Missing PersonalAccessToken from Target");
            Log.LogError(ex, "When migrating to Azure DevOps, you MUST provide a PAT to access the REST API for certain actions. For example, handling Work Item Type changes would be impossible without it.");
            Environment.Exit(-1);
        }
    }
}

---

Endpoint Configuration Example:

Your endpoint should be structured like this:

{
  "MigrationTools": {
    "Endpoints": {
      "Target": {
        "EndpointType": "TfsTeamProjectEndpoint",
        "Collection": "https://dev.azure.com/ContosoInc/",
        "Project": "Company Project",
        "Authentication": {
          "AuthenticationMode": "AccessToken",
          "AccessToken": "thisiswhereatokengoesanditalmostkindalookslikethis"
        },
        "ReflectedWorkItemIdField": "Custom.ReflectedWorkItemId",
        "AllowCrossProjectLinking": false,
        "LanguageMaps": {
          "AreaPath": "Area",
          "IterationPath": "Iteration"
        }
      }
    }
  }
}

---

Supporting Multiple Endpoints:

To support multiple endpoints, you can use named entries:

{
  "MigrationTools": {
    "Endpoints": {
      "Target": {
        "EndpointType": "TfsTeamProjectEndpoint",
        "Collection": "https://dev.azure.com/ContosoInc/",
        "Project": "Company Project",
        "Authentication": {
          "AuthenticationMode": "AccessToken",
          "AccessToken": "thisiswhereatokengoesanditalmostkindalookslikethis"
        },
        "ReflectedWorkItemIdField": "Custom.ReflectedWorkItemId",
        "AllowCrossProjectLinking": false,
        "LanguageMaps": {
          "AreaPath": "Area",
          "IterationPath": "Iteration"
        }
      },
      "Alpha": {
        "EndpointType": "TfsTeamProjectEndpoint",
        "Collection": "https://dev.azure.com/ContosoInc/",
        "Project": "Company Project",
        "Authentication": {
          "AuthenticationMode": "AccessToken",
          "AccessToken": "thisiswhereatokengoesanditalmostkindalookslikethis"
        },
        "ReflectedWorkItemIdField": "Custom.ReflectedWorkItemId",
        "AllowCrossProjectLinking": false,
        "LanguageMaps": {
          "AreaPath": "Area",
          "IterationPath": "Iteration"
        }
      },
      "Beta": {
        "EndpointType": "TfsTeamProjectEndpoint",
        "Collection": "https://dev.azure.com/ContosoInc/",
        "Project": "Company Project",
        "Authentication": {
          "AuthenticationMode": "AccessToken",
          "AccessToken": "thisiswhereatokengoesanditalmostkindalookslikethis"
        },
        "ReflectedWorkItemIdField": "Custom.ReflectedWorkItemId",
        "AllowCrossProjectLinking": false,
        "LanguageMaps": {
          "AreaPath": "Area",
          "IterationPath": "Iteration"
        }
      }
    }
  }
}

---

This structure allows you to define multiple endpoints with the flexibility to customize their configurations as needed.

[nkdagility-actions-bot[bot]]

Hi there,

ACTION REQUIRED

In order for us to help you will need to provide both the log, and the configration file that you are using.

Generate and Attach Log

Steps:

1. In the Config file, change LogLevel to Debug.
2. Identify a work item ID that exhibits this issue and create a query that only selects this work item.
3. Delete any existing copy of that work item from the target.
4. Run the migration.
5. Comment here with the full Log attached as well as the full (redacted) configration file that you are using.

Your assistance is greatly appreciated!

Thanks, Azure DevOps Migration Tools team

[nachofridemberg]

Hi similar isse was turned up here.
Testing version 16.0.6 some error ocurs which said "-The AccessToken must not be null or empty when AuthenticationMode is set to 'AccessToken'...." but the accesstoken is set it. The weird in this case is error ocurs just in targe validation not source. Below drop of an extract of the log.

> _The configuration entry at 'MigrationTools:Endpoints:Target' did not pass validation!
>  Please check the following failures:-The AccessToken must not be null or empty when AuthenticationMode is set to 'AccessToken'. You must provide a PAT to use 'AccessToken' as the authentication m
> ode. You can set this through the config at 'MigrationTools:Endpoints:{name}:Authentication:AccessToken', or you can set an environment variable of 'MigrationTo_

[MrHinsh]

If you are recieving the error then the system is seeing the field as empty.

Can you share your config? Most of the time it's a "}" in the wrong place which messes up the nesting ..

[nachofridemberg]

I'm really sure about the code, but confirm at follow i drop off my configuration model without the my data:

 {
  "Serilog": {
    "MinimumLevel": "Information"
  },
  "MigrationTools": {
    "Version": "16.0",
    "Endpoints": {
      "Source": {
        "EndpointType": "TfsTeamProjectEndpoint",
        "Collection": "https://dev.azure.com/myorgxxx/",
        "Project": "mysourceprojectxxx",
        "Authentication": {
          "AuthenticationMode": "AccessToken",
          "NetworkCredentials": {
            "Domain": "",
            "UserName": "",
            "Password": "** removed as a secret ***"
          },
          "AccessToken": "myAccessTokenxxxxxxisTheSameAsTargetPAT"
        },
        "ReflectedWorkItemIdField": "Custom.ReflectedWorkItemId",
        "LanguageMaps": {
          "AreaPath": "Area",
          "IterationPath": "Iteration"
        }
      },
      "Target": {
        "EndpointType": "TfsTeamProjectEndpoint",
        "Collection": "https://dev.azure.com/myorgxxx/",
        "Project": "mytargetprojectxxx",
        "Authentication": {
          "AuthenticationMode": "AccessToken",
          "NetworkCredentials": {
            "Domain": "",
            "UserName": "",
            "Password": "** removed as a secret ***"
          },
          "AccessToken": "myAccessTokenxxxxxxisTheSameAsSourcetPAT"
        },
        "ReflectedWorkItemIdField": "Custom.ReflectedWorkItemId",
        "LanguageMaps": {
          "AreaPath": "Area",
          "IterationPath": "Iteration"
        }
      }
    },
    "Processors": [
      {
        "ProcessorType": "TfsWorkItemMigrationProcessor",
        "Enabled": false,
        "UpdateCreatedDate": true,
        "UpdateCreatedBy": true,
        "WIQLQuery": "SELECT [System.Id] FROM WorkItems WHERE [System.TeamProject] = @TeamProject AND [System.WorkItemType] NOT IN ('Test Suite', 'Test Plan','Shared Steps','Shared Parameter','Feedback Request') ORDER BY [System.ChangedDate] desc",
        "FixHtmlAttachmentLinks": true,
        "WorkItemCreateRetryLimit": 5,
        "FilterWorkItemsThatAlreadyExistInTarget": false,
        "GenerateMigrationComment": true,
        "SourceName": "Source",
        "TargetName": "Target"
      }
    ],
    "CommonTools": {
      "FieldMappingTool": {
        "Enabled": true,
        "FieldMaps": [
          {
            "FieldMapType": "FieldLiteralMap",
            "targetField": "Custom.SomeField",
            "value": "New field value",
            "ApplyTo": [
              "SomeWorkItemType"
            ]
          }
        ]
      }
    }
  }
}

[MrHinsh]

If I run your config it passes validation!

[nachofridemberg]

Any advice?

[denniselit]

@MrHinsh I've update the link with the file I think you refer to

[MrHinsh]

Running with v16.0.6 and the config above I get passed the validation and end up at 401 trying to connect to ADO.

2024-11-05_09-55-48.mp4

I added some debug output to the latest preview that outputs the sucess/fail on the validators...

[09:53:34 DBG] [16.0.7-Preview.3] TfsLanguageMapOptions::Validate::Success
[09:53:34 DBG] [16.0.7-Preview.3] TfsAuthenticationOptions::Validate::Success
[09:53:34 DBG] [16.0.7-Preview.3] TfsEndpointOptionsValidator::Validate::Success
[09:53:34 INF] [16.0.7-Preview.3] Migrating all Nodes before the Processor run.
[09:53:34 DBG] [16.0.7-Preview.3] TfsLanguageMapOptions::Validate::Success
[09:53:34 DBG] [16.0.7-Preview.3] TfsAuthenticationOptions::Validate::Success
[09:53:34 DBG] [16.0.7-Preview.3] TfsEndpointOptionsValidator::Validate::Success

[denniselit]

@MrHinsh Thank you for your video, I have now tested the suggestions provided by you but without any success. I shared the config and the log file.
migration2024110511.log
configuration.json

[MrHinsh]

@denniselit the config you are providing here is not the config tha the tool is using! In the log you have 2 fieldmapps, and in the config that you are providing you have only 1.

Can you provide C:\Users\U\AppData\Local\Microsoft\WinGet\Packages\nkdAgility.AzureDevOpsMigrationTools.Preview_Microsoft.Winget.Source_8wekyb3d8bbwe\configuration.json as this is the one that is running!

---

devopsmigration is runs relative to the install location and not the location that you are running the tool. This is due to bugs in portable support in winget that are fixed in the next version.

[denniselit]

@MrHinsh Your comment above helped me to locate the issue, I had to provide the full path to the file I used (was located in C:\temp\configuration.json and not C:\Users\U\AppData\Local\Microsoft\WinGet\Packages\nkdAgility.AzureDevOpsMigrationTools.Preview_Microsoft.Winget.Source_8wekyb3d8bbwe\configuration.json)

Now it works for me!

[Joan-fr]

@MrHinsh I found the cause of the error. It was really simple : I had not provided the full path of the configuration.json file ...
Had written "devopsmigration execute --config .\configuration. json" instead of "devopsmigration execute C:\users\username\configuration.json. Working now!

[nachofridemberg]

Same. It neccesary full path.
Thanks

[MrHinsh]

Im seeing if I can make this easier....

  hostBuilder.ConfigureLogging((context, logBuilder) =>
   {
   })
  .ConfigureAppConfiguration(builder =>
  {
      //builder.SetBasePath(AppContext.BaseDirectory);
      builder.AddJsonFile(Path.Combine(AppContext.BaseDirectory, "appsettings.json"), optional: false);
      if (!string.IsNullOrEmpty(configFile) && File.Exists(configFile))
      {
          builder.AddJsonFile(configFile);
      }
      builder.AddEnvironmentVariables();
      builder.AddCommandLine(args);
  });

I'm trying to remove the builder.SetBasePath(AppContext.BaseDirectory); as well as remove the default config so we would get a "missing" error.