Characters lost in strings?

[kine]

Version

• I confirm that I am using the latest version

Source Version

Azure DevOps Service

Target Version

Azure DevOps Service

Relevant configuration

{
  "Serilog": {
    "MinimumLevel": "Information"
  },
  "MigrationTools": {
    "Version": "16.0",
    "Endpoints": {
      "Source": {
        "EndpointType": "TfsTeamProjectEndpoint",
        "Collection": "https://XXXX.visualstudio.com/",
        "Project": "XXXX DEMO",
        "Authentication": {
          "AuthenticationMode": "AccessToken",
          "AccessToken": "XXXX"
        },
        "ReflectedWorkItemIdField": "Custom.ReflectedWorkItemIdUrl",
        "LanguageMaps": {
          "AreaPath": "Area",
          "IterationPath": "Iteration"
        }
      },
      "Target": {
        "EndpointType": "TfsTeamProjectEndpoint",
        "Collection": "https://dev.azure.com/XXXX-XXXX",
        "Project": "XXXX Demo",
        "Authentication": {
          "AuthenticationMode": "AccessToken",
          "AccessToken": "XXXXX"
        },
        "ReflectedWorkItemIdField": "Custom.ReflectedWorkItemIdUrl",
        "LanguageMaps": {
          "AreaPath": "Area",
          "IterationPath": "Iteration"
        }
      }
    },
    "Processors": [
      {
        "ProcessorType": "TfsWorkItemMigrationProcessor",
        "Enabled": true,
        "UpdateCreatedDate": false,
        "UpdateCreatedBy": false,
        "WIQLQuery": "SELECT [System.Id] FROM WorkItems WHERE [System.TeamProject] = @TeamProject AND [System.WorkItemType] NOT IN ('Test Suite', 'Test Plan','Shared Steps','Shared Parameter','Feedback Request') ORDER BY [System.ChangedDate] desc",
        "FixHtmlAttachmentLinks": true,
        "WorkItemCreateRetryLimit": 5,
        "FilterWorkItemsThatAlreadyExistInTarget": false,
        "SourceName": "Source",
        "TargetName": "Target"
      }
    ],
    "CommonTools": {
      "TfsChangeSetMappingTool": {
        "Enabled": false,
        "ChangeSetMappingFile": null
      },
      "TfsGitRepositoryTool": {
        "Enabled": false,
        "Mappings": null
      },
      "FieldMappingTool": {
        "Enabled": false,
        "FieldMaps": []
      },
      "TfsNodeStructureTool": {
        "Enabled": true,
        "Areas": {
          "Filters": [],
          "Mappings": {}
        },
        "Iterations": {
          "Filters": [],
          "Mappings": {}
        },
        "ShouldCreateMissingRevisionPaths": false,
        "ReplicateAllExistingNodes": false
      },
      "TfsTeamSettingsTool": {
        "Enabled": "False",
        "MigrateTeamCapacities": "True",
        "MigrateTeamSettings": "True",
        "Teams": [],
        "UpdateTeamSettings": "True"
      }
    }
  }
}

Relevant log output

No response

What happened?

In some cases, local characters are lost during the migration. It seems that it is lost when comment is migrated on the work item. This is example of source work items - both have same text in source project:

After migration they looks like this:

You can see that User Story have no local characters in the title anymore.
When I compare the history for the user story, the chracters are lost when the comment was added on the User Story:

Debug in Visual Studio

• Visual Studio Debug

[MrHinsh]

The https://nkdagility.com/learn/azure-devops-migration-tools/Reference/Tools/StringManipulatorTool/ is handling this...

Solution:

Add your own string manipulator that contains characters that you want to keep.

{
  "MigrationTools": {
    "Version": "16.0",
    "CommonTools": {
      "StringManipulatorTool": {
        "Enabled": "True",
        "Manipulators": [
          {
            "$type": "RegexStringManipulator",
            "Description": "Remove invalid characters from the end of the string",
            "Enabled": "True",
            "Pattern": "[^( -~)\n\r\t]+",
            "Replacement": ""
          }
        ],
        "MaxStringLength": "1000000"
      }
    }
  }
}

This will drop any characters that are not in [^( -~)\n\r\t]+ and generally prevents non-aski chars...

You can add to this list and see #2411 as an example.

[kine]

Thanks for the info, I will try this workaround.

But then question is, why it is not applied to all workitems but only on some?

[kine]

Just want to confirm that adding e.g. \S into the pattern to keep all non-space chars solved the problem.

[MrHinsh]

That I can't tell you. It should be applied to every string field on every work item.

[kine]

Seems it is not in some cases (then this will be the bug). But anyway, thanks for your help, much appreciated. My problem was solved.

[MrHinsh]

Just want to confirm that adding e.g. \S into the pattern to keep all non-space chars solved the problem.

This may result is errors if you migrate ADO to ADO. Its possible to put characters in ADO that are not supported in the Tfs Object Model/

[satano]

I run into the same issue. Why is this tool needed and enabled by default? When I am migrating work items, I expect that if something is in the source, it can exist in the target and hence I do not want/need to remove invalid characters. Especially, when the removed characters are not invalid at all. So would not be better if this tool was disabled by default?

Or if I ask the other way around – what problems can I have if I will disable this tool?

[MrHinsh]

We started to run into a number of errors where people had put emojis or other characters in the work items. The Tfs Object Model is nearly twenty years old and does not support that.

ADO now also supports strings longer than 1000000 which is also not supported in Tfs Object Model.

This was added as the default 9 months ago so that most folks would not have this issue. Its easily overridden for advanced users.

[MrHinsh]

I'm not opposed to changing the default! We could remove the default manipulator and just have the length.

[kine]

I am not sure what are the numbers where Tfs is used versus ADO2ADO transfers. For me, if someone is still working on TFS, he is advance user and should be able to handle this, if it is documented well. I expect that there is bunch of users doing "just" migration ADO2ADO who knows nothing around and just needs to run the tool to e.g. move to another account or project. I will support the change of default to be more "harmless" for these cases and add warning when TFS is used...