VSBuild@1 fails to build appx WITHOUT the task updating

[Skintkingle]

Description

COPIED FROM Azure Pipelines Issue

Last week, we were using VSBuild@1 (version 1.245.0), last used Friday 13th September at 8pm UTC to successfully build and package an APPX.
On sunday (and today) we are using the same version (1.245.0), and the build is now failing with the following error:

fatal error CMF1106: failed to open input PDB file for reading (PDB error code = 13)

I have raised this here INSTEAD of azure-pipelines-tasks because the task hasn't changed between the successful execution and the failure, so I don't beleive it's the task that has changed to make this fail, rather the hosting environment/agent

The task is configured as such (and has been like this for many months):

- task: VSBuild@1
  displayName: Build Our Product
  inputs:
    platform: 'x64' 
    solution: 'Our Solution.sln'
    msbuildArgs: '/p:AppxBundlePlatforms="x64" /p:AppxBundle=Always /p:UapAppxPackageBuildMode=SideLoadOnly /p:AppxPackageSigningEnabled=false'

Critically - we can run this on the same commit that previously passed, and it now fails.

This is preventing us from releasing any changes and as such, for us, is critical to get a fix or workaround

Wondering if the fix deployed for this event was the cause

Which would be related to #10601

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

20240912.1.0

Is it regression?

yes, 20240811.1.0

Expected behavior

For vsbuild@1 to build appx packages

Actual behavior

failed with the above error

Repro steps

_GenerateAppxSymbolPackage:
  C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VisualStudio\v17.0\AppxPackage\\x64\mspdbcmf.exe /check "D:\a\1\s\Our App\bin\x64\Debug\net8.0-windows10.0.19041.0\win-x64\msixpublish\Our App.pdb" 
  Microsoft (R) FastLink To Full PDB Converter Version 14.41.33923.0
  Copyright (C) Microsoft Corporation.  All rights reserved.
  
  D:\a\1\s\Our App\bin\x64\Debug\net8.0-windows10.0.19041.0\win-x64\msixpublish\Our App.pdb : fatal error CMF1106: failed to open input PDB file for reading (PDB error code = 13)
##[error]C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VisualStudio\v17.0\AppxPackage\Microsoft.AppXPackage.Targets(3715,5): Error MSB6006: "mspdbcmf.exe" exited with code 1106.
##[debug]Processed: ##vso[task.logissue type=Error;sourcepath=C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VisualStudio\v17.0\AppxPackage\Microsoft.AppXPackage.Targets;linenumber=3715;columnnumber=5;code=MSB6006;]"mspdbcmf.exe" exited with code 1106.
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VisualStudio\v17.0\AppxPackage\Microsoft.AppXPackage.Targets(3715,5): error MSB6006: "mspdbcmf.exe" exited with code 1106. [D:\a\1\s\OurApp.Package\OurApp.Package.wapproj]

[Wuppie007]

Also run into this is issue.

The last succesfully completed pipeline for me was with runner image version 20240811.1.0. At this image the "Fastlink to Full PDB Converter" version was 14.33.31424.0.

[kishorekumar-anchala]

Hi @Skintkingle ,

Thank you for bringing this issue to us. We are looking into this issue and will update you on this issue after investigating.

[Skintkingle]

Thanks for looking into this, This is currently stopping us from releasing and as such is quite a critical blocker for us. Do you have any ideas of a workaround for the time being?

[kishorekumar-anchala]

HI @Skintkingle ,

Could you please try to. build with windows-2019 image .

[Skintkingle]

Hi @kishorekumar-anchala
Unfortunately we can't use windows-2019 as our app is a .Net 8 app and the visual studio version installed for windows-2019 can't build .Net 8 projects. Its MSBuild is too old. :(

We have also tried downloading .Net 8 with the UseDotNet@2 Task, but fundamentally, given this installs the dotnet tool for that .Net version, and not MSBuild, we're left in the same situation.

[josephsleiman16]

Also run into this issue and also unable to use windows-2019 image which only supports .NET 5.0 and lower. This is preventing us from releasing and is also a critical blocker for us, so any better workaround or permanent fix would be great.

[yosefmah]

This is impacting us too. We are unable to build or deploy updates to our internal applications so it is very critical for us to get a workaround at least for the time being.

[kishorekumar-anchala]

HI @Skintkingle ,

Could you please share your yaml file to investigate further.

[Skintkingle]

Sure, it is very simple though.

I've also removed everything after the failing step (as it's never got to) :)

pool:
  vmImage: windows-latest

steps:
- task: NuGetAuthenticate@1
  displayName: Authenticate with NuGet feed

- task: DotNetCoreCLI@2
  displayName: Restore Our Solution
  inputs:
    command: 'restore'
    projects: 'Our Solution.sln'
    feedsToUse: 'select'
    vstsFeed: '<ourInternalFeed>'

- task: DotNetCoreCLI@2
  displayName: Build Our Web App
  inputs:
    command: 'build'
    projects: 'OurApp.Web/OurApp.Web.csproj'
    
 - task: VSBuild@1
  displayName: Build Our Product
  inputs:
    platform: 'x64' 
    solution: 'Our Solution.sln'
    msbuildArgs: '/p:AppxBundlePlatforms="x64" /p:AppxBundle=Always /p:UapAppxPackageBuildMode=SideLoadOnly /p:AppxPackageSigningEnabled=false'

[kishorekumar-anchala]

Hi @Skintkingle ,

In VsBuild@1 task , kindly modify platform value to Any CPU from x64 and try . it works for me. thank you !

[josephsleiman16]

I have practically the same yaml file and the same error occurring on the exact same task. Changing the platform from 'x64' to 'Any CPU' is not compatible with our solution as it is not configured to build with Any CPU, so this is not a valid fix or workaround, even. This is becoming a more urgent issue as it is almost a week now we cannot deploy releases.

[Skintkingle]

This also doesn't work for us. We get the same error when platform is set to Any CPU

[kishorekumar-anchala]

HI @Skintkingle @josephsleiman16 ,

Might be issue with the Vstask@1 , have you tried to change the version of vsbuild to 2 ? could you please try this , as sometimes it will fix the issue.

and also confirm what is the image version you're using ? 20240912.1.0 ?

[Skintkingle]

there is only VSBuild@1. The task is unchanged between it working and it not working.

I can confirm these tests this morning were run against a newer version of the runner: 20240915.1.0

[kishorekumar-anchala]

Hi @Skintkingle , Could you please raise issue with azure tasks team. https://github.com/microsoft/azure-pipelines-tasks

[Skintkingle]

If you sincerely beleive that it's a tasks issue and not an image issue (given the runner image has changed between it working and not working, but the task hasn't changed)

[Skintkingle]

I am skeptical that it's the tasks fault though. Please don't stop investigating.

[Skintkingle]

Further to this, I have tried using MSBuild@1 rather than VSBuild@1 on the slim chance it's the tasks fault, and I get exactly the same error at exactly the same point in the MSBuild task. This has almost certainly go to do with the build environment (or its' configuration or available tooling or something like that)

The MSBuild task looked like this:

- task: MSBuild@1
  displayName: Build Our Product with MSBuild
  inputs:
    solution: 'Our Solution.sln'
    msbuildVersion: '17.0'
    msbuildArchitecture: 'x64'
    platform: 'x64'
    msbuildArguments: '/p:AppxBundlePlatforms="x64" /p:AppxBundle=Always /p:UapAppxPackageBuildMode=SideLoadOnly /p:AppxPackageSigningEnabled=false'

[josephsleiman16]

Agree with @Skintkingle, this is almost certainly a build issue with the runner image changing. Our builds were working on Friday on runner image 20240811.1.0 and then on monday, using runner image 20240912.1.0 they stopped working, despite being the exact same commits using the exact same yaml pipelines and tasks.

[yosefmah]

Agreed here as well. Does not seem to be an issue with the task at all from our perspective.

[Skintkingle]

@kishorekumar-anchala are you still looking into this or not anymore?

[Wuppie007]

To add more information, I've added the command /p:AppxSymbolPackageEnabled=False, to not run into the "PDB file error".

- task: MSBuild@1
  inputs:
    solution: '**/msix project.wapproj'
    platform: AnyCPU
    configuration: Release
    msbuildArguments: '
     /p:OutputPath=NonPackagedApp
     /p:UapAppxPackageBuildMode=SideLoadOnly  
     /p:AppxBundle=Never
     /p:AppxSymbolPackageEnabled=False
     /p:AppxPackageOutput=$(Build.ArtifactStagingDirectory)\outputApp.msix 
     /p:AppxPackageSigningEnabled=false'
  displayName: 'Package the App'

This results in the following error: (I have removed some paths and names because of company names etc)

_GenerateAppxPackageFile:
  C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\x64\MakeAppx.exe pack /l /h sha256 /f obj\Release\package.map.txt /o /p D:\a\1\a\AppAttach\<omitted>.msix  
  <omitted>-> D:\a\1\a\AppAttach\<omitted>.msix
##[error]C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VisualStudio\v17.0\AppxPackage\Microsoft.AppXPackage.Targets(6573,5): Error APPX0002: Task 'GenerateAppInstallerFile' failed. Could not find a part of the path 'D:\a\1\s\<omitted>\trunk\<omitted>\AppPackages\<omitted>_neutral.appinstaller'.

I've checked the last succesfull pipeline, and the used buildnummer fro MakeAppx.exe is unchanged (10.0.22621.0)

[Skintkingle]

adding `/p:AppxSymbolPackageEnabled=false' actually worked around the issue, for us. So thanks for that idea @Wuppie007 ! It's definitely a workaround though. So I'm still keen to see a proper resolution

[Wuppie007]

It seems to be that de parameter /p:AppxPackageOutput=$(Build.ArtifactStagingDirectory)\outputApp.msix is causing trouble. When removing this, the pipeline succeeds. But then there are several files created instead of one *.msix file

[Skintkingle]

How is this going?

[CatalinPrata]

I have the same issue, no changes to the pipeline or task and the build started to fail with the exact same error. The last image version that worked was version: '20240811.1.0', now it is at '20240915.1.0' and it fails.
I have to mention that I am using CmdLine@2 task with Cake to build my UWP project.

Any updates on this?

[kishorekumar-anchala]

Currently investigation is going on , we will provide the RCA once it done. thank you !

[yosefmah]

A small update here. It turns out we also get the same error when publishing locally with pdb files included (not something we've had to try for a long time).

After a lot of trial and error with the MSVC tools, it looks like version 14.39.33519 was the latest working version. Installing that version and adding the following to the csproj works locally to publish with symbols:

 <PropertyGroup>
  <PdbCmfx64ExeFullPath>C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.39.33519\bin\Hostx64\x64\mspdbcmf.exe</PdbCmfx64ExeFullPath>
  <PdbCmfx86ExeFullPath>C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.39.33519\bin\Hostx64\x86\mspdbcmf.exe</PdbCmfx86ExeFullPath>
 </PropertyGroup>

As per @Wuppie007's comment above, our latest successful run was in runner image 20240811.1.0 (which our agent was still using as per 12/09/2024) so I'm thinking it might have been a Visual Studio upgrade some time after that which has caused this. Possibly 20240825.1.1 which upgraded Visual Studio from 17.10.35122.118 to 17.11.35219.272 and the MSVC tools alongside it.

[Skintkingle]

I'm interested if this is still being looked into for the image? Have you taken this to the relevant department internally at microsoft following yosefmah's investigation?

[yosefmah]

So after much trial and error, we've narrowed it down very specifically to our use of Fody.PropertyChanged. Disabling this weaver seems to then get rid of the error. Nothing has changed with that package, and given that it works fine with previous versions I do feel the issue lies with the VS upgrade.

I have raised an issue here at the Visual Studio Community.

If anyone else is still having the issue, would appreciate an upvote for visibility there. I'm also interested to see if others have also used Fody.PropertyChanged or if it is something similar which triggers the problem.

[josephsleiman16]

Is there any update on this?

[derekryms]

I am also running into these same issues. Our pipeline was running fine last week, but died this week in the same way with no changes. Setting AppxSymbolPackageEnabled to false worked to get rid of the "Error MSB6006: "mspdbcmf.exe" exited with code -1073741571" but we still have "Error APPX0002: Task 'GenerateAppInstallerFile' failed. Could not find a part of the path ..."

Any updates or work arounds?

[derekryms]

I went ahead and created a new WPF project and WAP project and the APPX0002 error issues persisted, so I created a ticket.

https://developercommunity.visualstudio.com/t/WAP-Project-Error-APPX0002:-Task-Genera/10760537

[derekryms]

Just found a temporary fix for the Appx0002 error.

@Wuppie007 Try to manually create the AppPackages directory before building the wap project.

[Skintkingle]

What's the progress on this issue?

[kishorekumar-anchala]

Hi @Skintkingle ,

The issue is under investigation , i request you to track the updates from community team on it . thank you !

https://developercommunity.visualstudio.com/t/Regression-in-MSVC-tools-1711-when-pu/10755007

[kishorekumar-anchala]

We're closing the issue as it it not belongs to runner images. thank you !

you can track the updates from below.
https://developercommunity.visualstudio.com/t/Regression-in-MSVC-tools-1711-when-pu/10755007