Too many retries performed when using cmdlets #2534

nickp85 · 2024-01-17T21:40:05Z

Thanks for reporting the bug. Please ensure you've gone through the following checklist before opening an issue:

Make sure you can reproduce this issue using the latest released version of Microsoft.Graph or Microsoft.Graph.Beta.
Please search the existing issues to see if there has been a similar issue filed.
For issues related to authentication and service errors, please refer to our troubleshooting guide. For service issues, please open a question at https://developer.microsoft.com/graph/support.

Describe the bug
Using Invoke-MgGraphRequest to make custom Graph requests to query devices in Intune. Has been working for years and within the last week, the requests are returning "Too many retries performed". I have tried to change the retry count or retry delay but nothing has helped. Eventually I started getting "task has been canceled" instead of "too many retries performed". The issue occurs the same with v1.0 or beta API. I use the beta API since that matches what Microsoft uses for the production Intune console (why I don't know).

The API chunks the response into 1000 at a time so the error is random. Sometimes it's immediate, other times it takes 3-4 runs of the while loop for it to fail

To Reproduce
Steps to reproduce the behavior:

Connect to MgGraph
Execute this code to grab all of our devices

# Graph API version and call to make
$graphApiVersion = "beta"
$Resource = "deviceManagement/managedDevices"
$filter = "`?`$filter=managementAgent eq 'mdm' or managementAgent eq 'easmdm'"

# API call URI
$uri = "https://graph.microsoft.com/$graphApiVersion/$Resource$filter"

# Make the API call using HTTP GET with the token in the header
$DevicesResponse = (Invoke-MgGraphRequest Get -Uri $uri -OutputType PSObject)

# Store the response
$devices = $DevicesResponse.value

# Get all the devices over 1000
$DevicesNextLink = $DevicesResponse."@odata.nextLink" 
While ($null -ne $DevicesNextLink) {            
    $DevicesResponse = (Invoke-MgGraphRequest Get -Uri $DevicesNextLink -OutputType PSObject)
    $DevicesNextLink = $DevicesResponse."@odata.nextLink"
    $devices += $DevicesResponse.value
}

See error returned
Invoke-MgGraphRequest : Too many retries performed

Expected behavior
The device data is put into the $devices variable

Debug Output

Run the problematic command with -Debug and paste the resulting debug stream below.
⚠ ATTENTION: Be sure to remove any sensitive information that may be in the logs.

DEBUG: GET
/beta/deviceManagement/managedDevices?$filter=managementAgent+eq+%27mdm%27+or+managementAgent+eq+%27easmdm%27&$skiptoken=LastDeviceName%3d%27Ben%25e2%2580%2599s%2520iPhone%27%2cLast
DeviceId%3d%27e230406f-f8ef-4dac-8eb1-f352c2b7ec63%27 HTTP/1.1
HTTP: graph.microsoft.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Microsoft Windows 10.0.19045; en-US) PowerShell/2024.0.0 Invoke-MgGraphRequest

Confirm
Continue with this operation?
[Y] Yes [A] Yes to All [H] Halt Command [S] Suspend [?] Help (default is "Y"): y
Invoke-MgGraphRequest : Too many retries performed
At line:4 char:25

... Response = (Invoke-MgGraphRequest Get -Uri $DevicesNextLink -OutputTy ...
```
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
- CategoryInfo : InvalidOperation: (:) [Invoke-MgGraphRequest], InvalidOperationException
- FullyQualifiedErrorId : NotSpecified,Microsoft.Graph.PowerShell.Authentication.Cmdlets.InvokeMgGraphRequest

Module Version

Please run Get-Module Microsoft.Graph* after cmdlet execution and paste the output below.
If a module cannot be installed or imported, please run Get-Module -ListAvailable and paste the output.
ModuleType Version Name ExportedCommands

Script 2.11.1 Microsoft.Graph.Authentication {Add-MgEnvironment, Connect-MgGraph, Disconnect-MgGraph, Get-MgContext...}

Environment Data

Please run $PSVersionTable and paste the output below. If running the Docker container image, indicate the tag of the image used and the version of Docker engine.

Name Value

PSVersion 5.1.19041.3803
PSEdition Desktop
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
BuildVersion 10.0.19041.3803
CLRVersion 4.0.30319.42000
WSManStackVersion 3.0
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

SeniorConsulting · 2024-01-18T01:10:11Z

Hi Nickp85,

If I may ask a simple question:
Is there a reason you aren't using something like:
$Devices = Get-MgBetaDeviceManagementManagedDevice -filter "managementAgent eq 'mdm' or managementAgent eq 'easmdm'" -all

nickp85 · 2024-01-18T01:12:34Z

Hi Nickp85,

If I may ask a simple question:

Is there a reason you aren't using something like:

$Devices = Get-MgBetaDeviceManagementManagedDevice -filter "managementAgent eq 'mdm' or managementAgent eq 'easmdm' -all"

The script is years old and it was easier to change from Invoke-RestMethod to Invoke-MgGraphRequest since I already had all the other code laid out to make the requests

SeniorConsulting · 2024-01-18T01:21:08Z

Fair enough. I agree that there was a time before the SDK came into existence. It wasn't terribly fun managing access tokens, REST requests and such.

However, I would try to steer you toward using the cmdlet above, as it'll take care of the (1000) chunking for you, step off when it comes to the HTTP 429 responses for as long as it takes etc. Hopefully that single string should be able to replace all of the code you've posted above, which should make support/maintenance a little easier too.

nickp85 · 2024-01-18T01:22:51Z

Fair enough. I agree that there was a time before the SDK came into existence. It wasn't terribly fun managing access tokens, REST requests and such.

However, I would try to steer you toward using the cmdlet above, as it'll take care of the (1000) chunking for you, step off when it comes to the HTTP 429 responses for as long as it takes etc. Hopefully that single string should be able to replace all of the code you've posted above, which should make support/maintenance a little easier too.

Is the output identical to what I'm getting today? A JSON output?

SeniorConsulting · 2024-01-18T01:30:23Z

It'll be in a PSObject form (System.Array).

From the text you posted above, it looked as though you were interested in returning PSObjects too.

$DevicesResponse = (Invoke-MgGraphRequest Get -Uri $uri -OutputType PSObject)

However, if you must return this in JSON, you could do something like:

$Devices = Get-MgBetaDeviceManagementManagedDevice -filter "managementAgent eq 'mdm' or managementAgent eq 'easmdm'" -all
$JSON = $Devices | ConvertTo-JSON -Depth 5

However, I am only able to see the information you've supplied, and I lack the full context of what you're doing. I can only recommend that you give the read-only cmdlets a whirl to experiment with.

nickp85 · 2024-01-18T01:32:37Z

You're right, I have PSObject and convert to JSON later

It's also been said I should try the "export APIs"?

nickp85 · 2024-01-18T18:03:43Z

It'll be in a PSObject form (System.Array).

From the text you posted above, it looked as though you were interested in returning PSObjects too.

$DevicesResponse = (Invoke-MgGraphRequest Get -Uri $uri -OutputType PSObject)

However, if you must return this in JSON, you could do something like:
$Devices = Get-MgBetaDeviceManagementManagedDevice -filter "managementAgent eq 'mdm' or managementAgent eq 'easmdm'" -all
$JSON = $Devices | ConvertTo-JSON -Depth 5
However, I am only able to see the information you've supplied, and I lack the full context of what you're doing. I can only recommend that you give the read-only cmdlets a whirl to experiment with.

unfortunately the different cmdlet did not work either.

$devices = Get-MgBetaDeviceManagementManagedDevice -filter "managementAgent eq 'mdm' or managementAgent eq 'easmdm'" -all -debug

Get-MgBetaDeviceManagementManagedDevice : One or more errors occurred.
At line:1 char:1

$devices = Get-MgBetaDeviceManagementManagedDevice -filter "managemen ...

  + CategoryInfo          : NotSpecified: (:) [Get-MgBetaDevic...agedDevice_List], AggregateException
  + FullyQualifiedErrorId : System.AggregateException,Microsoft.Graph.Beta.PowerShell.Cmdlets.GetMgBetaDeviceManagementManagedDevice_List

nickp85 · 2024-01-18T18:15:09Z

Running again I again get the retries error

Get-MgBetaDeviceManagementManagedDevice : Too many retries performed
At line:1 char:1

$devices = Get-MgBetaDeviceManagementManagedDevice -filter "managemen ...

  + CategoryInfo          : NotSpecified: (:) [Get-MgBetaDevic...agedDevice_List], InvalidOperationException
  + FullyQualifiedErrorId : Microsoft.Graph.Beta.PowerShell.Cmdlets.GetMgBetaDeviceManagementManagedDevice_List

SeniorConsulting · 2024-01-18T23:56:33Z

Wow, that's a little surprising. I'm not sure I've got the right questions to ask to continue this. It does sound like a Graph API based problem, rather than the SDK - but I can't be certain. I'm not convinced whether running your cmdlet with a debug command would be of use either, because it sounds like it collects several iterations (pages of 1000), before failing.

I'll leave this for someone else to pick up and point you in the right direction, but I'll keep tabs on this just in case. A piece of advice I have here is to jump on over to the Graph API support location and raise the same over there. If the SDK devs determine that it's an API service issue, you'll have a head start on that ticket. The API support links can be found here:
https://developer.microsoft.com/en-us/graph/support

Nohvah22 · 2024-01-19T14:31:36Z

I opened a case yesterday for the same exact issue and I am leveraging 'Get-MgBetaDeviceManagementManagedDevice'. I have operational scripts that rely on it that have been working for a year and this week they are all failing consistently. This occurs with Graph Powershell v1.0 and beta. Also, I have no issues pulling back user and device objects with 'get-MgUser' and 'get-mgdevice'... this seems to be only affecting the managed device space (Intune devices). You're not alone in this and I have heard nothing from Microsoft yet. I provided them my debug output as well.

Underdoge · 2024-01-19T16:47:08Z

Having the same issue but with the non-Beta cmdlet Get-MgDeviceManagementManagedDevice.

gmenziesint · 2024-01-24T10:52:39Z

Similar issue - Started happening around the 21st / 22nd of Jan, we've had this runbooks for well over six months with no issues until this.

$devices=Get-MgBetaDeviceManagementManagedDevice -Filter "(OperatingSystem eq 'Android' and OperatingSystem eq 'iOS')" -All
Get-MgBetaDeviceManagementManagedDevice_List: Too many retries performed

Nohvah22 · 2024-01-24T14:45:56Z

That is the exact behavior I get and my case is ongoing with no further details unfortunately.

CBSDNetworkTeam · 2024-01-25T14:33:38Z

I also have the exact same issue i have operation scripts that run the Get-MgDeviceManagementManagedDevice -All command. they worked fine upto 1/17/24 on 1/18/24 this command started only pulling 2000 results. when it used to pull 19000+ results.

Get-MgDeviceManagementManagedDevice : Too many retries performed
At line:1 char:1

$ManagedDevices = Get-MgDeviceManagementManagedDevice -All

  + CategoryInfo          : NotSpecified: (:) [Get-MgDeviceManagementManagedDevice_List], InvalidOperationException
  + FullyQualifiedErrorId : Microsoft.Graph.PowerShell.Cmdlets.GetMgDeviceManagementManagedDevice_List

using the old intune powershell module i get this error when i run Get-IntuneManagedDevice | Get-MSGraphAllPages command:

Get-MSGraphNextPage : 504 Gateway Timeout

My Code has not changed. this has worked fine for many months.

I put in a ticket with Microsoft and it hasnt received much response.

Clawz2k · 2024-01-26T17:41:50Z

Getting the same issue.

Tried the following with Graph Explorer:
https://graph.microsoft.com/v1.0/deviceManagement/managedDevices
responds fine.

https://graph.microsoft.com/v1.0/deviceManagement/managedDevices?$filter=operatingSystem eq 'iOS'
fails with the following error:
{
"error": {
"code": "UnknownError",
"message": "{"Message":"{\ \"_version\": 3,\ \"Message\": \"An error has occurred - Operation ID (for customer support): 00000000-0000-0000-0000-000000000000 - Activity ID: ee4e1210-b639-2562-b4b7-73b928aba827 - Url: https://fef.msua01.manage.microsoft.com/TrafficGateway/TrafficRoutingService/DeviceFE/StatelessDeviceFEService/deviceManagement/managedDevices?api-version=2023-07-04&$filter=operatingSystem+eq+%27iOS%27\\\",\\ \"CustomApiErrorPhrase\": \"\",\ \"RetryAfter\": null,\ \"ErrorSourceService\": \"\",\ \"HttpHeaders\": \"{}\"\}"}",
"innerError": {
"date": "2024-01-26T17:40:50",
"request-id": "8b54afd8-d0bf-49b7-8766-93c2edd00af3",
"client-request-id": "ee4e1210-b639-2562-b4b7-73b928aba827"
}
}
}

Microsoft has given us an workaround of using the export API to get the data, but would want to know when this issue has been resolved.
https://learn.microsoft.com/en-us/mem/intune/fundamentals/reports-export-graph-apis

Sample of the code I'm using for the time being.

$Body = @"
{
    "reportName": "Devices",
    "filter":"(CategoryName eq 'UK' or CategoryName eq 'Ireland')",
    "localizationType": "LocalizedValuesAsAdditionalColumn",
    "format": "json",
    "select": [
        "DeviceId",
        "ReferenceId",
        "DeviceName",
        "OSVersion",
        "LastContact",
        "CompliantState",
        "UPN",
        "SerialNumber",
        "UserName",
        "UserEmail",
        "Model",
        "OS"
    ]
}
"@
$ID = (Invoke-MSGraphRequest -Url "/deviceManagement/reports/exportJobs" -HttpMethod POST -Content $Body).id
Sleep 15
$Status = Invoke-MSGraphRequest -Url "/deviceManagement/reports/exportJobs/$ID" -HttpMethod GET
While ($Status.status -ne "completed") {
    Sleep 30
    $Status = Invoke-MSGraphRequest -Url "/deviceManagement/reports/exportJobs/$ID" -HttpMethod GET
}
Invoke-WebRequest $Status.url -OutFile Temp.zip
Sleep 15
Expand-Archive Temp.zip -DestinationPath .
Sleep 15
$Devices = (Get-Content "$($ID).json" | ConvertFrom-Json).values | Where-Object {$_.OS -like "iOS*"} | Select-Object @{N='id';E={$_.DeviceId}},@{N='azureADDeviceId';E={$_.ReferenceId}},DeviceName,OSVersion,@{N='lastSyncDateTime';E={[datetime]$_.LastContact}},@{N='complianceState';E={$_.CompliantState}},@{N='userPrincipalName';E={$_.UPN}},SerialNumber,@{N='userDisplayName';E={$_.UserName}},Model,@{N='emailAddress';E={$_.UserEmail}} | Where-Object {$_.azureADDeviceID -ne "00000000-0000-0000-0000-000000000000"}
Remove-Item Temp.zip
Remove-Item "$($ID).json"

tkalligo · 2024-01-30T14:19:09Z

I am having the same problem with Get-MgDeviceManagementDeviceCompliancePolicyDeviceStatus. I've tried various maxretry values and various pagesizes between 100 and 999 but it did not make a difference. This code has been running since May of 2023 and started having this problem on 1/26. If I try the same query in graph explorer, it errors as well with a generic error.

{
"error": {
"code": "UnknownError",
"message": "",
"innerError": {

jcahudson · 2024-01-30T15:34:54Z

Just tossing another hat in the ring here. I am having the same issue with a script that deletes a list of mobile devices from a CSV in Intune. This script was updated for the new MSGraph module syntax last year, and I noticed it had stopped working on 1/25.

In said script, I pull a list of devices from Intune by running Get-MgDeviceManagementManagedDevice -All then piping that output into a CSV. When that code is run, the following error is shown:
Get-MgDeviceManagementManagedDevice : Too many retries performed

I have tried running the above command with a -Debug added to see what it is doing. I can see it querying devices, but after going through a number of them it stops and errors out.
I also tried setting the maximum retries to 10 using Set-MgRequestContext -MaxRetry 10 but no change.

Briggidy · 2024-01-31T15:58:16Z

I'm also hitting this, but found there is a service advisory

Some admins are unable to utilize the Graph API to fetch Microsoft Intune user devices.
IT711139, Last updated: Jan 31, 2024, 8:03 AM EST
Estimated start time: Jan 18, 2024, 5:58 AM EST

CBSDNetworkTeam · 2024-01-31T17:26:03Z

I'm also hitting this, but found there is a service advisory

Some admins are unable to utilize the Graph API to fetch Microsoft Intune user devices. IT711139, Last updated: Jan 31, 2024, 8:03 AM EST Estimated start time: Jan 18, 2024, 5:58 AM EST

can you link directly to the service advisory? i cannot find it anywhere.

Underdoge · 2024-01-31T22:30:52Z

I'm also hitting this, but found there is a service advisory
Some admins are unable to utilize the Graph API to fetch Microsoft Intune user devices. IT711139, Last updated: Jan 31, 2024, 8:03 AM EST Estimated start time: Jan 18, 2024, 5:58 AM EST

can you link directly to the service advisory? i cannot find it anywhere.

https://admin.microsoft.com/AdminPortal/home#/servicehealth/:/alerts/IT711139/undefined

timayabi2020 · 2024-02-01T16:56:21Z

Hi @nickp85, after doing some investigations, It would be best to log this issue with the service owner.
For API related issues/questions we are not best placed to give an answer. Kindly raise an issue here https://developer.microsoft.com/en-us/graph/support so that the API owner can respond to it.

Nohvah22 · 2024-02-01T17:29:17Z

As of today, my case has been open for two weeks, been escalated to the Intune product group and no details have been provided on the status on their end. Thank goodness our users can still enroll and check in, but our operational automation has been busted since 1/14.

Briggidy · 2024-02-02T16:01:17Z

@Underdoge https://admin.microsoft.com/Adminportal/Home?#/servicehealth/:/alerts/IT711139

the latest status:

Current status: We're continuing to observe and check our applied scale optimizations and monitoring telemetry enhancements, however, due to the complexity of the issue, this is taking longer than expected to produce a timeline for mitigation.

Start time: Monday, January 15, 2024 at 10:00 AM EST

Root cause: The Microsoft Graph API is experiencing timeout issues, which is causing affected admins to see failed queries that indicate too many retries have been performed.

Next update by: Friday, February 2, 2024 at 1:00 PM EST

nkasco · 2024-02-02T19:05:10Z

@Underdoge https://admin.microsoft.com/Adminportal/Home?#/servicehealth/:/alerts/IT711139

the latest status:

Current status: We're continuing to observe and check our applied scale optimizations and monitoring telemetry enhancements, however, due to the complexity of the issue, this is taking longer than expected to produce a timeline for mitigation.

Start time: Monday, January 15, 2024 at 10:00 AM EST

Root cause: The Microsoft Graph API is experiencing timeout issues, which is causing affected admins to see failed queries that indicate too many retries have been performed.

Next update by: Friday, February 2, 2024 at 1:00 PM EST

It seems this might be affecting some WUfB DS Driver related queries too, that then also manifest themselves as never ending syncs in Intune.

It is related to this resource in my case: https://learn.microsoft.com/en-us/graph/api/resources/windowsupdates-applicablecontent?view=graph-rest-beta

Which is called via:
https://learn.microsoft.com/en-us/graph/api/windowsupdates-deploymentaudience-get?view=graph-rest-beta&tabs=http

Perhaps on the backend the APIs are using the get devices queries, but regardless this too manifests itself with the "Invoke-MgGraphRequest : Too many retries performed" error

It does seem to have to do with some level of record count, as this happens in multiple tenants, but only with policies with around 25+ devices in them. The policies with only a few devices seem to work as expected.

Nohvah22 · 2024-02-02T19:06:13Z

As of two hours ago, service has been restored if you want to test.

nkasco · 2024-02-02T19:12:23Z

As of two hours ago, service has been restored if you want to test.

Thanks for the info, just retried and still seems to be an issue for me. Retries are set to 5.

CBSDNetworkTeam · 2024-02-02T19:28:58Z

As of two hours ago, service has been restored if you want to test.

Thanks for the info, just retried and still seems to be an issue for me. Retries are set to 5.

i just tested Get-MgDeviceManagementManagedDevice -all. It initially stalled. then i opened a new powershell console window, authenticated and ran the command again and its pulling 20k+ records without issues now. its very slow though.

nickp85 · 2024-02-05T17:21:50Z

It appears Microsoft made some adjustments to the API and it is now pulling back all records without error.

ericohlin · 2024-02-06T05:39:24Z

Observation... they changed the pagination limit from 1,000 results to 500 per page.

CBSDNetworkTeam · 2024-02-06T13:54:51Z

Observation... they changed the pagination limit from 1,000 results to 500 per page.

Well that explains why this happened. why would they put this limit in? it makes this command run crazy slow now. So before if I had 20k Intune devices I could complete it iterating through 20 pages of 1000 results. Now it has to run iterating through 40 pages. running the "get next page" command every time and waiting for a response. it also ignores the "-PageSize 999" and only gives you 500 results.

tkalligo · 2024-03-28T11:46:56Z

Did anyone see this problem start happening again? It started again on my side two days ago without any changes. It's been fine since it "disappeared" early Feb.

Nohvah22 · 2024-03-28T13:15:47Z

Did anyone see this problem start happening again? It started again on my side two days ago without any changes. It's been fine since it "disappeared" early Feb.

I'm so sorry, I am not seeing it right now. Just ran my scripts successfully. I saw it ONE time last Wednesday... just one time and ran fine a second time. If it's consistent for you, definitely open a case.

ericohlin · 2024-03-28T19:57:16Z

Did anyone see this problem start happening again? It started again on my side two days ago without any changes. It's been fine since it "disappeared" early Feb.

Same as @Nohvah22. I saw it a few times last week, "504 Gateway Timeout", but it went away and hasn't returned.

TheKrisSodroski · 2024-05-31T13:58:05Z

Received this today with a rather innocuous command. I assume the service is just down temporarily?

Get-MgIdentityGovernancePrivilegedAccessGroupEligibilityScheduleRequest -Filter "status eq 'Provisioned'" -Debug

DEBUG: [CmdletBeginProcessing]: - Get-MgIdentityGovernancePrivilegedAccessGroupEligibilityScheduleRequest begin       
processing with parameterSet 'List'.

Confirm
Continue with this operation?
[Y] Yes  [A] Yes to All  [H] Halt Command  [S] Suspend  [?] Help (default is "Y"): a
DEBUG: [Authentication]: - AuthType: 'Delegated', TokenCredentialType: 'InteractiveBrowser', ContextScope:
'CurrentUser', AppName: 'Microsoft Graph Command Line Tools'.
DEBUG: [Authentication]: - Scopes: [Application.ReadWrite.All, Directory.AccessAsUser.All, Directory.ReadWrite.All,   
Group.Read.All, Group.ReadWrite.All, GroupMember.Read.All, GroupMember.ReadWrite.All, Mail.Send, Mail.Send.Shared,    
openid, PrivilegedAccess.Read.AzureADGroup, PrivilegedAccess.ReadWrite.AzureADGroup,
PrivilegedEligibilitySchedule.Read.AzureADGroup, PrivilegedEligibilitySchedule.ReadWrite.AzureADGroup, profile,       
RoleManagement.ReadWrite.Directory, RoleManagementPolicy.Read.AzureADGroup,
RoleManagementPolicy.ReadWrite.AzureADGroup, User.Read, User.Read.All, User.ReadWrite.All, email].
DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://graph.microsoft.com/v1.0/identityGovernance/privilegedAccess/group/eligibilityScheduleRequests?$filter=status 
 eq 'Provisioned'

Headers:
FeatureFlag                   : 00000043
Cache-Control                 : no-store, no-cache
User-Agent                    : Mozilla/5.0,(Windows NT 10.0; Microsoft Windows 10.0.22621;
en-US),PowerShell/5.1.22621.2506
Accept-Encoding               : gzip
SdkVersion                    : graph-powershell/2.12.0
client-request-id             : 8bb86b2a-135c-4d84-980e-f4b7091403d2

Body:


DEBUG: [CmdletException]: Received exception with message 'InvalidOperationException - Too many retries performed :  
  at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.RetryHandler.<SendRetryAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.RetryHandler.<SendAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.CompressionHandler.<SendAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Graph.PowerShell.Authentication.Handlers.AuthenticationHandler.<SendAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at
Microsoft.Graph.PowerShell.IdentityGovernance.<IdentityGovernancePrivilegedAccessGroupListEligibilityScheduleRequest_ 
Call>d__3865.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at
Microsoft.Graph.PowerShell.IdentityGovernance.<IdentityGovernancePrivilegedAccessGroupListEligibilityScheduleRequest_ 
Call>d__3865.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at
Microsoft.Graph.PowerShell.IdentityGovernance.<IdentityGovernancePrivilegedAccessGroupListEligibilityScheduleRequest> 
d__3863.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at
Microsoft.Graph.PowerShell.Cmdlets.GetMgIdentityGovernancePrivilegedAccessGroupEligibilityScheduleRequest_List.<Proce 
ssRecordAsync>d__80.MoveNext()'

Confirm
Too many retries performed
[Y] Yes  [A] Yes to All  [H] Halt Command  [S] Suspend  [?] Help (default is "Y"): a

EmilienCourt · 2024-06-14T16:56:34Z

Hi,

I'm getting this exact issue when calling New-MgBetaSecurityAuditLogQuery.

New-MgBetaSecurityAuditLogQuery -FilterStartDateTime $startDate -FilterEndDateTime $endDate -DisplayName "test" -ErrorAction Stop -Verbose -Debug
DEBUG: [CmdletBeginProcessing]: - New-MgBetaSecurityAuditLogQuery begin processing with parameterSet 'CreateExpanded'.
DEBUG: [Authentication]: - AuthType: 'AppOnly', TokenCredentialType: 'ClientCertificate', ContextScope: 'Process', AppName: '<REDACTED>'.
DEBUG: [Authentication]: - Scopes: [AuditLogsQuery.Read.All, AuditLog.Read.All].

Confirm
Are you sure you want to perform this action?
Performing the operation "New-MgBetaSecurityAuditLogQuery_CreateExpanded" on target "Call remote 'POST /security/auditLog/queries' operation".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): Y
DEBUG: [CmdletException]: Received exception with message 'InvalidOperationException - Too many retries performed :    at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.RetryHandler.SendRetryAsync(HttpResponseMessage response, RetryHandlerOption retryOption, CancellationToken cancellationToken, ActivitySource activitySource)
   at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.RetryHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.CompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Graph.PowerShell.Authentication.Handlers.AuthenticationHandler.SendAsync(HttpRequestMessage httpRequestMessage, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.Graph.Beta.PowerShell.Security.SecurityAuditLogCreateQuery_Call(HttpRequestMessage request, Func`3 on2Xx, Func`3 onDefault, IEventListener eventListener, ISendAsync sender)
   at Microsoft.Graph.Beta.PowerShell.Security.SecurityAuditLogCreateQuery_Call(HttpRequestMessage request, Func`3 on2Xx, Func`3 onDefault, IEventListener eventListener, ISendAsync sender)
   at Microsoft.Graph.Beta.PowerShell.Security.SecurityAuditLogCreateQuery(IDictionary headers, IMicrosoftGraphSecurityAuditLogQuery body, Func`3 on2Xx, Func`3 onDefault, IEventListener eventListener, ISendAsync sender)
   at Microsoft.Graph.Beta.PowerShell.Cmdlets.NewMgBetaSecurityAuditLogQuery_CreateExpanded.ProcessRecordAsync()'
New-MgBetaSecurityAuditLogQuery_CreateExpanded: Too many retries performed

Might be worth re-opening this issue.

(ping @petrhollayms @nickp85)

alpixdotro · 2025-02-03T23:32:59Z

Getting the same issue with:
$Uri = "https://graph.microsoft.com/v1.0/admin/sharepoint/settings"
Invoke-MgGraphRequest -Uri $Uri -Method Patch -Body $updateBody

but also with:
Update-MgAdminSharepointSetting -SharingAllowedDomainList $allowedDomains
Update-MgAdminSharepointSetting_UpdateExpanded: Too many retries performed. More than 3 retries encountered while sending the request. (HTTP request failed with status code: GatewayTimeout.▼♦
T�1♫�0♀@ѻx��MH��☺�ؠ
HEZ�P��0��65H+�I¶↕��N�z�f♥☼�g��♠�Z��$� ��,y��?Q�(&k�@��K�♣G�♦Y�►Y∟F-↓��3�%�B��g;��r▼�.��♠!V▲∟v�#:>(�‼�D.�}(♫�m��♥H��) (HTTP request failed with status code: GatewayTimeout.▼♦
T�1♫�0♀@ѻx��4M��☺�Ġ
HEZ�P��(0��:W�9+D8�[��↔<tY��♠t0��w�♠Y��ɐCCh�l$��BU�/]V�2D��→�F�^t��.♥�!dd▲%��]3�>iY�&b�∟,R2��▲��<��1H�F↕��▼��♥��↨��) (HTTP request failed with status code: GatewayTimeout.▼♦
T�1♫�0♀@ѻx�%7nK��☺\Ǡ
HEZ�P��0��:UH+�↕�˭L�r�f☼�g��♫�R��$��eٻ'�!y$>yNL�"8��|ټ��!☺w↕%�▲E��-��►.◄)�5M eUp��ʂ⌂��♀Y��႑��h�RC∟�♀�m��♥ʭ�S�) (HTTP request failed with status code: GatewayTimeout.▼♦
T�9♫�0►@ѻL��+�{n�2F◄'�"��◄P��☺�1☼��2D8�k�_��☻�,��☺►0��'q��OW�,�B�'��h↑�x��T!☻9%��x�FꂹI��b��e[(��⌂+�F��C�أ�Abn-aR�Qac�☼��♥gс|�)

I've tried reinstalling Microsoft.Graph modules, attempted in both Poweshell 5+ and 7+ as well.
Same things happens if script is run in Azure Automation runbook.

Microsoft.Graph version 2.25 and 2.24 same issue

alpixdotro · 2025-02-04T00:02:52Z

Did any of you guys try this ? Or find a way to make it work?

        $headers = @{
            "Content-Type" = "application/json"
            "Retry-After" = "20"
        }
        # HOLY SHIT THIS WORKS
        
        Invoke-MgGraphRequest -Uri $Uri -Method Patch -Body $updateBody -Headers $headers

Let me know if this header works for you .

L.E: For me it worked for around 300 domains, when I tried with 1260 domains, got the same error

timayabi2020 added the Status: Under Investigation label Jan 29, 2024

timayabi2020 added Service issue Needs: Author Feedback and removed Status: Under Investigation labels Feb 1, 2024

nickp85 closed this as completed Feb 5, 2024

microsoft-github-policy-service bot added Needs: Attention 👋 and removed Needs: Author Feedback labels Feb 5, 2024

petrhollayms removed the Needs: Attention 👋 label May 31, 2024

Too many retries performed when using cmdlets #2534

Too many retries performed when using cmdlets #2534

Comments

nickp85 commented Jan 17, 2024 • edited Loading

SeniorConsulting commented Jan 18, 2024 • edited Loading

nickp85 commented Jan 18, 2024

SeniorConsulting commented Jan 18, 2024

nickp85 commented Jan 18, 2024

SeniorConsulting commented Jan 18, 2024

nickp85 commented Jan 18, 2024

nickp85 commented Jan 18, 2024 • edited Loading

nickp85 commented Jan 18, 2024

SeniorConsulting commented Jan 18, 2024

Nohvah22 commented Jan 19, 2024

Underdoge commented Jan 19, 2024

gmenziesint commented Jan 24, 2024 • edited Loading

Nohvah22 commented Jan 24, 2024

CBSDNetworkTeam commented Jan 25, 2024

Clawz2k commented Jan 26, 2024 • edited Loading

tkalligo commented Jan 30, 2024 • edited Loading

jcahudson commented Jan 30, 2024

Briggidy commented Jan 31, 2024

CBSDNetworkTeam commented Jan 31, 2024

Underdoge commented Jan 31, 2024

timayabi2020 commented Feb 1, 2024

Nohvah22 commented Feb 1, 2024

Briggidy commented Feb 2, 2024

nkasco commented Feb 2, 2024

Nohvah22 commented Feb 2, 2024

nkasco commented Feb 2, 2024

CBSDNetworkTeam commented Feb 2, 2024 • edited Loading

nickp85 commented Feb 5, 2024

ericohlin commented Feb 6, 2024 • edited Loading

CBSDNetworkTeam commented Feb 6, 2024

tkalligo commented Mar 28, 2024

Nohvah22 commented Mar 28, 2024

ericohlin commented Mar 28, 2024

TheKrisSodroski commented May 31, 2024 • edited Loading

EmilienCourt commented Jun 14, 2024 • edited Loading

alpixdotro commented Feb 3, 2025

alpixdotro commented Feb 4, 2025 • edited Loading

nickp85 commented Jan 17, 2024 •

edited

Loading

SeniorConsulting commented Jan 18, 2024 •

edited

Loading

nickp85 commented Jan 18, 2024 •

edited

Loading

gmenziesint commented Jan 24, 2024 •

edited

Loading

Clawz2k commented Jan 26, 2024 •

edited

Loading

tkalligo commented Jan 30, 2024 •

edited

Loading

CBSDNetworkTeam commented Feb 2, 2024 •

edited

Loading

ericohlin commented Feb 6, 2024 •

edited

Loading

TheKrisSodroski commented May 31, 2024 •

edited

Loading

EmilienCourt commented Jun 14, 2024 •

edited

Loading

alpixdotro commented Feb 4, 2025 •

edited

Loading