Merge pull request #4611 from KoenZomers/GetPnPFolderItemLargeList

Optimized `Get-PnPFolderItem`, `Get-PnPFileInFolder` and `Get-PnPFolderInFolder` to support document libries with over 5,000 items

CHANGELOG.md

@@ -43,6 +43,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Added `-Files` parameter for `Send-PnPMail` cmdlet to allow files to be downloaded from SharePoint and then sent as attachments.
 - Added `-Force` parameter to `Set-PnPPropertyBagValue` cmdlet to toggle NoScript status of the site.
 - Added `-Batch` parameter to `Invoke-PnPGraphMethod` cmdlet to allow adding request in a batch.
+- Added `-List` parameter to `Get-PnPFolderItem`, `Get-PnPFileInFolder` and `Get-PnPFolderInFolder` which allows them to work with a document library containing more than 5,000 items [#4611](https://github.com/pnp/powershell/pull/4611)
 
 ### Changed
 

documentation/Get-PnPFileInFolder.md

@@ -24,10 +24,17 @@ Get-PnPFileInFolder [-FolderSiteRelativeUrl <String>] [-ItemName <String>] [-Rec
 Get-PnPFileInFolder [-Identity <FolderPipeBind>] [-ItemName <String>] [-Recurse] [-Includes <String[]>] [-ExcludeSystemFolders] [-Verbose] [-Connection <PnPConnection>] 
 ```
 
+### Folder via list pipebind
+```powershell
+Get-PnPFileInFolder [-List <ListPipeBind>] [-ItemType <String>] [-ItemName <String>] [-Includes <String[]>] [-Verbose] [-Connection <PnPConnection>] 
+```
+
 ## DESCRIPTION
 
 This cmdlet allows listing of all the files in a folder. It can optionally also list all files in the underlying subfolders.
 
+When working with a document library containing over 5,000 items in it, you will need to use the -List parameter to specify the document library in order to be able to retrieve the files in it. It will always recursively retrieve all files in the document library. You can also use -Includes to fetch additional properties of the files.
+
 ## EXAMPLES
 
 ### EXAMPLE 1
@@ -165,6 +172,20 @@ Accept pipeline input: False
 Accept wildcard characters: False
 ```
 
+### -List
+The document library to retrieve the files from. This parameter is required when working with document libraries containing over 5,000 items.
+
+```yaml
+Type: ListPipeBind
+Parameter Sets: Folder via list pipebind
+
+Required: False
+Position: 0
+Default value: None
+Accept pipeline input: True
+Accept wildcard characters: False
+```
+
 ### -Recurse
 A switch parameter to include files of all subfolders in the specified folder
 

documentation/Get-PnPFolderInFolder.md

@@ -24,10 +24,17 @@ Get-PnPFolderInFolder [-FolderSiteRelativeUrl <String>] [-ItemName <String>] [-E
 Get-PnPFolderInFolder [-Identity <FolderPipeBind>] [-ItemName <String>] [-ExcludeSystemFolders] [-Includes <String[]>] [-Recurse] [-Verbose] [-Connection <PnPConnection>] 
 ```
 
+### Folder via list pipebind
+```powershell
+Get-PnPFolderInFolder [-List <ListPipeBind>] [-ItemType <String>] [-ItemName <String>] [-Includes <String[]>] [-Verbose] [-Connection <PnPConnection>] 
+```
+
 ## DESCRIPTION
 
 This cmdlet allows listing of all the subfolders of a folder. It can optionally also list all folders in the underlying subfolders.
 
+When working with a document library containing over 5,000 items in it, you will need to use the -List parameter to specify the document library in order to be able to retrieve the Folders in it. It will always recursively retrieve all folders in the document library. You can also use -Includes to fetch additional properties of the folders.
+
 ## EXAMPLES
 
 ### EXAMPLE 1
@@ -165,6 +172,20 @@ Accept pipeline input: False
 Accept wildcard characters: False
 ```
 
+### -List
+The document library to retrieve the folders from. This parameter is required when working with document libraries containing over 5,000 items.
+
+```yaml
+Type: ListPipeBind
+Parameter Sets: Folder via list pipebind
+
+Required: False
+Position: 0
+Default value: None
+Accept pipeline input: True
+Accept wildcard characters: False
+```
+
 ### -Recurse
 A switch parameter to include folders of all subfolders in the specified folder
 

documentation/Get-PnPFolderItem.md

@@ -19,15 +19,22 @@ List files and/or subfolders in a folder
 Get-PnPFolderItem [-FolderSiteRelativeUrl <String>] [-ItemType <String>] [-ItemName <String>] [-Recursive] [-Verbose] [-Connection <PnPConnection>] 
 ```
 
-### Folder via pipebind
+### Folder via folder pipebind
 ```powershell
 Get-PnPFolderItem [-Identity <FolderPipeBind>] [-ItemType <String>] [-ItemName <String>] [-Recursive] [-Verbose] [-Connection <PnPConnection>] 
 ```
 
+### Folder via list pipebind
+```powershell
+Get-PnPFolderItem [-List <ListPipeBind>] [-ItemType <String>] [-ItemName <String>] [-Includes <string[]>] [-Verbose] [-Connection <PnPConnection>] 
+```
+
 ## DESCRIPTION
 
 This cmdlet allows listing of all the content in a folder. It can be used to list all files and folders in a folder and optionally all its subfolders.
 
+When working with a document library containing over 5,000 items in it, you will need to use the -List parameter to specify the document library in order to be able to retrieve the Files and Folders in it. It will always recursively retrieve all files and folders in the document library. It is still possible to use -ItemType to restrict the results to only files or folders. You can also use -Includes to fetch additional properties of the files and folders. Start the property name with "File." or "Folder." followed by the property name of the file or folder. For example, to include the file version history report, pass in -Includes "File.VersionExpirationReport".
+
 Use [Get-PnPFileInFolder](Get-PnPFileInFolder.md) to retrieve only files and [Get-PnPFolderInFolder](Get-PnPFolderInFolder.md) to retrieve only folders allowing additional properties of the returned items to be requested.
 
 ## EXAMPLES
@@ -154,6 +161,20 @@ Accept pipeline input: False
 Accept wildcard characters: False
 ```
 
+### -List
+The document library to retrieve the files and folders from. This parameter is required when working with document libraries containing over 5,000 items.
+
+```yaml
+Type: ListPipeBind
+Parameter Sets: Folder via list pipebind
+
+Required: False
+Position: 0
+Default value: None
+Accept pipeline input: True
+Accept wildcard characters: False
+```
+
 ### -Recursive
 A switch parameter to include contents of all subfolders in the specified folder
 

src/Commands/Base/GetManagedAppId.cs

@@ -1,18 +1,13 @@
 using System;
 using System.Management.Automation;
-using System.Net;
-using Microsoft.SharePoint.Client;
-using PnP.Framework.Utilities;
-
-using PnP.PowerShell.Commands.Enums;
 
 namespace PnP.PowerShell.Commands.Base
 {
     [Cmdlet(VerbsCommon.Get, "PnPManagedAppId")]
     [OutputType(typeof(PSCredential))]
     public class GetManagedAppId : PSCmdlet
     {
-        [Parameter(Mandatory = true)]
+        [Parameter(Mandatory = true, Position = 0)]
         public string Url;
 
         protected override void ProcessRecord()
@@ -25,7 +20,7 @@ protected override void ProcessRecord()
             }
             else
             {
-                WriteError(new ErrorRecord(new System.Exception("AppId not found"), "APPIDNOTFOUND", ErrorCategory.AuthenticationError, this));
+                WriteError(new ErrorRecord(new Exception("AppId not found"), "APPIDNOTFOUND", ErrorCategory.AuthenticationError, this));
             }
         }
     }

src/Commands/Base/RemoveManagedAppId.cs

@@ -1,14 +1,13 @@
 using System;
 using System.Management.Automation;
 
-
 namespace PnP.PowerShell.Commands.Base
 {
     [Cmdlet(VerbsCommon.Remove, "PnPManagedAppId")]
     [OutputType(typeof(void))]
     public class RemoveManagedAppId : PSCmdlet
     {
-        [Parameter(Mandatory = true)]
+        [Parameter(Mandatory = true, Position = 0)]
         public string Url;
 
         [Parameter(Mandatory = false)]
@@ -17,20 +16,20 @@ public class RemoveManagedAppId : PSCmdlet
         protected override void ProcessRecord()
         {
             Uri uri = new Uri(Url);
-            var appId = Utilities.CredentialManager.GetAppId((uri.ToString()));
+            var appId = Utilities.CredentialManager.GetAppId(uri.ToString());
             if (appId != null)
             {
                 if (Force || ShouldContinue($"Remove App Id: {Url}?", Properties.Resources.Confirm))
                 {
                     if (!Utilities.CredentialManager.RemoveAppid(uri.ToString()))
                     {
-                        WriteError(new ErrorRecord(new System.Exception($"AppId for {Url} not removed"), "APPIDNOTREMOVED", ErrorCategory.WriteError, Url));
+                        WriteError(new ErrorRecord(new Exception($"AppId for {Url} not removed"), "APPIDNOTREMOVED", ErrorCategory.WriteError, Url));
                     }
                 }
             }
             else
             {
-                WriteError(new ErrorRecord(new System.Exception($"AppId not found for {Url}"), "APPIDNOTFOUND", ErrorCategory.ObjectNotFound, Url));
+                WriteError(new ErrorRecord(new Exception($"AppId not found for {Url}"), "APPIDNOTFOUND", ErrorCategory.ObjectNotFound, Url));
             }
         }
     }

src/Commands/Base/SetManagedAppId.cs

@@ -1,17 +1,13 @@
 using System;
 using System.Management.Automation;
-using System.Security;
-using PnP.Framework.Utilities;
-
-using PnP.PowerShell.Commands.Enums;
 
 namespace PnP.PowerShell.Commands.Base
 {
     [Cmdlet(VerbsCommon.Set, "PnPManagedAppId")]
     [OutputType(typeof(void))]
     public class SetManagedAppId : PSCmdlet
     {
-        [Parameter(Mandatory = true)]
+        [Parameter(Mandatory = true, Position = 0)]
         public string Url;
 
         [Parameter(Mandatory = true)]

src/Commands/Files/GetFileInFolder.cs

@@ -1,7 +1,9 @@
 using System;
 using System.Collections.Generic;
 using System.Linq;
+using System.Linq.Expressions;
 using System.Management.Automation;
+using System.Xml.Linq;
 using Microsoft.SharePoint.Client;
 using PnP.Framework.Utilities;
 using PnP.PowerShell.Commands.Base.PipeBinds;
@@ -15,6 +17,7 @@ namespace PnP.PowerShell.Commands.Files
     public class GetFileInFolder : PnPWebRetrievalsCmdlet<File>
     {
         private const string ParameterSet_FOLDERSBYPIPE = "Folder via pipebind";
+        private const string ParameterSet_LISTSBYPIPE = "Folder via list pipebind";
         private const string ParameterSet_FOLDERBYURL = "Folder via url";
 
         [Parameter(Mandatory = false, Position = 0, ValueFromPipeline = true, ParameterSetName = ParameterSet_FOLDERBYURL)]
@@ -23,20 +26,36 @@ public class GetFileInFolder : PnPWebRetrievalsCmdlet<File>
         [Parameter(Mandatory = false, Position = 0, ValueFromPipeline = true, ParameterSetName = ParameterSet_FOLDERSBYPIPE)]
         public FolderPipeBind Identity;
 
+        [Parameter(Mandatory = false, Position = 0, ValueFromPipeline = true, ParameterSetName = ParameterSet_LISTSBYPIPE)]
+        public ListPipeBind List;
+
         [Parameter(Mandatory = false)]
         public string ItemName = string.Empty;
 
-        [Parameter(Mandatory = false)]
-        public SwitchParameter Recurse;
+        [Alias("Recurse")]
+        [Parameter(Mandatory = false, ParameterSetName = ParameterSet_FOLDERBYURL)]
+        [Parameter(Mandatory = false, ParameterSetName = ParameterSet_FOLDERSBYPIPE)]
+        public SwitchParameter Recursive;
 
-        [Parameter(Mandatory = false)]
+        [Parameter(Mandatory = false, ParameterSetName = ParameterSet_FOLDERBYURL)]
+        [Parameter(Mandatory = false, ParameterSetName = ParameterSet_FOLDERSBYPIPE)]
         public SwitchParameter ExcludeSystemFolders;          
 
         protected override void ExecuteCmdlet()
         {
             CurrentWeb.EnsureProperty(w => w.ServerRelativeUrl);
 
-            var contents = GetContents(FolderSiteRelativeUrl);
+            IEnumerable<File> contents = null;
+            if (ParameterSetName == ParameterSet_LISTSBYPIPE)
+            {
+                // Get the files from the list, supporting large lists
+                contents = GetContentsFromDocumentLibrary(List.GetList(CurrentWeb));
+            }
+            else
+            {
+                // Get the files from the file system, not supporting large lists
+                contents = GetContentsByUrl(FolderSiteRelativeUrl);
+            }
 
             if (!string.IsNullOrEmpty(ItemName))
             {
@@ -46,7 +65,48 @@ protected override void ExecuteCmdlet()
             WriteObject(contents, true);
         }
 
-        private IEnumerable<File> GetContents(string FolderSiteRelativeUrl)
+        private IEnumerable<File> GetContentsFromDocumentLibrary(List documentLibrary)
+        {
+            var query = CamlQuery.CreateAllItemsQuery();
+            var queryElement = XElement.Parse(query.ViewXml);
+
+            var rowLimit = new XElement("RowLimit");
+            rowLimit.SetAttributeValue("Paged", "TRUE");
+            rowLimit.SetValue(1000);
+            queryElement.Add(rowLimit);
+
+            query.ViewXml = queryElement.ToString();
+
+            List<File> results = [];
+
+            do
+            {
+                var listItems = documentLibrary.GetItems(query);
+                // Call ClientContext.Load() with and without retrievalExpressions to load FieldValues, otherwise no fields will be loaded (CSOM behavior)
+                ClientContext.Load(listItems);
+                ClientContext.Load(listItems, items => items.Include(item => item.FileSystemObjectType, 
+                                                                    item => item.Id, 
+                                                                    item => item.DisplayName, 
+                                                                    item => item["FileLeafRef"],
+                                                                    item => item["FileRef"], 
+                                                                    item => item.File));
+
+                if(ParameterSpecified(nameof(Includes)))
+                {
+                    var expressions = Includes.Select(i => (Expression<Func<ListItem, object>>)Utilities.DynamicExpression.ParseLambda(typeof(ListItem), typeof(object), $"File.{i}", null)).ToArray();
+                    ClientContext.Load(listItems, items => items.Include(expressions));
+                }
+                ClientContext.ExecuteQueryRetry();
+
+                results.AddRange(listItems.Where(item => item.FileSystemObjectType == FileSystemObjectType.File).Select(item => item.File));
+
+                query.ListItemCollectionPosition = listItems.ListItemCollectionPosition;
+            } while (query.ListItemCollectionPosition != null);
+
+            return results;
+        }
+
+        private IEnumerable<File> GetContentsByUrl(string FolderSiteRelativeUrl)
         {
             Folder targetFolder = null;
             if (string.IsNullOrEmpty(FolderSiteRelativeUrl) && ParameterSetName == ParameterSet_FOLDERSBYPIPE && Identity != null)
@@ -81,7 +141,7 @@ private IEnumerable<File> GetContents(string FolderSiteRelativeUrl)
 
             files = ClientContext.LoadQuery(targetFolder.Files.IncludeWithDefaultProperties(RetrievalExpressions)).OrderBy(f => f.Name);
 
-            if (Recurse)
+            if (Recursive)
             {
                 if(ExcludeSystemFolders.ToBool())
                 {
@@ -96,15 +156,15 @@ private IEnumerable<File> GetContents(string FolderSiteRelativeUrl)
 
             IEnumerable<File> folderContent = files;
 
-            if (Recurse && folders.Count() > 0)
+            if (Recursive && folders.Count() > 0)
             {
                 foreach (var folder in folders)
                 {
                     var relativeUrl = folder.ServerRelativeUrl.Remove(0, CurrentWeb.ServerRelativeUrl.Length);
 
                     WriteVerbose($"Processing folder {relativeUrl}");
 
-                    var subFolderContents = GetContents(relativeUrl);
+                    var subFolderContents = GetContentsByUrl(relativeUrl);
                     folderContent = folderContent.Concat(subFolderContents);
                 }
             }