2.0.0.0

Added Instagram downloading, filter by site, channels groups, change folder function, imgur compatibility, special folders, deleting with keeping data, Reddit saved posts downloading
Fixed limited twitter downloading, suspended profiles
Updated download algo
Concat sites editors into a single form
Updated Reddit downloading algo
Fixed saved function in video downloader
Some improvements

Author: AAndyProgram

Changelog.md

@@ -1,3 +1,20 @@
+# 2.0.0.0
+
+- Added
+  - **Instagram**
+  - Filter by site
+  - Group for regular channels in the main window
+  - Ability to change user/collection path
+  - Imgur albums downloading
+  - NSFW Imgur content bypass (requires 'CilentID')
+  - Special user folder
+  - Remove user while keeping data
+  - Disabled overriding user preferences when creating a new user if it already exists in the destination (in case of deleting a user with saving data).
+  - **Saved Reddit posts downloading**
+- Fixed
+  - Suspended profiles do not change status if the profile is no longer suspended
+  - Limited download for Twitter not implemented
+
 # 1.0.1.0
 
 - Added

README.md

@@ -1,8 +1,8 @@
 # Social networks crawler
 
-Program for downloading photo and video from Reddit and Twitter
+Program for downloading photo and video from Reddit, Twitter and Instagram
 
-Enjoying the tool? Considering adding to my coffee fund :)
+Do you like this program? Consider adding to my coffee fund by making a donation to show your support. :)
 
 [![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/andyprogram)
 
@@ -12,9 +12,10 @@ Enjoying the tool? Considering adding to my coffee fund :)
   - Reddit galleries of images;
   - Redgifs hosted videos (https://www.redgifs.com/);
   - Reddit hosted videos (downloading Reddit hosted video is going through ffmpeg);
-  - Twitter images;
-  - Twitter videos.
-- Parse channel and view data.
+  - Twitter images and videos;
+  - Instagram images and videos.
+- Parse [channel and view data](https://github.com/AAndyProgram/SCrawler/wiki/Channels).
+- Download [saved Reddit posts](https://github.com/AAndyProgram/SCrawler/wiki/Home#saved-posts).
 - Add users from parsed channel.
 - Labeling users.
 - Filter exists users by label or group.
@@ -24,41 +25,49 @@ Enjoying the tool? Considering adding to my coffee fund :)
 
 ## Reddit
 
-The program parsing all user's posts, gathering pictures' MD5 hash and compare with existing for remove duplicates. Then media will be downloaded.
+The program parses all user posts, obtain MD5  images hash and compares them with existing ones to remove duplicates. Then the media will be downloaded.
 
-## Twitter
+## Twitter and Instagram
 
-The program parsing all user's posts and compare file names with existing for remove duplicates. Then media will be downloaded.
+The program parses all user posts and compares file names with existing ones to remove duplicates. Then the media will be downloaded.
+
+You can read about Instagram restrictions [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram-limits)
 
 # Requirements:
 
 - Windows 7, 8, 9, 10, 11 with NET Framework 4.6.1 or higher
 - Authorization cookies and tokens for Twitter (if you want to download data from Twitter)
-- ffmpeg library for download Reddit hosted videos (you can download it from the [official repo](https://github.com/GyanD/codexffmpeg/releases/tag/2021-01-12-git-ca21cb1e36) or [from my first release](https://github.com/AAndyProgram/SCrawler/releases/download/1.0.0.0/ffmpeg.zip))
+- Authorization cookies Instagram (if you want to download data from Instagram)
+- ffmpeg library for downloading videos hosted on Reddit (you can download it from the [official repo](https://github.com/GyanD/codexffmpeg/releases/tag/2021-01-12-git-ca21cb1e36) or [from my first release](https://github.com/AAndyProgram/SCrawler/releases/download/1.0.0.0/ffmpeg.zip))
 - Don't put program in the ```Program Files``` system folder (this is portable program and program settings are stored in the program folder)
-- Just unpack program archive in any folder you want, copy ```ffmpeg.exe``` into and enjoy. :-)
+- Just unzip the program archive to any folder, copy the file ```ffmpeg.exe``` into it and enjoy. :)
+
+# Updating
+
+Just download [latest](https://github.com/AAndyProgram/SCrawler/releases/latest) version and unpack it into the program folder. Before starting a new version, I recommend making a backup copy of the program settings folder.
 
 # Settings and usage
 
 The program has an intuitive interface.
 
-Just add user profile and press ```Start downloading``` button.
+Just add a user profile and click the ```Start downloading``` button.
 
-Users can be added by patterns:
+You can add users by patterns:
+- https://www.instagram.com/SomeUserName
 - https://twitter.com/SomeUserName
 - https://reddit.com/user/SomeUserName
 - https://reddit.com/r/SomeSubredditName
 - u/SomeUserName
 - r/SomeSubredditName
-- SomeUserName (in this case you must to choose user site)
+- SomeUserName (in this case, you need to select the user's site)
 - SomeSubredditName
 
-More about users and subreddits adding [here](https://github.com/AAndyProgram/SCrawler/wiki/Users)
+Read more about adding users and subreddits [here](https://github.com/AAndyProgram/SCrawler/wiki/Users)
 
 **Full guide you can find [here](https://github.com/AAndyProgram/SCrawler/wiki)**
 
 ## Using program as just video downloader
 
-Create a shortcut for the program. Open shortcut properties. On the ```Shortcut``` tab in ```Target``` field just add ```v``` at the end through the space.
+Create a shortcut for the program. Open shortcut properties. In the ```Shortcut``` tab, in the ```Target``` field, just add the letter ```v``` at the end across the space.
 
 Example: ```D:\Programs\SCrawler\SCrawler.exe v```

SCrawler.sln

@@ -11,6 +11,8 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution
 	ProjectSection(SolutionItems) = preProject
 		.gitignore = .gitignore
 		Changelog.md = Changelog.md
+		Info\InstaAlgo.txt = Info\InstaAlgo.txt
+		Info\InstagramInfo.txt = Info\InstagramInfo.txt
 		README.md = README.md
 		Info\RedditUrlsInfo.txt = Info\RedditUrlsInfo.txt
 		ToDo.txt = ToDo.txt

SCrawler/API/Base/Declarations.vb

@@ -0,0 +1,13 @@
+' Copyright (C) 2022  Andy
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Namespace API.Base
+    Friend Module Declarations
+        Friend ReadOnly LNC As New ListAddParams(LAP.NotContainsOnly)
+    End Module
+End Namespace

SCrawler/API/Base/SiteSettings.vb

@@ -25,49 +25,160 @@ Namespace API.Base
                 _Path.Value = NewFile
             End Set
         End Property
+        Friend ReadOnly Property InstaHash As XMLValue(Of String)
+        Friend ReadOnly Property InstaHashUpdateRequired As XMLValue(Of Boolean)
+        Friend ReadOnly Property InstagramDownloadingErrorDate As XMLValue(Of Date)
+        Friend Property InstagramLastApplyingValue As Integer? = Nothing
+        Friend ReadOnly Property InstagramReadyForDownload As Boolean
+            Get
+                With InstagramDownloadingErrorDate
+                    If .ValueF.Exists Then
+                        Return .ValueF.Value.AddMinutes(If(InstagramLastApplyingValue, 10)) < Now
+                    Else
+                        Return True
+                    End If
+                End With
+            End Get
+        End Property
+        Friend Property InstagramTooManyRequestsReadyForCatch As Boolean = True
+        Friend Sub InstagramTooManyRequests(ByVal Catched As Boolean)
+            With InstagramDownloadingErrorDate
+                If Catched Then
+                    If Not .ValueF.Exists Then
+                        .Value = Now
+                        If InstagramTooManyRequestsReadyForCatch Then
+                            InstagramLastApplyingValue = If(InstagramLastApplyingValue, 0) + 10
+                            InstagramTooManyRequestsReadyForCatch = False
+                            MyMainLOG = $"Instagram downloading error: too many requests. Try again after {If(InstagramLastApplyingValue, 10)} minutes..."
+                        End If
+                    End If
+                Else
+                    .ValueF = Nothing
+                    InstagramLastApplyingValue = Nothing
+                End If
+            End With
+        End Sub
+        Friend ReadOnly Property Temporary As XMLValue(Of Boolean)
+        Friend ReadOnly Property DownloadImages As XMLValue(Of Boolean)
+        Friend ReadOnly Property DownloadVideos As XMLValue(Of Boolean)
+        Friend ReadOnly Property GetUserMediaOnly As XMLValue(Of Boolean)
+        Friend ReadOnly Property SavedPostsUserName As XMLValue(Of String)
         Private ReadOnly SettingsFile As SFile
-        Friend Sub New(ByVal s As Sites, ByRef _XML As XmlFile, ByVal GlobalPath As SFile)
+        Friend Sub New(ByVal s As Sites, ByRef _XML As XmlFile, ByVal GlobalPath As SFile,
+                       ByRef _Temp As XMLValue(Of Boolean), ByRef _Imgs As XMLValue(Of Boolean), ByRef _Vids As XMLValue(Of Boolean))
             Site = s
             SettingsFile = $"{SettingsFolderName}\Responser_{s}.xml"
             Responser = New WEB.Response(SettingsFile)
 
             If SettingsFile.Exists Then
                 Responser.LoadSettings()
             Else
-                If Site = Sites.Twitter Then
-                    With Responser
-                        .ContentType = "application/json"
-                        .Accept = "*/*"
-                        .CookiesDomain = "twitter.com"
-                        .Decoders.Add(SymbolsConverter.Converters.Unicode)
-                        With .Headers
-                            .Add("sec-ch-ua", " Not;A Brand" & Chr(34) & ";v=" & Chr(34) & "99" & Chr(34) & ", " & Chr(34) &
-                                 "Google Chrome" & Chr(34) & ";v=" & Chr(34) & "91" & Chr(34) & ", " & Chr(34) & "Chromium" &
-                                 Chr(34) & ";v=" & Chr(34) & "91" & Chr(34))
-                            .Add("sec-ch-ua-mobile", "?0")
-                            .Add("sec-fetch-dest", "empty")
-                            .Add("sec-fetch-mode", "cors")
-                            .Add("sec-fetch-site", "same-origin")
-                            .Add(Header_Twitter_Token, String.Empty)
-                            .Add("x-twitter-active-user", "yes")
-                            .Add("x-twitter-auth-type", "OAuth2Session")
-                            .Add(Header_Twitter_Authorization, String.Empty)
+                Select Case Site
+                    Case Sites.Twitter
+                        With Responser
+                            .ContentType = "application/json"
+                            .Accept = "*/*"
+                            .CookiesDomain = "twitter.com"
+                            .Decoders.Add(SymbolsConverter.Converters.Unicode)
+                            With .Headers
+                                .Add("sec-ch-ua", " Not;A Brand" & Chr(34) & ";v=" & Chr(34) & "99" & Chr(34) & ", " & Chr(34) &
+                                     "Google Chrome" & Chr(34) & ";v=" & Chr(34) & "91" & Chr(34) & ", " & Chr(34) & "Chromium" &
+                                     Chr(34) & ";v=" & Chr(34) & "91" & Chr(34))
+                                .Add("sec-ch-ua-mobile", "?0")
+                                .Add("sec-fetch-dest", "empty")
+                                .Add("sec-fetch-mode", "cors")
+                                .Add("sec-fetch-site", "same-origin")
+                                .Add(Header_Twitter_Token, String.Empty)
+                                .Add("x-twitter-active-user", "yes")
+                                .Add("x-twitter-auth-type", "OAuth2Session")
+                                .Add(Header_Twitter_Authorization, String.Empty)
+                            End With
                         End With
-                    End With
-                ElseIf Site = Sites.Reddit Then
-                    Responser.CookiesDomain = "reddit.com"
-                    Responser.Decoders.Add(SymbolsConverter.Converters.Unicode)
-                End If
+                    Case Sites.Reddit
+                        Responser.CookiesDomain = "reddit.com"
+                        Responser.Decoders.Add(SymbolsConverter.Converters.Unicode)
+                    Case Sites.Instagram : Responser.CookiesDomain = "instagram.com"
+                End Select
                 Responser.SaveSettings()
             End If
-            _Path = New XMLValue(Of SFile)("Path", SFile.GetPath($"{GlobalPath.PathWithSeparator}{Site}"),
-                                           _XML, {SettingsCLS.Name_Node_Sites, Site.ToString}, XMLValue(Of SFile).ToFilePath)
+
+            Dim n() As String = {SettingsCLS.Name_Node_Sites, Site.ToString}
+            _Path = New XMLValue(Of SFile)("Path", SFile.GetPath($"{GlobalPath.PathWithSeparator}{Site}"), _XML, n, XMLValue(Of SFile).ToFilePath)
             _Path.ReplaceByValue("Path", {Site.ToString})
             _XML.Remove(Site.ToString)
+
+            Temporary = New XMLValue(Of Boolean)
+            Temporary.SetExtended("Temporary", False, _XML, n)
+            Temporary.SetDefault(_Temp)
+
+            DownloadImages = New XMLValue(Of Boolean)
+            DownloadImages.SetExtended("DownloadImages", True, _XML, n)
+            DownloadImages.SetDefault(_Imgs)
+
+            DownloadVideos = New XMLValue(Of Boolean)
+            DownloadVideos.SetExtended("DownloadVideos", True, _XML, n)
+            DownloadVideos.SetDefault(_Vids)
+
+            If Site = Sites.Twitter Then
+                GetUserMediaOnly = New XMLValue(Of Boolean)("GetUserMediaOnly", True, _XML, n)
+                GetUserMediaOnly.ReplaceByValue("TwitterDefaultGetUserMedia", n)
+            Else
+                GetUserMediaOnly = New XMLValue(Of Boolean)
+            End If
+
+            If Site = Sites.Instagram Then
+                InstaHash = New XMLValue(Of String)("InstaHash", String.Empty, _XML, n)
+                InstaHashUpdateRequired = New XMLValue(Of Boolean)("InstaHashUpdateRequired", True, _XML, n)
+                If (InstaHash.IsEmptyString Or InstaHashUpdateRequired) And Responser.Cookies.ListExists Then GatherInstaHash()
+                InstagramDownloadingErrorDate = New XMLValue(Of Date) With {.ToStringFunction = Function(ss, vv) AConvert(Of String)(vv, Nothing)}
+                InstagramDownloadingErrorDate.SetExtended("InstagramDownloadingErrorDate", Now.AddYears(-10), _XML, n)
+            Else
+                InstaHash = New XMLValue(Of String)
+                InstaHashUpdateRequired = New XMLValue(Of Boolean)
+            End If
+            If Site = Sites.Reddit Then
+                SavedPostsUserName = New XMLValue(Of String)("SavedPostsUserName", String.Empty, _XML, n)
+            Else
+                SavedPostsUserName = New XMLValue(Of String)
+            End If
         End Sub
         Friend Sub Update()
             Responser.SaveSettings()
         End Sub
+        Friend Function GatherInstaHash() As Boolean
+            Try
+                Dim rs As New RegexStructure("=" & Chr(34) & "([^" & Chr(34) & "]+?ConsumerLibCommons[^" & Chr(34) & "]+?.js)" & Chr(34), 1) With {
+                    .UseTimeOut = True,
+                    .MatchTimeOutSeconds = 10
+                }
+                Dim r$ = Responser.GetResponse("https://instagram.com",, EDP.ThrowException)
+                If Not r.IsEmptyString Then
+                    Dim hStr$ = RegexReplace(r, rs)
+                    If Not hStr.IsEmptyString Then
+                        Do While Left(hStr, 1) = "/" : hStr = Right(hStr, hStr.Length - 1) : Loop
+                        hStr = $"https://instagram.com/{hStr}"
+                        r = Responser.GetResponse(hStr,, EDP.ThrowException)
+                        If Not r.IsEmptyString Then
+                            rs = New RegexStructure("generatePaginationActionCreators.+?.profilePosts.byUserId.get.+?queryId:.([\d\w\S]+?)" & Chr(34), 1) With {
+                                .UseTimeOut = True,
+                                .MatchTimeOutSeconds = 10
+                            }
+                            Dim h$ = RegexReplace(r, rs)
+                            If Not h.IsEmptyString Then
+                                InstaHash.Value = h
+                                InstaHashUpdateRequired.Value = False
+                                Return True
+                            End If
+                        End If
+                    End If
+                End If
+                Return False
+            Catch ex As Exception
+                InstaHashUpdateRequired.Value = True
+                InstaHash.Value = String.Empty
+                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, "[SiteSettings.GaterInstaHash]", False)
+            End Try
+        End Function
 #Region "IDisposable Support"
         Private disposedValue As Boolean = False
         Protected Overridable Overloads Sub Dispose(ByVal disposing As Boolean)