2.0.0.2

Site settings have been expanded, some functions and dependencies have been changed.
Removed unused elements in UserDataBase, added additional xml fields, added error executor.
Created a basic download function.
Added Instagram saved posts and 429 bypass.
Added channel statistics.
Added site redgifs.
Updated sites algorithms.
Other improvements.
Updated downloader algorithm.

Author: AAndyProgram

CONTRIBUTING.md

@@ -1,6 +1,6 @@
 # Contributor's Guide
 
-I welcome pull requests! Follow these steps to contribute:
+I welcome requests! Follow these steps to contribute:
 
 1. Find an [issue](https://github.com/AAndyProgram/SCrawler/issues) that needs assistance.
 2. Let me know you are working on it by posting a comment on the issue.
@@ -21,11 +21,22 @@ I welcome pull requests! Follow these steps to contribute:
 2. If you don't find anything, create a new issue with your request. I usually reply as soon as possible (within the next few hours).
     - If I'm interested in a site you want to add, it may be added in future releases.
       - If the site has an API that does not require authorization, it may be added in the coming releases.
-      - You can make it faster by posting a link to the API. I don't use OAuth authentication in my application, so if it's not too hard to make a new parsing algorithm without OAuth authorization, I can start developing it in the coming days. Otherwise, I need time to figure out how to do it.
+      - You can make it faster by posting a link to the API. **I don't use OAuth authentication** in my application, so if it's not too hard to make a new parsing algorithm **without OAuth** authorization, I can start developing it in the coming days. Otherwise, I need time to figure out how to do it.
       - If the site does not have an API that does not require authorization, this may take some time.
+	- If you will be posting request urls **without OAuth** authentication, I might consider adding your site if I have time.
     - If I'm **not** interested in the site you want to add, you can pay to have it added by making a donation of approximately $10. **But before that, you still need to create an issue. If I'm not interested, you can offer me a deal to develop it for money. I'll check the site you want to add, check the availability of the API and tell you how much time I need to develop it and the price. If you agree, I will do it.** [![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/andyprogram)
 
 
 # Sites I will never develop
 
 - Facebook
+
+# Sites requested by users
+
+- TikTok
+  - API for receiving data without authorization was not found. Therefore, I don't have time to start developing this site parsing algorithm. If anyone knows of requests that may collect data without OAuth authentication, please let me know.
+  
+# Contact me
+
+[Element messenger](https://element.io/): @andyprogram:matrix.org
+https://matrix.to/#/@andyprogram:matrix.org

Changelog.md

@@ -1,3 +1,25 @@
+# 2.0.0.2
+
+**This is the last release that supports program settings of version 1.0.0.4 and lower. Compatibility of program settings with version 1.0.0.4 and lower will be removed in future releases. It is strongly recommended that you upgrade to this release before future releases. Otherwise, you will have to configure the program settings again. If your program version is 1.0.1.0 or higher, you should not pay attention to this message.**
+
+- Added
+  - Tray icon
+  - Close program to tray
+  - Close confirmation dialog
+  - **Separated thread for downloading Instagram profiles**
+  - **Wait timers to bypass Instagram error "Too Many Requests" (429)**
+  - **Downloading saved Instagram posts** *(requires a second InstaHash)*
+  - Downloading saved posts (from Reddit and Instagram) form
+  - Tray notification when download is complete (Instagram notification separate from other)
+  - Downloading not downloaded Instagram posts when a 429 error is encountered and/or the user stops downloading
+  - Separate progress bar for downloading Instagram profiles
+  - Clear information about downloaded profiles of the current session in the "Download info form"
+  - Increased the number of Instagram posts (from 12 to 50) received per request
+  - Channels' statistics
+  - **RedGisf profiles support**
+- Fixed
+  - The program was showing incorrect information about the total numbers of images and videos downloaded when a Reddit user was created from a channel
+
 # 2.0.0.1
 
 - Added

README.md

@@ -1,6 +1,6 @@
 # Social networks crawler
 
-Program for downloading photo and video from Reddit, Twitter and Instagram
+A program to download photo and video from Reddit, Twitter, Instagram, [etc](#supported-sites).
 
 Do you like this program? Consider adding to my coffee fund by making a donation to show your support. :)
 
@@ -11,23 +11,30 @@ Do you like this program? Consider adding to my coffee fund by making a donation
   - Reddit images;
   - Reddit galleries of images;
   - Redgifs hosted videos (https://www.redgifs.com/);
-  - Reddit hosted videos (downloading Reddit hosted video is going through ffmpeg);
+  - Reddit hosted videos (downloading Reddit hosted video is going through ffmpeg (**ffmpeg only works with the x64 program**));
   - Twitter images and videos;
   - Instagram images and videos.
 - Parse [channel and view data](https://github.com/AAndyProgram/SCrawler/wiki/Channels).
-- Download [saved Reddit posts](https://github.com/AAndyProgram/SCrawler/wiki/Home#saved-posts).
+- Download [saved Reddit and Instagram posts](https://github.com/AAndyProgram/SCrawler/wiki/Home#saved-posts).
 - Add users from parsed channel.
 - Labeling users.
 - Filter exists users by label or group.
 - Selection of media types you want to download (images only, videos only, both)
 
+# Supported sites
+
+- Reddit
+- Twitter
+- Instagram
+- RedGifs
+
 # How does it works:
 
 ## Reddit
 
 The program parses all user posts, obtain MD5  images hash and compares them with existing ones to remove duplicates. Then the media will be downloaded.
 
-## Twitter and Instagram
+## Other sites
 
 The program parses all user posts and compares file names with existing ones to remove duplicates. Then the media will be downloaded.
 
@@ -42,7 +49,7 @@ Read [here](https://github.com/AAndyProgram/SCrawler/blob/main/CONTRIBUTING.md#h
 - Windows 7, 8, 9, 10, 11 with NET Framework 4.6.1 or higher
 - Authorization cookies and tokens for Twitter (if you want to download data from Twitter)
 - Authorization cookies Instagram (if you want to download data from Instagram)
-- ffmpeg library for downloading videos hosted on Reddit (you can download it from the [official repo](https://github.com/GyanD/codexffmpeg/releases/tag/2021-01-12-git-ca21cb1e36) or [from my first release](https://github.com/AAndyProgram/SCrawler/releases/download/1.0.0.0/ffmpeg.zip))
+- ffmpeg library for downloading videos hosted on Reddit (you can download it from the [official repo](https://github.com/GyanD/codexffmpeg/releases/tag/2021-01-12-git-ca21cb1e36) or [from my first release](https://github.com/AAndyProgram/SCrawler/releases/download/1.0.0.0/ffmpeg.zip)). **ffmpeg only works with the x64 version of the program.**
 - **Don't put program in the ```Program Files``` system folder (this is portable program and program settings are stored in the program folder)**
 - **Just unzip the program archive to any folder, copy the file ```ffmpeg.exe``` into it and enjoy. :)**
 
@@ -69,6 +76,7 @@ You can add users by patterns:
 - https://twitter.com/SomeUserName
 - https://reddit.com/user/SomeUserName
 - https://reddit.com/r/SomeSubredditName
+- https://www.redgifs.com/users/SomeUserName
 - u/SomeUserName
 - r/SomeSubredditName
 - SomeUserName (in this case, you need to select the user's site)
@@ -83,3 +91,8 @@ Read more about adding users and subreddits [here](https://github.com/AAndyProgr
 Create a shortcut for the program. Open shortcut properties. In the ```Shortcut``` tab, in the ```Target``` field, just add the letter ```v``` at the end across the space.
 
 Example: ```D:\Programs\SCrawler\SCrawler.exe v```
+
+# Contact me
+
+[Element messenger](https://element.io/): @andyprogram:matrix.org
+https://matrix.to/#/@andyprogram:matrix.org

SCrawler.sln

@@ -11,13 +11,8 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution
 	ProjectSection(SolutionItems) = preProject
 		.gitignore = .gitignore
 		Changelog.md = Changelog.md
-		Info\InstaAlgo.txt = Info\InstaAlgo.txt
-		Info\InstagramInfo.txt = Info\InstagramInfo.txt
 		README.md = README.md
-		Info\RedditUrlsInfo.txt = Info\RedditUrlsInfo.txt
 		ToDo.txt = ToDo.txt
-		Info\TwitterNewAlgo.txt = Info\TwitterNewAlgo.txt
-		Info\TwitterUrlsInfo.txt = Info\TwitterUrlsInfo.txt
 	EndProjectSection
 EndProject
 Global

SCrawler/API/Base/SiteSettings.vb

@@ -25,7 +25,9 @@ Namespace API.Base
                 _Path.Value = NewFile
             End Set
         End Property
+#Region "Instagram"
         Friend ReadOnly Property InstaHash As XMLValue(Of String)
+        Friend ReadOnly Property InstaHash_SP As XMLValue(Of String)
         Friend ReadOnly Property InstaHashUpdateRequired As XMLValue(Of Boolean)
         Friend ReadOnly Property InstagramDownloadingErrorDate As XMLValue(Of Date)
         Friend Property InstagramLastApplyingValue As Integer? = Nothing
@@ -40,7 +42,18 @@ Namespace API.Base
                 End With
             End Get
         End Property
-        Friend Property InstagramTooManyRequestsReadyForCatch As Boolean = True
+        Friend ReadOnly Property InstagramLastDownloadDate As XMLValue(Of Date)
+        Friend ReadOnly Property InstagramLastRequestsCount As XMLValue(Of Integer)
+        Private InstagramTooManyRequestsReadyForCatch As Boolean = True
+        Friend Function GetInstaWaitDate() As Date
+            With InstagramDownloadingErrorDate
+                If .ValueF.Exists Then
+                    Return .ValueF.Value.AddMinutes(If(InstagramLastApplyingValue, 10))
+                Else
+                    Return Now
+                End If
+            End With
+        End Function
         Friend Sub InstagramTooManyRequests(ByVal Catched As Boolean)
             With InstagramDownloadingErrorDate
                 If Catched Then
@@ -55,9 +68,14 @@ Namespace API.Base
                 Else
                     .ValueF = Nothing
                     InstagramLastApplyingValue = Nothing
+                    InstagramTooManyRequestsReadyForCatch = True
                 End If
             End With
         End Sub
+        Friend ReadOnly Property RequestsWaitTimer As XMLValue(Of Integer)
+        Friend ReadOnly Property RequestsWaitTimerTaskCount As XMLValue(Of Integer)
+        Friend ReadOnly Property SleepTimerOnPostsLimit As XMLValue(Of Integer)
+#End Region
         Friend ReadOnly Property Temporary As XMLValue(Of Boolean)
         Friend ReadOnly Property DownloadImages As XMLValue(Of Boolean)
         Friend ReadOnly Property DownloadVideos As XMLValue(Of Boolean)
@@ -98,6 +116,7 @@ Namespace API.Base
                         Responser.CookiesDomain = "reddit.com"
                         Responser.Decoders.Add(SymbolsConverter.Converters.Unicode)
                     Case Sites.Instagram : Responser.CookiesDomain = "instagram.com"
+                    Case Sites.RedGifs : Responser.CookiesDomain = "redgifs.com"
                 End Select
                 Responser.SaveSettings()
             End If
@@ -126,20 +145,30 @@ Namespace API.Base
                 GetUserMediaOnly = New XMLValue(Of Boolean)
             End If
 
+            CreateProp(InstaHashUpdateRequired, Sites.Instagram, "InstaHashUpdateRequired", True, _XML, n)
+            CreateProp(InstaHash, Sites.Instagram, "InstaHash", String.Empty, _XML, n)
+            If Site = Sites.Instagram AndAlso (InstaHash.IsEmptyString Or InstaHashUpdateRequired) AndAlso Responser.Cookies.ListExists Then GatherInstaHash()
+            CreateProp(InstaHash_SP, Sites.Instagram, "InstaHashSavedPosts", String.Empty, _XML, n)
+            CreateProp(InstagramLastDownloadDate, Sites.Instagram, "LastDownloadDate", Now.AddDays(-1), _XML, n)
+            CreateProp(InstagramLastRequestsCount, Sites.Instagram, "LastRequestsCount", 0, _XML, n)
+            CreateProp(RequestsWaitTimer, Sites.Instagram, "RequestsWaitTimer", 1000, _XML, n)
+            CreateProp(RequestsWaitTimerTaskCount, Sites.Instagram, "RequestsWaitTimerTaskCount", 1, _XML, n)
+            CreateProp(SleepTimerOnPostsLimit, Sites.Instagram, "SleepTimerOnPostsLimit", 60000, _XML, n)
             If Site = Sites.Instagram Then
-                InstaHash = New XMLValue(Of String)("InstaHash", String.Empty, _XML, n)
-                InstaHashUpdateRequired = New XMLValue(Of Boolean)("InstaHashUpdateRequired", True, _XML, n)
-                If (InstaHash.IsEmptyString Or InstaHashUpdateRequired) And Responser.Cookies.ListExists Then GatherInstaHash()
                 InstagramDownloadingErrorDate = New XMLValue(Of Date) With {.ToStringFunction = Function(ss, vv) AConvert(Of String)(vv, Nothing)}
                 InstagramDownloadingErrorDate.SetExtended("InstagramDownloadingErrorDate", Now.AddYears(-10), _XML, n)
             Else
-                InstaHash = New XMLValue(Of String)
-                InstaHashUpdateRequired = New XMLValue(Of Boolean)
+                InstagramDownloadingErrorDate = New XMLValue(Of Date)
             End If
-            If Site = Sites.Reddit Then
-                SavedPostsUserName = New XMLValue(Of String)("SavedPostsUserName", String.Empty, _XML, n)
+
+            SavedPostsUserName = New XMLValue(Of String)("SavedPostsUserName", String.Empty, _XML, n)
+        End Sub
+        Private Sub CreateProp(Of T)(ByRef p As XMLValue(Of T), ByVal s As Sites,
+                                     ByVal p_Name As String, ByVal p_Value As T, ByRef x As XmlFile, ByVal n() As String)
+            If Site = s Then
+                p = New XMLValue(Of T)(p_Name, p_Value, x, n)
             Else
-                SavedPostsUserName = New XMLValue(Of String)
+                p = New XMLValue(Of T)
             End If
         End Sub
         Friend Sub Update()

SCrawler/API/Base/Structures.vb

@@ -85,5 +85,21 @@ Namespace API.Base
                 Return v
             End Function
         End Structure
+        Friend Structure Sizes : Implements IComparable(Of Sizes)
+            Friend Value As Integer
+            Friend Data As String
+            Friend ReadOnly HasError As Boolean
+            Friend Sub New(ByVal _Value As String, ByVal _Data As String)
+                Try
+                    Value = _Value
+                    Data = _Data
+                Catch ex As Exception
+                    HasError = True
+                End Try
+            End Sub
+            Friend Function CompareTo(ByVal Other As Sizes) As Integer Implements IComparable(Of Sizes).CompareTo
+                Return Value.CompareTo(Other.Value) * -1
+            End Function
+        End Structure
     End Module
 End Namespace