diff --git a/docs/changelog.md b/docs/changelog.md index 830c08437..4421508d2 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -7,6 +7,40 @@ title: Changelog !!! note This is the new changelog, only the most recent builds. For all versions, see the [old changelog](old_changelog.html). +## [Version 584](https://github.com/hydrusnetwork/hydrus/releases/tag/v584) + +### misc + +* fixed a logical hole in the recent 'is this URL that is saying (deleted/already in db) trustworthy, or does it have weird mappings to other files?' pre-download check that was causing Pixiv, Kemono, and Twitter and any other multiple-URL Post URL Class to, on re-encountering the URL in a downloader, classify the underlying file URL as untrustworthy and re-download the files(l!!) +* the 'copy all' and paste buttons in the manage known urls dialog are replaced with icon buttons, and the copy button now copies the current selection if there is one +* the newish Regex input widget (the one that goes green/red based on current text validity) now propagates an Enter key into a dialog ok event when appropriate +* when you ctrl+double-click a taglist, the program now ensures the item under the mouse is selected before firing off the double-click nega-activation event. this is slightly awkward, but I hope it smoothes out the awkward moment where you want to invert a selection of tags but doing a normal ctrl+double-click on them causes the one of them to be deselected and then messes up the selection +* regex URL searches are now always the last job to run in a file query. if you mix in any other predicate like filesize or just some tag, your regex URL searches should run massively massively faster +* improved some boot error handling when Qt fails to import +* fixed the whack alignment of the 'filename'/'first directory'/etc.. checkbox-and-text-edit widgets in the filename tagging panel, and set 'namespace' placeholder text +* force-selecting a null value in a 'select one from this list of things' dialog should no longer raise errors +* thanks to a user, the new shimmie parser gets tags in a simpler, more reliable way + +### client api + +* added a new permission, `Commit Pending` (12), which allows you to see and commit pending content for each service +* added `/manage_services/get_pending_counts`, which basically returns the content of the client's 'pending' menu +* added `/manage_services/commit_pending`, which fires those commands off +* added `/manage_services/forget_pending`, which does the same 'forget' command on that menu +* added `/manage_file_relationships/remove_potentials`, which clears any known potential pairs off the given files +* the `/manage_pages/get_pages` and `/manage_pages/get_page_info` commands now return `is_media_page` boolean, which is a simple shorthand for 'not a page of pages' +* added the above to the Client API help +* wrote unit tests covering the above +* the client api version is now 66 + +### boring cleanup + +* fixed up how some lists deliver their underlying data to various methods +* `CallBlockingToQt` no longer spams encountered errors to the log--proper error handling should (and does now) occur elsewhere +* the way the initial focus is set on system predicate flesh-out panels (when you double-click on like 'system:dimensions' and get a bunch of sub-panels) is more sane. should be, fairly reliably, on the first editable panel's ok button. I want it to be on an editable widget in the panel in future, I think, but I need to do some behind the scenes stuff to make this work in a nicer way +* pulled some stuff out of `HydrusData`, mostly to `HydrusNumbers`, `HydrusLists`, and the new `HydrusProcess`, mostly for decoupling purposes +* renamed some `ConvertXToY` stuff to just `XToY` + ## [Version 583](https://github.com/hydrusnetwork/hydrus/releases/tag/v583) ### new @@ -322,34 +356,3 @@ title: Changelog * moved `ClientDuplicates` up to a new `duplicates` module and migrated some duplicate enums over to it from `ClientConstants` * removed an old method-wrapper hack that applied the 'load images with PIL' option. I just moved to a global that I set on init and update on options change * cleaned some duplicate checking code - -## [Version 574](https://github.com/hydrusnetwork/hydrus/releases/tag/v574) - -### local hashes cache - -* we finally figured out the 'update 404' issue that some PTR-syncing users were getting, where PTR processing would halt with an error about an update file not being available on the server. long story short, SQLite was sometimes crossing a wire in the database on a crash, and this week I add some new maintenance code to fix this and catch it in future -* the local hash cache has a bunch of new resync/recovery code. it can now efficiently recover from missing hash_ids, excess hash_ids, desynced hash_ids, and even repopulate the master hash table if that guy has missing hash_ids (which can happen after severe db damage due to hard drive failure). it records all recovery info to the log -* the normal _database->regenerate->local hashes cache_ function now works entirely in this new resync code, making it significantly faster (previously it just deleted and re-added everything). this job also gets a nicer popup with a summary of any problems found -* when the client recovers from a bad shutdown, it now runs a quick sync on the latest hash_ids added to the local hashes cache to ensure that desync did not occur. fingers crossed, this will work super fast and ensure that we don't get the 404 problem (or related hash_id cross-wire problems) again -* on repository processing failure and a scheduling of update file maintenance, we now resync the update files in the local hash cache, meaning the 404 problem, if it does happen again, will now fix itself in the normal recovery code -* on update, everyone is going to get a full local hash cache resync, just to catch any lingering issues here. it should now work super fast! -* fixed an issue where the local hash and tags caches would not fully reset desynced results on a 'regenerate' call until a client restart - -### misc - -* thanks to a user, the default twitter downloader I added last week now gets full-size images. if you spammed a bunch of URLs last week, I apologise: please do a search for 'imported within the last 7 days/has a twitter url/height=1200px' and then copy/paste the results' tweet URLs into a new urls downloader. because of some special twitter settings, you shouldn't have to set 'download the file even if known url match' in the file import options; the downloader will discover the larger versions and download the full size files with no special settings needed. once done, assuming the file count is the same on both pages, go back to your first page and delete the 1200px tall files. then repeat for width=1200px! -* the filetype selector in system:filetype now expands to eat extra vertical space if the dialog is resized -* the filetype selector in file import options is moved a bit and also now expands to eat extra vertical space -* thanks to a user, the Microsoft document recognition now has fewer false negatives (it was detecting some docs as zips) -* when setting up an import folder, the dialog will now refuse to OK if you set a path that is 1) above the install dir or db dir or 2) above or below any of your file storage locations. shouldn't be possible to set up an import from your own file storage folder by accident any more -* added a new 'apply image ICC Profile colour adjustments' checkbox to _options->media_. this simply turns off ICC profile loading and application, for debug purposes - -### boring cleanup - -* the default SQLite page size is now 4096 bytes on Linux, the SQLite default. it was 1024 previously, but SQLite now recommend 4096 for all platforms. the next time Linux users vacuum any of their databases, they will get fixed. I do not think this is a big deal, so don't rush to force this -* fixed the last couple dozen missing layout flags across the program, which were ancient artifacts from the wx->Qt conversion -* fixed the WTFPL licence to be my copyright, lol -* deleted the local booru service management/UI code -* deleted the local booru service db/init code -* deleted the local booru service network code -* on update, the local booru service will be deleted from the database diff --git a/docs/developer_api.md b/docs/developer_api.md index 76edd06c7..d70c861fc 100644 --- a/docs/developer_api.md +++ b/docs/developer_api.md @@ -13,7 +13,7 @@ hide: navigation ## API -In general, the API deals with standard UTF-8 JSON. POST requests and 200 OK responses are generally going to be a JSON 'Object' with variable names as keys and values obviously as values. There are examples throughout this document. For GET requests, everything is in standard GET parameters, but some variables are complicated and will need to be JSON encoded and then URL encoded. An example would be the 'tags' parameter on [GET /get\_files/search\_files](#get_files_search_files), which is a list of strings. Since GET http URLs have limits on what characters are allowed, but hydrus tags can have all sorts of characters, you'll be doing this: +In general, the API deals with standard UTF-8 JSON. POST requests and 200 OK responses are generally going to be a JSON 'Object' with variable names as keys and values obviously as values. There are examples throughout this document. For GET requests, everything is in standard GET parameters, but some variables are complicated and will need to be JSON-encoded and then URL-encoded. An example would be the 'tags' parameter on [GET /get\_files/search\_files](#get_files_search_files), which is a list of strings. Since GET http URLs have limits on what characters are allowed, but hydrus tags can have all sorts of characters, you'll be doing this: * Your list of tags: @@ -21,19 +21,19 @@ In general, the API deals with standard UTF-8 JSON. POST requests and 200 OK res [ 'character:samus aran', 'creator:青い桜', 'system:height > 2000' ] ``` -* JSON encoded: +* JSON-encoded: ```json ["character:samus aran", "creator:\\u9752\\u3044\\u685c", "system:height > 2000"] ``` -* Then URL encoded: +* Then URL-encoded: ``` %5B%22character%3Asamus%20aran%22%2C%20%22creator%3A%5Cu9752%5Cu3044%5Cu685c%22%2C%20%22system%3Aheight%20%3E%202000%22%5D ``` -* In python, converting your tag list to the URL encoded string would be: +* In python, converting your tag list to the URL-encoded string would be: ``` urllib.parse.quote( json.dumps( tag_list ), safe = '' ) @@ -128,7 +128,7 @@ Arguments: * `hash`: (selective, a hexadecimal SHA256 hash) * `hashes`: (selective, a list of hexadecimal SHA256 hashes) -In GET requests, make sure any list is percent-encoded. +In GET requests, make sure any list is percent-encoded JSON. Your `[1,2,3]` becomes `urllib.parse.quote( json.dumps( [1,2,3] ), safe = '' )`, and thus `file_ids=%5B1%2C%202%2C%203%5D`. ### **file domain** { id="parameters_file_domain" } @@ -358,6 +358,7 @@ Arguments: * 9 - Edit File Ratings * 10 - Manage Popups * 11 - Edit File Times + * 12 - Commit Pending ``` title="Example request" /request_new_permissions?name=my%20import%20script&basic_permissions=[0,1] @@ -2299,6 +2300,32 @@ Response: If there are no potential duplicate groups in the search, this returns an empty list. +### **POST `/manage_file_relationships/remove_potentials`** { id="manage_file_relationships_remove_potentials" } + +Remove all potential pairs that any of the given files are a part of. If you hit [/manage\_file\_relationships/get\_file\_relationships](#get-manage_file_relationshipsget_file_relationships) after this on any of these files, they will have no potential relationships, and any hashes that were potential to them before will no longer, conversely, refer to these files as potentials. + +Restricted access: +: YES. Manage File Relationships permission needed. + +Required Headers: +: + * `Content-Type`: application/json + +Arguments (in JSON): +: + * [files](#parameters_files) + +```json title="Example request body" +{ + "file_id" : 123 +} +``` + +Response: +: 200 with no content. + +If the files are a part of any potential pairs (with any files, including those you did not specify), those pairs will be deleted. This deletes everything they are involved in, and the files will not be queued up for a re-scan, so I recommend you only do this if you know you added the potentials yourself (e.g. this is regarding video files) or you otherwise have a plan to replace the deleted potential pairs with something more useful. + ### **POST `/manage_file_relationships/set_file_relationships`** { id="manage_file_relationships_set_file_relationships" } Set the relationships to the specified file pairs. @@ -2441,6 +2468,108 @@ Response: The files will be promoted to be the kings of their respective duplicate groups. If the file is already the king (also true for any file with no duplicates), this is idempotent. It also processes the files in the given order, so if you specify two files in the same group, the latter will be the king at the end of the request. +## Managing Services + +For now, this refers to just seeing and committing pending content (which you see in the main "pending" menubar if you have an IPFS, Tag Repository, or File Repository service). + +### **GET `/manage_services/get_pending_counts`** { id="manage_services_get_pending_counts" } + +_Get the counts of pending content for each upload-capable service. This basically lets you construct the "pending" menu in the main GUI menubar._ + +Restricted access: +: YES. Start Upload permission needed. + +Required Headers: n/a + +Arguments: n/a + +``` title="Example request" +/manage_services/get_pending_counts +``` + +Response: +: A JSON Object of all the service keys capable of uploading and their current pending content counts. + +```json title="Example response" +{ + "pending_counts" : { + "ae91919b0ea95c9e636f877f57a69728403b65098238c1a121e5ebf85df3b87e" : { + "pending_tag_mappings" : 11564, + "petitioned_tag_mappings" : 5, + "pending_tag_siblings" : 2, + "petitioned_tag_siblings" : 0, + "pending_tag_parents" : 0, + "petitioned_tag_parents" : 0 + }, + "3902aabc3c4c89d1b821eaa9c011be3047424fd2f0c086346e84794e08e136b0" : { + "pending_tag_mappings" : 0, + "petitioned_tag_mappings" : 0, + "pending_tag_siblings" : 0, + "petitioned_tag_siblings" : 0, + "pending_tag_parents" : 0, + "petitioned_tag_parents" : 0 + }, + "e06e1ae35e692d9fe2b83cde1510a11ecf495f51910d580681cd60e6f21fde73" : { + "pending_files" : 2, + "petitioned_files" : 0 + } + } +} +``` + +The keys are as in [/get\_services](#get_services). + +Each count here represents one 'row' of content, so for "tag_mappings" that is one (tag, file) pair and for "tag_siblings" one (tag, tag) pair. You always get everything, even if the counts are all 0. + +### **POST `/manage_services/commit_pending`** { id="manage_services_commit_pending" } + +_Start the job to upload a service's pending content._ + +Restricted access: +: YES. Start Upload permission needed. + +Required Headers: n/a + +Arguments (in JSON): +: +* `service_key`: (the service to commit) + +``` title="Example request body" +{ + "service_key" : "ae91919b0ea95c9e636f877f57a69728403b65098238c1a121e5ebf85df3b87e" +} +``` + +This starts the upload popup, just like if you click 'commit' in the menu. This upload could ultimately take one second or several minutes to finish, but the response will come back immediately. + +If the job is already running, this will return 409. If it cannot start because of a difficult problem, like all repositories being paused or the service account object being unsynced or something, it gives 422; in this case, please direct the user to check their client manually, since there is probably an error popup on screen. + +If tracking the upload job's progress is important, you could hit it again and see if it gives 409, or you could [/get\_pending\_counts](#manage_services_get_pending_counts) again--since the counts will update live as the upload happens--but note that the user may pend more just after the upload is complete, so do not wait forever for it to fall back down to 0. + +### **POST `/manage_services/forget_pending`** { id="manage_services_forget_pending" } + +_Forget all pending content for a service._ + +Restricted access: +: YES. Start Upload permission needed. + +Required Headers: n/a + +Arguments (in JSON): +: +* `service_key`: (the service to forget for) + +``` title="Example request body" +{ + "service_key" : "ae91919b0ea95c9e636f877f57a69728403b65098238c1a121e5ebf85df3b87e" +} +``` + +This clears all pending content for a service, just like if you click 'forget' in the menu. + +Response description: +: 200 and no content. + ## Managing Cookies This refers to the cookies held in the client's session manager, which you can review under _network->data->manage session cookies_. These are sent to every request on the respective domains. @@ -2663,6 +2792,7 @@ Response: "page_key" : "3b28d8a59ec61834325eb6275d9df012860a1ecfd9e1246423059bc47fb6d5bd", "page_state" : 0, "page_type" : 10, + "is_media_page" : false, "selected" : true, "pages" : [ { @@ -2670,6 +2800,7 @@ Response: "page_key" : "d436ff5109215199913705eb9a7669d8a6b67c52e41c3b42904db083255ca84d", "page_state" : 0, "page_type" : 6, + "is_media_page" : true, "selected" : false }, { @@ -2677,6 +2808,7 @@ Response: "page_key" : "40887fa327edca01e1d69b533dddba4681b2c43e0b4ebee0576177852e8c32e7", "page_state" : 0, "page_type" : 9, + "is_media_page" : true, "selected" : false }, { @@ -2684,6 +2816,7 @@ Response: "page_key" : "2ee7fa4058e1e23f2bd9e915cdf9347ae90902a8622d6559ba019a83a785c4dc", "page_state" : 0, "page_type" : 10, + "is_media_page" : false, "selected" : true, "pages" : [ { @@ -2691,6 +2824,7 @@ Response: "page_key" : "9fe22cb760d9ee6de32575ed9f27b76b4c215179cf843d3f9044efeeca98411f", "page_state" : 0, "page_type" : 7, + "is_media_page" : true, "selected" : true }, { @@ -2698,6 +2832,7 @@ Response: "page_key" : "2977d57fc9c588be783727bcd54225d577b44e8aa2f91e365a3eb3c3f580dc4e", "page_state" : 0, "page_type" : 6, + "is_media_page" : true, "selected" : false } ] @@ -2707,35 +2842,37 @@ Response: } ``` - `name` is the full text on the page tab. - - `page_key` is a unique identifier for the page. It will stay the same for a particular page throughout the session, but new ones are generated on a session reload. - - `page_type` is as follows: - - * 1 - Gallery downloader - * 2 - Simple downloader - * 3 - Hard drive import - * 5 - Petitions (used by repository janitors) - * 6 - File search - * 7 - URL downloader - * 8 - Duplicates - * 9 - Thread watcher - * 10 - Page of pages - - `page_state` is as follows: - - * 0 - ready - * 1 - initialising - * 2 - searching/loading - * 3 - search cancelled - - Most pages will be 0, normal/ready, at all times. Large pages will start in an 'initialising' state for a few seconds, which means their session-saved thumbnails aren't loaded yet. Search pages will enter 'searching' after a refresh or search change and will either return to 'ready' when the search is complete, or fall to 'search cancelled' if the search was interrupted (usually this means the user clicked the 'stop' button that appears after some time). - - `selected` means which page is currently in view. It will propagate down the page of pages until it terminates. It may terminate in an empty page of pages, so do not assume it will end on a media page. - - The top page of pages will always be there, and always selected. - +`name` is the full text on the page tab. + +`page_key` is a unique identifier for the page. It will stay the same for a particular page throughout the session, but new ones are generated on a session reload. + +`page_type` is as follows: + +* 1 - Gallery downloader +* 2 - Simple downloader +* 3 - Hard drive import +* 5 - Petitions (used by repository janitors) +* 6 - File search +* 7 - URL downloader +* 8 - Duplicates +* 9 - Thread watcher +* 10 - Page of pages + +`page_state` is as follows: + +* 0 - ready +* 1 - initialising +* 2 - searching/loading +* 3 - search cancelled + +Most pages will be 0, normal/ready, at all times. Large pages will start in an 'initialising' state for a few seconds, which means their session-saved thumbnails aren't loaded yet. Search pages will enter 'searching' after a refresh or search change and will either return to 'ready' when the search is complete, or fall to 'search cancelled' if the search was interrupted (usually this means the user clicked the 'stop' button that appears after some time). + +`is_media_page` is simply a shorthand for whether the page is a normal page that holds thumbnails or a 'page of pages'. Only media pages can have files (and accept [/manage\_files/add\_files](#manage_pages_add_files) commands). + +`selected` means which page is currently in view. It will propagate down the page of pages until it terminates. It may terminate in an empty page of pages, so do not assume it will end on a media page. + +The top page of pages will always be there, and always selected. + ### **GET `/manage_pages/get_page_info`** { id="manage_pages_get_page_info" } @@ -2767,6 +2904,7 @@ Response description "page_key" : "aebbf4b594e6986bddf1eeb0b5846a1e6bc4e07088e517aff166f1aeb1c3c9da", "page_state" : 0, "page_type" : 3, + "is_media_page" : true, "management" : { "multiple_watcher_import" : { "watcher_imports" : [ @@ -2827,12 +2965,11 @@ Response description } ``` - `name`, `page_key`, `page_state`, and `page_type` are as in [/manage\_pages/get\_pages](#manage_pages_get_pages). - - As you can see, even the 'simple' mode can get very large. Imagine that response for a page watching 100 threads! Turning simple mode off will display every import item, gallery log entry, and all hashes in the media (thumbnail) panel. - - For this first version, the five importer pages--hdd import, simple downloader, url downloader, gallery page, and watcher page--all give rich info based on their specific variables. The first three only have one importer/gallery log combo, but the latter two of course can have multiple. The "imports" and "gallery_log" entries are all in the same data format. - +`name`, `page_key`, `page_state`, and `page_type` are as in [/manage\_pages/get\_pages](#manage_pages_get_pages). + +As you can see, even the 'simple' mode can get very large. Imagine that response for a page watching 100 threads! Turning simple mode off will display every import item, gallery log entry, and all hashes in the media (thumbnail) panel. + +For this first version, the five importer pages--hdd import, simple downloader, url downloader, gallery page, and watcher page--all give rich info based on their specific variables. The first three only have one importer/gallery log combo, but the latter two of course can have multiple. The "imports" and "gallery_log" entries are all in the same data format. ### **POST `/manage_pages/add_files`** { id="manage_pages_add_files" } @@ -2840,7 +2977,7 @@ _Add files to a page._ Restricted access: : YES. Manage Pages permission needed. - + Required Headers: : * `Content-Type`: application/json @@ -2860,7 +2997,7 @@ The files you set will be appended to the given page, just like a thumbnail drag ``` Response: -: 200 with no content. If the page key is not found, this will 404. +: 200 with no content. If the page key is not found, it will 404. If you try to add files to a 'page of pages' (i.e. `is_media_page=false` in the [/manage\_pages/get\_pages](#manage_pages_get_pages) call), you'll get 400. ### **POST `/manage_pages/focus_page`** { id="manage_pages_focus_page" } diff --git a/docs/old_changelog.html b/docs/old_changelog.html index 762938e05..ddaf0d975 100644 --- a/docs/old_changelog.html +++ b/docs/old_changelog.html @@ -34,6 +34,37 @@

changelog