Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Start installing mods while downloads are still in progress #4249

Merged
merged 12 commits into from
Nov 6, 2024

Conversation

HebaruSan
Copy link
Member

@HebaruSan HebaruSan commented Nov 6, 2024

Motivations

  • Currently the module installer downloads all of the uncached mods in a changeset before it starts installing anything. It would be more time-efficient to start installing some mods while others are still downloading, when feasible.
  • Currently a mod's SHA256 hash is calculated after we finish downloading it, which means we receive all of those bytes over the network, save them to disk, and then load them back from disk into memory to hash them. This can take a while for large files. It would be better to calculate the hash while the download is in progress, since we have to wait for the bytes to come across the network anyway.
  • Currently ckan available always lists all the available mods. If you have some idea of what you're looking for (e.g., I wanted to list the "OPX" mods for testing), it would be nice to be able to apply a filter.

Problems

  • The GUI progress screen's per-mod progress bars can sometimes get stuck at nearly-complete instead of disappearing when their task finishes
  • If you Shift-click the All filter in GUI, an exception is thrown
  • Mac OS has a default keybind for F10, which conflicts with ConsoleUI's use of that key
  • If you try to install a mod that recommends a DLC that you don't have installed, you get an error that says "CKAN can't install expansion 'Making History' for you.", even though you're not trying to install it.

Causes

  • The progress bars are driven by Progress<> objects, which run code on a thread pool and can be de-allocated before that actually happens if the task that created them finishes
  • Shift-clicking All doesn't make sense as an operation because the intersection of a set with All is the same set you started with (n.b., in contrast to the other filters, there is no tooltip advertising this key option), but there is code partially implementing it anyway. This partial implementation passes an empty list to EditModSearches.MergeSearches, which expects at least one element.
  • The relationship resolver throws an exception if it finds a DLC in its traversal of relationships, but as of Parallelize for performance, relationship resolver improvements #3917, that's how we find recommendations.

Changes

  • Now downloads and installs are performed in parallel. A downloaded mod will only be installed once all of its dependencies have been installed, to preserve the exFAT-friendly timestamp ordering from Install dependencies first #3667. For upgrades, the downloads begin in parallel with removals and continue into the installs if they take that long. This should reduce the overall time required for installs and upgrades. Upon a download error or user cancellation, the installs are reverted but any completed downloads remain cached.
    Fixes [Feature] Install the mods while downloading others #3793.
  • The progress screen is fairly drastically refactored to be closer to how I imagined it ought to have worked all along:
    • Where previously only the main progress bar showed the rate and percentage in a separate label located above it, now every progress bar shows those details in an internal label.
      Known issue: The internal progress bar label is not compatible with "marquee" style progress bars (where instead of reflecting a percentage completion, it just plays an animation). The label is shown for one frame of animation and then disappears. I think this is OK for now since we don't use the labels with marquee style to display actual useful information. This might be something we can address in the future with more messing around with the WinForms APIs.
    • The overall progress bar now combines total progress on all downloads and installs since they are no longer two separate stages
    • Per-mod progress bars don't disappear anymore but instead are scrolled out of view, with the active progress bars kept at the bottom of the list and the pane sized to fit them
    • The user can scroll through the progress bars if they wish and move the splitter that separates them from the log messages box
    • Previously only downloading and validating had per-mod progress bars, but now installing and removing steps also have them
  • Now the Progress<> objects are replaced with ProgressImmediate<> objects that run their callbacks right away instead of in a thread pool, so we no longer lose updates
  • Now the progress bar captions show the approximate remaining time, which we compute by dividing the remaining bytes by the average of two rates: the total bytes per second of the download since it started, and the bytes per second over the last three seconds.
    Fixes [Feature]: Show remaining download time #4160.
  • Now the ResumingWebClient's constructor accepts an optional HashAlgorithm parameter. If supplied, it will be used to calculate the hash of the incoming byte stream as received from the server. Upon completion, the resulting hash is propagated outwards to the NetAsyncModulesDownloader where it is checked against the mod's hash metadata. This eliminates hashing from the download validation step (but we still need that step for a few things like checking that the file is a valid ZIP, etc.). This should speed up installs a bit.
  • Now when a mod recommends a non-installed DLC, the install won't error out, and the DLC will show up in the recommendations list, as intended.
    Tests are created to exercise this and catch future regressions.
    Fixes [Bug]: MakingHistory recommendation halts install instead of appearing in recommendations list #4242.
  • Now Shift-clicking the All filter works exactly like clicking it normally.
    Fixes [Bug]: Unhandled exception when adding 'All' to search "Sequence contains no elements" #4244.
  • Now ConsoleUI supports Ctrl+T in addition to F10 to open the menu, and the screen tip on Mac is changed from F10 to Ctrl+T.
    Fixes [Feature]: Configurable keys for menu/actions in ConsoleUI #4248.
  • Now ckan available supports a globbing pattern (meaning you can use ? and * as wildcards) to filter the list of available mods

@HebaruSan HebaruSan added Bug Something is not working as intended Enhancement New features or functionality GUI Issues affecting the interactive GUI Cmdline Issues affecting the command line Core (ckan.dll) Issues affecting the core part of CKAN Tests Issues affecting the internal tests Network Issues affecting internet connections of CKAN ConsoleUI Issues affecting the interactive console UI Performance Something's slower than it should be labels Nov 6, 2024
@HebaruSan HebaruSan merged commit 49c79d2 into KSP-CKAN:master Nov 6, 2024
3 checks passed
@HebaruSan HebaruSan deleted the feature/install-pipeline branch November 6, 2024 04:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something is not working as intended Cmdline Issues affecting the command line ConsoleUI Issues affecting the interactive console UI Core (ckan.dll) Issues affecting the core part of CKAN Enhancement New features or functionality GUI Issues affecting the interactive GUI Network Issues affecting internet connections of CKAN Performance Something's slower than it should be Tests Issues affecting the internal tests
Projects
None yet
1 participant