ghostmirror v0.9.20

Introduction:

We've all experienced moments when there seemed to be no packages to update, only to realize days later that the issue was an out-of-sync mirror.
Despite the numerous tools available on Arch Linux for managing mirrors, none of them fully met my expectations.
So, I set out to solve this problem.
While writing the software, I realized that by continuously adding features, ghostmirror has become a great tool for the mirror maintainers themselves.
In fact, besides displaying mirrors that have errors, it can investigate and show the possible causes that generated the error.

Description:

What does GhostMirror do?
It compares the mirror databases with the local database and provides a detailed description of whether the mirror's packages are more or less up-to-date compared to our local database.
It can analyze the mirrors and display in-depth errors or the names of packages that are not updated.
Thanks to the custom sorting mode, it can create a list of mirrors based on each user's needs.
If you don't have time to update the mirror list manually, by adding a single command-line argument, the systemd service will be automatically activated. Feel free to forget about the mirrors.

Install:

You can use your favorite AUR helper to install it.

$ yay -S ghostmirror

otherwise you can find PKGBUILD in distro dir

$ cd distro && makepkg -sirc

Usage:

You can use the software in three ways: manually, automatically, or for investigation.

Manually

In this mode, you will perform all the steps manually.

First step

in the first step you need to search a good quantity of mirrors.
-P for get progress and output colors, -o for get output table.
-c for select country, for example Italy,Germany,France
-l where do you save list
-L max numbers of output mirror in list
-S sort mode, first add state, in this mode remove error mirror, after you can add outofdate, in this mode display first the mirror sync, can also add morerecent to ensure you never go out of sync, and at the end, you can add ping to try to prioritize the closest ones.

$ ghostmirror -PoclLS Italy,Germany,France ./mirrorlist.new 30 state,outofdate,morerecent,ping

Second step

Now, instead of taking the mirrors from the global list, we will better evaluate the mirrors found in the first step.
While the first step will be performed only once or rarely, this step will be the one you repeat periodically.
-P -o -l is same the previous step
-m tell software to use a local mirror list, and -u for use only uncommented mirror
-s for apply a real test for mirror speed
-S change a sort mode, remove ping, add extimated for get more stable mirror and speed for reorder speed

$ ghostmirror -PmuolsS  ./mirrorlist.new ./mirrorlist.new light state,outofdate,morerecent,extimated,speed

Last step

now you save your old mirrorlist

# cp /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist.bak

now save new mirrorlist

# cp ./mirrorlist.new /etc/pacman.d/mirrorlist

Automatically

In this mode, you will perform the second step automatically.

Prepare

you need manual make dir for a new location of mirrorlist, need location where user can edit this without root privilege

$ mkdir ~/.config/ghostmirror

now you need to inform pacman where you have stored mirrorlist
edit file /etc/pacman.conf, search and replace this line and change with your username.

[core]
Include = /home/<username>/.config/ghostmirror/mirrorlist

[extra]
Include = /home/<username>/.config/ghostmirror/mirrorlist

First step

same manually mode you need create a big good mirrorlist

$ ghostmirror -PoclLS Italy,Germany,France ~/.config/ghostmirror/mirrorlist 30 state,outofdate,morerecent,ping

Last step

the arguments,lsS, you pass to ghostmirror at this point it's saved for auto reuse it in service.
so, the only difference with manually second step is -D option, this enable ghostmirror.timer and loginctl linger

$ ghostmirror -PoDumlsS  ~/.config/ghostmirror/mirrorlist ~/.config/ghostmirror/mirrorlist light state,outofdate,morerecent,extimated,speed

Use of systemd

show the timer

$ systemctl --user list-timers

force start ghostmirror before timer ellapsed

$ systemctl --user ghostmirror.service

End

Now you can forget about mirrors forever.

Investigation

It's be simple, add -i option and it's show the list of error server with motivation.

Reference

the software accept long options with -- and assign value without space but with =

--option
--option value
--option=value

accept short option with - or multiple option, followed by value

-o
-o value
-abc valueA valueB valueC

All Options

-a --arch

select arch, default 'x86_64'

-m --mirrorfile

use mirror file instead of downloading mirrorlist, without -m and -u ghostmirror download mirrorlist and search in all mirror.
you can use -m /etc/pacman.d/mirrorlist.pacnew if you not want downloading mirrorlist but used local list.

-c --country

select country from mirrorlist.
if mirrorlist is not same the default mirrorlist and not have ##Country or mixed mirror, the selection country fail

-C --country-list

show all possibile country

-u --uncommented

use only uncommented mirror, by default use commented and uncommented mirror.
with -m is simple to check local mirrolist -mu /etc/pacman.d/mirrorlist.

-d --downloads

set numbers of parallel download, default '4'.
if you abuse the download is more simple to fail, 1,4,8,16 is good value, >16 you can try but is not very affidable

-O --timeout

set timeout in seconds for not reply mirror, default 5s

-p --progress

show progress, default false

-P --progress-colors

same -p but with -o add color table

-o --output

enable table output, with -P display with colors

-s --speed

test speed for downloading:
light: download one small package ~6MiB
normal: download light + normal package ~250Mib
heavy: download light+normal+heavy packege ~350MiB, total download >500Mib

-S --sort

sort result for any of fields display in table, mutiple fields supported.
country and mirror is sorted by name
proxy first false, last true
state first success last error
outofdate, retry and ping, display first less value
uptodate, morerecent, sync, speed and extimated, display first great value

-l --list

save new mirrorlist in file passed as argument.
special name, stdout, can be used for write to stdout file.

-L --max-list

set max numbers of output mirrors

-T --type

select mirrors type, http,https,all

-i --investigate

search mirror errors to detect the problem.
can select mode: error, outofdate, all.
error: investigate only on error.
outofdate: investigate only on outofdate package.
all: same passing -i error,outofdate

-D --systemd

auto manager systemd.timer.
when you pass this option the software activate login linger if not ebabled.
auto create ghostmirror.service and ghostmirror.timer
the config.service start ghostmirror in the same mode you haved executed it, with only differences that need -l.
for exaples if you execute: -DmuldsS 16 light extimated,speed
the service is always start with 16 parallel downloads, speed light and extimated,speed sort.
for change you can simple repeat a command.
the expire timer is the first element in table and is dinamic, can change every time.

-t --time

in systemd timer whend extimate date is ellapsed can set a time. Set specific hh:mm:ss when start service, default 00:00:00.
validate input with systemd-analyzer calendar hh:mm:ss before use

-f --fixed-time

in systemd timer use fixed time instead of extimated time.
validate input with systemd-analyzer calendar hh:mm:ss before use

-h --help

display this

Manual Build

This section is for developer

Require:

libcurl, zlib, systemd-libs

Build:

$ meson setup build
$ cd build
$ ninja

Debug:

for enable very verbose output.

$ meson configure -Debug=4

warning this is only for contributor, enable auto versioning and auto push

$ meson configure -Developer=true

State:

• v0.9.20 bash completion(request 4), fix environment(issue 5)
• v0.9.19 merge man page, check on generate man page, test.
• v0.9.18 returned to the old method of the pre-calculated and fixed date in the calculation of time, theoretically it should never fail, I had unnecessarily complicated things
• v0.9.17 fix unable restart unit if not file exists
• v0.9.16 systemd oncalendar does not work as expected, changed the time calculation and forcing method, remove warning sign
• v0.9.15 systemd oncalendar does not work as expected, changed the time calculation and forcing method
• v0.9.14 custom time and fixed time, specific option for path, wrong indentation
• v0.9.13 pkgbuild doc, thanks, fix possible issue by scan-build, valgrind success (only remain memory when exit from software, its not problem), this is first candidate for stable version
• v0.9.12 more doc and wiki.
• v0.9.11 the prev version not build, missing gm.h. add screenshot.
• v0.9.10 little more doc, change conf for waiting nss-lockup.service, removed opt show unknonw option at end of argument, opt usage show is array
• v0.9.9 support all flags to systemd service
• v0.9.8 more precise and elegant stability
• v0.9.7 there was a piece of test code left
• v0.9.5 strong systemd: better config for autorestart, auto use local db. get local pacman database if many fail remote local database. can use manually RESTART_COUNT=999. Versioning config for automatic reconfigure when new version is released
• v0.9.4 doc have a problem, need to write the doc
• v0.9.3 doc have a problem, need to write the doc
• v0.9.2 local mirror now is get from correct but not perfect location
• v0.9.1 unmask service, mkalldir, wrong argument to sd-bus, end first systemd test
• v0.9.0 warning this version not have systemd tested, change option name, t->d, o->O, o->table output, D->systemd linger, tar pax error, sort stability->extimated
• v0.8.4 add internal error for more precise investigation and better output, fix sync interrupt threads not initialized, change another time time out
• v0.8.3 investigation, add basic support for simple server test, not affidable need more and more test
• v0.8.2 ping field and sort, better status sort, prepare for investigation
• v0.8.1 merge wwmdownload now resolve follow location, add proxy in sort and table, fatal error if not downloaded local database
• v0.8.0 concept of stability, full sort this fix die with speed, internal rfield back to struct field, www followlocation, dynamic time retry, default tout 20
• v0.7.1 fix wrong -T description, set max numbers of output mirrors, unique mirrors.
• v0.7.0 add speed type, light, normal, heavy. fix speed benchmark delay to double
• v0.6.0 add select mirror type, http or https or all
• v0.5.8 fix sort to manage new field, not tested
• v0.5.7 merge newversion and notexists in morerecent, highlighted local
• v0.5.6 assertion boolean value
• v0.5.5 fix assertion isheap not exists
• v0.5.4 some testing
• v0.5.3 upload on aur
• v0.5.2 colorized output, switch to array fields, compact table code, remove some magic
• v0.5.1 better progress, if not country set mirrors country is search on remote mirrorlist
• v0.5.0 add all fields sorting
• v0.4.2 only for developer, auto follow tags
• v0.4.1 mem nullterm, info stored in list, meson developer message
• v0.4.0 --list create a list with Server=mirror, country+retry in info
• v0.3.1 autogit, first tag
• v0.3.0 sorting result
• v0.2.2 more stable, removed useless code, auto versioning.
• v0.2.1 check current mirror used and show
• v0.2.0 use mirrorlist for get current remote mirror in use in this mode not required -Syu, calcolate speed mirror
• v0.1.0 remove useless options, check if server have all files in db, little better output
• v0.0.0 first alpha version, be careful

Bug:

try to writing many