Lua script for checking the health of Devuan Linux package mirrors.
As the name would suggest, the ultimate goal of apt-panopticon is to probe into every nook and cranny of Devuans apt distribution system, to find out what is breaking, and where. Graphing history and sending alerts as needed. At some later stage it's likely to be generalised for other apt based distros.
This is currently under development, not everything has been written yet. Some of this document mentions some of the things that are not written yet.
apt-panopticon is a Lua script used by the Devuan mirror admins (maybe, if they like it) to check the health of Devuan Linux package mirrors. Originally there was bash scripts for this job, then Evilham wrote some Python scripts, now onefang has written it in Lua. We all have different tastes in languages. lol
The main difference is that this Lua version tries to do everything, and will be maintained. Currently the shell scripts and Python scripts are actually being used I think. Evilham asked me to write this, after I badgered him about his Python scripts. It should also be much easier to use, the previous scripts needed some work before you could run them, this one you just download and run.
After a discussion, it is likely that apt-panopticon will be used to automatically decide which Devuan mirrors are in the DNS round robin. The details of that are yet to be determined.
You can get the source code from https://sledjhamr.org/cgit/apt-panopticon/ (main repo) and https://git.devuan.dev/onefang/apt-panopticon (Devuan repo). You can get the cgp graphing source code from https://sledjhamr.org/cgit/apt-panopticon_cgp/ (main repo) and https://git.devuan.dev/onefang/apt-panopticon_cgp (Devuan repo).
The issue tracker is at https://sledjhamr.org/mantisbt/project_page.php?project_id=13
Download the source. You may want to put the apt-panopticon.lua script
in someplace like
/usr/local/bin and make sure it is executable.
It should run on any recent Linux, you'll need to have the following installed -
- dig, part of BIND. On Debian based systems it'll be in the dnsutils package.
- flock, on Debian based systems it'll be in the util-linux package.
- ionice, on Debian based systems it'll be in the util-linux package.
- LuaSocket, on Debian based systems it'll be in the lua-socket package.
- md5sum and sha256, on Debian based systems they'll be in the coreutils package.
- xz, on Debian based systems it'll be in the xz-utils package.
If you want to have lots of graphs, also install https://sledjhamr.org/cgit/apt-panopticon_cgp/.
For the apt-panopticon_cgp package, which is used to show the detailed graphs, you'll need a web server that supports PHP. apt-panopticon_cgp includes some support files for running PHP via CGI, which more web servers support. You'll need php-cgi for that. It's been tested with recent versions of Apache 2 and Lightttpd.
This is a suggestion for installation on a Devuan based web server.
Install apt-panopticon and apt-panopticon_cgp there, so you end up with -
The script update_apt-panopticon is an example script for updating
everything, including commented out commands to update the source code.
The file apt-panopticron is an example crontab file for updating everything once every ten minutes. They assume your web server user is www-data with a group of www-data, and you have a mirror user called mirrors. For mirror operators, that mirrors user would be the owner of the mirror files. You can change these to suite yourself.
Once everything is updated, /var/www/html/apt-panopticon/results/Report-web.html will point to the main web page, and there will be links that point to the detailed graphs.
Note that two runs of apt-panopticon have to happen ten minutes apart at least in order to see any data on the graphs.
If you had already been running apt-panopticon for a while and have lots of data collected, the apt-panopticon-update-data.lua script can go through all of that and feed it to RRD / update the files.
These examples assume you are running it from the source code directory.
A directory will be created called
results, it'll be full of log files
and any files that get downloaded.
Note that unlike typical commands, you can't run single character options together, so this is wrong -
$ ./apt-panopticon.lua -vvv
Instead do this -
$ ./apt-panopticon.lua -v -v -v
Just run the script to do all of the tests -
Which will print any CRITICAL errors. If you don't want to see errors -
$ ./apt-panopticon.lua -q
If you want to see other errors, warnings, and even more details as well
(as usual, the more
-v options, the more details) -
$ ./apt-panopticon.lua -v
Or use the usual options for the help and version number (not written yet) -
$ ./apt-panopticon.lua -h $ ./apt-panopticon.lua --help $ ./apt-panopticon.lua --version
To run the tests on a specific mirror, for example pkgmaster.devuan.org -
$ ./apt-panopticon.lua pkgmaster.devuan.org
You can use the
--tests option to tune which tests are run, for example
to stop IPv6 tests, coz you don't have IPv6 -
$ ./apt-panopticon.lua --tests=-IPv6
To do the same, but not run the HTTPS tests either -
$ ./apt-panopticon.lua --tests=-IPv6,-https
To only run the HTTP integrity tests, only on IPv6 -
$ ./apt-panopticon.lua --tests=http,Integrity,IPv6
The basic test is to find all the IPs for a mirror, including any CNAMES, then send HTTP HEAD requests to those IPs, with HOST headers for that mirror, and follow any redirections, doing the same for those redirections. Unless a specific mirror is given on the command line, the mirror_list.txt file from pkgmaster.devuan.org is used to select mirrors to test.
The --tests= option can be used to adjust the list of tests performed.
- IPv4, perform the tests with IPv4 addresses (A records)
- IPv6, perform the tests with IPv6 addresses (AAAA records)
- ftp, test FTP protocol access, check for the existence of the file instead of a HTTP HEAD.
- http, test HTTP protocol access.
- https, test HTTPS protocol access.
- rsync, test RSYNC protocol access.
- DNS-RR, Checks if the IPs for this mirror are part of the DNS-RR.
- Protocol, warn if the protocol changed during a redirect.
- Redirects test bad redirects, redirecting /DEVUAN/ to deb.devuan.org.
- URL-Sanity, add gratuitous multiple slashes to the URLs.
- Integrity, check hashes and PGP signatures.
- Updated, check Release dates and updated packages.
Note that apt-panopticon will detect if you have no IPv6 conectivity, and disable IPv6 tests automatically.
The old tests include a "DNS-RR" test, I'm not sure what that is. I guess it checks if the mirror responds properly if it's accessed via it's DNS RR (round robin) IP, and a HOST header of deb.devuan.org. If no other mirror is listed on the command line, we start with deb.devuan.org and check all of it's IPs, which are the DNS RR mirrors anyway.
The mirror_list.txt file also used to select which protocols to test for each mirror, it will only test those protocols the mirror lists as supporting. Actually it'll test the ones not supported as well, but mark them differently in the results.
Print the help text.
Print the version.
Print more verbose output. Normally only CRITICAL messages are printed. -v will print ERROR messages as well, -v -v WARNING messages, -v -v -v INFO messages, and -v -v -v -v DEBUG messages. All messages are logged regardless.
Print no output messages.
Keep any results from the previous runs, instead of deleting them before running the tests. This may or may not be obsolete.
-4 and -6
Used internally to pass around flags to make sure curl only tries to use IPv4 or IPv6 as appropriate.
-o and -r
Used internally to keep track of -origin server and -redirect server.
--bandwidth, --low, --medium, --high, --more, and --all
Enable and disable tests that use more or less bandwidth. --bandwidth=x where x is a digit between 0 and 4, and the other options are aliases for those numbers. So --bandwidth=0 is the same as --low. --bandwidth=2 or --high is the default.
Use the php.cgi versions of the URLs for the graphs.
--maxtime, --retries, --timeout, and --timeouts
Various paramaters to determine how long to wait for downloads to complete, how many retries to attempt for downloads that fail, how long to wait for a connection, and how many timeouts to tolerate.
The mirror to use as a reference for the tests, the default is pkgmaster.devuan.org.
The name of the DNS round robin domain, the default is deb.devuan.org.
Select which reports to generate. The arguments are comma separated. A negative argument deselects a report.
Select which tests to run. The arguments are comma separated. A negative argument deselects a test. Examples are given above.