Browse Source
First pass at making the acquire system capable of downloading files based on configuration rather than hardcoded entries. It is now possible to instruct 'deb' and 'deb-src' sources.list lines to download more than just Packages/Translation-* and Sources files. Details on how to do that can be found in the included documentation file.debian/1.8.y
David Kalnischkies
7 years ago
8 changed files with 376 additions and 192 deletions
@ -0,0 +1,142 @@ |
|||
# Acquire additional files in 'update' operations |
|||
|
|||
The download and verification of data from multiple sources in different |
|||
compression formats, with partial downloads and patches is an involved |
|||
process which is hard to implement correctly and securely. |
|||
|
|||
APT frontends share the code and binaries to make this happen in libapt |
|||
with the Acquire system, supported by helpers shipped in the apt package |
|||
itself and additional transports in individual packages like |
|||
apt-transport-https. |
|||
|
|||
For its own operation libapt needs or can make use of Packages, Sources |
|||
and Translation-* files, which it will acquire by default, but |
|||
a repository might contain more data files (e.g. Contents) a frontend |
|||
might want to use and would therefore need to be downloaded as well |
|||
(e.g. apt-file). |
|||
|
|||
This file describes the configuration scheme such a frontend can use to |
|||
instruct the Acquire system to download those additional files. |
|||
|
|||
Note that you can't download files from other sources. It must be files |
|||
in the same repository and below the Release file. The Release file must |
|||
also contain hashes for the file itself as well as for the compressed |
|||
file if wanted, otherwise a download isn't even tried! |
|||
|
|||
|
|||
# The Configuration Stanza |
|||
|
|||
The Acquire system uses the same configuration settings to implement the |
|||
files it downloads by default. These settings are the default, but if |
|||
they would be written in a configuration file the configuration |
|||
instructing the Acquire system to download the Packages files would look |
|||
like this (see also apt.conf(5) manpage for configuration file syntax): |
|||
|
|||
APT::Acquire::Targets::deb::Packages { |
|||
URI "$(COMPONENT)/binary-$(ARCHITECTURE)/Packages"; |
|||
ShortDescription "Packages"; |
|||
Description "$(SITE) $(RELEASE)/$(COMPONENT) $(ARCHITECTURE) Packages"; |
|||
|
|||
flatURI "Packages"; |
|||
flatDescription "$(SITE) $(RELEASE) Packages"; |
|||
|
|||
Optional "false"; |
|||
}; |
|||
|
|||
All files which should be downloaded (nicknamed 'Targets') are mentioned |
|||
below the APT::Acquire::Targets scope. 'deb' is here the type of the |
|||
sources.list entry the file should be acquired for. The only other |
|||
supported value is hence 'deb-src'. Beware: You can't specify multiple |
|||
types here and you can't download the same URI for multiple types! |
|||
|
|||
After the type you can pick any valid and unique string which preferable |
|||
refers to the file it downloads (In the example we picked 'Packages'). |
|||
This string is never shown or used. |
|||
|
|||
All targets have three main properties you can define: |
|||
* URI: The identifier of the file to be downloaded as used in the |
|||
Release file. It is also the relative location of the file from the |
|||
Release file. You can neither download from a different server |
|||
entirely (absolute URI) nor should you try to access directories above |
|||
the Release file (e.g. "../../"). |
|||
* ShortDescription: Very short string intended to be displayed to the |
|||
user e.g. while reporting progress. apt will e.g. use this string in |
|||
the last line to indicate progress of e.g. the download of a specific |
|||
item. |
|||
* Description: A preferable human understandable and readable identifier |
|||
of which file is acquired exactly. Mainly used for progress reporting |
|||
and error messages. apt will e.g. use this string in the Get/Hit/Err |
|||
progress lines. |
|||
|
|||
Additional optional properties: |
|||
* flat{URI,Description}: APT supports two types of repositories: |
|||
dists-style repositories which are the default and by far the most |
|||
common which are named after the fact that the files are in an |
|||
elaborated directory structure. In contrast a flat-style repositories |
|||
lumps all files together in one directory. Support for these flat |
|||
repositories exists mainly for legacy purposes only. It is therefore |
|||
recommend to not set these values. |
|||
* Optional: The default value is 'true' and should be kept at this |
|||
value. If enabled the acquire system will skip the download if the |
|||
file isn't mentioned in the Release file. Otherwise this is treated as |
|||
a hard error and the update process fails. |
|||
|
|||
|
|||
Note that the acquire system will automatically choose to download |
|||
a compressed file if it is available and uncompress it for you, just as |
|||
it will also use pdiff patching if provided by the repository and |
|||
enabled by the user. You only have to ensure that the Release file |
|||
contains the information about the compressed files/pdiffs to make this |
|||
happen. NO properties have to be set to enable this. |
|||
|
|||
# More examples |
|||
|
|||
The stanzas for Translation-* files as well as for Sources files would |
|||
look like this: |
|||
|
|||
APT::Acquire::Targets { |
|||
deb::Translations { |
|||
URI "$(COMPONENT)/i18n/Translation-$(LANGUAGE)"; |
|||
ShortDescription "Translation-$(LANGUAGE)"; |
|||
Description "$(SITE) $(RELEASE)/$(COMPONENT) Translation-$(LANGUAGE)"; |
|||
|
|||
flatURI "$(LANGUAGE)"; |
|||
flatDescription "$(SITE) $(RELEASE) Translation-$(LANGUAGE)"; |
|||
}; |
|||
|
|||
deb-src::Sources { |
|||
URI "$(COMPONENT)/source/Sources"; |
|||
ShortDescription "Sources"; |
|||
Description "$(SITE) $(RELEASE)/$(COMPONENT) Sources"; |
|||
|
|||
flatURI "Sources"; |
|||
flatDescription "$(SITE) $(RELEASE) Sources"; |
|||
|
|||
Optional "false"; |
|||
}; |
|||
}; |
|||
|
|||
# Substitution variables |
|||
|
|||
As seen in the examples, properties can contain placeholders filled in |
|||
by the acquire system. The following variables are known; note that |
|||
unknown variables have no default value nor are they touched: They are |
|||
printed literally. |
|||
|
|||
* $(SITE): An identifier of the site we access, e.g. |
|||
"http://example.org/". |
|||
* $(RELEASE): This is usually an archive- or codename, e.g. "stable" or |
|||
"stretch". Note that flat-style repositories do not have a archive- |
|||
or codename per-se, so the value might very well be just "/" or so. |
|||
* $(COMPONENT): as given in the sources.list, e.g. "main", "non-free" or |
|||
"universe". Note that flat-style repositories again do not really |
|||
have a meaningful value here. |
|||
* $(LANGUAGE): Values are all entries (expect "none") of configuration |
|||
option Acquire::Languages, e.g. "en", "de" or "de_AT". |
|||
|
|||
These values are defined both for 'deb' as well as 'deb-src' targets. |
|||
'deb' targets additional have the variable: |
|||
|
|||
* $(ARCHITECTURE): Values are all entries of configuration option |
|||
APT::Architectures (potentially modified by sources.list options), |
|||
e.g. "amd64", "i386" or "armel". |
|||
@ -0,0 +1,54 @@ |
|||
#!/bin/sh |
|||
set -e |
|||
|
|||
TESTDIR=$(readlink -f $(dirname $0)) |
|||
. $TESTDIR/framework |
|||
|
|||
setupenvironment |
|||
configarchitecture 'amd64' |
|||
configcompression '.' 'gz' |
|||
|
|||
buildsimplenativepackage 'foo' 'amd64' '1' 'unstable' |
|||
|
|||
setupaptarchive --no-update |
|||
changetowebserver |
|||
|
|||
testsuccessequal "Get:1 http://localhost:8080 unstable InRelease [$(stat -c%s aptarchive/dists/unstable/InRelease) B] |
|||
Get:2 http://localhost:8080 unstable/main Sources [$(stat -c%s aptarchive/dists/unstable/main/source/Sources.gz) B] |
|||
Get:3 http://localhost:8080 unstable/main amd64 Packages [$(stat -c%s aptarchive/dists/unstable/main/binary-amd64/Packages.gz) B] |
|||
Get:4 http://localhost:8080 unstable/main Translation-en [$(stat -c%s aptarchive/dists/unstable/main/i18n/Translation-en.gz) B] |
|||
Reading package lists..." aptget update |
|||
|
|||
testempty find rootdir/var/lib/apt/lists -name '*Contents*' |
|||
|
|||
cat > rootdir/etc/apt/apt.conf.d/content-target.conf <<EOF |
|||
APT::Acquire::Targets::deb::Contents { |
|||
URI "\$(COMPONENT)/Contents-\$(ARCHITECTURE)"; |
|||
ShortDescription "Contents"; |
|||
Description "\$(SITE) \$(RELEASE)/\$(COMPONENT) \$(ARCHITECTURE) Contents"; |
|||
}; |
|||
EOF |
|||
|
|||
testsuccessequal "Hit http://localhost:8080 unstable InRelease |
|||
Get:1 http://localhost:8080 unstable/main amd64 Contents [$(stat -c%s aptarchive/dists/unstable/main/Contents-amd64.gz) B] |
|||
Reading package lists..." aptget update |
|||
|
|||
testequal 'rootdir/var/lib/apt/lists/localhost:8080_dists_unstable_main_Contents-amd64' find rootdir/var/lib/apt/lists -name '*Contents*' |
|||
testsuccess cmp 'rootdir/var/lib/apt/lists/localhost:8080_dists_unstable_main_Contents-amd64' 'aptarchive/dists/unstable/main/Contents-amd64' |
|||
|
|||
# no automatic uncompress based on the name please, |
|||
# only if we downloaded a compressed file, but target was uncompressed |
|||
cat > rootdir/etc/apt/apt.conf.d/content-target.conf <<EOF |
|||
APT::Acquire::Targets::deb::Contents { |
|||
URI "\$(COMPONENT)/Contents-\$(ARCHITECTURE).gz"; |
|||
ShortDescription "Contents.gz"; |
|||
Description "\$(SITE) \$(RELEASE)/\$(COMPONENT) \$(ARCHITECTURE) Contents.gz"; |
|||
}; |
|||
EOF |
|||
|
|||
testsuccessequal "Hit http://localhost:8080 unstable InRelease |
|||
Get:1 http://localhost:8080 unstable/main amd64 Contents.gz [$(stat -c%s aptarchive/dists/unstable/main/Contents-amd64.gz) B] |
|||
Reading package lists..." aptget update |
|||
|
|||
testequal 'rootdir/var/lib/apt/lists/localhost:8080_dists_unstable_main_Contents-amd64.gz' find rootdir/var/lib/apt/lists -name '*Contents*' |
|||
testsuccess cmp 'rootdir/var/lib/apt/lists/localhost:8080_dists_unstable_main_Contents-amd64.gz' 'aptarchive/dists/unstable/main/Contents-amd64.gz' |
|||
Loading…
Reference in new issue