So far we set the distribution/codename to $release. Now it's
possible to point distribution/codename to a different value by
using $RELEASE_DISTRIBUTION.
Usage example:
export release=0.42-update
export RELEASE_DISTRIBUTION=jessie
will result (with default settings) in a repository under
/srv/repository/release/0.42-update/ with a setting of
"Codename: jessie" inside
/srv/repository/release/0.42-update/conf/distributions
Development sponsored by Sipwise GmbH, recorded as
MT#17701 in customers' ticket system.
To allow pushing packages to multiple release repositories
we can now use something like:
RELEASE_REPOSITORIES="/srv/repository/release/ce/${release} /srv/repository/release/pro/${release}"
iff $release is set.
Development sponsored by Sipwise GmbH, recorded as
MT#17701 in customers' ticket system.
During release rebuilds we way too often see:
| W: Failed to fetch https://[snip]/autobuild/dists/release-trunk-jessie/main/binary-amd64/Packages Hash Sum mismatch
during the `cowbuilder --update` runs.
apt versions >=1.1 are supposed to be improve that situation,
but until we have this apt version available everywhere we
need to find workarounds.
This applies only in cases where the debian/changelog is explicitly
set to "UNRELEASED" which makes the generate-git-snapshot use plain
'dch' instead of 'gbp dch' or 'git-dch' which in turn passes the
generated version string directly as the new package version.
This is implemented as an additional option to avoid breaking
existing users which are just happy with the current behaviour.
Since git-buildpackage v0.6.24 the `git-buildpackage` and
`git-dch` commands no longer exist, instead we need to use
`gbp buildpackage` and `gbp dch`.
Closes: #799183
Thanks: Daniele E. Domenichelli <daniele.domenichelli@gmail.com> for the bug report
Quoting from adt-run(1):
| --source dsc
| [...]
| The ordering is significant: each --source option
| should precede options whose dependencies are to be
| satisfied by the binaries it produces.
|
| --binary deb
| [...]
| The ordering is significant, as for --source. In
| particular, if a subsequent source package will build
| a binary of the same name, that will be used from then
| on, and deb will be ignored.
So we need to refer to the *.deb files *before* the --build-tree
in the command line, otherwise autopkgtest/adt-run installs the
Debian packages from a present Debian repository instead of
testing the Debian packages under test (being produced as part of
the running *-binaries job).
While at it drop the workaround to identify Debian package files,
instead let's just use all the *.deb files available in
/tmp/buildd/.
Thanks: Axel Beckert <abe@debian.org> for spotting the issue + Martin Pitt <mpitt@debian.org> for helping resolve this issue
option to just check the files listed in the provided
argument of the --file-list option, useful to check
only modified files instead of checking all files
in the source tree
Otherwise when e.g. building Debian/jessie on Ubuntu 14.04 causes:
| ERROR: ld.so: object 'libeatmydata.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
The REPOSITORY variable is used to set the main/base path for the
(reprepro) repositories (defaulting to /srv/repository). This
REPOSITORY setting could be defined via /etc/jenkins/debian_glue.
Since we are sourcing /etc/jenkins/debian_glue in our scripts
this overwrites a possibly existing environment variables named
REPOSITORY, preventing us to overwrite the REPOSITORY setting on
demand as needed as soon as REPOSITORY is configured in
/etc/jenkins/debian_glue.
To fix that behaviour we hereby rename the REPOSITORY variable to
DEFAULT_REPOSITORY while being backwards compatible. If you have
configured REPOSITORY via /etc/jenkins/debian_glue please rename
it to DEFAULT_REPOSITORY instead. In an upcoming release we might
drop backwards compatibility.
Thanks: Jean Baptiste Favre for the bug report
eatmydata support gets enabled by default if it's installed on
the host system and when building for a recent Debian/Ubuntu
version (Debian/jessie + Ubuntu/vivid or newer). Usage can be
forced via USE_EATMYDATA=true and disabled via
USE_EATMYDATA=false.
ccache support can be enabled via USE_CCACHE=true
Thanks to Franco (nextime) Lanza for the initial patch in
https://github.com/mika/jenkins-debian-glue/pull/125
If multiple builds are running at the same time then the
/var/cache/pbuilder/build directory shows up as shared bindmounts
in all environments. Besides being ugly this also causes the
first build environment to fail to cleanly unmount it and
therefore failing the build.
As a workaround the pbuilderrc could include the following
snippet, as suggested by Philipp Hahn:
| mount () {
| case "$1" in
| -obind) /bin/mount --make-private "$@" ;;
| *) /bin/mount "$@" ;;
| esac
| }
Leave this decision up to the user though (to avoid any possible
side effects depending on the user's environment which we can't
control).
The only reason why /var/cache/pbuilder/build should be needed as
bindmount at all is pbuilder-hookdir/C10shell. This pbuilder hook
copies the according build directory away on failing builds (and
if requested to do so). If this feature is requested and it
fails to access /var/cache/pbuilder/build then inform the user
that /var/cache/pbuilder/build should be added to the
USER_BINDMOUNTS configuration setting.
See http://lists.alioth.debian.org/pipermail/pbuilder-maint/2015-August/005368.html
Thanks: Philipp Hahn <hahn@univention.de> for the bug report and analysis
We need variables like COWBUILDER_BASE for usage with
the lockfile variables, otherwise when PROVIDE_ONLY is set
we fail with:
| /usr/bin/build-and-provide-package: line 130: build_lockfile: unbound variable
The parallel tools support different options, so we need
to check for the according version. But since moreutils
might be present because of other packages depending on
them we should support it as alternative.
When upgrading from jenkins-debian-glue <=0.13 we run into:
| Unpacking jenkins-debian-glue (0.14.0+0~20150822132303.252~1.gbp3da4c8) over (0.13.0+0~20150727152008.249~1.gbp026ace) ...
| dpkg: error processing archive /var/cache/apt/archives/jenkins-debian-glue_0.14.0+0~20150822132303.252~1.gbp3da4c8_all.deb (--unpack):
| trying to overwrite '/usr/bin/checkbashism_tap', which is also in package jenkins-debian-glue-buildenv-taptools 0.13.0+0~20150727152008.249~1.gbp026ace
By using --disable-$checkname it's possible to disable specific
checks as needed. Also if a tool isn't present instead of
spitting out error messages instead just ignore the check.
When uploading towards Debian we shouldn't have to deal with
so many Debian packages, instead provide one single package
which depends on all relevant packages and if someone
wants to have just a specific subset installed on their
system they should do so through their own metapackage or
configuration management.
Thanks: Christian Hofstaedtler <christian@hofstaedtler.name>
Without root permissions we might not have write access to
/var/cache/pbuilder/, so instead use /var/run/lock/ as base
directory for writing our lockfiles.
During running cowbuilder builds we shouldn't update the
underlying cowbuilder base.cow because the hardlinks might
bring the build environment into an inconsistent state.
Also we shouldn't run multiple update operations in parallel,
since this suffers from the same problem.
Therefore we have to implement an according blocking mechanism
between a) parallel update runs and b) concurrent update and
build runs.
Thanks: Christian Hofstaedtler <christian@hofstaedtler.name> for testing + feedback and Victor Seva <linuxmaniac@torreviejawireless.org> for helping with the actual implementation
autopkgtest's adt-run has a defined exit status, quoting adt-run(1):
| EXIT STATUS
| 0 all tests passed
| 2 at least one test skipped
| 4 at least one test failed
| 6 at least one test failed and at least one test skipped
| 8 no tests in this package
| 12 erroneous package
| 16 testbed failure
| 20 other unexpected failures including bad usage
So don't just check for exit code '4', but also for 2, 6 and 8.
Thanks: Jan Alexander Steffens <jan.steffens@gmail.com> for reporting
Introduce the env variable JENKINS_DEBIAN_GLUE_QUIET to turn off the
bash option 'set -x' in various scripts.
To mute them, simply set the variable to any value.
At a customer of mine when executing repository_checker with its
--validate-incoming option to ensure that the incoming directory
is in a sane state we noticed problems with different compression
defaults. With the recent switch to Debian/jessie for the build
infrastructure we noticed that *.tar.xz files are left behind:
| 12:12:59 Checking for leftover files in incoming directory /srv/repository/release/release-mr0.42/incoming/release-mr0.42:
| [...]
| 12:12:59 /srv/repository/release/release-mr0.42/incoming/release-mr0.42/ngcp-ngcpcfg_0.26.1.1+0~mr0.42.1.tar.xz
| [...]
| 12:12:59 Leftover files found. Needs investigation.
This is caused by the fact that executing generate-git-snapshot
on Debian/jessie by default generates a *.tar.xz file nowadays,
quoting dpkg-source(1) :
| -Zcompression, --compression=compression
|
| Specify the compression to use for created files
| (tarballs and diffs). Note that this option will not
| cause existing tarballs to be recompressed, it only
| affects new files. Supported values are: gzip, bzip2,
| lzma and xz. The default is xz for formats 2.0 and
| newer, and gzip for format 1.0. xz is only supported
| since dpkg 1.15.5.
During the binary package build another tarball gets created via
dpkg-buildpackage's -sa option (to include the sources). In the
binary package build for an older release (like Debian/wheezy)
this won't default to *.tar.xz though. Therefore we end up with
an additional tarball, usually being *.tar.gz. Now we have a
*.tar.gz and a *.tar.xz next to each other and when simply
copying all files from binaries/ directory to the incoming
repository we end up with one unneeded tarball which is
identified as "leftover file(s) in incoming directory".
We sadly can't just default to build with dpkg-buildpackages' -b
option because then we don't get the source files in *any* binary
package build. The source files need to be present at least once
though to end up properly in the archive.
NOTE: When uploading artifacts e.g. via dput (instead of simply
copying them around) then this isn't a problem at all since
*.changes refers to the files that should be uploaded (and unused
files aren't included).
Since we execute this *only* when running through the release
repository steps (see release_repos()) it should be safe to
enable it by default. To disable this behaviour
DROP_UNUSED_DEBFILES=false still can be used.
Michael Prokop