The recent commit to make sure quilt can't break the build (git
commit 63d6c47) made a switch from dpkg-source to
dpkg-buildpackage. dpkg-buildpackage generates a *_source.changes
file which isn't present when using dpkg-source. While this file
is nice it doesn't provide any real value for us, but what's more
important, it might break existing setups:
If all files are copied around from one Jenkins job to another
then in the final jenkins job the changes file from the binaries
job and the source.changes one end up together. If those are
passed to other tools then things might break (see github issue
67 as an example). So let's just remove the file after the
dpkg-buildpackage call to provide the used behaviour.
When using the dpkg 3.0 (quilt) format (see debian/source/format)
there might be quilt patches still applied in the source tree,
either because of a broken build or under certain - though not yet
clearly identified - situations which seem to end with a
"diff: standard output: Broken pipe" message in the build log.
This then might end up in a build failure like:
| + cd /var/lib/jenkins/workspace/glusterfs-source
| + dpkg-source -i -I -b source
| dpkg-source: info: using source format `3.0 (quilt)'
| diff: standard output: Broken pipe
| diff: standard output: Broken pipe
| dpkg-source: info: building glusterfs using existing ./glusterfs_3.2.7.orig.tar.gz
| dpkg-source: info: local changes detected, the modified files are:
| source/ChangeLog
| source/doc/gluster.8
| source/doc/glusterfs.8
| source/libglusterfs/src/common-utils.c
| source/rpc/rpc-lib/src/rpcsvc.c
| source/xlators/mgmt/glusterd/src/glusterd-rpc-ops.c
| dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/glusterfs_3.2.7-2~bpo60+1.diff.3Wkisr
| dpkg-source: info: you can integrate the local changes with dpkg-source --commit
While everything has been cleanly integrated in debian/patches/
the build process fails because it looks like there have been
modifications to upstream sources whereas it's just wrong
information about patches applied.
So instead make sure we enter a clean git repository before doing
any changes (this can be skipped via SKIP_GIT_CLEANUP=true) and
also unapply all present quilt patches (this can be skipped via
SKIP_QUILT_CLEANUP=true). Finally replace the dpkg-source call
with dpkg-buildpackage and its --source-option=--unapply-patches
feature. While the combination of the git cleanup, quilt cleanup
and dpkg-source->dpkg-buildpackage is more than explicitely
necessary this provides more flexibility if needed.
Thanks: Raphael Hertzog <hertzog@debian.org> for his dpkg related feedback
This makes only the appropriate part of lintian output to go to test
case. The error line is put as is into Stacktrace (i.e. <failure> tag)
and "Standard Output" is filled in with lintian tag information (if not
disabled).
This patch improves representation of lintian results in Jenkins. All
the lintian results are now in package called "lintian" to separate them
from other aggregated tests. The next level of grouping in Jenkins are
classes and for those package name from lintian output is used. The
spaces in package field in lintian output are preserved, as Jenkins
doesn't care about "class" names with spaces. Finally lintian tag and
arguments (i.e. rest of line) form the test case name.
First thing first, original conversion creating intermediate REXML::Text
object and then converting it to string results in a more complex
wrapper around REXML::Text::normalize, which actually does the job which
is simply translating some characters to XML entities.
Although I know which characters are going to be converted
(REXML::DocType::DEFAULT_ENTITIES which is: ", &, < and >) and those
should not make difference in the parts of output being parsed it still
feels better for me to parse raw unmodified lintian output.
So instead of converting lintian output at once this change XML-escapes
string where needed. As additional bonus original lintian margins are
kept in output (previously those were squeezed).
The changes from commits 7a974f4f and 21b8527d don't work when
running on Ruby 1.8:
| /usr/bin/lintian-junit-report:85:in `popen': can't convert Array into String (TypeError)
| from /usr/bin/lintian-junit-report:85
This patch adds support for passing options directly to lintian,
allowing things like
lintian-junit-report \
--lintian-opt=-Xsource-copyright,copyright-file *.deb
for those who like to break Debian Policy on purpose :). To be sure that
arguments are passed correctly lintian process is now executed with
IO::popen.
Provide at least some basic informational project description
for display at https://github.com/mika/jenkins-debian-glue
While at it rename README.org to README.md since we don't
include our official docs at that place any longer.
* Do not test if the file exist. It blocks the usage of 'export PRE_SOURCE_HOOK=/home/git/foo.sh version branch'
* Use the same declaration between generate-git-snapshot & generate-svn-snapshot
* Match the similar declaration in the git & svn scripts
If the PBUILDER_CONFIG configuration is pointing to a file then
its argument is used for pbuilder's --configfile option.
For example when using the following option in /etc/jenkins/debian_glue:
PBUILDER_CONFIG=/etc/jenkins/pbuilderrc
and the following configuration is present in /etc/jenkins/pbuilderrc:
CCACHEDIR=/var/cache/pbuilder/ccache
then jenkins-debian-glue and its cowdancer system will use the
provided directory as ccache to speed up builds.
Development sponsored by Sipwise GmbH.
There's no point in running the reprepro command if there's nothing
to do, even though neither the perl nor the reprepro command line
would complain about it.
There might be packages left in the repos:
| # reprepro -b /srv/repository/ clearvanished
| There are still packages in 'foo/bar|main|amd64', not removing (give --delete to do so)!
| There are still packages in 'foo/bar|main|i386', not removing (give --delete to do so)!
| There are still packages in 'foo/bar|main|source', not removing (give --delete to do so)!
So get rid of the repos, no matter what.
When providing user specific repositories it's important
to also get rid of them (e.g. when the $user/$feature branch
gets removed), so make this procedure as simple as possible.
If the changelog contains the string "Version" then when
identifying the original version number this can go wrong
obviously, e.g.:
| + version_information
| ++ dpkg-parsechangelog --count 1
| ++ awk '/Version/ {print $2}'
| + ORIG_VERSION='1.17.2
| Version'
Adjust the dpkg-parsechangelog command line to what's being
used in all the other command lines.
Starting with Debian/wheezy /usr/local/sbin, /usr/sbin and /sbin
aren't available inside $PATH of non-root users under sudo if
Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
is not defined in /etc/sudoers.
Previously, it was doing "cd .." to move up one level, which doesn't
work correctly if $SOURCE_DIRECTORY is multiple levels deep, e.g.
SOURCE_DIRECTORY=source/subproject
While testing jenkins-debian-glue-piuparts builds the following showed
up in the logs:
| [...]
| 8m7.3s DUMP:
| (Reading database ... 6686 files and directories currently installed.)
| Removing sudo ...
| You have asked that the sudo package be removed,
| but no root password has been set.
| Without sudo, you may not be able to gain administrative privileges.
|
| If you would prefer to access the root account with su(1)
| or by logging in directly,
| you must set a root password with "sudo passwd".
|
| If you have arranged other means to access the root account,
| and you are sure this is what you want,
| you may bypass this check by setting an environment variable
| (export SUDO_FORCE_REMOVE=yes).
|
| Refusing to remove sudo.
| dpkg: error processing sudo (--purge):
| subprocess installed pre-removal script returned error exit status 1
| Errors were encountered while processing:
| sudo
| [...]
While piuparts doesn't report this as an error it's still quite
confusing. The scripts in /etc/piuparts/scripts/ handle issues like
this one, so if SCRIPTSDIR is not set then run it with
/etc/piuparts/scripts/ enabled by default.
Thanks: Holger Levsen for the hint
For some reason current versions of Jenkins with TAP plugin fail to
retrieve the artifacts and the jenkins-debian-glue-piuparts job fails
with:
| Unable to find a build for artifact copy from: jenkins-debian-glue-binaries
| Build step 'Copy artifacts from another project' marked build as failure
| TAP Reports Processing: START
| Looking for TAP results report in workspace using pattern: piuparts.tap
| Did not find any matching files.
The 'Parameter filters' option from the 'Copy artifacts from another
project' setting gets set to "architecture=$architecture" when using
the 'projectName' tag. By using
'jenkins-debian-glue-binaries/architecture=$architecture' inside the
'project' tag the 'Parameter filters' option is empty again and does
what it is supposed to do.
On Debian/wheezy LANG and LC_ALL are empty by default (and therefore
defaulting to 'C' locale), running piuparts_tap therefore fails with:
| /usr/bin/piuparts_tap:26:in `gsub': invalid byte sequence in US-ASCII (ArgumentError)
| from /usr/bin/piuparts_tap:26:in `<main>'
The real fix would be to set LANG/LC_ALL accordingly, but we can't
control this from inside the piuparts_tap script, so let's
work around this issue by setting Encoding::UTF_8 explicitely.
Thanks: Christian Hofstaedtler <ch@zeha.at> for feedback
Jenkins depends on a working Java runtime package but has just
'java2-runtime' in its depends (which is a virtual package),
this results in a broken default installation. So make sure
a Java runtime package is picked up by explicitely installing
default-jre-headless.
Thanks: Cyril Brulebois for the bug report
Reported in https://github.com/mika/jenkins-debian-glue/issues/47
Compare:
| % git-buildpackage -nc --git-force-create --git-ignore-new -S -us -uc --git-verbose --git-builder=/bin/true --git-cleaner=/bin/true
| gbp:debug: ['git', 'rev-parse', '--show-cdup']
| gbp:debug: ['git', 'rev-parse', '--is-bare-repository']
| gbp:debug: /bin/true [] []
| gbp:debug: ['git', 'symbolic-ref', 'HEAD']
| gbp:error: Currently not on a branch
with:
| % git-buildpackage -nc --git-force-create --git-ignore-new -S -us -uc --git-verbose --git-builder=/bin/true --git-cleaner=/bin/true --git-ignore-branch
| gbp:debug: ['git', 'rev-parse', '--show-cdup']
| gbp:debug: ['git', 'rev-parse', '--is-bare-repository']
| gbp:debug: /bin/true [] []
| gbp:debug: ['git', 'symbolic-ref', 'HEAD']
| gbp:debug: ['git', 'show-ref', 'refs/heads/pristine-tar']
| gbp:debug: ['git', 'ls-tree', 'upstream/1.7.4']
| gbp:error: upstream/1.7.4 is not a valid treeish
using a recent git-buildpackage version.
The --git-ignore-branch option allows to build packages
from detached HEAD state, also see
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=661598
As the option is already present (at least) in gbp 0.5.10
this should be a safe way how to prevent problems with
recent versions of gbp.
Verified against git-buildpackage 0.6.0~git20130506 and 0.5.10.
At least when using the 3.0 quilt format in a Debian package there
might be changes left around in the working directory after
dpkg-source's execution:
| + dpkg-source -i -I -b source
| dpkg-source: info: using source format `3.0 (quilt)'
| dpkg-source: warning: patches have not been applied, applying them now (use --no-preparation to override)
| dpkg-source: info: applying 0001-Added-assertTrue-assertFalse-and-assertNotEquals-fun.patch
| dpkg-source: info: applying 0002-Fix-typo.patch
| dpkg-source: info: applying 0003-Added-assertItemsEquals-function.patch
| dpkg-source: info: building lua-unit using existing ./lua-unit_1.5.orig.tar.gz
| dpkg-source: info: building lua-unit in lua-unit_1.5-1~bpo70+1+0~1368438751.21+wheezy~1.gbp63ac0c.debian.tar.gz
| dpkg-source: info: building lua-unit in lua-unit_1.5-1~bpo70+1+0~1368438751.21+wheezy~1.gbp63ac0c.dsc
| ++ readlink -f debian/changelog
| + git checkout -- /var/lib/jenkins/workspace/lua-unit-source/source/debian/changelog
| + git checkout origin/pristine-tar
| error: You have local changes to 'luaunit.lua'; cannot switch branches.
| Build step 'Execute shell' marked build as failure
So avoid failing builds because of that and force switching
branch.
In some cases the '-I' param to dpkg-source is not enough. In this case
builds fail since dpkg-source complains about modified files.
This additionally adds (previously used) '-i' to the params, which
fixes those cases.
Michael Prokop