amprolla is an apt repository merger originally intended for use with the Devuan infrastructure. This version is the third iteration of the software.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

93 lines
3.5 KiB

#!/bin/sh
5 years ago
# See LICENSE file for copyright and license details.
5 years ago
# Orchestration of incremental updates
# Make sure these correlate to lib/config.py
AMPROLLA_SOURCE_DIR="${AMPROLLA_SOURCE_DIR:-/home/amprolla/amprolla}"
REPO_ROOT="${REPO_ROOT:-/home/amprolla/amprolla}"
Fix locking by using flock instead of tombstone files. Necessary changes to orchestrate.sh: The way I understand things, there are three processes (A,B,C) A. amprolla_update is run with orchestrate.sh (very often!): 1. there is always a consistent working set in merged which usually points to merged-production 2. before amprolla_update, merged switches to merged-staging (why here and not after amprolla_update?) 3. amprolla_update works against -volatile, during this process that directory is not necessarily fully consistent 4. after amprolla_update, merged-volatile is synchronised to merged-production 5. merged switches to merged-production 6. merged-volatile is synchronised to merged-staging 7. merged-production is synchronised to pkgmaster B. Sometimes amprolla_merge is run C. Sometimes amprolla_merge_contents + amprolla_merge are run The *intent* of the implemented locks appears to be that only one process out of A,B,C is active at a time, but the reality is that this is not formally provided, timing just happens to match, but shouldn't be relied upon: Existence of an active lock does prevent A, B or C from starting. But in A the lock is only active while point A.3. is executed, everything else runs without an active lock, so if e.g. A.4 to A.7 take long enough that A.4 is running again while A.7 is happening, bogus data will be synchronised to pkgmaster. Since A.7 is a network operation and A.5 and A.6 are disk operations, delays *could* happen and they could lead to this scenario (maybe it has happened at some point). That's why I added the support for the --no-lock-I-am-sure argument to amprolla_update, and instead obtain the lock as a step A.0 that is valid throughout all process A in the orchestrate.sh script. With that, I think there is now only need for 2 directories: -volatile and -production, with A being redefined as: A. amprolla_update is run with orchestrate.sh (very often!): 0. obtain amprolla lock and exit if it can't be obtained 1. there is always a consistent working set in merged which usually points to merged-production 2. amprolla_update --no-lock-I-am-sure works against -volatile, during this process that directory is not necessarily fully consistent 3. merged switches to merged-volatile 4. merged-volatile is synchronised to merged-production 5. merged switches to merged-production 7. merged-production is synchronised to pkgmaster I have adapted orchestrate.sh and therefore merged-staging is not used with these patches and could be removed in theory.
3 years ago
AMPROLLA_LOCK="${AMPROLLA_LOCK:-/run/lock/amprolla.lock}"
RSYNC_URL="${RSYNC_URL:-mirror@pkgmaster.devuan.org:/home/mirror/devuan}"
#
# Pseudo-booleans, will be tested with -n (see test(1)).
# This allows the admin to easily run certain steps or all of them.
#
# Enabled by default, can be disabled in the environment
DO_AMPROLLA_UPDATE="${DO_AMPROLLA_UPDATE:-YES}"
DO_AMPROLLA_PUSH="${DO_AMPROLLA_PUSH:-YES}"
# Disabled by default, can be enabled in the environment
DO_AMPROLLA_MERGE_TRANSLATIONS="${DO_AMPROLLA_MERGE_TRANSLATIONS:-}"
DO_AMPROLLA_MERGE_CONTENTS="${DO_AMPROLLA_MERGE_CONTENTS:-}"
DO_AMPROLLA_MERGE="${DO_AMPROLLA_MERGE:-}"
# Force amprolla_merge if admin requested to run at least one of:
# amprolla_merge_contents or amprolla_merge_translations
if [ -n "${DO_AMPROLLA_MERGE_CONTENTS}${DO_AMPROLLA_MERGE_TRANSLATIONS}" ]; then
DO_AMPROLLA_MERGE="YES"
fi
Fix locking by using flock instead of tombstone files. Necessary changes to orchestrate.sh: The way I understand things, there are three processes (A,B,C) A. amprolla_update is run with orchestrate.sh (very often!): 1. there is always a consistent working set in merged which usually points to merged-production 2. before amprolla_update, merged switches to merged-staging (why here and not after amprolla_update?) 3. amprolla_update works against -volatile, during this process that directory is not necessarily fully consistent 4. after amprolla_update, merged-volatile is synchronised to merged-production 5. merged switches to merged-production 6. merged-volatile is synchronised to merged-staging 7. merged-production is synchronised to pkgmaster B. Sometimes amprolla_merge is run C. Sometimes amprolla_merge_contents + amprolla_merge are run The *intent* of the implemented locks appears to be that only one process out of A,B,C is active at a time, but the reality is that this is not formally provided, timing just happens to match, but shouldn't be relied upon: Existence of an active lock does prevent A, B or C from starting. But in A the lock is only active while point A.3. is executed, everything else runs without an active lock, so if e.g. A.4 to A.7 take long enough that A.4 is running again while A.7 is happening, bogus data will be synchronised to pkgmaster. Since A.7 is a network operation and A.5 and A.6 are disk operations, delays *could* happen and they could lead to this scenario (maybe it has happened at some point). That's why I added the support for the --no-lock-I-am-sure argument to amprolla_update, and instead obtain the lock as a step A.0 that is valid throughout all process A in the orchestrate.sh script. With that, I think there is now only need for 2 directories: -volatile and -production, with A being redefined as: A. amprolla_update is run with orchestrate.sh (very often!): 0. obtain amprolla lock and exit if it can't be obtained 1. there is always a consistent working set in merged which usually points to merged-production 2. amprolla_update --no-lock-I-am-sure works against -volatile, during this process that directory is not necessarily fully consistent 3. merged switches to merged-volatile 4. merged-volatile is synchronised to merged-production 5. merged switches to merged-production 7. merged-production is synchronised to pkgmaster I have adapted orchestrate.sh and therefore merged-staging is not used with these patches and could be removed in theory.
3 years ago
# Don't do anything if we can't lock
flock -n -x "${AMPROLLA_LOCK}" /bin/sh << EOF
# Ensure directories exist
[ -d "${REPO_ROOT}/merged-volatile" ] || mkdir "${REPO_ROOT}/merged-volatile"
[ -d "${REPO_ROOT}/merged-production" ] || mkdir "${REPO_ROOT}/merged-production"
# Each Amprolla step will exit with an error code if something goes wrong
Fix locking by using flock instead of tombstone files. Necessary changes to orchestrate.sh: The way I understand things, there are three processes (A,B,C) A. amprolla_update is run with orchestrate.sh (very often!): 1. there is always a consistent working set in merged which usually points to merged-production 2. before amprolla_update, merged switches to merged-staging (why here and not after amprolla_update?) 3. amprolla_update works against -volatile, during this process that directory is not necessarily fully consistent 4. after amprolla_update, merged-volatile is synchronised to merged-production 5. merged switches to merged-production 6. merged-volatile is synchronised to merged-staging 7. merged-production is synchronised to pkgmaster B. Sometimes amprolla_merge is run C. Sometimes amprolla_merge_contents + amprolla_merge are run The *intent* of the implemented locks appears to be that only one process out of A,B,C is active at a time, but the reality is that this is not formally provided, timing just happens to match, but shouldn't be relied upon: Existence of an active lock does prevent A, B or C from starting. But in A the lock is only active while point A.3. is executed, everything else runs without an active lock, so if e.g. A.4 to A.7 take long enough that A.4 is running again while A.7 is happening, bogus data will be synchronised to pkgmaster. Since A.7 is a network operation and A.5 and A.6 are disk operations, delays *could* happen and they could lead to this scenario (maybe it has happened at some point). That's why I added the support for the --no-lock-I-am-sure argument to amprolla_update, and instead obtain the lock as a step A.0 that is valid throughout all process A in the orchestrate.sh script. With that, I think there is now only need for 2 directories: -volatile and -production, with A being redefined as: A. amprolla_update is run with orchestrate.sh (very often!): 0. obtain amprolla lock and exit if it can't be obtained 1. there is always a consistent working set in merged which usually points to merged-production 2. amprolla_update --no-lock-I-am-sure works against -volatile, during this process that directory is not necessarily fully consistent 3. merged switches to merged-volatile 4. merged-volatile is synchronised to merged-production 5. merged switches to merged-production 7. merged-production is synchronised to pkgmaster I have adapted orchestrate.sh and therefore merged-staging is not used with these patches and could be removed in theory.
3 years ago
# In that case we should stop the run.
# Notice we locked in this script, so we don't need the amprolla lock
# to run an amprolla step. We can't even get the lock if this code runs.
# amprolla_merge_translations must happen before merge
if [ -n "${DO_AMPROLLA_MERGE_TRANSLATIONS}" ]; then
python3 "$AMPROLLA_SOURCE_DIR/amprolla_merge_translations.py" --no-lock-I-am-sure || exit 1
fi
# amprolla_merge_contents must happen before merge
if [ -n "${DO_AMPROLLA_MERGE_CONTENTS}" ]; then
python3 "$AMPROLLA_SOURCE_DIR/amprolla_merge_contents.py" --no-lock-I-am-sure || exit 1
fi
# amprolla_merge must happen after merge_contents and merge_translations
if [ -n "${DO_AMPROLLA_MERGE}" ]; then
python3 "$AMPROLLA_SOURCE_DIR/amprolla_merge.py" --no-lock-I-am-sure || exit 1
fi
# amprolla_update is the very last step
if [ -n "${DO_AMPROLLA_UPDATE}" ]; then
python3 "$AMPROLLA_SOURCE_DIR/amprolla_update.py" --no-lock-I-am-sure || exit 1
fi
# Actually propagate any changes
if [ -n "${DO_AMPROLLA_PUSH}" ]; then
# Switch /merged to merged-volatile
ln -snf "$REPO_ROOT"/merged-volatile "$REPO_ROOT"/merged
# Here merged-production is 'dirty' (not necessarily consistent)
printf "rsyncing volatile to production... "
rsync --delete -raX "$REPO_ROOT"/merged-volatile/* "$REPO_ROOT"/merged-production
echo "done!"
# Switch /merged back to merged-production, as it should be unless synchronising
ln -snf "$REPO_ROOT"/merged-production "$REPO_ROOT"/merged
printf "rsyncing production to pkgmaster... "
rsync --delete -raX \
"$REPO_ROOT"/merged-production/ "${RSYNC_URL}/merged"
echo "done!"
# handle obsolete package logs
cat "$REPO_ROOT"/log/*-oldpackages.txt | sort | uniq > "$REPO_ROOT"/log/oldpackages.txt
_logfiles="libsystemd bannedpackages"
mkdir -p "$REPO_ROOT"/log/t
for i in \$_logfiles; do
sort "$REPO_ROOT/log/\${i}.txt" | uniq > "$REPO_ROOT/log/t/\${i}.txt"
done
cp -f "$REPO_ROOT"/log/t/*.txt "$REPO_ROOT"/log/
rsync "$REPO_ROOT"/log/t/*.txt "${RSYNC_URL}"
rsync "$REPO_ROOT"/log/oldpackages.txt "$REPO_ROOT"/log/amprolla.txt "${RSYNC_URL}"
fi
Fix locking by using flock instead of tombstone files. Necessary changes to orchestrate.sh: The way I understand things, there are three processes (A,B,C) A. amprolla_update is run with orchestrate.sh (very often!): 1. there is always a consistent working set in merged which usually points to merged-production 2. before amprolla_update, merged switches to merged-staging (why here and not after amprolla_update?) 3. amprolla_update works against -volatile, during this process that directory is not necessarily fully consistent 4. after amprolla_update, merged-volatile is synchronised to merged-production 5. merged switches to merged-production 6. merged-volatile is synchronised to merged-staging 7. merged-production is synchronised to pkgmaster B. Sometimes amprolla_merge is run C. Sometimes amprolla_merge_contents + amprolla_merge are run The *intent* of the implemented locks appears to be that only one process out of A,B,C is active at a time, but the reality is that this is not formally provided, timing just happens to match, but shouldn't be relied upon: Existence of an active lock does prevent A, B or C from starting. But in A the lock is only active while point A.3. is executed, everything else runs without an active lock, so if e.g. A.4 to A.7 take long enough that A.4 is running again while A.7 is happening, bogus data will be synchronised to pkgmaster. Since A.7 is a network operation and A.5 and A.6 are disk operations, delays *could* happen and they could lead to this scenario (maybe it has happened at some point). That's why I added the support for the --no-lock-I-am-sure argument to amprolla_update, and instead obtain the lock as a step A.0 that is valid throughout all process A in the orchestrate.sh script. With that, I think there is now only need for 2 directories: -volatile and -production, with A being redefined as: A. amprolla_update is run with orchestrate.sh (very often!): 0. obtain amprolla lock and exit if it can't be obtained 1. there is always a consistent working set in merged which usually points to merged-production 2. amprolla_update --no-lock-I-am-sure works against -volatile, during this process that directory is not necessarily fully consistent 3. merged switches to merged-volatile 4. merged-volatile is synchronised to merged-production 5. merged switches to merged-production 7. merged-production is synchronised to pkgmaster I have adapted orchestrate.sh and therefore merged-staging is not used with these patches and could be removed in theory.
3 years ago
EOF