Devuan deployment of britney2
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

272 lines
12 KiB

  1. A short introduction to migrations
  2. ==================================
  3. This is a technical introduction to how britney
  4. handles migrations. Being an introduction, it deliberately
  5. oversimplifies certain things at the expense of accuracy.
  6. It also covers common migration issues and how to fix
  7. them in :doc:`solutions-to-common-policy-issues`.
  8. The document is primarily aimed at contributors for
  9. distributions that want to understand the basics of
  10. britney and its migration rules.
  11. The documentation also aspires to be a general purpose document
  12. for britney that is applicable for multiple distributions.
  13. However, it does reference distribution-specific practises in
  14. some examples to prevent the documentation from becoming too
  15. abstract. Furthermore, the document assumes familiarity with
  16. Debian-based distribution practises and terminology (such as
  17. "suites" and "source package").
  18. A high level overview of britney and migrations
  19. -----------------------------------------------
  20. The purpose of britney is to (semi-)automatically select
  21. a number of migration items from a series of source suites
  22. (e.g. Debian unstable) that are ready to migrate to
  23. the target suite (e.g. Debian testing).
  24. The definition of "ready" can be summarized as satisfying all
  25. of the following points:
  26. 1. The migration items pass a number of policies for the target
  27. suite. Most of these policies are basically that the
  28. migration items do not regress on selected QA checks.
  29. * An item satisfying this part is called a `valid candiate`.
  30. 2. Installability will not regress as a result of
  31. migrating the migration items.
  32. * An item that (also) satisfies this part will be selected
  33. for migration.
  34. The keyword in both points being *regress*. If a package has an
  35. existing issue in the target suite, the item including a new version
  36. of that package is generally allowed to migrate if it has the same
  37. issue (as it is not a regression).
  38. This only leaves the definition of a migration items. They come
  39. in several variants defined in the next section.
  40. Migration items
  41. ---------------
  42. Internally, britney groups packages into migration items based on a
  43. few rules. There are several kinds of migration items and this
  44. document will only describe the source migration item.
  45. A source migration item is one upload of a source package, with
  46. associated binary packages once built.
  47. Once a new version of a source package appears in the source suite,
  48. britney will track it with a source migration item. As the
  49. binary packages are built and uploaded, they will be included into the
  50. migration item and various QA checks/policies will be applied to the
  51. item.
  52. Once britney deems the item ready, it will attempt to
  53. migrate the item (i.e. source with its binaries) to the
  54. target suite.
  55. As implied earlier, there are several other migration types,
  56. not covered in this document. They deal with cases like removals,
  57. rebuilds of existing binaries, etc.
  58. Migration phase 1: Policies / Excuses
  59. -------------------------------------
  60. To begin with, britney will apply a number of policies to
  61. all migration items. Each policy will rate each migration
  62. item and the combined results will be added into one of
  63. britney's output documents known as the "excuses" (exists in
  64. an HTML and a YAML variant). A migration item that passes all
  65. applicable policies will be labelled as a `valid candidate` in
  66. the excuses and continue to the next phase.
  67. The policies gives exactly one verdict to each item, some of
  68. these verdicts are:
  69. * The item passes the policy.
  70. * The policy is waiting for test suites before providing a
  71. pass/fail result (temporary failure).
  72. * The item fails the policy and the failure is believed to
  73. be "permanent" (given no external changes).
  74. * The item does not pass the policy, but britney has
  75. insufficient information to determine if the failure is
  76. persistent or not.
  77. It is important to note that all verdicts are based on the current
  78. data that britney has access to. This mean that without any change
  79. to the items themselves:
  80. 1. Items that passed originally may fail in a later britney run.
  81. 2. Likewise, items may go from a "permanent failure" to a pass.
  82. This can be seen in the following example case:
  83. 1. A new version of package is uploaded.
  84. * Britney processes the package and concludes that there no blocking bugs,
  85. so the package passes the bug policy.
  86. 2. Then before it migrates, someone files a blocking bug against
  87. the new version.
  88. * Britney reprocesses the package and now concludes it has a regression in
  89. the bug policy (i.e. the policy verdict goes from "pass" to "permanent fail").
  90. 3. The bug is examined and it is determined that the bug also affects the
  91. version in the target suite. The bug tracker is updated to reflect this.
  92. * Britney reprocesses the package again and now concludes there is a blocking
  93. bug, but it is not a regression (since it also affects the target suite).
  94. This means the policy verdict now go from "fail" to "pass".
  95. This is also applicable to e.g. the piuparts policy, where if the test is
  96. rescheduled on the piuparts side and the result changes as a result of that.
  97. Finally, the people running the britney instance can overrule any
  98. policy by applying a britney hint (see :ref:`hints` for more details), if they deem it
  99. necessary. One caveat here is that not all policies can be overridden
  100. directly and some will require the "ignore all policies"-hint (known
  101. as the `force`-hint).
  102. Since most policies are defined based on regressions,
  103. a hinted migration generally implies that the problem will not
  104. prevent future migrations for newer versions of the same source
  105. package (assuming that the problem is deterministic).
  106. Migration phase 2: Installability regression testing
  107. ----------------------------------------------------
  108. For the migration items that pass the previous phase, britney
  109. will do a test migration to see if anything becomes uninstallable.
  110. This is a more expensive test to ensure the migration does not cause
  111. installability regressions.
  112. The status of this phase is *not* included in the excuses. To debug
  113. problems here, the britney log file has to be examined. This requires
  114. a bit more technical insight as it has not been polished as much as
  115. the excuses.
  116. Confirming a migration
  117. ^^^^^^^^^^^^^^^^^^^^^^
  118. To start with; if a migration is accepted and "committed" (i.e. it will not
  119. be rolled back), britney will include in a line starting with ``final:`` like
  120. in this example::
  121. Apparently successful
  122. final: -cwltool,-libtest-redisserver-perl,-pinfo,-webdis,hol88
  123. start: 41+0: a-4:i-27:a-1:a-1:a-1:m-0:m-3:m-1:p-1:s-2
  124. orig: 41+0: a-4:i-27:a-1:a-1:a-1:m-0:m-3:m-1:p-1:s-2
  125. end: 41+0: a-4:i-27:a-1:a-1:a-1:m-0:m-3:m-1:p-1:s-2
  126. SUCCESS (182/177)
  127. The above example is a regular migration run where 4 source removal migration
  128. items and one source migration item where accepted (those listed on the
  129. ``final:`` line). The rest of the information are various statistical counters
  130. which are useful for other purposes beyond the scope of this document.
  131. When debugging a migration for an item that passed the previous phase, if the
  132. item appears on a ``final:`` line like that, then it is migrated. That is, the
  133. problem is most likely that the britney run crashes later or the britney's
  134. output is not committed to the archive (for reasons outside britney's control).
  135. On the flip side, if the migration item of interest does *not* appear in a
  136. final line, then the migration was rejected (or rolled back).
  137. Reminder: Migration items generally use the name of the source package. There
  138. are exceptions to that "rule" (but they are not common cases covered by this
  139. document).
  140. Debugging failed migration attempts
  141. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  142. Start by confirming that the migration item was not accepted (as described
  143. in the above section). If the migration item does not appear on a ``final:`` line,
  144. then we need to debug the actual migration attempts. Migration attempts look
  145. something like this::
  146. trying: -webdis
  147. accepted: -webdis
  148. ori: 41+0: a-4:i-27:a-1:a-1:a-1:m-0:m-3:m-1:p-1:s-2
  149. pre: 41+0: a-4:i-27:a-1:a-1:a-1:m-0:m-3:m-1:p-1:s-2
  150. now: 41+0: a-4:i-27:a-1:a-1:a-1:m-0:m-3:m-1:p-1:s-2
  151. all: -pinfo -webdis
  152. [...]
  153. trying: libaws
  154. skipped: libaws (0, 165, 11)
  155. got: 45+0: a-4:i-27:a-5:a-1:a-1:m-0:m-3:m-1:p-1:s-2
  156. * arm64: libaws-bin, libaws17.2.2017, libaws3.3.2.2-dev, liblog4ada3-dev
  157. [...]
  158. Trying easy from autohinter: asis/2017-1 dh-ada-library/6.12 [...]
  159. start: 41+0: a-4:i-27:a-1:a-1:a-1:m-0:m-3:m-1:p-1:s-2
  160. orig: 41+0: a-4:i-27:a-1:a-1:a-1:m-0:m-3:m-1:p-1:s-2
  161. easy: 261+0: a-26:i-49:a-23:a-23:a-23:m-22:m-25:m-23:p-23:s-24
  162. * amd64: asis-programs, libasis2017, libasis2017-dev, libaws-bin, [...]
  163. * i386: asis-programs, libasis2017, libasis2017-dev, libaws-bin, [...]
  164. * arm64: asis-programs, libasis2017, libasis2017-dev, libaws-bin, [...]
  165. * armel: asis-programs, libasis2017, libasis2017-dev, libaws-bin, [...]
  166. [...]
  167. FAILED
  168. This example has one succeeding migration (``-webdis``) and one failing
  169. (``libaws``) plus finally a failed ``easy``-hint with several packages.
  170. Both of the two first are "single item" migrations (i.e. the attempt only
  171. includes a single item in isolation). However, Britney can do multi-item
  172. migrations (even outside hints).
  173. Please keep in mind that items can attempted multiple times and accepted in a
  174. later attempt. It is not always immediately obvious, which attempt is better
  175. for debugging. When in doubt, it is *usually* easiest to look at the attempt
  176. with the least amount of new uninstallable packages.
  177. In the libaws example, a total of 4 binary packages become
  178. uninstallable on the architecture ``arm64``. Here is the output again
  179. with this information high lighted::
  180. migration item(s) being attempted
  181. vvvvvv
  182. trying: libaws
  183. skipped: libaws (0, 165, 11)
  184. got: 45+0: a-4:i-27:a-5:a-1:a-1:m-0:m-3:m-1:p-1:s-2
  185. * arm64: libaws-bin, libaws17.2.2017, libaws3.3.2.2-dev, liblog4ada3-dev
  186. ^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  187. ||||| The binary packages becoming uninstallable (here 4)
  188. Affected architecture (here "arm64")
  189. Please note that britney is lazy and will often reject an item after proving
  190. that there is a regression on a single architecture. So in the above example,
  191. we are not actually sure whether this problem is architecture specific. For
  192. `easy`-hints, the information is presented slightly different::
  193. Trying easy from autohinter: asis/2017-1 dh-ada-library/6.12 [...]
  194. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  195. migration item(s) being attempted
  196. [... several lines of statistics from start, before and after ...]
  197. * amd64: asis-programs, libasis2017, libasis2017-dev, libaws-bin, [...]
  198. ^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  199. ||||| The binary packages becoming uninstallable on amd64
  200. Affected architecture (here "amd64")
  201. * i386: asis-programs, libasis2017, libasis2017-dev, libaws-bin, [...]
  202. ^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  203. ||||| The binary packages becoming uninstallable on i386
  204. Affected architecture (here "i386")
  205. [... more architectures with binary packages becoming uninstallable ...]
  206. While this tells us what britney tried to migrate and what would break (become
  207. uninstallable) as a result, it is not very helpful at explaining *why*
  208. things break. If there are few broken packages, it is often a question of
  209. looking for ``Breaks``-relations or ``Depends``-relations with upper bounds on
  210. versions / on old packages being removed. Alternatively, there are also tools
  211. like ``dose-debcheck``, which attempts to analyse and explain problems like this.