Archive Admin museum

All this content is saved from the Wiki Archive Admin page

The museum

This section houses all the content which is no longer up-to-date, but might still be of historical interest. All of this content is now deprecated and you should refer to the main Archive Administration documentation.

Useful runes

This section contains some copy-and-paste shell tricks to ease recurring jobs.

reprocess-failed-to-move

In some cases, binary packages fail to move from the incoming queue to the accepted queue. To fix this, run: ~lp_buildd/reprocess-failed-to-move as lp_buildd

Reprocessing failed source uploads

If a source upload fails for spurious/transient reasons (you can check in pepo:/srv/launchpad.net/production-logs/lp_queue/process-upload.log), ask webops to locate the appropriate upload-* directory and corresponding .distro file in /srv/launchpad.net/ubuntu-queue/ (they will probably be in either failed or rejected) and move them both back to /srv/launchpad.net/ubuntu-queue/incoming/.

Other archives

extras.ubuntu.com is not managed by the Ubuntu Archive Administration team, but is a PPA owned by the Application Review Board.

Useful web pages

Equally useful to the tools are the various auto-generated web pages in ubuntu-archive’s public_html that can give you a feel for the state of the Archive.

  • component-mismatches.txt

    As described above, this lists the differences between the Archive and the output of the Germinate script. It shows up packages that are in the wrong place, or need seeding.

  • germinate-output

    This is the output of the germinate script, split up into each release of each flavour of Ubuntu.

  • priority-mismatches.txt

    Shows discrepancies between priorities of packages and where they probably should go according to the seeds.

  • architecture-mismatches.txt

    Shows override discrepancies between architectures, which are generally bugs.

  • http://people.canonical.com/~ubuntu-archive/testing/xenial_probs.html

    Generated by the half-hourly run of britney and indicates packages that are uninstallable on Xenial, usually due to missing dependencies or problematic conflicts. note, this link 404s

  • http://people.canonical.com/~ubuntu-archive/testing/xenial_outdate.html

    Lists differences between binary and source versions in the Archive. This often shows up both build failures (where binaries are out of date for particular architectures) but also where a binary is No longer Built from the Source. note, this link 404s

  • ~ubuntu-archive/NBS/ and ~ubuntu-archive/nbs.html

    This contains a list of binary packages which are Not Built from Source (NBS) any more. The files contain the list of reverse dependencies of those packages (output of checkrdepends -b). These packages need to be removed eventually, thus all reverse dependencies need to be fixed. This is updated half-hourly.

Chroot management

Important

Please note that chroot management is something generally handled by Canonical IS (and specifically by Adam Conrad). The following section documents the procedures required should one have to, for instance, remove all the chroots for a certain suite to stop the build queue in its tracks while a breakage is hunted down and fixed, but please don’t take this as an open invitation to mess with the buildd chroots at will.

Soyuz stores one chroot per (suite, architecture).

manage-chroot allows the following actions upon a specified chroot:

$ ./manage-chroot
Usage: manage-chroot -a ARCHITECTURE [options] <get|info|remove|set>

manage-chroot: error: You must specify an architecture.

Downloading (get) an existing chroot:

$ ./manage-chroot [-s SUITE] <-a ARCH> get

The chroot will be downloaded and stored in the local disk name as ‘chroot-<DISTRIBUTION>-<SERIES>-<ARCHTAG>.tar.bz2’.

Uploading (add/update) a new chroot:

$ ./manage-chroot [-s SUITE] <-a ARCH> set -f <CHROOT_FILE>

The new chroot will be immediately used for the next build job in the corresponding architecture.

Disabling (remove) an existing chroot:

$ ./manage-chroot [-s SUITE] <-a ARCH> remove

Important

Unless you have plans for creating a new chroots from scratch, it’s better to download them to disk before the removal (recover is possible, but involves direct DB access)

No builds will be dispatched for architectures with no chroot, the build-farm will continue functional for the rest of the system.

Useful tools

There are other useful tools in your PATH which are invaluable.

Publishing security uploads from the ubuntu-security private PPA

Security uploads in Soyuz are first built, published, and tested in the Security Team’s private PPA. To un-embargo them, the Security Team use a tool that re-publishes them to the primary archive. This is now self-service, although you may occasionally be asked to adjust overrides, which can be done in the usual way.

Publishing packages from the ubuntu-security-proposed public PPA to -proposed

This works the same as the ubuntu-mozilla-security PPA, above. E.g., using lp:ubuntu-qa-tools/security-tools/unembargo:

$ unembargo --ppa=ubuntu-security-proposed -r xenial --pocket=proposed gnupg

Copying PPA kernels to -proposed

With the new Stable Release Cadence, kernels are built in the Kernel Team PPA and then pocket copied to the proposed pocket in the main archive once they are ACK-ed.

The two most useful URLs for this process are the Kernel SRU Report, where you will find links to the tracking bugs (the magnifying glasses), as well as the status of each package in each release, and the Kernel SRU workflow, which outlines in fine detail how each bug task should be handled, what the states mean, etc. The executive summary of the previous link is “only operate on tasks assigned to your team, and only when they’re in the Confirmed state, and when you do, please reassign to yourself and set the task to Fix Released when you’re done”.

The Pending SRU report has a section for the kernel PPA which shows all newer kernels in the PPA, provides click-able links to open all bugs (with separate CVE bugs), and copy-and-paste-able copy-proposed-kernel and sru-accept commands (both in ubuntu-archive-tools).

To publish linux version 2.6.32-27.49 from the canonical-kernel-team PPA to the -proposed pocket of Ubuntu’s lucid release, you would do the following:

  • Click on the “open bugs” button and check:

    • that the bugs are in a reasonable state, i.e. they target the right source package (e.g. linux vs. linux-ec2)

    • that the bugs are fixed in the development release (or at least upstream)

    • that the changes are limited to bug fixes, and are in general within the boundaries of the Stable Release Updates policy. They should ideally have a task for the stable release the SRU targets. Note that this button does not open CVE bugs, as they don’t get verification or other tracking (there is a separate button for opening them, if desired).

  • Find the tracking bug by either traversing through the list of bugs, or opening the .changes file and looking at the top (the release bug is pointed out there). Ensure that there is a proper stable task, and that the main (development release) task is invalid.

  • Run the copy-proposed-kernel command to copy it to -proposed. Once the kernels are copied to -proposed and accepted, update the bug to close the “Promote-to-proposed” task and assign it to yourself.

    However, with some flavours like -ec2 or armel kernels, which are mostly just a merge with the main linux kernel, it is too much overhead to add -ec2 tasks to all the bugs.

  • Due to current limitations of Launchpad, new packages copied from a PPA into the archive sometimes go to universe. As a result, please verify the overrides for all packages copied to -proposed, otherwise these packages might become uninstallable when they are ultimately copied to -updates/-security. find-bin-overrides from lp:ubuntu-qa-tools can help with this. You use it like so:

    find-bin-overrides <pocket to compare to> <target pocket> <ubuntu release> <source package>=<version in pocket to compare to>,<old abi>,<new abi>
```.

E.g., suppose there is a new kernel in `hardy-proposed` with a new ABI of
`2.6.24-30` and you want to get a list of overrides for the new kernel based
on the old ABI of `2.6.24-16` in the `2.6.24.12-16.34` version from the
`-release` pocket of Hardy. For the
`linux-restricted-modules-2.6.24 source package`, you might use:

```bash
$ find-bin-overrides release proposed hardy \
linux-restricted-modules-2.6.24=2.6.24.12-16.34,2.6.24-16,2.6.24-30
## hardy/linux-restricted-modules-2.6.24
./change-override -c multiverse -s hardy-proposed fglrx-kernel-source ...
./change-override -c restricted -s hardy-proposed avm-fritz-firmware-2.6.24-30 ...

    For the linux source package, you might use:

    $ find-bin-overrides release proposed hardy linux=2.6.24-16.30,2.6.24-16,2.6.24-30
## hardy/linux
...

    You may also specify --show-main to also show the the change-override command to move things to main. This can be useful if you know the overrides are very wrong. See find-bin-overrides -h for details.

TODO: A process/script similar to kernel-overrides should be developed to make sure overrides are properly handled for binaries not in main. Is find-bin-overrides from lp:ubuntu-qa-tools good enough?

Copying security uploads to -updates

Security uploads are distributed from a single system, security.ubuntu.com (consisting of one or a small number of machines in the Canonical datacenter). While this ensures much quicker distribution of security updates than is possible from a mirror network, it places a very high load on the machines serving security.ubuntu.com, as well as inflating Canonical’s bandwidth expenses very substantially. Every new installation of a stable release of Ubuntu is likely to be shortly followed by downloading all security updates to date, which is a significant ongoing cost.

To mitigate this, we periodically copy security uploads to the -updates pocket, which is distributed via the regular mirror network. In fact, the pooled packages associated with -security are mirrored too, but mirrored -security entries are not in the default /etc/apt/sources.list to avoid causing even more HTTP requests on every apt-get update. This is a cheap operation, and has no effect on the timely distribution of security updates, other than to reduce the load on central systems.

The copy-report tool lists all security uploads that need to be copied to -updates. If the package in question is not already in -updates, it can be copied without further checks. Otherwise, copy-report will extract the changelogs (which may take a little while) and confirm that the package in -security is a descendant of the package in -updates. If that is not the case, it will report that the package needs to be merged by hand.

The output of the tool looks like this:

$ ./copy-report
The following packages can be copied safely:
--------------------------------------------

copy-package -y -b -s feisty-security --to-suite feisty-updates -e 8.3.5-6ubuntu2.1 tk8.3
copy-package -y -b -s feisty-security --to-suite feisty-updates -e 8.4.14-0ubuntu2.1 tk8.4

The block of output under “The following packages can be copied safely:” may be copied and pasted in its entirety (or run copy-report --safe). If there is a block headed “The following packages need to be merged by hand:”, then make sure that the security team is aware of those cases.

Note that copy-report --safe is run from a cron job as the ubuntu-archive user on ubuntu-archive-toolbox, and so does not typically need to be run by hand.

Handling updates to partner

Updates to the partner archive follow a similar process as packages in the Ubuntu archive. Specifically, packages are first built in the -proposed pocket, then they need to be copied to the -release pocket after they are built and then cleaned up from the -proposed pocket. The process is as follows (examples show adobe-flashplugin being updated for Xenial):

  1. Review the package in the “unapproved” queue, and if appropriate, accept the package into partner for the -proposed pocket (in this example, -proposed for Xenial)

  2. When the package builds on all architectures, copy it to the partner release pocket (this makes it available to users). E.g.:

$ ./copy-package -b --auto-approve --from=ubuntu/partner -s xenial-proposed --to-suite xenial adobe-flashplugin

or to do all at once:

$ for i in xenial bionic focal groovy; do ./copy-package -b --auto-approve --from=ubuntu/partner -s $i-proposed --to-suite $i adobe-flashplugin ; done

  1. Remove the package from proposed in the partner archive. E.g.:

$ ./remove-package -A ubuntu/partner -m "copied to release" -s xenial-proposed adobe-flashplugin

or to do all at once:

$ for i in xenial bionic focal groovy; do ./remove-package -A ubuntu/partner -m "copied to release" -s $i-proposed adobe-flashplugin ; done

The museum attic

This stuff was already old in the old wiki. This will be deleted outright as it has either been replaced or folded into other sections.

Syncs

Syncing packages with Debian used to be a reasonably common request. It is now self-service for developers using syncpackage. If you are asked to process a sync in the old style, you can use the -s option to syncpackage to do so in the name of the requester.

Most updates available in Debian are automatically synced before Debian Import Freeze, but packages that were previously in Ubuntu will not be automatically reintroduced. To process these interactively:

$ ./auto-sync --new-only

To sync from any source that is not automatically imported by Launchpad, use the syncpackage tool on a .dsc file. Any uploader can do this; it does not require being an Archive Administrator.

Likewise, backports are now self-service for members of ubuntu-backporters using backportpackage.

Useful tools

There are other useful tools in your PATH which are invaluable.

Archive state checks

rmadison (in the devscripts package, run on your client system) examines the current state of the archive for a given binary/source package:

$ rmadison dpkg
      dpkg | 1.10.22ubuntu2 |         warty | source, amd64, i386, powerpc
      dpkg | 1.10.22ubuntu2.1 | warty-security | source, amd64, i386, powerpc
      dpkg | 1.10.27ubuntu1 |         hoary | source, amd64, i386, ia64, powerpc, sparc
      dpkg | 1.10.27ubuntu1.1 | hoary-security | source, amd64, i386, ia64, powerpc, sparc
      dpkg | 1.10.27ubuntu2 | hoary-updates | source, amd64, i386, ia64, powerpc, sparc
      dpkg | 1.13.10ubuntu4 |        breezy | source, amd64, hppa, i386, ia64, powerpc, sparc
      dpkg | 1.13.11ubuntu5 |        dapper | source, amd64, hppa, i386, ia64, powerpc, sparc

$ rmadison dselect
   dselect | 1.10.22ubuntu2 |         warty | amd64, i386, powerpc
   dselect | 1.10.22ubuntu2.1 | warty-security | amd64, i386, powerpc
   dselect | 1.10.27ubuntu1 |         hoary | amd64, i386, ia64, powerpc, sparc
   dselect | 1.10.27ubuntu1.1 | hoary-security | amd64, i386, ia64, powerpc, sparc
   dselect | 1.10.27ubuntu2 | hoary-updates | amd64, i386, ia64, powerpc, sparc
   dselect | 1.13.10ubuntu4 |        breezy | amd64, hppa, i386, ia64, powerpc, sparc
   dselect | 1.13.11ubuntu5 |        dapper | amd64, hppa, i386, ia64, powerpc, sparc

Or when used with -S and a source package, the source and every binary built by it:

$ rmadison -S dpkg
      dpkg | 1.10.22ubuntu2 |         warty | source, amd64, i386, powerpc
      dpkg | 1.10.22ubuntu2.1 | warty-security | source, amd64, i386, powerpc
      dpkg | 1.10.27ubuntu1 |         hoary | source, amd64, i386, ia64, powerpc, sparc
      dpkg | 1.10.27ubuntu1.1 | hoary-security | source, amd64, i386, ia64, powerpc, sparc
      dpkg | 1.10.27ubuntu2 | hoary-updates | source, amd64, i386, ia64, powerpc, sparc
      dpkg | 1.13.10ubuntu4 |        breezy | source, amd64, hppa, i386, ia64, powerpc, sparc
      dpkg | 1.13.11ubuntu5 |        dapper | source, amd64, hppa, i386, ia64, powerpc, sparc
  dpkg-dev | 1.10.22ubuntu2 |         warty | all
  dpkg-dev | 1.10.22ubuntu2.1 | warty-security | all
  dpkg-dev | 1.10.27ubuntu1 |         hoary | all
  dpkg-dev | 1.10.27ubuntu1.1 | hoary-security | all
  dpkg-dev | 1.10.27ubuntu2 | hoary-updates | all
  dpkg-dev | 1.13.10ubuntu4 |        breezy | all
  dpkg-dev | 1.13.11ubuntu5 |        dapper | all
  dpkg-doc | 1.10.22ubuntu2 |         warty | all
  dpkg-doc | 1.10.22ubuntu2.1 | warty-security | all
  dpkg-doc | 1.10.27ubuntu1 |         hoary | all
  dpkg-doc | 1.10.27ubuntu1.1 | hoary-security | all
  dpkg-doc | 1.10.27ubuntu2 | hoary-updates | all
   dselect | 1.10.22ubuntu2 |         warty | amd64, i386, powerpc
   dselect | 1.10.22ubuntu2.1 | warty-security | amd64, i386, powerpc
   dselect | 1.10.27ubuntu1 |         hoary | amd64, i386, ia64, powerpc, sparc
   dselect | 1.10.27ubuntu1.1 | hoary-security | amd64, i386, ia64, powerpc, sparc
   dselect | 1.10.27ubuntu2 | hoary-updates | amd64, i386, ia64, powerpc, sparc
   dselect | 1.13.10ubuntu4 |        breezy | amd64, hppa, i386, ia64, powerpc, sparc
   dselect | 1.13.11ubuntu5 |        dapper | amd64, hppa, i386, ia64, powerpc, sparc

checkrdepends lists the reverse dependencies of a given binary:

$ checkrdepends -s zesty -b nm-applet

or source package:

$ checkrdepends -s zesty network-manager

NEW handling

A lot of churn in NEW comes from Debian imports. Since they already went through NEW in Debian, we should not waste too much time on it, so there are some tools.

  • The first thing you need to handle are native syncs. These are syncs performed via URLs like https://launchpad.net/ubuntu/zesty/+localpackagediffs or via the Launchpad API. You can recognize these in the LP queue pages because they have ‘(sync)’ in the name. In queue info, they show up as ‘X-’ (as opposed to ‘S-’ like normal source uploads). There are no changes files for these, so they cannot currently be fetched via queue fetch. To verify a native sync:

    1. Download the source package from Debian (e.g. via dget or apt-get source <pkg>=<version>)

    2. Download the imported dsc file from the Debian project in LP, e.g. https://launchpad.net/debian/sid/+source/pxe-kexec

    3. Compare the dsc file from Debian and from LP. Since both should be signed, if they are identical, then you know the package hasn’t been tampered with. You can also compare the full source package from Debian and LP if desired.

    Once verified, accept it normally via LP or ./queue accept <srcpkg>.

  • ./new-binary-debian-universe accepts all binary NEW packages whose source was imported from Debian and is in universe. With the --lintian option, it runs lintian over all the imported .debs while it runs, so you can watch the output and note all particularly worrisome issues. When you are happy, say “yes” at each confirmation prompt.

  • When unpacking a source package for source NEW checks, you should run suspicious-source. This is basically a find -type f which ignores all files with a known-safe name (such as *.c, configure, *.glade). Every file that it outputs should be checked for being the preferred form of modification, as required by the GPL. This makes it easier to spot PDFs and other binary-only files that are not accompanied by a source. The licensecheck command is also useful for verifying the license status of source packages.

Moving packages to -updates

Packages in -proposed can be moved to -updates once they are approved by someone from sru-verification, and have passed the minimum aging period of 7 days. Use the sru-release script from ubuntu-archive-tools for this:

$ ./sru-release xenial kdebase

Please see --help, you can also use this tool to copy packages to -security and to the current development release.

Note

Before copying a package to -security ping a member of the ubuntu-security team.

Publishing packages from the ubuntu-mozilla-security public PPA

Mozilla (Firefox and Thunderbird) uploads in Soyuz are first built, published, and tested in the Mozilla Security Team’s public PPA. To publish them into the main archive, use copy-package from ubuntu-archive-tools.

Note that pocket copies to the -security pocket should never be done without an explicit request from a member of the Ubuntu Security Team (Mozilla Security Team is not enough), and copies to the -proposed pocket should not be done without an explicit request from a member of the SRU Team. Keep in mind that firefox 2.0 and later (i.e. Hardy and later) will always have a corresponding xulrunner package that needs copying.

To publish firefox version 22.0+build2-0ubuntu0.12.04.2 from the ubuntu-mozilla-security PPA to the -security pocket of Ubuntu’s precise release, you would do the following:

$ ./copy-package -b --from=~ubuntu-mozilla-security/ubuntu/ppa -s precise --to=ubuntu --to-suite precise-security -e 22.0+build2-0ubuntu0.12.04.2 firefox

Alternatively, can use lp:ubuntu-qa-tools/security-tools/unembargo like so:

$ unembargo --ppa=ubuntu-mozilla-security -r xenial --pocket=proposed chromium-browser

Other copies

copy-package is a versatile tool and can be used for a number of other purposes. If you are doing something not documented here, please ask on #ubuntu-release first.

Archive Administrators technically (as in, per the ACLs) have the ability to copy into the RELEASE pocket of the current development series. Please do not do this; that pocket is owned by the proposed-migration tools, and those tools have built-in override facilities for the cases where they are wrong.

The release team has access to apply such overrides when needed.

Archive Administration shifts

This is a next try for establishing Archive Admin shifts during the week. During an AA shift, an Archive Admin allocates at least 2 hours to work on tasks requiring special ubuntu-archive permissions.

Current members with regular admin shifts are:

  • Monday:

  • Tuesday:

  • Wednesday: sil2100 (9:00–11:00 UTC)

  • Thursday: seb128 (10–12 CET)

  • Friday:

On an archive shift, the following items should be the main focus:

  • Stay open for any ubuntu-archive IRC requests, being available to help developers out with their everyday work

Additionally, performing the following should be considered:

  • Process pending archive bugs. This query often generates a Launchpad OOPS rather than succeeding; This query is more likely to succeed, but might miss some bugs. Most of those are syncs, removals, component fixes, but there might be other (less common) requests.

  • Process the NEW queues of the current development release and *-backports of all supported stable releases.

  • If we are not yet in the Debian Import Freeze, run process-removals to review/remove packages that were removed in Debian.

  • Clean up component-mismatches, and poke people to fix dependencies/write MIRs.

  • Remove NBS packages without reverse dependencies, and prod maintainers to rebuild/fix packages to eliminate reverse dependencies to NBS packages.

Outdated

Question

This section is actually marked on the AA page as being outdated. Should we keep it for historical interest?

Logging in

Outdated: this is now on ubuntu-archive-toolbox.internal and works completely differently.

A few operations still require direct SSH access to pepo.canonical.com; accounts are provided to some members of the team. (We are transitioning away from direct shell access, so new team members will not get accounts here, thereby encouraging us to complete the transition more quickly). Changes can only be made as the lp_archive user, to which you’ll have sudo access.

So to begin:

$ ssh pepo
$ sudo -u lp_archive -i

The -i is important as lp_archive’s .bashrc sets the right environment variables and makes sure the directory with all of the tools is placed in the PATH.