Everything you ever wanted to know about Stable Release Management in Debian
                          but were afraid to ask

                                 - or -

          How I Learned to Stop Worrying and Love the Queue Viewer


Introduction
============

Updates to stable
=================

We use distribution tags (e.g. "bookworm") to allow us to quickly determine
whether an update bug applies to stable or oldstable.

The workflow for an update is approximately:

- A bug is filed usertagged "pu" (user release.debian.org@packages.debian.org)
  with a minimal debdiff attached and a description of why the update is
  required / desired. For non-controversial updates, this may be directly
  followed by an upload.
- If any changes / more information are required, tag the bug "moreinfo" and
  iterate until a final diff is agreed
- Once go ahead is given, tag the bug "confirmed"
- Once the upload has occurred and passed initial sanity checks (see "after
  the upload"), tag the bug "pending"
- The package is accepted in to proposed-updates and autobuilt on any
  necessary architectures

Binary rebuilds ("binNMUs") in stable follow a similar workflow, except:

- The bug is usertagged "binnmu"
- The "confirmed" tag is added once the binNMU has been scheduled

A few notes (which could / should be tidied up):

- Added build-dependencies are only appropriate if they are required for
  resolving the issue under discussion.
- Modifying the patch system in use is generally not appropriate for an
  update in stable. This implies that if the package does not currently use
  a patch system then any changes required should be applied directly to the
  upstream source.
- If the issue applies to the version of the package in unstable, then it
  should be fixed there first.
- We encourage the use of the version suffix "+deb${distversion}u${sequence}"
  However, if the uploader wishes to use an
  alternative version which is closer to a "standard" version number then
  this is generally okay, so long as it is verified that there has never
  been an upload to Debian using that version number (this is one reason
  why the version suffixes mentioned are conventional).
- Many updates to stable will be made by people other than the maintainer.
  We do not require the use of NMU version numbers in these cases, nor that
  the changelog mention that it is an NMU.
- There are some classes of bug which we are happy to ignore (including using
  the dist-ignore BTS tags) for stable. These include license clarifications
  where it can be shown that a later correction to the license information
  applies to the stable package; this is particularly the case when the
  corrected information has been included in an upload to unstable.

Removals from stable
====================

Requests for removal from stable should be filed as bugs against release.d.o,
usertagged "rm" (user release.debian.org@packages.debian.org) with an
appropriate distribution tag applied. If removal of the package from both
stable and oldstable (when possible) is requested, this should be tracked
using two separate bugs.

Assuming the removal is agreed, the bug should be tagged "pending" and
retitled to follow ftp-master's standard form for removal bugs (almost always
using "RoM" or "RoQA", depending on who requested the removal). At this
point an entry should also be added to the special "REMOVALS" comment file, in
the format:

  #${bugnumber}: ${source} - ${reason}  

Be careful when adding entries to the file, as queue-viewer does not gracefully
handle any not in the expected format.

If an update for the package has been previously issued via the security
archive, those packages should also be removed, via a bug against the
ftp.debian.org psuedopackage. The report should note that the removal of the
package from stable is planned to be actioned during the point release.

After the upload
================

A mail with the diff is sent by queue viewer to stable-diffs@, so that
it can be checked. One way to do so is running interdiff between what got
reviewed (diff in BTS) and what got uploaded (diff in mail).

Processing is done by creating "comments" files. Those live on coccia under:
  /srv/release.debian.org/www/proposed-updates/${codename}_comments
  /srv/release.debian.org/www/proposed-updates/${codename}_comments

(A symlink from ~ to those helps.)

The filename format is $source_$version. The version string is the equivalent
of the "Version" header in .changes files, so should include the "+bX" suffix
for binNMUs.

For an accept, the first two lines of a comment file must be of the form:

OK
$source - $reasons

$reasons for this upload is a short summary of bug fixes, CVE IDs, etc.

One can add metadata to the end of a comment file, as key:value pairs
after a standard format signature separator; any text present after the
separator will also be hidden from the queue-viewer output.

A common use of this functionality of to indicate the number of the
BTS request for the update, in the form "pu-bug:NNNNNN". The bug will
automatically be tagged "pending" as part of accepting the upload.
Thus, a complete example file might be:

OK
$source - $reasons
-- 
pu-bug:123456

The "pending" mail will be sent using your shell account on coccia, so you
may want to set $DEBEMAIL (and possibly $DEBFULLNAME) in your environment.

There is currently one other supported metadata key, to allow specifying
the architectures for which a binNMU was scheduled, so that queue-viewer
can ensure that they are all present before the package is flagged for
acceptance. This is the "expect-arches" key, with the value being a
space-separated list of architectures.

Once it's done, make sure the queue viewer is happy:
  https://release.debian.org/proposed-updates/oldstable.html
  https://release.debian.org/proposed-updates/stable.html

If it is, ask dak to accept the package by running
  /srv/release.debian.org/tools/scripts/srm \
    accept pu $comment_filename

and supply your "sudo" password when prompted.

This will create an "ACCEPT" comment in ftp-master's queue, which will
renamed to ACCEPTED.source_version when ftp-masters have
taken it into account. A "NEW changes in stable-new" (or oldstable-new)
mail is sent to debian-release@ in that case.

[Note that historically the comments directories lived within ftp-master's
queues and were directly manipulated there. In order to separate the
ftp-master and release services, the ftp-master directories are now
only manipulated using SSH triggers, wrapped by the "srm"
script detailed above. A second script, run from release's crontab directly
prior to queue-viewer itself, ensures that any changes to the ftp-master
directories are reflected in queue-viewer's comment files.]

If a pu bug was filed (see "BTS handling") then it should be tagged
"pending" at this point, if this was not automatically done using the
"pu-bug" metadata field detailed above.

If for some reason you need to reject a package, modify the first line
of the comment file to be "NOTOK" and, as with accepts, signal dak to
action the comment by:

  /srv/release.debian.org/tools/scripts/srm reject pu \
    $comment_filename

to create the file in the ftp-master queue, which will be
renamed to REJECTED.source_version once processed and the actions taken
included in a mail to debian-release.

The reasons listed in the comment file will be used as part of the reject
message sent by dak.

If the reject was required because the package should have been uploaded
to the security archive instead of ftp-master, you should tell the uploader
to re-sign the .changes file before proceeding with the re-upload.
Otherwise, the later automatic sync from the security archive back to
ftp-master will fail due to the signature having been previously seen.

Once the rejection has occurred, if a new upload of the same version
is expected, you may wish to tidy up some metadata so that it gets
re-generated for the new version:

- mv ${codename}_comments/REJECTED.$foo ${codename}_comments/cruft
- remove or rename ~release/www/proposed-updates/${codename}_diffs/${foo}.debdiff
- remove ~release/www/proposed-updates/${codename}_debcheck/${foo}_*.debcheck

The above steps can be automated by using "srm cleanup", once the reject
has occurred

  /srv/release.debian.org/tools/scripts/srm cleanup pu \
    REJECTED.$foo

Security updates
----------------

As part of the release of a DSA, the updated packages are pushed from
security-master to ftp-master where, all things being well, dak will
copy them to p-u-NEW.

When looking at packages in the -NEW queue for which there is not yet
a comment file, queue-viewer will compare the package name and version
to those in DSA/list (from the secure-testing repository).  If a match
is found, it will automatically create a comment file, with status
"UNKNOWN".  After a quick manual check, this can be updated to "OK".

If for some reason the information in the DSA list is missing (and this
can't / won't be fixed) then you can manually create a comment file in
the same way as above, but using the format below for the second line.

DSA 1234 $source - $reasons

This generally only needs doing when -2 DSAs are published, or a DSA
is released for an issue which is not actually a security issue in the
updated package itself - for instance, to resolve a regression introduced
by a security update of another package.  In these cases, the updated
package information is often not recorded in DSA/list.

The queue status of a security upload will only change to "OK" with the
green background once packages are available for all of the architectures
on which the package is built in the base suite.  Until this point, it
will remain "in limbo", with the missing architectures highlighted.

_DO NOT_ accept a package that is "in limbo".  This will cause the buildd
network to believe that the remaining architectures need to be built in
stable and can lead to the main archive and the security archive serving
packages with the same name/version but different content.

debian-installer
----------------

The installer is a little special, as it is auto-BYHANDed (don't blame me,
I didn't invent the name) by dak.  As a side-effect, the images are installed
directly into dists/proposed-updates.  The packages are then accepted using
comments files as usual.

If there are no changes to the debian-installer source package itself, it
can be binNMUed in order to incorporate changes from other packages.  (An
exception to this is the first debian-installer build for a given release.
This must be a source upload, in order to enable the "build against
proposed-updates" switch in the code.)

Once the installer is built on all architectures, there will need to be an
upload of debian-installer-netboot-images.

stable-updates
==============

Sometimes an update is sufficiently important that it merits release to a
wider audience before the next point release.  This is handled via the
"stable-updates" suites.

Releasing a package via stable-updates first requires accepting it into
proposed-updates from the -NEW queue.  In most cases you will also want to
wait for the package to be built on all architectures before releasing it.
Note that missing architectures will not be automatically included, so you
will have to take care to add any that are not included in the initial
release.

Similarly to testing, the Release Team have the ability to "set" the current
content of stable-updates.  On respighi, start by making a copy of
/srv/release.debian.org/sets/bookworm-updates/current to a datetime-stamped
file in the same directory.  Add the source and binaries which form part of
the update, in the same style as the existing entries (as returned by
"dak control-suite" or "dak ls -f control-suite").  If the package has
previously been released via stable-updates, the entries for the old
version should be removed.
(hint: dak ls -f control-suite -s proposed-updates -S <pkg>)

Once you're ready to proceed with the release, symlink the new file to
"current" and, as the release user:

md5=$(md5sum /srv/release.debian.org/sets/bookworm-updates/current | awk '{print $1}')
ssh -2 -T -o BatchMode=yes -i ../ssh/dak_import_trigger_key \
  dak@fasolo.debian.org import_dataset.sh bookworm-updates $md5 < \
  /srv/release.debian.org/sets/bookworm-updates/current

This will update the suite on ftp-master, ready to be pushed to mirrors at
the next dinstall.

An announcement mail will also need to be prepared, and sent to
debian-stable-announce@lists.d.o.  Although it is not required to
GPG sign mail to that list, it is conventional to do so.  The list is
moderated, so you may need to poke one of the moderators* if your mail
does not appear on the list after a while.  It is preferred to have the
mail appear on the list before the dinstall in which the package is
released, so that users are not surprised by the availability of a new
package.

Hints:
 + doc/templates/stable-updates-mail.example is an example mail
   (which needs adjusting to suit)
 + The name at the top of the announcement should be the package author,
   not you

* As of 2023-12-18, the moderators are adsb, jcristau, jmw and kibi.

Point releases
==============

Before
------

Once the date is chosen, send an announcement mail to debian-release,
BCCed to the pointreleases alias, which will help ensure that various
other lists and teams are notified.

Update the website index (release.d.o:www/index.html) to reference the
announcement mail and date.

Update the release calendar:

  release.debian.org$ ./bin/ical-add-event.py \
    -s "buster point release 10.10" \
    -d "https://lists.debian.org/c72fdf3099474bbfc60474d0f52563bca17e6d7c.camel@adam-barratt.org.uk" \
    -D 20210619

  Commit and push the updated ICS


Once p-u-NEW is frozen
~~~~~~~~~~~~~~~~~~~~~~

Create /srv/release.debian.org/www/proposed-updates/${version}/${pointversion}/
on coccia - for example "/srv/release.debian.org/www/proposed-updates/9/9.5/

Generate an "upcoming point release" announcement mail, and send it to
debian-stable-announce@lists.debian.org.  Although it is not required to
GPG sign mail to that list, it is conventional to do so.  The list is
moderated, so you may need to poke one of the moderators if your mail
does not appear on the list after a while.

~release/tools/scripts/point-release-mail -d -s stable -e jinja \
   -t ~release/tools/scripts/TEMPLATE.p-u-frozen.jinja2

The -d flag indicates that a draft mail should be generated.  This is often
useful at this stage of the process, as it includes packages that are not
completely ready (although you will have to move them from the "needs
investigation" section to wherever they should be and then drop that
section and the other indications of it being a draft, before sending).

Prepare some sets of data which will be useful during the point release
- some for the release team, some for ftp-master.  Note that some of
these steps may be skipped if they do not apply to a particular point
release; filenames may need to be updated as appropriate.

State of stable at various stages of the point release (expected)
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

The "puprep" script from release-tools can be used to prepare the data
sets commonly used for each point release. It should be invoked as:

puprep X.Y suite package_arches contents_arches [skips]

where:
  - "X.Y" is the point release number (e.g. "9.5")
  - "suite" is either "stable" or "oldstable"
  - "packages_arches" is a comma-separated list of architectures for
    which Packages files diffs will be generated. Note that "source"
    is implicitly included in the list.
  - "contents_arches" is a comma-separated list of architectures for
    which Contents files diffs will be generated. Note that "source"
    and "all" are implicitly included in the list.
  - The optional "skips" is a comma-separated list of source packages
    present in proposed-updates but which should not be included in
    the point release for some reason (most commonly because the update
    has been found to be broken but could not be fixed in time).

In order to avoid issues with differing locales amongst users involved
in a point release, puprep generates all files with "LC_ALL=C" in
force. If any files are manually generated, this should also be the case
(either globally or applied to all invocations of "sort" and "comm").

puprep will automatically generate the state directory
/srv/release.debian.org/www/proposed-updates/X/X.Y if it does not exist,
and will automatically remove any existing "before" subfolder.

The output generated by puprep will include:
- TODO.${codename}
  - useful information suitable for merging into the TODO comment file
- "before" containing dists/suite with the installer images removed
- Sources-before, ${arch}-before
- Contents-source-before, Contents-${arch}-before
- ${suite}.cs
  - the contents of the suite before the point release, in "dak control-suite"
    format
- proposed-updates.cs
  - similar for p-u
- combined.cs
  - a simple addition of ${suite} and p-u
- removals.cs
- removals-debug.cs
  - packages to be removed from ${suite}{,-debug}
- removals.${codename}
  - a list of "dak rm" commands to apply the package removals
- removals-dryrun.${codename}
  - the results of dry running the package removals
  - this will raise some dependency issues which will not occur in practice,
    because they are
    - resolved by packages added to ${suite} during the point release
    - resolved by removals placed earlier in the list
- combined-removals.cs
  - the state of ${suite} after removals have been applied
- combined-removals-dominated.cs
  - the state of ${suite} after removals have been applied and domination
    has occurred, removing obsolete versions of packages
- propups.testing{,-debug}
- propups.unstable{,-debug}
  - packages requiring propagation to the mentioned suite
  - If the files are non-empty, add notes (including the filename)
    to TODO. It may be possible for the Release Team to perform prop-ups
    to testing before the point release, using britney. In such cases,
    dak will automatically also prop-up the package to unstable if
    required.

# In the absence of cruft, removals or new packages, these should have
# the same number of packages in
wc -l stable-before.cs combined-dominated.cs

Check the output of "dak cruft-report -s stable" and for stable-debug.
If there's anything relevant, or if you're aware of any changes not
flagged (for example, due to a kernel ABI change),  note it in to TODO
and consider adding a new cs file.

Note: the cruft report has some notable gaps, including removals on a
subset of architectures. Be sure to check all proposed changes for
dropped packages, changed architectures etc.

Source compliance (mostly d-i initrd contents)
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

If there is a new debian-installer included in the point release, the
source and binary packages need to be added to the -r0 suite. "puprep"
will produce a "${codename}-r0-additions.cs" file containing these.
Other packages can be added manually if required.

The debian-installer build produces a Built-Using field, so dak will
ensure that any other required packages are kept in the archive.

queue-viewer
=-=-=-=-=-=-

Once all builds for all packages are available:

- copy /srv/release.d.o/www/proposed-updates/stable* to
  /srv/release.d.o/www/proposed-updates/X/X.Y

- create /srv/release.d.o/www/proposed-updates/X/X.Y/COMMENTS
  and copy all files (i.e. *not* the "cruft" subdirectory) from
  /srv/release.d.o/www/proposed-updates/${codename}_comments/
  to it.
  
At some point between here and the point release, de-cron updates
to queue-viewer.


T-3 days (so generally Wednesday)
~~~~~~~~

Draft a full announcement, commit it to the publicity-announcements repository
(git@salsa.debian.org:publicity-team/announcements.git) and send 
a mail to press@debian.org, debian-publicity@lists.debian.org
CC stable@release.debian.org

~release/tools/scripts/point-release-mail -s stable -e jinja \
  -t ~release/tools/scripts/TEMPLATE.wml.jinja2

In some cases the dates for DSAs won't be automatically located and will be
listed as XXXX; these will need to be fixed by hand.  The auto-generated
package comments should be reviewed and tweaked as necessary.

If there's any particularly interesting kernel or installer changes included
in the point release, you may wish to add a section to the mail about these.

In case any further packages are accepted after this point, send a diff for
the text to the press team.


Special comments files
  - TODO
  - Removals

Tools
  - point-release-mail

Preparation

On the day
----------

De-cron britney runs which make changes to the archive. While the point release
is in progress, the packages files seen by britney may well not match the
archive (e.g. if any packages are prop-upped during the point release), which
can lead to britney failing to commit changes.

During
------

ftp-master will begin by disabling their crontab and making a backup of the
projectb database.

If there are any packages in p-u which should not be released during this
point release, make sure you tell them early in the process or they may
start proceeding on the basis that all packages should be included.

The archive side of a point release is driven by a script that is part of dak.
The script will fetch and process some of the files produced earlier by
puprep, asking the ftp-master to check the files or provide further details
if required.  The script runs on the main archive server.  In order that
you can follow along, the ftp-master should arrange for the output of their
interactive session and the dak log to be visible on coccia.

It may be useful to maintain a screen session during the process, with at
least:
- the dak log
- the ftp-master interactive session
- a shell to quickly run "sudo -u release /usr/local/bin/static-update-component
  release.debian.org-pu"

The ftp-master will often mention what they're doing as the point release
progresses.  You can follow much of what's going on by observing
the ftp-master outputs, along with the output of "dak ls" and
"dak control-suite" at various points.  Where you have previously prepared
"before" or "expected" files, try and capture a matching set of "after" or
"actual" data for comparison purposes.  If there are any differences between
the expected and actual data, reconcile these, prompting ftp-master if
necessary.

In each case, the "actual" archive state should be generated using
"dak control-suite -l stable | LC_ALL=C sort", directing the output to the
relevant file for checking / posterity.

Points at which you might want to check the archive state are:
- once all the packages from p-u have been added to stable
  - actual-combined.cs
  - pu-after.cs
- once removals have been processed
  - actual-combined-removals.cs
- once domination has occurred
  - actual-combined-dominated.cs
- if multiple rounds of domination are required, you may wish to capture
  a result for each

# Once p-u has been merged into stable
dak control-suite -l stable | LC_ALL=C sort > actual-combined.cs
diff -u combined.cs actual-combined.cs

# Once p-u has been cleaned up
dak control-suite -l proposed-updates | LC_ALL=C sort > pu-after.cs
diff -u toskip.cs pu-after.cs

# Post-domination and removals
dak control-suite -l stable | LC_ALL=C sort > actual-combined-dominated.cs
diff -u combined-removals-dominated.cs actual-combined-dominated.cs

Once any packages have been copied between suites and removed and any
required changes to the set of installer versions for stable have been made,
ftp-master will request that a check of the new state be made; domination
of stable and cruft removal will also have occurred at this point.

Assuming everything is okay, the new filesystem metadata files will start
being generated.  This will include packages, sources, contents and release
files.  The files will then be copied to coccia, populating a tree at
/srv/ftp-master.debian.org/tmp-release/.

ftp-master will again ask for a check, followed by a signature of
the top-level release files, using the appropriate stable release key.

The release-tools repository contains scripts named "diffpackages" and
"diffsources", which can be used to produce diffs between the packages and
sources files generated by puprep and those in tmp-release. These attempt
to present only those differences between the original and new states that
are likely to need checking, eliding common expected patterns and making
the presence of new or removed packages clearer.

For Contents files, simply uncompress the new file and diff it against the
original.

An example set of commands you may wish to run is:

: T=/srv/ftp-master.debian.org/tmp-release/dists/${codename}/main
: zcat ${T}/source/Sources.gz > Sources-after
: /srv/release.debian.org/tools/scripts/diffsources Sources-{before,after} > Sources.diff
: for arch in [packages file architectures]
  do
    zcat ${T}/binary-${arch}/Packages.gz > ${arch}-after
    /srv/release.debian.org/tools/scripts/diffpackages ${arch}-{before,after} > ${arch}.diff
  done
: for arch in source all [contents file architectures]
  do
    zcat ${T}/Contents-${arch}.gz > Contents-${arch}-after
    diff -adu Contents-${arch}-{before,after} > Contents-${arch}.diff
  done

Once the diffs have been generated, you may wish to trigger a staticsync
before beginning to check the diffs. This will help reduce the amount of
data that needs to be copied when the time comes to provide ftp-master with
the signatures, thus speeding up that part of the process.

When triggering the sync, be careful not to be running commands such as "vi"
or "view" on files within the directory. The sync uses rsync and will be unable
to access the .swp file and will thus fail.

As there are often several people checking diffs on the day, it is customary
to mention on #debian-release as diffs become available, and which files you
are looking at, to avoid duplication of effort.

While checking diffs, things you will want to confirm include:
- any expected package removals have occurred
- no unexpected package additions have occurred
- changes to metadata are as expected and "seem reasonable"

Anything unexpected, or that you're unsure about, should be mentioned on
#debian-release.

The Sources diff will likely contain several changes to the set of packages
that are flagged as "Extra-Source-Only: yes" (also referred to as "ESO", for
ease), primarily due to changes in the Built-Using: fields of other packages.
Although this pattern is expected in general, it is customary to note the
changes on #debian-release, e.g. as "we (lose|gain) an ESO ncurses".

Most updates to larger packages such as kernels and web browsers will cause a
lot of churn in Contents diffs, so you may wish to be prepared to quickly
visually skip past the specifics, as long as the general pattern seems OK.
Similarly, although we generate diffs for source package Contents files, they
are generally too noisy to be able to be usefully checked.

The signature over Release should be an armoured detached signature, named
Release-X.Y(-debug).gpg.  Signatures for InRelease should be made by
clear-signing the corresponding Release file and named
InRelease-X.Y(-debug).gpg.  All must use the same hash as the ftp-master
signatures (currently SHA256).

Example:
 gpg -a --personal-digest-preferences SHA256 --output InRelease-12.1-debug.gpg \
   --clearsign Release

Once everything appears satisfactory, sign the Release files and
push them to the preparation directory. ftp-master will fetch the signatures
via HTTP, so after copying them to coccia run a staticsync to push them out.

Bonus: if you can access the ftp-master system, extra checks could include:

- ftp/README should have the correct version number and release date.
- ftp/README.html should have the correct version number, release date and
  "Modified" date.
- ftp/dists/README should have the correct version number.
- ftp/dists/stable/Changelog show contain:
  - a header with the correct version number and release date.
  - a list of any removals.
  - the contents of the .changes from any included uploads.
- pick some random entries from the top-level Release file(s) and verify that
  the checksums listed match the corresponding files.

For each of the packages / sources / contents files which were prepared
before the point release, check a diff between the original and new files.

After
-----

Mail debian-security@lists.debian.org (as people have in the past tended to
panic slightly when updated packages appear without an announcement) and the
press team, to let them know that the point releases has finished.  This
should include whether the packages are being pushed out straight away or
waiting for a scheduled dinstall (and when that is).  The Images Team should
also be informed, so they can begin building images as soon as their mirror
is in sync; they will most likely already be in #debian-release, but if not
they can be found in #debian-cd.

If point releases for both stable and oldstable are being performed at the
same time, inform the Images Team after the first process is finished. The
press and -security@lists notifications should be made after both releases
are completed.

Create ${codename}_comments/cruft/<longversion> and move ${codename}_comments/ACCEPTED.*
in to it.  Also copy ${codename}_comments/{REMOVALS,TODO} to the new directory and then
empty the live versions.

If any updates were not included in the point release, move the ACCEPTED
comment files for them back out of the cruft directory.

Create ${codename}_{debcheck,diffs,lintian}/cruft/<longversion> and move
the files that correspond to the comments moved previously into the new
directory. From the comments directory, something like

VERSION=12.x
CODENAME=bookworm

mkdir ../${CODENAME}_debcheck/cruft/${VERSION}
mkdir ../${CODENAME}_diffs/cruft/${VERSION}
mkdir ../${CODENAME}_lintian/cruft/${VERSION}
for c in $(find cruft/${VERSION} -name "ACCEPTED.*" -printf "%f\n" | cut -d. -f2-)
do
  mv -v ../${CODENAME}_debcheck/${c}_* ../${CODENAME}_debcheck/cruft/${VERSION}
  mv -v ../${CODENAME}_diffs/${c}[_.]* ../${CODENAME}_diffs/cruft/${VERSION}
  # *.gz for lintian so .cache files get left in place
  mv -v ../${CODENAME}_lintian/${c}_*.gz ../${CODENAME}_lintian/cruft/${VERSION}
done

Re-enable the queue-viewer cron job.

Close each of the currently pending release.d.o bugs corresponding to the
just released packages, using <longversion> as the "fixed" version.

Update release.debian.org's index page to reflect the fact that the point
release has happened.

Re-enable the britney cron jobs which were disabled earlier.

Keep an eye out for any issues being mentioned on IRC or on -release /
-security.

Full releases
=============

Note that some of this section is not strictly SRM territory, but it is useful
to start the new cycle with the information listed available.

Preparation
-----------

Check the contents of testing-proposed-updates. Any packages that should not
move to p-u after the release should be removed beforehand.

On coccia:

WORKDIR=/srv/release.debian.org/www/proposed-updates/${MAJOR}/${MAJOR}.0
mkdir -p ${WORKDIR}
cd ${WORKDIR}
mkdir before
mkdir after
mkdir sigs

: > empty.cs

for suite in oldstable stable testing {,old}stable-new {${OLD_CODENAME},${STABLE_CODENAME}}-{updates,r0} {,oldstable-,testing-}proposed-updates{,-debug} {{,old}stable,testing}-debug
do
  dak control-suite -l $suite | LC_ALL=C sort > before/${suite}.cs
done

for suite in oldstable stable testing
do
  zcat /srv/mirrors/debian/dists/$suite/main/source/Sources.gz > before/main_sources_$suite
  zcat /srv/mirrors/debian/dists/$suite/main/binary-amd64/Packages.gz > before/main_amd64_$suite
  zcat /srv/mirrors/debian/dists/$suite/main/Contents-amd64.gz > before/main_Contents_amd64_$suite
done

decron queue-viewer, secconnect and fetch-qa-data

On release day
--------------

for suite in oldoldstable oldstable stable testing {,old}stable-new {${OLD_CODENAME},${STABLE_CODENAME}}-{updates,r0} {,{,old}oldstable-,testing-}proposed-updates{,-debug} {{,old,oldold}stable,testing}-debug
do
  dak control-suite -l $suite | LC_ALL=C sort > after/${suite}.cs
done

for suite in oldoldstable oldstable stable testing
do
  zcat /srv/mirrors/debian/dists/$suite/main/source/Sources.gz > after/main_sources_$suite
  zcat /srv/mirrors/debian/dists/$suite/main/binary-amd64/Packages.gz > after/main_amd64_$suite
  zcat /srv/mirrors/debian/dists/$suite/main/Contents-amd64.gz > after/main_Contents_amd64_$suite
done

Compare:
  diff -u before/main_sources_oldstable after/main_sources_oldoldstable
  diff -u before/main_sources_stable after/main_sources_oldstable
  diff -u before/main_sources_testing after/main_sources_stable
  diff -u before/main_sources_testing after/main_sources_testing

  diff -u before/main_amd64_oldstable after/main_amd64_oldoldstable
  diff -u before/main_amd64_stable after/main_amd64_oldstable
  diff -u before/main_amd64_testing after/main_amd64_stable
  diff -u before/main_amd64_testing after/main_amd64_testing

  diff -u before/main_Contents_amd64_oldstable after/main_Contents_amd64_oldoldstable
  diff -u before/main_Contents_amd64_stable after/main_Contents_amd64_oldstable
  diff -u before/main_Contents_amd64_testing after/main_Contents_amd64_stable
  diff -u before/main_Contents_amd64_testing after/main_Contents_amd64_testing

  diff -u before/oldstable.cs after/oldoldstable.cs
  diff -u before/stable.cs after/oldstable.cs
  diff -u before/testing.cs after/stable.cs
  diff -u before/testing.cs after/testing.cs

  diff -u before/stable-new.cs after/oldstable-new.cs
  diff -u empty.cs after/stable-new.cs

  diff -u before/oldstable-proposed-updates.cs after/oldoldstable-proposed-updates.cs
  diff -u before/proposed-updates.cs after/oldstable-proposed-updates.cs
  diff -u before/testing-proposed-updates.cs after/proposed-updates.cs
  diff -u before/testing-proposed-updates.cs after/testing-proposed-updates.cs

  diff -u {before,after}/${OLD_CODENAME}-updates.cs
  diff -u {before,after}/${OLD_CODENAME}-r0.cs
  diff -u {before,after}/${STABLE_CODENAME}-updates.cs
  diff -u {before,after}/${STABLE_CODENAME}-r0.cs

All of the above comparisons should generate empty diffs and no errors.

Procedures for signatures are similar to those for point releases. A set of
signatures will need to be produced for each of stable, oldstable and oldoldstable
(assuming the latter is still on ftp-master).

Sanity-check signatures:

  printf "\n\n"
  for suite in oldoldstable oldstable stable
  do
    for suffix in "" -debug
    do
      printf "${suite}${suffix}\n\n"
      gpg --no-options --no-default-keyring --keyring /usr/share/keyrings/debian-archive-keyring.gpg --verify Release-${suite}${suffix}{.gpg,}
      printf "\n${suite}${suffix} InRelease\n\n"
      gpg --no-options --no-default-keyring --keyring /usr/share/keyrings/debian-archive-keyring.gpg --verify InRelease-${suite}${suffix}.gpg
      printf "\n"
    done
    printf "\n"
  done


Name the signature files {,In}Release-${suitename}{,-debug}.gpg and
copy them to the "sigs" folder.

Ensure that
- the ${NEW_STABLE}-new suite exists, and is empty
- the ${NEW_STABLE}-r0 suite exists, and contains at least the released d-i
- queue/{,o-}p-u-new exist and contain the expected files (particularly,
  o-p-u-new/COMMENTS should contain the files that used to be in p-u-new/COMMENTS)
- processing for o-p-u-new has been re-enabled in dak
- the ${NEW_STABLE}-updates suite exists, and dak will accept updates to it (the
  "import_dataset.sh" script in dak needs to be updated to allow this)


Updates to SRM tools after a Debian release
===========================================

Ensure that ${NEW_STABLE}_{comments,diffs,debcheck,lintian} directories exist,
and are mode 2775, and owned by release:debian-release.

queue-viewer configuration [in release.d.o git/etc]
  - re-enable oldstable

queue-viewer wrapper [in release.d.o git/bin]
  - re-enable oldstable

fetch-qa-data
  - re-enable oldstable

sync-pu-comments
  - re-enable oldstable

tools/config.ini [aka release.d.o git/etc/tools.conf]
  - update release_architectures for testing

BTS
  - update usertags used for pu, opu and stable / oldstable rm bugs

If any packages were moved from t-p-u to p-u during the release, queue-viewer
will not see them until manual ACCEPTED comments files are created for them.

Run sync-pu-comments by hand and check that it works. Note that the file moves
on the ftp-master side may have updated the modification time on the comments
files, causing older comments to be re-synced. Check and clean up any older /
unnecessary files before proceeding.

Run each of:
- secconnect
- fetch-qa-data
- queue-viewer

by hand before re-enabling their respective cron jobs.

If fetch-qa-data generates errors, check with the piuparts team on the
status of the relevant suites / reports.

website updates
---------------

In the release.debian.org repository:
- www/index.html:
  - update the "next stable point release" line to reflect the new stable
    version number
  - re-add a note about the next oldstable point release
  - re-add a link to the o-p-u queue-viewer page
- www/proposed-updates/index.html:
  - re-add a link to the o-p-u queue-viewer pge