Administrator's Manual
######################

Introduction
************

The administrator's manual covers the package *installation*, *setup* and *maintenance* of a mini-buildd instance.

When mini-buildd runs, it's basically acts as a *web server*, with a django web application running on it.

mini-buildd's *configuration* consists of related django model instances, and their configuration is done via Django's
'admin' application. On the mini-buildd home page, just hit on `admin`_ (left top) to enter.

You will need to log in as Django user ``admin``, with the password you configured when installing the package (if you
chose an insecure password on package install time, now is the time to set a proper one via Django's user management).

All changes you do here finally wind up in the SQL database at ``~/config.sqlite``; this config not only represents mere
configuration, but also **state** (of ``~mini-buildd/``, and in case of chroots, even artifacts on the system, see
`Model statuses` below), so this file cannot be simply interchanged or copied.

Some of the models have a **status** attached to it.

This usually refers to a model's associated data on the system (which can be managed via actions in the configuration
interface):

====================== ====================================== ===========================================================
Model                  Associated prepared system data        File location (``~`` denoting mini-buildd's home path)
====================== ====================================== ===========================================================
*Daemon*               GnuPG Key                              ``~/.gnupg/``
*Repository*           Reprepro repository                    ``~/repositories/REPO_ID/``
*Chroot*               Chroot data and schroot configuration  - ``~/var/chroots/CODENAME/ARCH/``
                                                              - ``/etc/schroot/chroot.d/mini-buildd-CODENAME-ARCH.conf``
                                                              - Some backends (like LVM) may taint other system data
====================== ====================================== ===========================================================

Some other models also use the same status infrastructure, but the associated data is prepared internally in the model's
data (SQL database) only:

=========================== ==============================================================
Model                       Associated prepared data
=========================== ==============================================================
*AptKey, Uploader, Remote*  Public GnuPG Key
*Source*                    List of matching archives, selected info from Release file
=========================== ==============================================================


**Status semantics**:

============ ========================== ===============================================================================
Status       Check status               Semantic
============ ========================== ===============================================================================
*Removed*                               No associated data.
*Prepared*                              Associated data exists. With no flags, data is checked and in-sync.
                                        Special conditions to the data may apply:
*Prepared*   *Unchecked* (-)            Needs a manual *check* run to set things straight.
*Prepared*   *Changed* (*)              Model was changed, but the date is not yet updated. Needs
                                        a manual *prepare* run to set things straight.
*Prepared*   *Failed* (x)               Check failed.
*Prepared*   *Failed_Reactivate* (A)    Check failed, will be automatically activated again as soon
                                        as *check* succeeds again.
*Active*                                Prepared on the system, checked and activated.
============ ========================== ===============================================================================

**Status actions**:

Status actions can be called from a model's list view in Django's admin configuration.

=========== ============================================================================
Action      Semantic
=========== ============================================================================
Prepare     Create associated data on the system, or synchronize it with item changes.
Check       Check item and/or associated data.
Activate    Activate the item, or set the auto-activate flag.
Deactivate  Deactivate the item, or remove the auto-activate flag.
Remove      Remove associated data from system.
=========== ============================================================================

.. tip:: In the web configurator, you may use the ``PCA`` shortcut (``prepare``, ``check`` and ``activate``).

Installation
************

:debpkg:`mini-buildd` is available natively in and for Debian.

* ``Debian Backports`` may offer updated packages
* mini-buildd's `homepage`_ may offer alternate (Debian or Ubuntu) distributions to install on, or bleeding-edge package updates.

To install a mini-buildd instance, just install the Debian package:

.. code:: bash
	:class: root-code

	apt install mini-buildd

.. tip:: How much disk space do I need to run mini-buildd?

	For **building**, as a rule of thumb, you will need about::

		(<CODENAMES> * <ARCHS> * 0.3G) + (<CORES> * 4G)

	disk space free for building.

	F.e., if you support codename ``bullseye`` and ``buster`` builds for archs ``amd64`` and ``i386`` on a machine with
	``4`` cores::

		(2 * 2 * 0.3G) + (4 * 4G) = 17.2G

	Builds would still not run oo space if max default number of parallel builds is running (assuming base chroot size of
	0.3G and a (sort of worst case) Debian Source Tree size of 4G).

	No rule of thumb when you host **repositories**. Just make it big.

.. tip:: How to run mini-buildd in a systemd container (nspawn)?

	 Running the whole kerfuffle in it's own container may be beneficial:

	 * Avoid installing all dependencies on your main system
	 * Run a tainted/compatibility system (i.e., with changes you would want to avoid on your main system) -- like a builder-only system for urold distributions
	 * Run multiple instances on the same host (using different ports)

	 .. code-block:: console
			:caption: Example running mini-buildd in a bookworm systemd container as permanent service
			:class: root-code

			debootstrap --include=dbus bookworm /var/lib/machines/mini-buildd
			apt install mini-buildd-utils
			mkdir -p /etc/systemd/nspawn
			ln -s /usr/share/doc/mini-buildd-utils/examples/systemd-container.nspawn /etc/systemd/nspawn/mini-buildd.nspawn
			machinectl enable mini-buildd
			machinectl start mini-buildd
			machinectl shell mini-buildd   # Install mini-buildd
			# Now setup mini-buildd, either via web from interface from outside or from shell/mini-buildd-api from inside the container

	 .. literalinclude:: ../examples/mini-buildd-utils/systemd-container.nspawn
			:caption: /usr/share/doc/mini-buildd-utils/examples/systemd-container.nspawn

Configuration
=============

Package configuration (via debconf) include the *home path* of the mini-buildd user, the **administrator's password**
and *extra options* for the daemon run.

Usually, you set the admin password on the initial install, and just leave the rest on default values.

Of course, you can change your (debconf) settings anytime (including (re-)setting the admin password and changing
mini-buildd's home path) using:

.. code:: bash
	 :class: root-code

	 dpkg-reconfigure mini-buildd

Running encrypted
-----------------

.. versionadded:: 1.1.21

To run **HTTPS**, you need to adapt the ``--http-endpoint`` and possibly ``--hostname`` (if your certificate's host name
differs from ``hostname -f``) command line options (see :mbdcommand:`mini-buildd`):

.. code:: bash
	 :class: root-code

	 dpkg-reconfigure mini-buildd  # [--hostname=<CERT_HOSTNAME>] --http-endpoint="ssl:port=8066:privateKey=KEY_PATH:certKey=CERT_PATH"

The SSL private key file **must be readable** by user or group ``mini-buildd``.

.. tip:: How to enable encrypted uploads (FTPS)?

	If you happen to bootstrap via ``setup`` *from HTTPS*, the created Daemon instance per default already uses *FTPS*.

	To enable it later, adapt the FTP setup in the ``Daemon`` instance via :mbdpage:`admin`.

Self-signed certificates
~~~~~~~~~~~~~~~~~~~~~~~~

If you don't have/want certificates signed by official authorities, self signed certificates will also work -- but be sure...

- certifcate's CN is the same as your fqdn hostname (or what you set via the ``--hostname`` arg).
- self signed public cert get added to the system's certificate store for all machines you want to access the installation (see Debian package ``ca-certificates``).
- self signed public cert get added to web browsers used for mini-buildd
- [dev only] each instance and cert have a different hostname -- in case you run multiple instances on the same host (only makes sense for developer testing).

You may use this tool to maintain such a certificate:

.. code:: bash
	 :class: root-code

	 mini-buildd-self-signed-certificate create
	 dpkg-reconfigure mini-buildd  # [--hostname=<CERT_HOSTNAME>] --http-endpoint=ssl:port=8066:privateKey=/etc/ssl/mini-buildd/private/mini-buildd.key:certKey=/etc/ssl/mini-buildd/certs/mini-buildd.crt

Setup
*****

A ``setup`` is the configuration (Daemon, Sources, Repositories, etc.) as well as (potentially) their status (f.e.,
actual chroots or repositories on the system).

You may use the :apicall:`setup` to simplify this task, configure manually via :mbdpage:`admin` (or do a mix of both).

Visit :mbdpage:`setup` to do both via the ``Web Interface``.

API call ``setup``
==================

:apicall:`setup` is a versatile tool that can be used to

* *Inspect* current settings for changes against *setup's defaults*
* *Extend* your current settings
* *Bootstrap* a whole instance (except ``Remote`` handshaking)

*setup's default* is mini-buildd's internal "database" of known (Debian-based) distributions (see :py:mod:`.dist`). The
data varies from APT keys sources need down to special workarounds (older) distribution just need to work.

Configurations
--------------

*Explicit configurations* only consist of options
 ``identity``, ``ftp_endpoint``, ``archives``, ``sources``, ``distributions``, ``repositories``, ``chroots`` and ``remotes``.

At all times, there is the *automated current explicit configuration* that is generated from the actual objects of your
setting (see :mbdpage:`setup`, **Current**). This may be used to *check*, or as starting point to *extend*, your current
setting.

Example (*Debian* preset March 2023)::

	--identity myid --ftp-endpoint tcp6:port=8067 --archives http://archive.debian.org/debian-backports/ http://archive.debian.org/debian-security/ http://archive.debian.org/debian/ http://deb.debian.org/debian-security/ http://deb.debian.org/debian/ http://ftp.debian.org/debian/ http://security.debian.org/debian-security/ --sources sid bookworm bullseye --distributions sid:amd64+i386 bookworm:amd64+i386 bullseye:amd64+i386 --repositories test:Default:sid+bookworm+bullseye --chroots sid:amd64+i386 bookworm:amd64+i386 bullseye:amd64+i386 --chroots-backend Dir

*Implicit configurations* use only (non-explicit) options that provide some magic to compute explicit configurations.

Example (*Debian* preset)::

	--from-origins Debian --archives-from-origins --sources-from-origins --distributions-from-sources --repositories-from-distributions test --chroots-from-sources

If implicit and explicit options are mixed, values will be merged.

Inspect
-------

Just run *Current* to check your current setup:

1. Go to :mbdpage:`setup`:
	 1. Run *Current*, check upcoming results (won't yet change anything)
	 2. In result, check report for potential issues && optionally auto-fix all or individual objects

Extend
------

To extend your configuration, best use implicit options on top of your current configuration. As example, say you want
to add all LTS distributions to the default *Debian* setup:

1. Go to :mbdpage:`setup`:
	 1. Expand extra options in *Extend*
			1. ``--from-origins``: 'Debian:lts'
	 2. Run *Extend*, check upcoming results (won't yet change anything)
	 3. If satisfied, run API call again via 'update+pca all' (at bottom of result) -- this may take some time

In general, in *Extend*, you may disable any of the meta options

1. ``--distributions-from-sources``
2. ``--repositories-from-distributions``
3. ``--chroots-from-sources``

to steer what is being extended -- for example, disable extending ``Distributions`` on a builder-only instance.

Bootstrap
---------

Let's say, after initial install, you want to have a standard *Debian* setup:

1. On :mbdpage:`setup`: Run *Debian* bootstrap (this may take some time)
	 1. Optional: Expand arguments of *Debian* bootstrap, and customize right now (like adding your custom repository)
2. From main menu: Run :apicall:`start` to *accept incoming*
3. On :mbdpage:`repositories`: Run :apicall:`keyring_packages` (this may take some time)

You should end up with a fully functional standalone setup with one ``test`` Repository (allowing anonymous uploads).

Manual Setup
============

.. attention:: **Manual Setup Section**: Has not been revised since ``1.0.x``

Daemon
------

The Daemon model represents a configured mini-buildd instance. It is limited to have exactly one instance; when
activated, it means the internal FTP server is started acting on ``*.changes``.

Don't confuse this with the ``mini-buildd`` Unix daemon, which is always running when the mini-buildd Debian package is
installed, and always provides the HTTP server and web application.

The Daemon instance inside of mini-buildd provides the packager and builder engine (triggered by incoming via the FTP
server), and can be enabled/disabled inside mini-buildd.

#. Edit the one Daemon instance. Get the ``identity`` and the ``gnupg template`` right, changing these later will call for trouble.
	 * ``identity`` will hereafter be referred to as ``<ARCHIVE>``. Keyring packages will be named ``<ARCHIVE>-archive-keyring``.
	 * ``prepare`` will generate your instance ID (read: GnuPG key).
		 You may need to generate some **entropy** (install
		 :debpkg:`haveged` maybe) on the system if this stalls.
#. ``PCA`` the Daemon instance.

Source
------

This groups all models that determine what APT sources are available, and where to get them.

You will later interface with ``Source`` and ``PrioritySource`` when dealing with chroots and distributions.

A ``Source`` is usually identified sufficiently by ``Origin`` and ``Codename``.

When a ``Source`` is prepared, it will implicitly update ``Architectures`` and ``Components`` from the the ``Release``
file.

.. versionchanged:: 2.0.0

	 Defaults to "one number scheme" (like ``~bpo8``, ``~bpo11``) for mandatory version appendix.

	 The default 'codename version' is now only a single number for Debian distributions wheezy upwards (for example, '8'
	 instead of '80' for jessie), following the updated convention in Debian.

	 This change does not affect existing Source instances; to follow this recommendation in this case, you need to
	 reconfigure the current resp. Source, save and empty string for 'Codeversion override' (Extras), and "PCA" the
	 Source.

.. versionchanged:: 2.0.0

	 Defaults to prefix '~' to rolling (like ``~TRIXIE``) for mandatory version appendix.

	 The mandatory version appendix for rolling distributions now defaults to ~CODENAME (i.e., like "~SID" or "~STRETCH").

	 Using 'CODENAME' alone may call for versioning trouble once you convert the distribution to non-rolling (i.e., like
	 "STRETCH" is bigger for version comparison than "90" or "9").

	 This change does not affect existing instances; to follow this recommendation in this case, you need to reconfigure
	 the current resp.  rolling Distributions.

.. tip:: How to manually fix a failing check in a ``Source`` due to wrong/missing APT key(s)?

	 Run the failing ``check`` on that Source on :mbdpage:`admin`, and look for messages like "No accepted signature
	 found". These lines list the keys the resp. ``Release`` file is actually signed with.

	 Add at least one of these keys as ``AptKey`` (if it does not exists already), then add at least one of these to your
	 ``Source``.

	 A ``Source`` must have **at least one** valid key the Release file is signed with.

	 .. code-block:: console
			:caption: Example: Manually list all signatures a release file is signed on the console

			gpg --verify /var/lib/apt/lists/PATH_Release.gpg /var/lib/apt/lists/PATH_Release
			gpg --verify /var/lib/apt/lists/PATH_InRelease

On the System
-------------

The actual repositories are managed via ``reprepro``, and live in ``~/repositories`` -- each repository in its own
subdir.

You normally don't need to, but it's technically perfectly fine to manually do package manipulations (on the shell, as
user ``mini-buildd``) using ``reprepro`` commands. I this case, of course, it's in your power to meet or loosen
restrictions that otherwise mini-buildd inflicts on the repository.

You **must not** manually change any repository's *configuration* (i.e., on local disk) though, as these are
handled/overwritten by mini-buildd.

To be able to cope with multiple versions (``reprepro`` does only allow one package version per dist) each distribution
also has several additional `*-rollbackN` distributions configured.

Layout
------

It's **highly recommended** to just stick with one of the default Layouts, as produced by the ``Defaults wizard``.

In case you really need a custom layout, it's *recommended* not to change the Default Layouts, but to create a new
Layout profile with an appropriate name.

The Default Layout's semantics are outlined in :ref:`consumer:Default Layout`.

The Debian Developers Layout is meant for mimicking a layout like in Debian unstable (no version restriction, upload to
meta a distribution names like ``unstable``) to test build packages meant for Debian.

You will interface with Layouts in Repositories, determining what suites (and rollback distributions) are available,
which suites are uploadable, and which suites migrate, etc...

Stay In Sync With Default Layout Changes
----------------------------------------

In general, all the ``wizards`` never touch existing objects. This means, on existing systems, there is currently
(``1.0.x``) no way (unfortunately) to *stay on* or easily *upgrade to* the defaults as provided by mini-buildd wizards.

For those who deliberately want to upgrade to these (recommended) defaults, here are instructions how to do this
manually:

1.0.17: New Hotfix Suite
~~~~~~~~~~~~~~~~~~~~~~~~

.. versionadded:: 1.0.17

#. Enter the web application's configuration section `admin`_ and login as superuser ``admin``.
#. For Layouts ``Default`` and ``Default (no rollbacks)``
	 #. Enter the editor for that layout.
	 #. Add a new ``Suite Option`` (note: last entry shown in the list is for that purpose).
			- Rollback: ``4``
			- Suite: ``hotfix`` (note: you may need to add this; look for the green "+" sign below suite name.)
			- Uploadable: ``yes``
			- Experimental: ``no``
			- Migrates to: ``stable``
			- Not Automatic: ``yes``
			- But Automatic Upgrades: ``yes``
#. Re-index (PCA) all affected repositories.

Meta-Distributions
------------------

``Meta-Distributions`` can be set in a Layout's ``meta_distribution`` field.

Meta-Distributions may be seen as workaround to be able to upload (i.e., via ``debian/changelog``) to other
distributions than to the generic ``<codename>-<repoid>-<suite>`` format.

For example, the built-in "Debian Developers" Layout has mappings for ``unstable`` and ``experimental`` by default.

Note that these mappings are per Layout (and then, eventually, per Repository), but the final overall mapping must still
be unique for the whole mini-buildd instance (as we only have *one* incoming, and the incoming change's distribution
must be unambiguous).

So, when using this feature, this usually means:

* Make sure only *one* repository uses a Layout with Meta-Distributions configured (**recommended**).
* Make any meta mapping key appear only once in each used Layout.

.. versionchanged:: 1.0.25
	 Ambiguity of the global meta distribution map is now checked for (on repository checks and implicitly on package builds).

Distribution
------------

Distributions determines how and for what architectures a base distribution is to be build:

* What **base distribution**? (*sid, wheezy, lenny, ...*)
* With what **extra sources**? (*Debian Backports, ...*)
* What **components** to support? (*main, contrib, non-free, ...*)
* With what generic **build options**? (*resolver, lintian, ccache, eatmydata, ...*)
* For what **architectures**? (*i386, amd64, armel, ...*)

.. versionadded:: 2.0.0

	 ``Sbuild Config Blocks`` and ``Sbuild Setup Blocks``: Both sbuildrc config snippets as well as script snippets are now available as internal blocks to choose from -- See Model ``Distribution``, :mbdcommand:`mini-buildd-internals`

Repository
----------

A repository represents one apt repository managed via reprepro:

* What repository **identity**? ("codename-*identity*-suite")
* What mini-buildd **Layout**?  ("codename-identity-*suite*", supported suites and their semantics)
* What mini-buildd **Distributions**? ("*codename*-identity-suite")
* What **misc configuration** to use? (*reprepro, static GPG auth, notify, ...*)

Uploader
--------

Uploader instances are created automatically to each user profile. The administrator may activate GPG keys a user has
uploaded, and decide what repositories he is allowed to upload.

Chroot
------

Adding (active) chroots to your mini-buildd instance implicitly makes it a **builder**.

Preparing a chroots will both bootstrap it, and create configuration on the system so it can be used via ``schroot``.

You can chose amongst a number of schroot backends; to be able to be supported by mini-buildd, the backend must support
*snapshots* (compare ``man 5 schroot.conf``).

At the time (Oct 2016) of this writing, mini-buildd supports these backends:

============ ========================================= ================ ======== ======== ========================================================= ===============
Type         Options                                   Build size limit Speed    Extra fs Extra dependencies
============ ========================================= ================ ======== ======== ========================================================= ===============
Dir          **aufs**, overlayfs, unionfs, **overlay** No               Medium   No       Kernel support (aufs <= jessie, overlay >= stretch)       **Recommended**
File         compression                               No               Low      No       No
LVM          loop, given LVM setup                     Yes              Fast     Yes      LVM tools, Kernel support (dm, in Debian standard kernel)
BTRFS        none                                      No               ???      Yes      btrfs host file system, btrfs-progs
============ ========================================= ================ ======== ======== ========================================================= ===============

In short, we **recommend directory based chroots via aufs** using ``3.2.35 =< Debian Linux Kernel < 3.18`` (jessie-) and
**recommend directory based chroots via overlay** with ``kernels > 3.18`` (stretch+) as best compromise. It offers
acceptable speed, and no limits.

**File chroots** are also fine, they will just always work; you may think about configuring schroot to use a tmpfs for
its snapshots (if you have enough RAM), and use no compression to speed it up.

If you are in for speed, or just already have a LVM setup on your system, **LVM chroots** are good alternative, too.

Preparing chroots may take a while; if you cancel the HTTP request in your browser, preparation will continue anyway.

.. tip:: What's ``eatmydata`` and ``ccache``?

	You may configure Distributions with generic build options that may also affect the backend (like pre-installing
	``eatmydata``) or build (like configuring ``ccache`` to be used) speed. See ``Distributions and Repositories``.

.. tip:: My chroot creation fails due to missing arch in archive (partial mirror)?

	 This might occur, for example, if you use a (local) partial mirror (with debmirror or the like) as mini-buildd
	 archive that does not mirror the arch in question.

	 At the moment, all archives you add must provide all architectures you are going to support to avoid problems.

.. tip:: sudo fails with "sudo: no tty present and no askpass program specified"?

	 Make sure /etc/sudoers has this line::

		 #includedir /etc/sudoers.d

	 (This is sudo's Debian package's default, but the administrator might have changed it at some point.)

Remote
------

Remotes can interconnect a mini-buildd instance with another in a peer-to-peer fashion, i.e., you need to add a
respective remote instance on both two peers. When interconnected, these two instances automatically share their build
chroots.

To interconnect two mini-buildd instances

#. Add remote on instance0 for instance1; prepare, check, and activate it. Activation will initially fail, but it will be put on auto-reactivate (A).
#. Add remote on instance1 for instance0; prepare, check, and activate it. Activation will work as instance0 already knows us, and is on auto-reactivate.
#. Run Activate remote on instance0.

.. versionadded:: 2.0.0

	 There is now working a ``wake`` support (for suspending hosts).

Custom dput config snippet
==========================

Just put your custom snippet in ``~mini-buildd/etc/dput.cf``. If that file exists, it will be added to the output of :apicall:`dput_conf`.

Custom Event Hooks
==================

.. versionadded:: 2.0.0

Optional custom event hooks may be defined in ``~/etc/event-hooks``.

Any executable files matching ``[!_.]*[!~]`` glob pattern will be considered.

They will always be called with exactly one argument: The absolute file name of the saved (json) event file.

All scripts found will run in alphabetical sort order, in the context of ``mini-buildd`` (i.e., as user ``mini-buildd``).

mini-buildd is agnostic to whether these hooks do succeed, rather merely logs results to ``daemon.log``.

Layout example::

	00-do-this-first
	50-do-that
	99-do-that-last

Access via SSH
==============

Please see :mbdcommand:`mini-buildd-ssh-setup` for an help on an extra setup (outside mini-buildd) to provide access (to uploads and API calls with staff or admin authorization).

Workflows
*********

Build Keyring Packages
======================

Once you are satisfied with your setup, you should finish up by building keyring packages for your archive via the
:apicall:`keyring_packages`.

This is usually a one-timer, unless of course

- You changed/renewed your archive key.
- You added new distributions.
- You know there is an update of some sort to the internal (mini-buildd-internal) keyring package generator you are inclined to use.

In this case, just repeat the API call...

Handling of *ongoing* events (``PACKAGING`` and ``BUILDING``)
=============================================================

.. versionadded:: 2.0.0

When a ``changes`` is uploaded, it triggers an *ongoing* ``PACKAGING`` event until all ``buildresults`` have been received
(then resulting in either ``INSTALLED`` or ``FAILED``).

When a ``buildrequest`` is uploaded, it triggers an *ongoing* ``BUILDING`` event (as soon as a build slot is available)
until the build is finished (resulting in ``BUILT``). Sub-sequentially, the ``buildresult`` is uploaded to the
requesting instance.

Failing ``buildresult`` uploads are retried indefinitely (see :mbdpage:`crontab`), even when mini-buildd is *restarted*.

When mini-buildd is *restarted* (either the service or via :apicall:`stop`):

* *ongoing* ``PACKAGING`` events and subsequent potential ``buildresult`` uploads will be dismissed
* *ongoing* ``BUILDING`` events are being cancelled (resulting in a ``BUILT`` event and a (cancelled) ``buildresult`` scheduled for upload)

In case disaster strikes and some *ongoing* ``PACKAGING`` event turns orphan anyway, just restarting (at some
appropriate time) should do the trick.

Building on urold distributions
===============================

There are various reason why a seasoned distribution can no longer build -- ``Release`` files may become outdated, their
APT keys may expire, incompatibilities with the system mini-buildd is running on and many more.

See ``sbuild setup blocks`` (:mbdcommand:`mini-buildd-internals` or configuration of ``Distribution`` model) for what
mini-buildd has in store to maybe still get it to work.

squeeze
-------

``squeeze`` (as of April 2023) does have a 'working workarounds setup' -- but needs ``sbuild < 0.84`` to work.

In such a case, I'd suggest you set up one 'dirty' builder-only instance with sbuild pinned to that version:

.. literalinclude:: ../examples/mini-buildd/pin-sbuild-for-urold
	 :caption: /usr/share/doc/mini-buildd/examples/pin-sbuild-for-urold

Archiving urold distributions
=============================

Seasoned repository instances might host "urold" distributions that are no longer actively used. These distributions
might eat up valuable space on that instance, or slow down accessing the repository.

In case you want to get rid of these, this might be a reasonable workflow:

0. Preliminary: Data gathering && inform users
	 * List of ``to-be-archived codenames`` (no builds possible in future)
	 * New replacement ``archived URL``
	 * Date for this concerted action
	 * [Optional] You might want to install a patched ``debmirror`` from ``Hellfield`` (to mirror ``Contents`` files for experimental suites, see :debbug:`819925`)
1. :apicall:`stop`: Enter **maintenance mode**
2. :apicall:`debmirror`: Mirror all to-be-archived codenames. Keep all values on default except
	 * ``codenames=<to-be-archived codenames>``
	 * ``rollbacks=-1`` (to get all rollbacks)
3. Move partial mirror ``~mini-buildd/debmirror/`` away to the previously designated ``archived URL`` location
4. :mbdpage:`setup`: Edit all **Repositories** and move all now-archived distributions from ``Chosen`` to ``Available``
	 * [Optional] :mbdpage:`setup`: Remove (object *action*, not object *deletion*) now-maybe-unneeded **Chroots** (do this *before* deleting resp. *Sources*, else associated file data may not be removed when the object is deleted implicitly)
	 * [Optional] :mbdpage:`setup`: Delete now-maybe-unneeded **Sources** (this will in turn also delete resp. *PrioritySources*, *Chroots* and *Distributions*)
5. :mbdpage:`setup`: PCA all **Repositories** again (does repository re-index, i.e. actually removes the codenames from the repository)
6. :apicall:`start`: Open up again

Internal Crontab
================

.. versionadded:: 2.0.0

Periodic task are now handled internally, see :mbdpage:`crontab` to monitor runs or potentially debug problems.

I.e., there are no more 'system-cronned' service restart workaround that could cause problems in the past.

mini-buildd's File Layout
=========================

mini-buildd relies on having a user called ``mini-buildd``; the Unix home of this user is the home of your mini-buildd
instance, and the Unix daemon runs with its user id.

In mini-buildd's home, you will find this top level layout; i.e., these are handled by mini-buildd itself, and should
not be touched manually (unless you really know what you are doing, of course)::

	config.sqlite       mini-buildd's configuration.
	incoming/           Directory served by the ftpd.
	var/                Variable data: chroots, logs, temp directories, build directories.
	repositories/       Your valuable repositories.
	.gnupg/             The instance's GnuPG key ring.
	.django_secret_key  Some django shit we need.

When you **remove** the mini-buildd package **without purging**, it will remove system artifacts (see
``--remove-system-artifacts`` option, this currently affects chroots only) that can only be properly removed with
mini-buildd installed. Otherwise, mini-buildd's home (and, of course, the repositories) stay intact.

When you **purge** the mini-buildd package, all traces will be removed from the system, including your repositories.

Cruft Files
-----------

With newer versions, mini-buildd changed (internal) file paths several times -- and it can't be bothered to clean up
fully automatically after itself.

However, there is a little helper :mbdcommand:`mini-buildd-cruft` if you want to do it semi-automatic.

Logging and Debugging
=====================

.. tip:: Can I manually instantiate a build chroot to repeat/debug a failed build?

	Yes, see :mbdcommand:`mini-buildd-debug-build` -- you need to be administrator, though.

Per default, mini-buildd **logs**

* to mini-buildd's log file ``~/var/log/daemon.log``.
* to ``stderr`` (which usually ends up in *systemd's journal*).

The former is handled by mini-buildd itself, including rotating and access to it via API calls.

The latter is the same place where ``sbuild`` and friends put their logs by default.

You may control the **log level** via the ``--log-level``, and extra **debug options** via the ``--debug`` command line
flag.  The latter allows you to *keep temporary files*, enable python *exception traces* or to enable debug options for
*specific software components* used by mini-buildd. See :mbdcommand:`mini-buildd` for full documentation.

.. versionadded:: 1.0.19
	 Debug option *sbuild*.

Just use ``dpkg-reconfigure mini-buildd`` or edit
``/etc/default/mini-buildd`` to set any of these options
permanently.

Debug run in the console
------------------------

This example gives you a full treat of logs to the console (you may vary with the arguments to suit your needs):

.. code:: bash
	 :class: root-code

	 systemctl stop mini-buildd

.. code:: bash
	 :class: mini-buildd-code

	 PYTHONWARNINGS="default" /usr/sbin/mini-buildd --log-level DEBUG --debug=exception,http,webapp

HTTP access log
===============

Mini-buildd also keeps a standard HTTP access log in ``~/var/log/access.log``.

django: Avoid downgrades (does not start after downgrade)
=========================================================

mini-buildd usually is compatible with several django main versions (see control file). This, package-wise, allows for
downgrading django (maybe you want to go back from backports to stable for some reason).

This, however, will mostly always cause problems as the SQL database scheme of your app has already been updated.

In case this already has happened, you can only upgrade django again (or somehow try to manually downgrade mini-buildd's
SQL (~/config.sqlite) if you dare).

Import a foreign archive key to an existing mini-buildd instance
================================================================

1. Stop the mini-buildd service.
2. Become the mini-buildd user.
3. Manipulate the user's GPG keyring
	 * Be sure it contains exactly one key (pub+sec) when done.
4. (Re)start the mini-buildd service.
	 * Check that the Daemon key has actually changed (f.e., on the web home, right bottom).
5. Make a pseudo change to all repository instances.
	 * Just enter the repo editor, don't actually change anything, but do "save".
	 * This fixes the status to "Prepared (Changed)" (matching the external manipulation).
6. "PCA" ((re)prepare, check, create) all repositories.
	 * This should bring the new key to the reprepro indices.
7. Re-create keyring packages.

Possible problems fetching keys from key servers (gpg 2.1, 2.2)
===============================================================

Since gpg 2.1.22, 'use-tor' option is default. Afaiu, there is some magic in dirmgr now trying to autodetect if tor is
available, and then uses this (safer) option.

In practice, we have seen that receiving from keyserver has become unreliable, sometimes failing with::

	gpg: keyserver receive failed: Connection closed in DNS

and sometimes with::

	gpg: WARNING: Tor is not properly configured
	gpg: keyserver receive failed: Permission denied

and occasionally working fine.

.. seealso:: :debbug:`836266`

.. versionchanged:: 1.0.34,1.1.9

	 mini-buildd's internal importer now first tries to utilize keys from installed Debian or Ubuntu archive key packages
	 (and add 'Suggests:' for them) before reverting to 'recv' from the configured keyserver. Also, the keyserver import
	 is now being retried.

Ulimately, the GPG's defaults should be used, and ``dirmngr`` should be more reliable. If this bugs you however, you
might try the following options to mitigate the problem:

- Update ``tor``; on a stretch system, updating from 0.2.9.14 -> 0.3.2.9 seems to improve the success rate.
- Remove ``tor`` from the system.
- Use ``no-use-tor`` in ``dirmngr.conf``. This might eventually be an option when there is a systemwide default config for user-context dirmngr instances. Currently, that's not the case, and also no command line option to tunnel that through.
