Administrator's Manual
======================

The administrator's manual covers the package *installation*,
*setup* and *maintenance* of a mini-buildd instance.

Installation
------------

:debpkg:`mini-buildd` is available natively in and for Debian.

* ``Debian Backports`` may offer bugfix updates.
* mini-buildd's `homepage`_ may offer other alternate (Debian or Ubuntu) distributions to install on, or bleeding-edge updates.

To install a mini-buildd instance, just install the Debian package:

.. code:: bash
	:class: root-code

	apt install mini-buildd

.. tip:: How much disk space do I need to run mini-buildd?

	For **building**, as a rule of thumb, you will need about::

		(<CODENAMES> * <ARCHS> * 0.3G) + (<CORES> * 4G)

	disk space free for building.

	F.e., if you support codename ``bullseye`` and ``buster`` builds
	for archs ``amd64`` and ``i386`` on a machine with ``4`` cores::

		(2 * 2 * 0.3G) + (4 * 4G) = 17.2G

	Builds would still not run oo space if max default number of
	parallel builds is running (assuming base chroot size of 0.3G and a
	(sort of worst case) Debian Source Tree size of 4G).

	No rule of thumb when you host **repositories**. Just make it big.

Configuration
~~~~~~~~~~~~~

Package configuration (via debconf) include the *home path* of
the mini-buildd user, the **administrator's password** and
*extra options* for the daemon run.

Usually, you set the admin password on the initial install, and
just leave the rest on default values.

Of course, you can change your (debconf) settings anytime (including
(re-)setting the admin password and changing mini-buildd's home path)
using:

.. code:: bash
	 :class: root-code

	 dpkg-reconfigure mini-buildd

Running encrypted
.................

To run **HTTPS**, you need to adapt the ``--http-endpoint`` command line
option (see :mbdcommand:`mini-buildd`), for example by:

.. code:: bash
	 :class: root-code

	 dpkg-reconfigure mini-buildd  # Syntax example: 'ssl:port=8066:privateKey=KEY_PATH:certKey=CERT_PATH'

.. versionadded:: 1.1.21

.. important:: The SSL private key must be readable by user or group ``mini-buildd``.

.. tip:: How to enable encrypted uploads (FTPS)?

	If you happen to bootstrap via ``setup`` *from HTTPS*, the created
	Daemon instance per default already uses *FTPS*.

	To enable it later, adapt the FTP setup in the ``Daemon`` instance via :mbdpage:`admin`.

Self-signed certificates
````````````````````````

If you don't have/want certificates signed by official authorities, self signed certificates will also work -- but be sure...

- certifcate's CN is the same as your fqdn hostname (or what you set via the ``--hostname`` arg).
- self signed public get added to the system's certificate store for all machines you want to access the installation (see Debian package ``ca-certificates``).
- [dev only] each instance and cert have a different hostname -- in case you run multiple instances on the same host (only makes sense for developer testing).

You may use this tool to maintain such a certificate:

.. code:: bash
	 :class: root-code

	 mini-buildd-self-signed-certificate


Setup
-----

A ``setup`` is the configuration (Daemon, Sources, Repositories, etc.) as well as (potentially) their status (f.e., actual chroots or repositories on the system).

You may use the :apicall:`setup` to simplify this task. ``setup`` can be used to bootstrap a setup as well as maintenance (f.e., add new distributions, or update/check for mini-buildd's defaults).

`setup`_ provides a one stop for this task, including some presets for bootstrapping.

.. attention:: TODO: More testing and clarification needed in this section.


Bootstrap
~~~~~~~~~

After initial install, i.e., with no setup, just use the
:apicall:`setup`, either with default values or using one of the
presets in :mbdpage:`setup`.

The most likely 'setup' options to customize here from the getgo would
be ``identity`` and ``repositories``.

When the call has successfully finished, you should end up with --
assuming the defaults -- a fully functional Repository
``test``. Optionally, build the internal test packages via
:apicall:`test_packages`.

In case you went with the defaults for the ``repositories`` (only have
a "test" repo), just create a new ``Repository`` instance with the
actual ``ID`` you want to use for production.

Think about how you want to do upload authorization for that
production repository; see :ref:`administrator:Auth via SSH wrapper`.

Calling :apicall:`setup` is *mostly harmless*: idempotent, and it will
never change existing model instances unless explicitly using
``--update``.

Keyring Packages
~~~~~~~~~~~~~~~~

Once you are satisfied with your setup, you should finish up by
building keyring packages for your archive via the
:apicall:`keyring_packages`.

This is usually a one-timer, unless of course

- You changed/renewed your archive key.
- You added new distributions.
- You know there is an update of some sort to the internal (mini-buildd-internal) keyring package generator you are inclined to use.

In this case, just repeat the API call...


Maintenance
~~~~~~~~~~~~

TBD

Models
------

When mini-buildd runs, it's basically acts as a *web server*, with
a django web application running on it.

mini-buildd's *configuration* consists of related django model
instances, and their configuration is done via Django's 'admin'
application. On the mini-buildd home page, just hit on
`admin`_ (left top) to enter.

You will need to log in as Django user ``admin``, with the
password you configured when installing the package (if you
chose an insecure password on package install time, now is the
time to set a proper one via Django's user management).

All changes you do here finally wind up in the SQL database at
``~/config.sqlite``; this config not only represents mere
configuration, but also **state** (of ``~mini-buildd/``, and in
case of chroots, even artifacts on the system, see `Model
statuses` below), so this file cannot be simply interchanged or
copied.

Some of the models have a **status** attached to it.

This usually refers to a model's associated data on the system
(which can be managed via actions in the configuration
interface):

====================== ====================================== ===========================================================
Model                  Associated prepared system data        File location (``~`` denoting mini-buildd's home path)
====================== ====================================== ===========================================================
*Daemon*               GnuPG Key                              ``~/.gnupg/``
*Repository*           Reprepro repository                    ``~/repositories/REPO_ID/``
*Chroot*               Chroot data and schroot configuration  - ``~/var/chroots/CODENAME/ARCH/``
                                                              - ``/etc/schroot/chroot.d/mini-buildd-CODENAME-ARCH.conf``
                                                              - Some backends (like LVM) may taint other system data
====================== ====================================== ===========================================================

Some other models also use the same status infrastructure, but
the associated data is prepared internally in the model's data
(SQL database) only:

=========================== ==============================================================
Model                       Associated prepared data
=========================== ==============================================================
*AptKey, Uploader, Remote*  Public GnuPG Key
*Source*                    List of matching archives, selected info from Release file
=========================== ==============================================================


**Status semantics**:

============ ========================== ===============================================================================
Status       Check status               Semantic
============ ========================== ===============================================================================
*Removed*                               No associated data.
*Prepared*                              Associated data exists. With no flags, data is checked and in-sync.
                                        Special conditions to the data may apply:
*Prepared*   *Unchecked* (-)            Needs a manual *check* run to set things straight.
*Prepared*   *Changed* (*)              Model was changed, but the date is not yet updated. Needs
                                        a manual *prepare* run to set things straight.
*Prepared*   *Failed* (x)               Check failed.
*Prepared*   *Failed_Reactivate* (A)    Check failed, will be automatically activated again as soon
                                        as *check* succeeds again.
*Active*                                Prepared on the system, checked and activated.
============ ========================== ===============================================================================

**Status actions**:

Status actions can be called from a model's list view in
Django's admin configuration.

=========== ============================================================================
Action      Semantic
=========== ============================================================================
Prepare     Create associated data on the system, or synchronize it with item changes.
Check       Check item and/or associated data.
Activate    Activate the item, or set the auto-activate flag.
Deactivate  Deactivate the item, or remove the auto-activate flag.
Remove      Remove associated data from system.
=========== ============================================================================

.. tip:: In the web configurator, you may use the ``PCA`` shortcut (``prepare``, ``check`` and ``activate``).


Daemon
~~~~~~
The Daemon model represents a configured mini-buildd
instance. It is limited to have exactly one instance; when
activated, it means the internal FTP server is started acting on
``*.changes``.

Don't confuse this with the ``mini-buildd`` Unix daemon, which
is always running when the mini-buildd Debian package is
installed, and always provides the HTTP server and web
application.

The Daemon instance inside of mini-buildd provides the packager
and builder engine (triggered by incoming via the FTP server),
and can be enabled/disabled inside mini-buildd.

#. Edit the one Daemon instance. Get the ``identity`` and the ``gnupg template`` right, changing these later will call for trouble.
	 * ``identity`` will hereafter be referred to as ``<ARCHIVE>``. Keyring packages will be named ``<ARCHIVE>-archive-keyring``.
	 * ``prepare`` will generate your instance ID (read: GnuPG key).
		 You may need to generate some **entropy** (install
		 :debpkg:`haveged` maybe) on the system if this stalls.
#. ``PCA`` the Daemon instance.


Source
~~~~~~

This groups all models that determine what APT sources are
available, and where to get them.

You will later interface with ``Source`` and ``PrioritySource``
when dealing with chroots and distributions.

A ``Source`` is usually identified sufficiently by :term:`Origin` and :term:`Codename`.

When a ``Source`` is prepared, it will implicitly update
``Architectures`` and ``Components`` from the the ``Release`` file.

.. tip:: What APT keys is my Release file signed with?

	 Run s.th. like:

	 .. code:: bash

		 gpg --verify /var/lib/apt/lists/PATH_Release.gpg /var/lib/apt/lists/PATH_Release

	 for the Release in question to get a list of key ids the source
	 is actually signed with.

	 A source must must have **at least one** valid key the Release file is signed with.


On the System
~~~~~~~~~~~~~

The actual repositories are managed via ``reprepro``, and live
in ``~/repositories`` -- each repository in its own subdir.

You normally don't need to, but it's technically perfectly fine
to manually do package manipulations (on the shell, as user
``mini-buildd``) using ``reprepro`` commands. I this case, of
course, it's in your power to meet or loosen restrictions that
otherwise mini-buildd inflicts on the repository.

You **must not** manually change any repository's *configuration*
(i.e., on local disk) though, as these are handled/overwritten by
mini-buildd.

To be able cope with multiple versions (``reprepro`` does only allow
one package version per dist) each distribution also has several
additional `*-rollbackN` distributions configured.

Layout
~~~~~~

It's **highly recommended** to just stick with one of the
default Layouts, as produced by the ``Defaults wizard``.

In case you really need a custom layout, it's *recommended* not to
change the Default Layouts, but to create a new Layout profile
with an appropriate name.

The Default Layout's semantics are outlined in :ref:`consumer:Default Layout`.

The Debian Developers Layout is meant for mimicking a layout
like in Debian unstable (no version restriction, upload to meta
a distribution names like ``unstable``) to test build packages
meant for Debian.

You will interface with Layouts in Repositories, determining what
suites (and rollback distributions) are available, which suites
are uploadable, and which suites migrate, etc...

Stay In Sync With Default Layout Changes
........................................

In general, all the ``wizards`` never touch existing
objects. This means, on existing systems, there is currently
(``1.0.x``) no way (unfortunately) to *stay on* or easily
*upgrade to* the defaults as provided by mini-buildd wizards.

For those who deliberately want to upgrade to these
(recommended) defaults, here are instructions how to do this
manually:

1.0.17: New Hotfix Suite
````````````````````````

.. versionadded:: 1.0.17

#. Enter the web application's configuration section `admin`_ and login as superuser ``admin``.
#. For Layouts ``Default`` and ``Default (no rollbacks)``
	 #. Enter the editor for that layout.
	 #. Add a new ``Suite Option`` (note: last entry shown in the list is for that purpose).
			- Extra Options: ``Rollback: 4``
			- Suite: ``hotfix`` (note: you may need to add this; look for the green "+" sign below suite name.)
			- Uploadable: ``yes``
			- Experimental: ``no``
			- Migrates to: ``stable``
			- Not Automatic: ``yes``
			- But Automatic Upgrades: ``yes``
#. Re-index (PCA) all affected repositories.


Meta-Distributions
..................

``Meta-Distributions`` can be set in a Layout's "Extra Options".

Meta-Distributions may be seen as workaround to be able to
upload (i.e., via ``debian/changelog``) to other distributions
than to the generic ``<codename>-<repoid>-<suite>`` format.

For example, the built-in "Debian Developers" Layout has
mappings for ``unstable`` and ``experimental`` by default.

Note that these mappings are per Layout (and then, eventually,
per Repository), but the final overall mapping must still be
unique for the whole mini-buildd instance (as we only have *one*
incoming, and the incoming change's distribution must be
unambiguous).

So, when using this feature, this usually means:

* Make sure only *one* repository uses a Layout with Meta-Distributions configured (**recommended**).
* Make any meta mapping key appear only once in each used Layout.

.. versionchanged:: 1.0.25
	 Ambiguity of the global meta distribution map is now checked for (on repository checks and implicitly on package builds).


Distribution
~~~~~~~~~~~~

Distributions determines how and for what architectures a base
distribution is to be build:

* What **base distribution**? (*sid, wheezy, lenny, ...*)
* With what **extra sources**? (*Debian Backports, ...*)
* What **components** to support? (*main, contrib, non-free, ...*)
* With what generic **build options**? (*resolver, lintian, ccache, eatmydata, ...*)
* For what **architectures**? (*i386, amd64, armel, ...*)

.. error:: eatmydata: Builds fail when linked with openvc!

	 Only a problem in current (Jan 2014) *sid*. See :debbug:`733281`.


Repository
~~~~~~~~~~

A repository represents one apt repository managed via reprepro:

* What repository **identity**? ("codename-*identity*-suite")
* What mini-buildd **Layout**?  ("codename-identity-*suite*", supported suites and their semantics)
* What mini-buildd **Distributions**? ("*codename*-identity-suite")
* What **misc configuration** to use? (*reprepro, static GPG auth, notify, ...*)


Uploader
~~~~~~~~

Uploader instances are created automatically to each user
profile. The administrator may activate GPG keys a user has
uploaded, and decide what repositories he is allowed to upload.

.. important:: Don't add or delete Uploader instances manually!

	 These are bound to users, and come automatically when new users are
	 created. The administrator only changes these instances to grant
	 rights.

Chroot
~~~~~~
Adding (active) chroots to your mini-buildd instance implicitly
makes it a **builder**.

Preparing a chroots will both bootstrap it, and create
configuration on the system so it can be used via ``schroot``.

You can chose amongst a number of schroot backends; to be able
to be supported by mini-buildd, the backend must support
*snapshots* (compare ``man 5 schroot.conf``).

At the time (Oct 2016) of this writing, mini-buildd supports
these backends:

============ ========================================= ================ ======== ======== ========================================================= ===============
Type         Options                                   Build size limit Speed    Extra fs Extra dependencies
============ ========================================= ================ ======== ======== ========================================================= ===============
Dir          **aufs**, overlayfs, unionfs, **overlay** No               Medium   No       Kernel support (aufs <= jessie, overlay >= stretch)       **Recommended**
File         compression                               No               Low      No       No
LVM          loop, given LVM setup                     Yes              Fast     Yes      LVM tools, Kernel support (dm, in Debian standard kernel)
BTRFS        none                                      No               ???      Yes      btrfs host file system, btrfs-progs
============ ========================================= ================ ======== ======== ========================================================= ===============

In short, we **recommend directory based chroots via aufs**
using ``3.2.35 =< Debian Linux Kernel < 3.18`` (jessie-) and
**recommend directory based chroots via overlay** with ``kernels
> 3.18`` (stretch+) as best compromise. It offers acceptable
speed, and no limits.

**File chroots** are also fine, they will just always work; you
may think about configuring schroot to use a tmpfs for its
snapshots (if you have enough RAM), and use no compression to
speed it up.

If you are in for speed, or just already have a LVM setup on
your system, **LVM chroots** are good alternative, too.

Preparing chroots may take a while; if you cancel the HTTP request in your browser, preparation will continue anyway.

.. tip:: What's ``eatmydata`` and ``ccache``?

	You may configure Distributions with generic build options that may
	also affect the backend (like pre-installing ``eatmydata``) or build
	(like configuring ``ccache`` to be used) speed. See ``Distributions
	and Repositories``.

.. tip:: How do I use foreign-architecture chroots with qemu?

	 Tested with 'armel' (other architectures might work as well, but not tested).

	 Install these additional packages:

	 .. code:: bash
			:class: root-code

			apt install binfmt-support qemu-user-static

	 You will need a version of qemu-user-static with :debbug:`683205` fixed.

	 In the Chroot configuration, add a line::

		 Debootstrap-Command: /usr/sbin/qemu-debootstrap

	 to the extra options. That's it. Now just prepare && activate as usual.

.. error:: debootstrap fails for <=lenny chroots on >=jessie host kernel (uname)!

	 See :debbug:`642031`. This should ideally be worked around in debootstrap itself eventually.

	 mini-buildd comes with a workaround wrapper :mbdcommand:`mini-buildd-debootstrap-uname-2.6`. Just add::

		 Debootstrap-Command: /usr/sbin/mini-buildd-debootstrap-uname-2.6

	 to the chroot's extra options to work around it (the default
	 chroots created with the chroot wizard already include this
	 workaround for lenny and etch chroots, btw).

	 Fwiw, this is due to older libc6 packaging's preinst, which will
	 meekly fail if ``uname -r`` starts with a two-digit version;
	 i.e.::

		 FINE : 3.2.0-4-amd64      Standard wheezy kernel
		 FAILS: 3.10-2-amd64       Standard jessie/sid kernel
		 FAILS: 3.9-0.bpo.1-amd64  Wheezy backport of the jessie/sid kernel

.. error:: Fails to build "all" packages with "build archall" flag set to arch "x" in case DSP has >= 1 arch "all" and >=1 arch "y" binary package!

	 This is due to sbuild and in in more detail explained here :debbug:`706086`.

	 A bad one-package workaround would be to change the "build archall" flag to arch "y".

.. error:: LVM chroots fail running lvcreate with 'not found: device not cleared'!

	 Unclear (?). See :debbug:`705238` or http://lists.debian.org/debian-user/2012/12/msg00407.html .

	 "--noudevsync" workaround makes lvcreate work again, but the
	 chroot will not work later anyway later.

.. tip:: My chroot creation fails due to missing arch in archive (partial mirror)?

	 This might occur, for example, if you use a (local) partial
	 mirror (with debmirror or the like) as mini-buildd archive that
	 does not mirror the arch in question.

	 At the moment, all archives you add must provide all architectures you are
	 going to support to avoid problems.

.. tip:: sudo fails with "sudo: no tty present and no askpass program specified"?

	 Make sure /etc/sudoers has this line::

		 #includedir /etc/sudoers.d

	 (This is sudo's Debian package's default, but the
	 administrator might have changed it at some point.)


Remote
~~~~~~

Remotes can interconnect a mini-buildd instance with another in
a peer-to-peer fashion, i.e., you need to add a respective
remote instance on both two peers. When interconnected, these
two instances automatically share their build chroots.

To interconnect two mini-buildd instances

#. Add remote on instance0 for instance1; prepare, check, and activate it. Activation will initially fail, but it will be put on auto-reactivate (A).
#. Add remote on instance1 for instance0; prepare, check, and activate it. Activation will work as instance0 already knows us, and is on auto-reactivate.
#. Run Activate remote on instance0.

.. important:: Be sure to use the exact same host names as given in the resp. instance's Daemon configuration!


Advanced Topics
---------------

Custom Event Hooks
~~~~~~~~~~~~~~~~~~

Optional custom event hooks may be defined in ``~/etc/event-hooks``.

Any executable files matching ``[!_.]*[!~]`` glob pattern will be considered.

They will always be called with exactly one argument, the file name of the saved (json) event file.

All scripts found will run in alphabetical sort order, in the context of ``mini-buildd`` (i.e., as user ``mini-buildd``).

mini-buildd is agnostic to whether these hooks do succeed, rather merely logs results to ``daemon.log``.

Layout example::

	00-do-this-first
	50-do-that
	99-do-that-last


Internal Crontab
~~~~~~~~~~~~~~~~

.. versionadded:: 2.0.0

Periodic task are now handled internally, see :mbdpage:`crontab` to
monitor runs or potentially debug problems.

I.e., there are no more 'system-cronned' service restart workaround
that could cause problems in the past.


The mini-buildd user
~~~~~~~~~~~~~~~~~~~~
mini-buildd relies on having a user called ``mini-buildd``; the
Unix home of this user is the home of your mini-buildd instance,
and the Unix daemon runs with its user id.

In mini-buildd's home, you will find this top level layout; i.e.,
these are handled by mini-buildd itself, and should not be
touched manually (unless you really know what you are doing, of
course)::

	config.sqlite       mini-buildd's configuration.
	incoming/           Directory served by the ftpd.
	var/                Variable data: chroots, logs, temp directories, build directories spool.
	repositories/       Your valuable repositories.
	.gnupg/             The instance's GnuPG key ring.
	.mini-buildd.pid    Unix daemon PID file.
	.django_secret_key  Some django shit we need.

When you **remove** the mini-buildd package **without purging**,
it will remove system artifacts (see
``--remove-system-artifacts`` option, this currently affects
chroots only) that can only be properly removed with mini-buildd
installed. Otherwise, mini-buildd's home (and, of course, the
repositories) stay intact.

When you **purge** the mini-buildd package, all traces will be
removed from the system, including your repositories.


Logging and Debugging
~~~~~~~~~~~~~~~~~~~~~

.. tip:: Can I manually instantiate a build chroot to repeat/debug a failed build?

	Yes, see :mbdcommand:`mini-buildd-debug-build` -- you need to be administrator, though.

Per default, mini-buildd **logs**

* to mini-buildd's log file ``~/var/log/daemon.log``.
* via syslog, facility USER (which usually ends up in ``/var/log/user.log``).

The former is handled by mini-buildd itself, including rotating
and access to it via API calls.

The latter is the same place where ``sbuild`` and friends put
their logs by default.

You may control the **log level** via the ``--verbose``, and extra
**debug options** via the ``--debug`` command line flag.  The latter
allows you to *keep temporary files*, enable python *exception traces*
or to enable debug options for *specific software components* used by
mini-buildd. See :mbdcommand:`mini-buildd` for full documentation.

.. versionadded:: 1.0.19
	 Debug option *sbuild*.

Just use ``dpkg-reconfigure mini-buildd`` or edit
``/etc/default/mini-buildd`` to set any of these options
permanently.

Debug run in the console
........................

This example gives you a full treat of logs to the console (you
may vary with the arguments to suit your needs):

.. code:: bash
	 :class: root-code

	 systemctl stop mini-buildd

.. code:: bash
	 :class: mini-buildd-code

	 PYTHONWARNINGS="default" /usr/sbin/mini-buildd --foreground --verbose --verbose --debug=exception,http,webapp

Profiling run on the console
............................

.. code:: bash
	 :class: root-code

	 systemctl stop mini-buildd

.. code:: bash
	 :class: mini-buildd-code

	 python3 -m cProfile -o mini-buildd.profile /usr/sbin/mini-buildd --foreground --verbose


Auth via SSH wrapper
~~~~~~~~~~~~~~~~~~~~

See :mbdcommand:`mini-buildd-ssh-uploader`
:mbdcommand:`mini-buildd-ssh-client` -- i.e., yes, look at the
instructions in these script.

HTTP access log
~~~~~~~~~~~~~~~
Mini-buildd also keeps a standard HTTP access log in ``~/var/log/access.log``.


Keyring and test packages
~~~~~~~~~~~~~~~~~~~~~~~~~

On mini-buildd's home, you will find action buttons to
create+build keyring packages, as well as running test packages.


linux.deb >= 4.8.5: You may need to re-enable vsyscall
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In Debian kernel packages since ``4.8.4-1~exp1``::

	[ Ben Hutchings ]
	* [amd64] Enable LEGACY_VSYSCALL_NONE instead of LEGACY_VSYSCALL_EMULATE.
		This breaks (e)glibc 2.13 and earlier, and can be reverted using the kernel
		parameter: vsyscall=emulate

I.e.: When running the Debian standard kernel and your mini-buildd instance needs
to support ``wheezy`` or earlier, you need to re-enable this (in ``/etc/default/grub``).

You may use ``local-vsyscall-emulate.cfg`` in ``examples`` (a symlink
should do) for this purpose.

On any running kernel, this is a poor man's check to see if
vsyscall is still enabled on your system:

.. code:: bash

	grep "\[vsyscall\]" /proc/self/maps

.. seealso:: :debbug:`847154`, linux package's ``NEWS`` file.


django: Avoid downgrades (does not start after downgrade)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mini-buildd usually is compatible with several django main
versions (see control file). This, package-wise, allows for
downgrading django (maybe you want to go back from backports to
stable for some reason).

This, however, will mostly always cause problems as the SQL
database scheme of your app has already been updated.

In case this already has happened, you can only upgrade django
again (or somehow try to manually downgrade mini-buildd's SQL
(~/config.sqlite) if you dare).


Cruft in ~/var/log
~~~~~~~~~~~~~~~~~~

With mini-buildd <= 1.0.17, rejected packages where logged here
too, which may have lead to cruft that is never cleaned up. You
may run:

.. code:: bash
	 :class: mini-buildd-code

	 mini-buildd-reject-cleanup


Import a foreign archive key to an existing mini-buildd instance
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1. Stop the mini-buildd service.
2. Become the mini-buildd user.
3. Manipulate the user's GPG keyring
	 * Be sure it contains exactly one key (pub+sec) when done.
4. (Re)start the mini-buildd service.
	 * Check that the Daemon key has actually changed (f.e., on the web home, right bottom).
5. Make a pseudo change to all repository instances.
	 * Just enter the repo editor, don't actually change anything, but do "save".
	 * This fixes the status to "Prepared (Changed)" (matching the external manipulation).
6. "PCA" ((re)prepare, check, create) all repositories.
	 * This should bring the new key to the reprepro indices.
7. Re-create keyring packages.

.. important:: The Daemon instance does not touch the GPG setup once it's created -- *unless you do an explicit remove* on the instance.


Possible problems fetching keys from key servers (gpg 2.1, 2.2)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Since gpg 2.1.22, 'use-tor' option is default. Afaiu, there is some
magic in dirmgr now trying to autodetect if tor is available, and then
uses this (safer) option.

In practice, we have seen that receiving from keyserver has become
unreliable, sometimes failing with::

	gpg: keyserver receive failed: Connection closed in DNS

and sometimes with::

	gpg: WARNING: Tor is not properly configured
	gpg: keyserver receive failed: Permission denied

and occasionally working fine.

.. seealso:: :debbug:`836266`

.. versionchanged:: 1.0.34,1.1.9
	 mini-buildd's internal importer now first tries to utilize keys
	 from installed Debian or Ubuntu archive key packages (and add
	 'Suggests:' for them) before reverting to 'recv' from the
	 configured keyserver. Also, the keyserver import is now being
	 retried.

Ulimately, the GPG's defaults should be used, and ``dirmngr`` should
be more reliable. If this bugs you however, you might try the
following options to mitigate the problem:

- Update ``tor``; on a stretch system, updating from 0.2.9.14 -> 0.3.2.9 seems to improve the success rate.
- Remove ``tor`` from the system.
- Use ``no-use-tor`` in ``dirmngr.conf``. This might eventually be an option when there is a systemwide default config for user-context dirmngr instances. Currently, that's not the case, and also no command line option to tunnel that through.

Migrate packages from 0.8.x
~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. tip:: Can I skip migrating from 0.8.x?

	Yes -- a much simpler solution might be to just copy the old
	repository directory (``~/rep``) somewhere and serve it via some
	standard web server, and just continue to use it along with your
	new repo as long as needed.

This roughly explains the steps needed to upgrade a mini-buildd
0.8.x installation to 1.0.x with **transferring the packages
from the old 0.8.x repositories over**, so you can continue with
the new 1.0.x repos only:

1. Upgrade mini-buildd from 0.8.x to 1.0.

	 Chances are this might have already implicitly happened,
	 with some update.

	 You will then have 1.0 up and running, and ye olde 0.8.x
	 repositories still available as read-only apt repositories.

	 Just be sure you don't **purge** the old package, and then
	 install 1.0, as this will remove the whole old repository.

2. Configure mini-buildd 1.0.

	 This means you should, in the end, have a 1.0 repository with
	 the **same identity** as the old 0.8.x repository, and with
	 all distributions you want to migrate.

3. Import packages.

	 Become mini-buildd user, and got to the new 1.0 reprepro
	 repository you want to import to, and use the importer
	 script to migrate packages:

	 .. code:: bash
			:class: mini-buildd-code

			cd repositories/REPOID
			mini-buildd-import-08x ~/rep/squeeze-REPOID-experimental squeeze-REPOID-experimental
			mini-buildd-import-08x ~/rep/squeeze-REPOID squeeze-REPOID-unstable

	 This example is for squeeze; repeat the imports for all base
	 distributions you want to migrate.

	 Thusly, ye olde ``*-experimental`` distribution will be
	 migrated to the distribution with the same name in 1.0. Ye
	 olde ``squeeze-REPOID`` goes to
	 ``squeeze-REPOID-unstable``. For the latter, multiple package
	 version will be automatically installed to the new *rollback
	 distributions* (which are needed with reprepro to support
	 multiple package versions).

4. (Optional) Fix up package status.

	 All the migrated packages are now in 1.0 "unstable"
	 distribution; you may think of bulk-migrating them all to
	 "stable", if that were your semantics for the 0.8.x
	 non-experimental distributions.

Eventually, when everything is updated, you may of course
(re)move the old 0.8.x directory ``~/rep/``.

.. seealso:: :debbug:`790292`
