==================
Uploading packages
==================

Task type changes
=================

There is a new ``Wait`` :ref:`task type <explanation-tasks>`.  This
represents a step in a workflow where debusine needs to wait until something
else (user interaction or some other part of debusine).  The task name
determines what debusine is waiting for.

Wait tasks may involve running some code in debusine to perform additional
checks or process input provided by the user; this is handled inline in the
scheduler, in the same way as workflow orchestrators.  (If this becomes a
performance limitation then it may be moved into a Celery worker instead.)

.. _task-make-source-package-upload:

MakeSourcePackageUpload task
============================

This worker task makes a ``debian:upload`` artifact from a
``debian:source-package`` artifact.  This involves unpacking the source
package and running ``dpkg-genchanges`` on it.

The ``task_data`` for this task may contain the following keys:

* ``input`` (required): a dictionary describing the input data:

  * ``source_artifact`` (:ref:`lookup-single`, required): a
    ``debian:source-package`` artifact

* ``since_version`` (string, optional): include changelog information from
  all versions strictly later than this version in the ``.changes`` file;
  the default is to include only the topmost changelog entry
* ``target_distribution`` (string, optional): override the target
  ``Distribution`` field in the ``.changes`` file to this value; the default
  is to use the distribution from the topmost changelog entry

The output is a ``debian:upload`` artifact with ``extends`` and
``relates-to`` relationships to the input source package artifact (to match
the behaviour of ``debusine import-debian-artifact``).

.. _task-merge-uploads:

MergeUploads task
=================

This worker task combines multiple ``debian:upload`` artifacts into a single
one, in preparation for uploading them together.  This involves running
``mergechanges`` on them, or the equivalent.

The ``task_data`` for this task may contain the following keys:

* ``input`` (required): a dictionary describing the input data:

  * ``uploads`` (:ref:`lookup-multiple`, required): a list of
    ``debian:upload`` artifacts

The output is a ``debian:upload`` artifact with ``extends`` relationships to
each of the input upload artifacts.

.. _task-debsign:

Debsign task
============

This is a :ref:`signing task <signing-app>` that signs a :ref:`debian:upload
<artifact-upload>` artifact on a signing worker.  It is separate from the
:ref:`task-sign` because signing uploads is a customized operation involving
signing multiple files and possibly updating checksums in the ``.changes``
file to match the signed versions of other files.

The ``task_data`` for this task may contain the following keys:

* ``unsigned`` (:ref:`lookup-single`, required): the ``debian:upload``
  artifact whose contents should be signed
* ``key`` (:ref:`lookup-single`, required): the ``debusine:signing-key``
  artifact to sign the upload with, which must have purpose ``openpgp``

The output will be provided as a :ref:`debian:upload <artifact-upload>`
artifact.

.. _task-external-debsign:

ExternalDebsign task
====================

This wait task blocks until a user provides a signature for an upload.

The ``task_data`` for this task may contain the following keys:

* ``unsigned`` (:ref:`lookup-single`, required): the ``debian:upload``
  artifact whose contents should be signed

Running this task does not do anything, but the task data informs the client
that it needs to use the
``/api/1.0/work-request/<int:work_request_id>/external-debsign/`` view below
to record a ``debian:upload`` artifact as an output of the task.  The
containing workflow should normally then use an event reaction to add that
output to a suitable collection (usually the workflow's internal
collection).

The task does not verify the signature, since it doesn't necessarily have
the public key available.  It remains the responsibility of whatever would
normally verify the signature (e.g. an external upload queue) to do so.

.. _task-package-upload:

PackageUpload task
==================

This server-side task uploads Debian packages to an upload queue.

It is the equivalent of running ``dput``, but since other parts of debusine
ensure that the upload is well-formed, there's no need for most of the
complexity of ``dput`` and we can avoid needing a worker environment for it.

The ``task_data`` for this task may contain the following keys:

* ``input`` (required): a dictionary describing the input data:

  * ``upload`` (:ref:`lookup-single`, required): a ``debian:upload``
    artifact

* ``target`` (required): the upload queue, as an ``ftp://`` or ``sftp://``
  URL

The implementation should take care to use a suitable connection timeout.
An SSH private key should be provided in the ``~/.ssh/`` directory of the
user running debusine.

.. _workflow-package-upload:

Workflow ``package_upload``
===========================

This workflow signs and uploads source and/or binary packages to an upload
queue.  It is normally expected to be used as a sub-workflow.

* ``task_data``:

  * ``source_artifact`` (:ref:`lookup-single`, optional): a
    ``debian:source-package`` or ``debian:upload`` artifact representing the
    source package (the former is used when the workflow is started based on
    a ``.dsc`` rather than a ``.changes``)
  * ``binary_artifacts`` (:ref:`lookup-multiple`, optional): a list of
    ``debian:upload`` artifacts representing the binary packages
  * ``merge_uploads`` (boolean, defaults to False): if True, merge the
    uploads and create a single ``PackageUpload`` task to upload them all
    together; if False, create a separate ``PackageUpload`` task for each
    upload
  * ``since_version`` (string, optional): passed to
    :ref:`task-make-source-package-upload` if ``source_artifact`` is a
    ``debian:source-package``
  * ``target_distribution`` (string, optional): passed to
    :ref:`task-make-source-package-upload` if ``source_artifact`` is a
    ``debian:source-package``
  * ``key`` (:ref:`lookup-single`, optional): the ``debusine:signing-key``
    artifact to sign the upload with, which must have purpose ``openpgp``
  * ``require_signature`` (boolean, defaults to True): whether the upload
    must be signed
  * ``target`` (required): the upload queue, as an ``ftp://`` or ``sftp://``
    URL

At least one of ``source_artifact`` and ``binary_artifacts`` must be set.

The workflow creates the following tasks, each of which has a dependency on
the previous one in sequence, using event reactions to store output in the
workflow's internal collection for use by later tasks:

* if ``source_artifact`` is a ``debian:source-package`` artifact: a
  :ref:`task-make-source-package-upload` (with ``since_version`` and
  ``target_distribution``) to build a corresponding ``.changes`` file
* if ``merge_uploads`` is True and there is more than one source and/or
  binary artifact: a :ref:`task-merge-uploads` to combine them into a single
  upload
* for each upload (or for the single merged upload, if merging):

  * if ``key`` is provided: a :ref:`task-debsign` to have debusine sign the
    upload with the given key
  * if ``key`` is not provided and ``require_signature`` is True: an
    :ref:`task-external-debsign` to wait until a user provides a signature,
    which debusine will then include with the upload
  * a :ref:`task-package-upload`, to upload the result to the given upload
    queue

API changes
===========

A new ``/api/1.0/work-request/<int:work_request_id>/external-debsign/`` view
allows handling the new ``ExternalDebsign`` wait task.

On ``GET``, the view returns a response including the unsigned
``debian:upload`` artifact ID.  The client downloads the files from this
artifact that need to be signed (``.changes``, plus whichever of ``.dsc``
and ``.buildinfo`` exist), runs ``debsign`` on them, creates a new
``debian:upload`` artifact, and uploads the files to it.  (This relies on
`some changes to avoid needing to reupload unchanged files
<https://salsa.debian.org/freexian-team/debusine/-/merge_requests/1081>`__.)
It then sends a ``POST`` request back to the view, including the signed
artifact ID.

On ``POST``, the view works as follows, in a transaction:

* check that the work request is ``WAIT/ExternalDebsign`` and is running,
  and otherwise return HTTP 400
* check that the signed files only include files with permitted suffixes
  (``.changes``, ``.dsc``, and ``.buildinfo``), and otherwise return HTTP
  400
* inspect the work request's task data to find the unsigned artifact, and
  look up the signed artifact from the request data, returning HTTP 400 if
  either of these fails
* add a ``relates-to`` relation from the signed artifact to the unsigned
  artifact
* link the signed artifact to the work request
* mark the work request completed
* return HTTP 200

UI changes
==========

Wait tasks will generally require some form of UI change.  In particular,
when showing a blocked ``ExternalDebsign`` work request, the web UI advises
the user to run a new client command (e.g. ``debusine provide-signature
<work request ID>``, which downloads the unsigned upload, signs it, and
uploads it back to the ``external-debsign`` API view above.

Pipeline considerations
=======================

The :ref:`package_upload workflow <workflow-package-upload>` will typically
be used as a sub-workflow of a smart "pipeline" workflow.  There are three
main use cases:

* Upload a source package to debusine, have it tested, and then have
  debusine pass on that upload to an external upload queue.

  In this case, the ``package_upload`` workflow's task data should set
  ``source_artifact`` to the source package and leave ``binary_artifacts``
  empty.

* Upload a source package to debusine, have it tested, and then have
  debusine upload both the source and all built binaries to an external
  upload queue.  (For example, this is useful when uploading a package to
  Debian that will land in the NEW queue, since Debian currently requires
  binaries for NEW uploads.)

  In this case, the ``package_upload`` workflow's task data should set
  ``source_artifact`` to the source package, set ``binary_artifacts`` to a
  list of :ref:`single lookups <lookup-single>` matching each of the binary
  uploads from the super-workflow's internal collection (e.g.
  ``[internal@collections/name:build-all,
  internal@collections/name:build-amd64]``, and set ``merge_uploads`` to
  True.

* debusine acts as a build daemon, building a source package for a number of
  architectures and uploading each of them as soon as the builds finish.

  In this case, the ``package_upload`` workflow's task data should leave
  ``source_artifact`` unset, set ``binary_artifacts`` to a list of
  :ref:`single lookups <lookup-single>` matching each of the binary uploads
  from the super-workflow's internal collection, and set ``merge_uploads``
  to False.

Generic code for creating child work requests using artifacts from the
workflow's internal collection adds appropriate dependencies on the work
requests that are expected to provide those artifacts.

If the parent workflow needs some kind of manual validation step to complete
before starting the upload (typically in the case of manual uploads but not
when acting as a build daemon), it should add a dependency from the
``package_upload`` sub-workflow to the validation workflow step.  The
``package_upload`` sub-workflow will be populated before validation is
complete (since the root workflow handles population of all its
sub-workflows), but it will not start running until validation is complete.
