Uploading packages

Task type changes

There is a new Wait task type. This represents a step in a workflow where debusine needs to wait until something else (user interaction or some other part of debusine). The task name determines what debusine is waiting for.

Wait tasks may involve running some code in debusine to perform additional checks or process input provided by the user; this is handled inline in the scheduler, in the same way as workflow orchestrators. (If this becomes a performance limitation then it may be moved into a Celery worker instead.)

MakeSourcePackageUpload task

This worker task makes a debian:upload artifact from a debian:source-package artifact. This involves unpacking the source package and running dpkg-genchanges on it.

The task_data for this task may contain the following keys:

  • input (required): a dictionary describing the input data:

    • source_artifact (Single lookup, required): a debian:source-package artifact

  • since_version (string, optional): include changelog information from all versions strictly later than this version in the .changes file; the default is to include only the topmost changelog entry

  • target_distribution (string, optional): override the target Distribution field in the .changes file to this value; the default is to use the distribution from the topmost changelog entry

The output is a debian:upload artifact with extends and relates-to relationships to the input source package artifact (to match the behaviour of debusine import-debian-artifact).

MergeUploads task

This worker task combines multiple debian:upload artifacts into a single one, in preparation for uploading them together. This involves running mergechanges on them, or the equivalent.

The task_data for this task may contain the following keys:

  • input (required): a dictionary describing the input data:

The output is a debian:upload artifact with extends relationships to each of the input upload artifacts.

Debsign task

This is a signing task that signs a debian:upload artifact on a signing worker. It is separate from the Sign task because signing uploads is a customized operation involving signing multiple files and possibly updating checksums in the .changes file to match the signed versions of other files.

The task_data for this task may contain the following keys:

  • unsigned (Single lookup, required): the debian:upload artifact whose contents should be signed

  • key (Single lookup, required): the debusine:signing-key artifact to sign the upload with, which must have purpose openpgp

The output will be provided as a debian:upload artifact.

ExternalDebsign task

This wait task blocks until a user provides a signature for an upload.

The task_data for this task may contain the following keys:

  • unsigned (Single lookup, required): the debian:upload artifact whose contents should be signed

Running this task does not do anything, but the task data informs the client that it needs to use the /api/1.0/work-request/<int:work_request_id>/external-debsign/ view below to record a debian:upload artifact as an output of the task. The containing workflow should normally then use an event reaction to add that output to a suitable collection (usually the workflow’s internal collection).

The task does not verify the signature, since it doesn’t necessarily have the public key available. It remains the responsibility of whatever would normally verify the signature (e.g. an external upload queue) to do so.

PackageUpload task

This server-side task uploads Debian packages to an upload queue.

It is the equivalent of running dput, but since other parts of debusine ensure that the upload is well-formed, there’s no need for most of the complexity of dput and we can avoid needing a worker environment for it.

The task_data for this task may contain the following keys:

  • input (required): a dictionary describing the input data:

  • target (required): the upload queue, as an ftp:// or sftp:// URL

The implementation should take care to use a suitable connection timeout. An SSH private key should be provided in the ~/.ssh/ directory of the user running debusine.

Workflow package_upload

This workflow signs and uploads source and/or binary packages to an upload queue. It is normally expected to be used as a sub-workflow.

  • task_data:

    • source_artifact (Single lookup, optional): a debian:source-package or debian:upload artifact representing the source package (the former is used when the workflow is started based on a .dsc rather than a .changes)

    • binary_artifacts (Multiple lookup, optional): a list of debian:upload artifacts representing the binary packages

    • merge_uploads (boolean, defaults to False): if True, merge the uploads and create a single PackageUpload task to upload them all together; if False, create a separate PackageUpload task for each upload

    • since_version (string, optional): passed to MakeSourcePackageUpload task if source_artifact is a debian:source-package

    • target_distribution (string, optional): passed to MakeSourcePackageUpload task if source_artifact is a debian:source-package

    • key (Single lookup, optional): the debusine:signing-key artifact to sign the upload with, which must have purpose openpgp

    • require_signature (boolean, defaults to True): whether the upload must be signed

    • target (required): the upload queue, as an ftp:// or sftp:// URL

At least one of source_artifact and binary_artifacts must be set.

The workflow creates the following tasks, each of which has a dependency on the previous one in sequence, using event reactions to store output in the workflow’s internal collection for use by later tasks:

  • if source_artifact is a debian:source-package artifact: a MakeSourcePackageUpload task (with since_version and target_distribution) to build a corresponding .changes file

  • if merge_uploads is True and there is more than one source and/or binary artifact: a MergeUploads task to combine them into a single upload

  • for each upload (or for the single merged upload, if merging):

    • if key is provided: a Debsign task to have debusine sign the upload with the given key

    • if key is not provided and require_signature is True: an ExternalDebsign task to wait until a user provides a signature, which debusine will then include with the upload

    • a PackageUpload task, to upload the result to the given upload queue

API changes

A new /api/1.0/work-request/<int:work_request_id>/external-debsign/ view allows handling the new ExternalDebsign wait task.

On GET, the view returns a response including the unsigned debian:upload artifact ID. The client downloads the files from this artifact that need to be signed (.changes, plus whichever of .dsc and .buildinfo exist), runs debsign on them, creates a new debian:upload artifact, and uploads the files to it. (This relies on some changes to avoid needing to reupload unchanged files.) It then sends a POST request back to the view, including the signed artifact ID.

On POST, the view works as follows, in a transaction:

  • check that the work request is WAIT/ExternalDebsign and is running, and otherwise return HTTP 400

  • check that the signed files only include files with permitted suffixes (.changes, .dsc, and .buildinfo), and otherwise return HTTP 400

  • inspect the work request’s task data to find the unsigned artifact, and look up the signed artifact from the request data, returning HTTP 400 if either of these fails

  • add a relates-to relation from the signed artifact to the unsigned artifact

  • link the signed artifact to the work request

  • mark the work request completed

  • return HTTP 200

UI changes

Wait tasks will generally require some form of UI change. In particular, when showing a blocked ExternalDebsign work request, the web UI advises the user to run a new client command (e.g. debusine provide-signature <work request ID>, which downloads the unsigned upload, signs it, and uploads it back to the external-debsign API view above.

Pipeline considerations

The package_upload workflow will typically be used as a sub-workflow of a smart “pipeline” workflow. There are three main use cases:

  • Upload a source package to debusine, have it tested, and then have debusine pass on that upload to an external upload queue.

    In this case, the package_upload workflow’s task data should set source_artifact to the source package and leave binary_artifacts empty.

  • Upload a source package to debusine, have it tested, and then have debusine upload both the source and all built binaries to an external upload queue. (For example, this is useful when uploading a package to Debian that will land in the NEW queue, since Debian currently requires binaries for NEW uploads.)

    In this case, the package_upload workflow’s task data should set source_artifact to the source package, set binary_artifacts to a list of single lookups matching each of the binary uploads from the super-workflow’s internal collection (e.g. [internal@collections/name:build-all, internal@collections/name:build-amd64], and set merge_uploads to True.

  • debusine acts as a build daemon, building a source package for a number of architectures and uploading each of them as soon as the builds finish.

    In this case, the package_upload workflow’s task data should leave source_artifact unset, set binary_artifacts to a list of single lookups matching each of the binary uploads from the super-workflow’s internal collection, and set merge_uploads to False.

Generic code for creating child work requests using artifacts from the workflow’s internal collection adds appropriate dependencies on the work requests that are expected to provide those artifacts.

If the parent workflow needs some kind of manual validation step to complete before starting the upload (typically in the case of manual uploads but not when acting as a build daemon), it should add a dependency from the package_upload sub-workflow to the validation workflow step. The package_upload sub-workflow will be populated before validation is complete (since the root workflow handles population of all its sub-workflows), but it will not start running until validation is complete.