.. _tutorial-getting-started:

=============================
Getting started with debusine
=============================

In this tutorial you will discover two of debusine's building blocks:
:ref:`work requests <explanation-work-requests>` and :ref:`artifacts
<explanation-artifacts>`. The former lets you schedule the execution of
different kind of :ref:`tasks <explanation-tasks>` on remote workers,
while the latter is a simple way to represent a set of files managed by
debusine.

For the purpose of this tutorial, we will use debusine's command line
interface to:

* create a build environment
* upload a source package
* build binary packages out of the uploaded source package and in the build
  chroot that got created

Pre-requisites
--------------

You need to have access to a working debusine instance:

* If you are a Debian developer, you can use `debusine.debian.net
  <https://debusine.debian.net>`_. You can login there with your
  salsa.debian.org account, and it will automatically create
  your debusine account.
* Otherwise, please follow the instructions from
  :ref:`tutorial-install-debusine`. Once completed, the debusine instance
  will be available under the ``debusine.internal`` hostname and this is
  the name that we will continue to use here.

Install the client and configure it
-----------------------------------

You should first install the ``debusine-client`` package (if needed
configure APT with one of the :ref:`supported package repositories
<debusine-package-repositories>`):

.. code-block:: console

   $ sudo apt install debusine-client

Now it's time to create yourself a :ref:`token <create-api-token>`
that the client will use to connect to the server. For this open your
web browser and visit the URL of your debusine instance
(e.g. ``http://debusine.internal`` or
https://debusine.debian.net). Then login with your user credentials
(or with the salsa.debian.org authentication) and open the user menu
(in the top right corner, or behind the "hamburger" menu) and click on
the *Tokens* page.

Click the link *create a new token* and input a comment describing the
token's intended usage (for example "On my laptop") and click the *Create*
button. You will receive the token, which you should copy before exiting
the page as the token will only be shown once.

Finally, we can create debusine-client's configuration file. It indicates
the server URL and the token to use (please adjust the DEBUSINE_URL to
match your specific case and paste the value of the token in the
TOKEN variable):

.. code-block:: console

   $ DEBUSINE_URL="http://debusine.internal" # or DEBUSINE_URL="https://debusine.debian.net"
   $ TOKEN="1e81fb5f00a489344ddfe4e34588c84d202a97ca87cc136d15d17cb3d1eb4e82"
   $ DEBUSINE_FQDN=$(basename $DEBUSINE_URL)
   $ SCOPE=debusine
   $ mkdir -p ~/.config/debusine/client
   $ cat >~/.config/debusine/client/config.ini <<END
   [General]
   default-server = $DEBUSINE_FQDN

   [server:$DEBUSINE_FQDN]
   api-url = $DEBUSINE_URL/api
   scope = $SCOPE
   token = $TOKEN
   END

More information about the debusine command line interface is available
in :ref:`debusine-cli`.

Create a build environment
--------------------------

If you're using debusine.debian.net, you can skip this step, as suitable
build environments have already been created for you.

Otherwise, start by creating a :ref:`debian:environments
<collection-environments>` collection on your server:

.. code-block:: console

    $ sudo -u debusine-server debusine-admin create_collection \
        debian debian:environments </dev/null

You'll also need to have the necessary permissions to create workflow
templates.  We don't yet have a good interface for granting this permission, so
for now you'll need to insert this into the database by hand:

.. code-block:: console

    $ sudo -u debusine-server debusine-admin shell
    from django.contrib.auth import get_user_model
    from django.contrib.auth.models import Permission
    User = get_user_model()
    user = User.objects.get(username="YOUR-USER-NAME")
    permission = Permission.objects.get_by_natural_key(
        "add_workflowtemplate", "db", "workflowtemplate"
    )
    user.user_permissions.add(permission)

Then create an :ref:`update_environments <workflow-update-environments>`
workflow template and start it running:

.. code-block:: console

    $ debusine create-workflow-template \
        update-debian-environments update_environments <<END
    vendor: "debian"
    targets:
    - codenames: ["trixie"]
      architectures: ["amd64"]
      backends: ["unshare"]
      mmdebstrap_template:
        bootstrap_options:
          variant: "minbase"
        bootstrap_repositories:
        - mirror: "http://deb.debian.org/debian"
          components: ["main"]
    - codenames: ["trixie"]
      architectures: ["amd64"]
      variants: ["sbuild"]
      backends: ["unshare"]
      mmdebstrap_template:
        bootstrap_options:
          variant: "buildd"
        bootstrap_repositories:
        - mirror: "http://deb.debian.org/debian"
          components: ["main"]
    END

    $ debusine create-workflow update-debian-environments <<END
    {}
    END

Once this workflow finishes (which will take a few minutes), you should have
a ``debian:environments`` collection populated with some useful base
tarballs for ``trixie/amd64`` that can be used with the ``unshare`` backend:
a default variant containing only essential and required packages, and an
``sbuild`` variant that also contains build-essential packages.  These can
be :ref:`looked up by name <lookup-syntax>`.  If you wish, you can vary the
``targets`` dictionary to build different environments, or automate this
workflow to run regularly.

Create an artifact by uploading a source package
------------------------------------------------

The low level ``debusine create-artifact`` command can be used to create
any arbitrary artifact but when it comes to Debian source packages (.dsc)
or Debian uploads (.changes), debusine offers a more convenient
interface with ``debusine import-debian-artifact [FILE|URL]``. You can
refer to a local file or to a remote URL.

For instance, you can create and upload an artifact for the "hello" source
package with:

.. code-block:: console

  $ debusine import-debian-artifact http://deb.debian.org/debian/pool/main/h/hello/hello_2.10-3.dsc
  [...]
  message: New artifact created in http://debusine.internal/api in workspace System with id 536.
  artifact_id: 536

Artifacts can be provided as input to some debusine tasks.  We will shortly
use this artifact to schedule a package build.

Submit a work request to build binary packages
----------------------------------------------

Submitting a work request is asking the debusine server to schedule some
specific task on one of its available workers. Debusine can execute
many different :ref:`tasks <task-reference>`.

Among the available tasks, the :ref:`sbuild <task-sbuild>` one can build
binary packages out of a source package (provided through the parameter
``source_artifact`` below the ``input`` key) and a build chroot (provided
through the parameter ``environment``).

Each task requires a certain number of key-value parameters that are fed to
``debusine create-work-request TASKNAME`` as YAML data on the standard
input. Try this, taking care to refer to the ID of the artifact that we
created in the previous step (``536`` for the source package in this
example):

.. code-block:: console

    $ debusine create-work-request sbuild << END
    build_components:
    - any
    - all
    host_architecture: amd64
    input:
      source_artifact: 536
    environment: debian/match:codename=trixie:variant=sbuild
    END

This outputs some YAML structured information:

.. code-block:: yaml

  result: success
  message: Work request registered on http://debusine.internal/api with id 315.
  work_request_id: 315

At this point, the task has not been executed yet but it has been accepted
and will be processed as soon as a worker becomes available. You can
follow the status of the work request through the web interface
(click on *Work requests* in the top menu) or with ``debusine
show-work-request WORK_REQUEST_ID``.

.. code-block:: console

    $ debusine show-work-request 315
    id: 315
    created_at: 2024-01-24 17:02:31.855184+00:00
    started_at: 2024-01-24 17:02:31.937001+00:00
    completed_at: null
    duration: null
    status: running
    result: ''
    worker: 1
    task_name: sbuild
    task_data:
    […]
    artifacts: []
    workspace: System

In the above example, the task is currently running. Note that if the
status is ``pending`` it indicates that no ``debusine-worker`` is
currently available to run the task. Check that the ``debusine-worker`` is
connected, has been approved and has ``mmdebstrap`` installed.

Once it has completed (as indicated by ``status: completed``), you will
see different values: the result field will be either ``success``,
``failure`` (task ran but returned a failure) or ``error`` (something
unexpected happened), and more information will be available in the
``artifacts`` key in particular:

.. code-block:: console

    $ debusine show-work-request 315
    id: 315
    created_at: 2024-01-24 17:02:31.855184+00:00
    started_at: 2024-01-24 17:02:31.937001+00:00
    completed_at: 2024-01-24 17:05:48.457239+00:00
    duration: 196
    status: completed
    result: success
    […]
    artifacts:
    - id: 537
      workspace: System
      category: debian:package-build-log
    […]
    - id: 538
      workspace: System
      category: debian:binary-package
    […]
    - id: 541
      workspace: System
      category: debusine:work-request-debug-logs
    […]

The artifacts listed correspond to sets of files generated by the task, and
each task will typically generate the same kind of files as output.  In the
case of the ``sbuild`` task, you will get :ref:`debian:upload
<artifact-upload>`, :ref:`debian:binary-packages <artifact-binary-packages>`
and :ref:`debian:package-build-log <artifact-package-build-log>` artifacts.
You will also get a :ref:`debusine:work-request-debug-logs
<artifact-work-request-debug-logs>` artifact containing various files
generated by debusine to help troubleshoot issues with the task.

The generated artifacts can be browsed and downloaded from the web
interface. But they can also be downloaded from the command line
with the debusine client:

.. code-block:: console

    $ debusine download-artifact 538
    Downloading artifact and uncompressing into /home/debian
    hello_2.10-3_amd64.deb

On top of the contained files, the artifact category also defines the
structure of the metadata that is associated to the artifact. You can
inspect those metadata and the file listing on the web interface or on
the command line with ``debusine show-artifact ARTIFACT_ID`` (here ``540``
is the artifact ID of the ``debian:upload`` artifact created by the
task):

.. code-block:: console

    $ debusine show-artifact 540
    id: 540
    workspace: System
    category: debian:upload
    created_at: '2024-01-24T17:05:04.975882+00:00'
    data:
      type: dpkg
      changes_fields:
        Date: Mon, 26 Dec 2022 16:30:00 +0100
    […]
    download_tar_gz_url: http://debusine.internal/artifact/539/?archive=tar.gz
    files_to_upload: []
    expire_at: null
    files:
      hello_2.10-3_amd64.buildinfo:
        size: 5511
        checksums:
          sha256: 422aef340c827d2ed2b38c353f660b70e754509bc0ddb0952975090d9f25caaa
        type: file
        url: http://debusine.internal/artifact/539/hello_2.10-3_amd64.buildinfo
      hello_2.10-3_amd64.changes:
        size: 1889
        checksums:
          sha256: d5d694b42b94587d38a5f883fe1fc5d44368ffe974ac3d506d55bcbef0ab0767
        type: file
        url: http://debusine.internal/artifact/539/hello_2.10-3_amd64.changes
      hello_2.10-3_amd64.deb:
        size: 53084
        checksums:
          sha256: 069754b87d7a546253554813252dacbd7a53e959845cc9f6e8f4c1c8fe3746c5
        type: file
        url: http://debusine.internal/artifact/539/hello_2.10-3_amd64.deb
      hello-dbgsym_2.10-3_amd64.deb:
        size: 35096
        checksums:
          sha256: 1550fcd93105a3cf8fddfc776fda0fbebb51dd7c2d2286eeabc43cb37896ad1e
        type: file
        url: http://debusine.internal/artifact/539/hello-dbgsym_2.10-3_amd64.deb
