==========================
Coordinating sub-workflows
==========================

The design of workflows allows for multiple sub-workflows to be composed
into a higher-level workflow.  This is an important and powerful feature,
but there are many subtleties involved.

General scheme
==============

Cooperation between workflows is defined at the level of workflows.
Individual work requests should not concern themselves with this; they
should be designed to take inputs using lookups and produce output artifacts
that are linked to the work request.

Sub-workflow coordination takes place through the workflow's internal
collection (which is shared among all sub-workflows of the same root
workflow), providing a mechanism for some work requests to declare that they
will provide certain kinds of artifacts which may then be required by work
requests in other sub-workflows.

On the providing side, workflows use the
:ref:`action-update-collection-with-artifacts` event reaction to add
relevant output artifacts from work requests to the internal collection, and
create :ref:`promises <bare-data-promise>` to indicate to other workflows
that they have done so.  Providing workflows choose item names in the
internal collection; it is the responsibility of workflow designers to
ensure that they do not clash, and workflows that provide output artifacts
have a optional ``prefix`` field in their task data to allow multiple
instances of the same workflow to cooperate under the same root workflow.

On the requiring side, workflows look up the names of artifacts they require
in the internal collection; each of those lookups may return nothing, or a
promise including a work request ID, or an artifact that already exists, and
they may use that to determine which child work requests they create.  They
use :ref:`lookups <lookup-syntax>` in their child work requests to refer to
items in the internal collection (e.g.
``internal@collections/name:build-amd64``), and add corresponding
dependencies on work requests that promise to provide those items.

The ``Workflow`` (or ``WorkRequest``, if appropriate) class gains helper
methods for use by workflow orchestrators to declare that a work request
provides output (adding an event reaction and a promise) and to declare that
a work request requires input (adding any necessary dependencies based on
its task data).  These help to make both the providing and requiring sides
work in uniform ways.

Orchestration changes
=====================

The ``populate`` method of sub-workflows is no longer called directly by the
scheduler.  Instead, the root workflow's ``populate`` method calls
``populate`` on any sub-workflows it creates.

This means that sub-workflows may depend on other steps within the root
workflow while still being fully populated in advance of being able to run.
A workflow that needs more information before being able to populate child
work requests should use :ref:`workflow callbacks <workflow-callback>` to
run the workflow orchestrator again when it is ready.  (For example, a
workflow that creates a source package and then builds it may not know which
work requests it needs to create until it has created the source package and
can look at its ``Architecture`` field.)

.. _bare-data-promise:

Category ``debusine:promise``
=============================

This bare data item represents an artifact that will eventually be provided
as the output of some existing work request.  The bare data item may contain
additional properties of said artifact to help differentiate them.  These
items are created by workflows when they populate their work request graphs.

Workflows add promises to their internal collection using the same name as
they use for the expected artifact.

* Data:

  * ``promise_work_request_id`` (integer, required): the ID of the work
    request that is expected to fulfil this promise
  * ``promise_workflow_id`` (integer, required): the ID of the workflow work
    request that created this promise
  * ``promise_category`` (string, required): the category of the artifact
    that is expected to be provided
  * ``promise_name`` (string, required): the name of this item in the
    workflow's internal collection (duplicated here in order to allow
    reconstructing lookups that will result in either this promise or this
    promised artifact once it exists)

Other data items should match the per-item data structure used when adding
the promised artifact to the workflow's internal collection, which is
determined by the parent workflow based on its protocol for communicating
with other workflows.  This allows constructing lookups that match either a
promise or the promised artifact.  Names starting with ``promise_`` are
reserved.

When a work request is retried, it must update the work request ID in any
associated promises.

debusine:workflow-internal collection changes
=============================================

The :ref:`debusine:workflow-internal collection
<collection-workflow-internal>` may now have per-item data, whose structure
is defined by workflows using the
:ref:`action-update-collection-with-artifacts` or
:ref:`action-update-collection-with-data` event reactions.  The
``variables`` or ``data`` fields respectively are copied into per-item data.
Names starting with ``promise_`` are reserved.

This allows matching promises or promised artifacts using workflow-defined
criteria.
