Built-in modules

A module gives a workflow the ability to do something outside itself. It exposes variables, actions and Starlark files, and a workflow definition imports them with load("module:<name>", ...). GitLab Relay (formerly KAS) compiles four modules into its own binary.

Every action call returns a future, and the result arrives where the future is gathered. Built-in names are reserved: a remote module that registers one of them is refused.

gitlab

Calls the GitLab REST API from inside AutoFlow.

load("module:gitlab", "call_api", "post_value")

call_api

call_api(method, path, query = None, headers = None, body = None)

method and path are positional. query is a dict of string or bytes values, or lists of them, and refuses sensitive values because query parameters end up in the URL. headers is a dict whose values may be plain or sensitive strings or bytes, or lists of those. body is a plain or sensitive string or bytes; Content-Type is yours to set through headers.

The result is always a four-element tuple.

A non-2xx response is data, not an error: it arrives as an ordinary status, so the workflow decides what it means. Only a failure to complete the request at all fills the fourth element.

def main(w, project_id, issue_iid, token):
    status, _headers, body, error = gather(call_api(
        "POST",
        "/api/v4/projects/%d/issues/%d/notes" % (project_id, issue_iid),
        headers = {
            "Authorization": "Bearer " + token,
            "Content-Type": "application/json",
        },
        body = json.encode({"body": "done"}),
    ))
    if error != None:
        fail("note: %s" % error)
    if status != 201:
        fail("note: API returned %d: %s" % (status, body))

Three rules follow from where the call runs.

• The request carries no AutoFlow JWT, so authentication has to come from headers. A token passed to main as a sensitive value concatenates into a header with +.
• Any method other than GET, HEAD, PUT, DELETE, OPTIONS, TRACE and QUERY gets an Idempotency-Key header set to the invocation’s idempotency key. It is applied after your headers, so it cannot be overridden. Every attempt of one call presents the same key.
• The response body is read up to 240 KiB, which is the 256 KiB result cap minus a 16 KiB framing margin for the status and the headers. A larger body comes back as a transport failure.

post_value

post_value(path, value, headers = None)

path is positional and the rest are keyword arguments; value is required. The action POSTs a JSON body of {"value": <value>, "channel_tokens": [...]} and returns the same four-tuple as call_api. Content-Type is always application/json and Idempotency-Key is always set; neither can be overridden through headers.

value accepts any AutoFlow value, including a channel nested at any depth. AutoFlow exchanges the channel tokens of the invocation for tokens that bind no principal and puts them in channel_tokens, so the recipient can later send a value back into that channel with SendToWorkflowChannel. That is how a workflow hands GitLab a place to answer.

reply = channel()
gather(post_value(
    path,
    value = {"question": "approve?", "reply": reply},
    headers = headers,
))
answer = gather(reply)

policy

Asks the governance engine whether an operation may proceed.

load("module:policy",
     "evaluate", "ALLOW", "DENY", "REQUIRE_APPROVAL", "UNDECIDABLE")

The four variables are the strings a verdict can take. evaluate takes keyword arguments only.

evaluate(trigger, resource, context = None)

trigger names the event the decision is about, by convention reverse-DNS. resource names what the decision is about. Both are required and non-empty. context is a string holding a JSON object, which the policies read as Rego’s input document; an absent one is an empty document.

The result is a dict with two keys, decision_id and verdict. A dict rather than a tuple, so later additions do not break existing workflow definitions.

def main(w):
    decision = gather(evaluate(
        trigger = "com.gitlab.cd.deployment_promoted",
        resource = "organizations/9/rollouts/7",
        context = json.encode({"environment": {"tier": "production"}}),
    ))
    if decision["verdict"] != ALLOW:
        fail("refused: %s" % decision["decision_id"])

event

Publishes a CloudEvents 1.0 event on the GitLab events platform.

load("module:event", "emit")

gather(emit(
    topic = "com.gitlab.cd.flow",
    type = "com.gitlab.cd.step_started",
    data = json.encode({"step": "migrate"}),
))

emit(topic, type, data = None) takes keyword arguments only and resolves to None. topic and type are 1 to 255 bytes; type is reverse-DNS by convention. data must be a plain or sensitive string, plain or sensitive bytes, or a proto message; anything else is an error, because a wire format is not guessed on the workflow’s behalf.

AutoFlow stamps the event itself: id is the invocation’s idempotency key, so retries of one logical emit carry one id, and source is urn:gitlab:autoflow:<workflow_key>, so provenance cannot be forged.

gitlab-function

Runs a CD Function inside AutoFlow. The module is developed in the argo-rollout repository and compiled into Relay from there.

load("module:gitlab-function", "run")

outputs = gather(run(
    function = "builtin://manifest.pin_images",
    inputs = {"manifest": manifest, "images": images},
))

run(function, inputs = None, timeout = None) takes keyword arguments only and returns the function’s outputs as a dict. function must use the builtin:// scheme; anything else, or an unknown name, is an error. timeout must be positive and defaults to five minutes.

Six functions are registered: gitlab_project.read_files, gitlab_project.commit, manifest.pin_images, manifest.set_canary_strategy, argo.sync and argo.promote. The two that reach GitLab read their token from GITLAB_TOKEN in AutoFlow’s own environment and fail up front when it is unset.

How modules reach workflows

Today every module a workflow definition can load is built into Relay: it is registered in the binary and served in-process, which is why the four above need no configuration beyond their own dependencies. New capabilities currently ship the same way, as Go modules compiled in, whether they live in this repository or in another one, as gitlab-function does.

The contract between a module and AutoFlow is a gRPC and protobuf protocol, ModuleAPI and ModuleTunnelAPI in pkg/autoflow/rpc/rpc.proto, so a module can be written in any language. A remote module connects as an agent and serves actions over a reverse tunnel.

The server side of this exists, but the credentials a third-party module would need to register are not available yet, so remote modules are not something you can deploy today.

The Go SDK in pkg/autoflow is a convenience implementation of the protocol, not the only way to write a module; a module built with it serves either path, built in or remote. Whatever the language, a module publishes this descriptor:

message ModuleDescriptor {
  google.protobuf.FileDescriptorSet descriptor_set = 1;
  repeated Var vars = 2;
  repeated Action actions = 3;
  repeated File files = 4;
}

An implementation must deduplicate on the invocation’s idempotency key and must answer a duplicate with the result the first execution produced: AutoFlow may re-deliver an invocation.

More detail

• Modules in the design doc: the descriptor, the value mapping, error semantics and module files.
• The pkg/autoflow SDK: protos and Go helpers for writing a module.
• Write a module: the protocol step by step and a worked example with the SDK.