Prefect is a workflow automation system. QHub integrates Prefect with a feature flag as follows (in the top level):

  enabled: true

You can also specify a particular image for Prefect with the image key as follows:

  enabled: true
  image: prefecthq/prefect:0.14.22-python3.8

There are a bunch of components in getting Prefect working for you, here is a brief description of them:

  1. Create a free Prefect cloud account here:

  2. Create a Service Account and an API key for the same and add this to the CI secrets as TF_VAR_prefect_token:

    • In GitHub: Set it in Secrets (

    • In GitLab: Set it as Variables (

  3. Create a project in the Prefect Cloud Dashboard. Alternatively from CLI:

prefect create project 'your-prefect-project-name'

The TF_VAR_prefect_token API key is set as PREFECT__CLOUD__AGENT__AUTH_TOKEN environment variable in the agent. It is used while deploying Prefect Agent so that it can connect to Prefect Cloud and query flows.

Prefect Cloud

Prefect Cloud is a fully hosted, production-ready backend for Prefect Core. Checkout prefect documentation to know more.

Prefect Agent

Prefect Agents is a lightweight processes for orchestrating flow runs. Agents run inside a user’s architecture, and are responsible for starting and monitoring flow runs.

During operation the agent process queries the Prefect API for any scheduled flow runs, and allocates resources for them on their respective deployment platforms.

When you enable prefect via qhub-config.yml prefect agent is deployed on the QHub’s kubernetes cluster, which querys the Prefect Cloud for flow runs.


Prefect agent can only orchestrate your flows, you need an actual flow to run via prefect agent. The API for the same can be found in the prefect documentation Here is a simple example from their official doc:

from prefect import task, Task, Flow
import random

def random_number():
    return random.randint(0, 100)

def plus_one(x):
    return x + 1

with Flow('My Functional Flow') as flow:
    r = random_number()
    y = plus_one(x=r)


The Prefect Storage interface encapsulates logic for storing flows. Each storage unit is able to store multiple flows (with the constraint of name uniqueness within a given unit).

The API documentation for the same can be found in the prefect documentation

Example: Creating, Building and Register Flow

Below is a complete example of

  • creating couple of workflows

  • Adding them to a storage (docker in this case)

  • Building and pushing that storage to a docker registry

  • Registering the flows to prefect cloud

import logging
import time

import prefect
from prefect import Flow, task
from import Docker

PREFECT_PROJECT = "your-prefect-project-name"
# Name of  your docker repository
DOCKER_REGISTRY = "quansight"

def hello_world():
    """Sample flow to make sure prefect works."""
    logger = prefect.context.get("logger")"Running Hello World!")

def init_and_register_flows():
    storage = Docker(
            "env_var": "value"
    prefect_flows = [

    add_flows_to_storage(storage, prefect_flows)
    register_flows(prefect_flows, build=False)"Everything done.!")

def add_flows_to_storage(storage, flows):"Add flows: {flows} to storage: {storage}")
    flow_storage_paths = [storage.add_flow(flow) for flow in flows]"flows path in storage: {flow_storage_paths}")
    return storage

def register_flows(flows, **kwargs):"Registering flows: {flows}")
    registered_flows = [
        flow.register(project_name=PREFECT_PROJECT, **kwargs) for flow in flows
    ]"Registered flows: {registered_flows}")
    return registered_flows

def setup_logging(level=logging.INFO):
    """Setup standard logging format."""
        format="%(asctime)s %(levelname)9s %(lineno)4s %(module)s: %(message)s"

def main():

if __name__ == "__main__":

Running your flows

Now that you have Prefect Agent running in QHub Kubernetes cluster, you can now run your flows from either of the two ways:

  • Triggering manually from the Prefect Cloud dashboard.

  • Running them on a schedule by adding a parameter to you flow. You can read more about it in the prefect docs.