Day-to-day Maintenance#

All modifications to the infrastructure should be done with GitHub Pull-Requests.

Common Modifications#

Modifying docker images: jupyterlab, jupyterhub, dask-workers#

The docker images used for dask-worker and jupyterlab user environments are pulled from a docker container registry. The images are built based on the images specified in the image folder. There are three images that are currently built

jupyterlab :: modification of jupyterlab instances for each user
dask-worker :: modification of dask workers and dask scheduler
jupyterhub :: the jupyterhub server (allows for customization of hub UI)
conda-store :: Environment management tool for QHub

Each docker image is customized with its respective directory (for example image/Dockerfile.jupyterlab -> image/jupyterlab/*. For jupyterlab the environment is located at image/jupyterlab/environment.yaml. Thus to add a package to the environment simply submit a pull request with the new package.

At this current point in time once a user submits a pull request to create the given docker image and the PR is accepted with images built, a PR must follow that adds the image to the qhub deployment. This can be done by modifying infrastructure/variables.tf and the configuration file.

Adding additional worker nodegroups#

Adding additional nodegroups can be done by editing the configuration file. While a general, user, and worker nodegroup are required you may create any additional node group. Take for example the Digital Ocean configuration.

digital_ocean:
  region: nyc3
  kubernetes_version: "1.18.8-do.0"
  node_groups:
    general:
      instance: "s-2vcpu-4gb"
      min_nodes: 1
      max_nodes: 1
    user:
      instance: "s-2vcpu-4gb"
      min_nodes: 1
      max_nodes: 5
    worker:
      instance: "g-2vcpu-8gb"
      min_nodes: 1
      max_nodes: 5

To add a node group for a node group called worker-high-mem simply add to the configuration. The same applies for AWS, GCP, and DO.

digital_ocean:
  region: nyc3
  kubernetes_version: "1.18.8-do.0"
  node_groups:
    general:
      instance: "s-2vcpu-4gb"
      min_nodes: 1
      max_nodes: 1
    user:
      instance: "g-2vcpu-8gb"
      min_nodes: 1
      max_nodes: 4
    worker:
      instance: "g-2vcpu-8gb"
      min_nodes: 1
      max_nodes: 4
    worker-high-memory:
      instance: "m6-2vcpu-16gb"
      min_nodes: 1
      max_nodes: 4

Setting specific JupyterLab profile to run on a nodegroup#

Sometimes we would like a profile to execute on nodes that are not in the normal nodegroup. In the example above we created a high memory node group. To make the jupyterlab profile small worker use the high memory nodegroup do the following.

profiles:
  jupyterlab:
    - display_name: Small Instance
      description: Stable environment with 1 cpu / 1 GB ram
      access: yaml
      groups:
        - admin
      kubespawner_override:
        cpu_limit: 1
        cpu_guarantee: 1
        mem_limit: 1G
        mem_guarantee: 1G
        image: "quansight/qhub-jupyterlab:v2022.11.1"

We add the node_selector attribute. Note that for AWS, GCP, and DO they have different keys for the nodegroup name.

AWS :: eks.amazonaws.com/nodegroup
GCP :: cloud.google.com/gke-nodepool
DO :: doks.digitalocean.com/node-pool

Since we are using digital ocean in this example we then need to set the following.

profiles:
  jupyterlab:
    - display_name: Small Instance
      description: Stable environment with 1 cpu / 1 GB ram
      access: yaml
      groups:
        - admin
      kubespawner_override:
        cpu_limit: 1
        cpu_guarantee: 1
        mem_limit: 1G
        mem_guarantee: 1G
        image: "quansight/qhub-jupyterlab:v2022.11.1"
        node_selector:
          "doks.digitalocean.com/node-pool": worker-high-memory

Setting specific dask workers to run on a nodegroup#

Suppose we want a specific dask worker profile to run on a specific node group. Here we demonstrate annotating the DO example configuration.

profiles:
  dask_worker:
    "Small Worker":
      worker_cores_limit: 1
      worker_cores: 1
      worker_memory_limit: 1G
      worker_memory: 1G
      image: "quansight/qhub-dask-worker:v2022.11.1"

Dask-gateway takes additional configuration for the scheduler pods and workers. Remember similar to assigning node groups to specific jupyterlab instances we must get the key for the node pool.

AWS :: eks.amazonaws.com/nodegroup
GCP :: cloud.google.com/gke-nodepool
DO :: doks.digitalocean.com/node-pool

Since we are using digital ocean in this example we then need to set the following.

profiles:
  dask_worker:
    "Small Worker":
      worker_cores_limit: 1
      worker_cores: 1
      worker_memory_limit: 1G
      worker_memory: 1G
      image: "quansight/qhub-dask-worker:v2022.11.1"
      scheduler_extra_pod_config:
        nodeSelector:
          "doks.digitalocean.com/node-pool": worker-high-memory
      worker_extra_pod_config:
        nodeSelector:
          "doks.digitalocean.com/node-pool": worker-high-memory

General Modifications#

The infrastructure was designed with the goal in mind that each module is orthogonal.

Preemptible and Spot instances on QHub

Monitoring