Troubleshooting and further setup#

This guide aims to provide useful information to developers in the detection and correction of issues within QHub.

General Troubleshooting#

To minimize the occurrence of errors on your QHub application, please follow the best practices described on the Installation, Setup and Usage sections.

Solutions for common problems#

Getting Kubernetes Context#

Digital Ocean#

To get the kubernetes context to interact with a Digital Ocean cluster use the following instructions.

Download Digital Ocean command line utility
Create Digital Ocean API Token likely already done
doctl access via api token doctl auth init
doctl kubernetes cluster kubeconfig save "<project-name>-<namespace>"

After completing these steps. kubectl should be able to access the cluster.

Google Cloud Platform#

To get the kubernetes context to interact with a GCP use the following instructions.

Download the GCP SDK
Login to GCP via gcloud init
gcloud container clusters get-credentials <project-name>-<namespace> --region <region>

After completing these steps. kubectl should be able to access the cluster.

Amazon Web Services#

To get the kubernetes context to interact with a AWS use the following instructions.

Download the aws command line
Create AWS Access Key and Secret Key likely already done
aws eks --region <region-code> update-kubeconfig --name <project-name>-<namespace>

After completing these steps. kubectl should be able to access the cluster.

Debug your Kubernetes cluster#

k9s is a terminal-based UI to manage Kubernetes clusters that aims to simplify navigating, observing, and managing your applications in k8s. k9s continuously monitors Kubernetes clusters for changes and provides shortcut commands to interact with the observed resources becoming a fast way to review and resolve day-to-day issues in Kubernetes. It’s definitely a huge improvement to the general workflow, and a best-to-have tool for debugging your Kubernetes cluster sessions.

Installation can be done on macOS, Windows, and Linux. Instructions for each operating system can be found here. Complete the installation to follow along.

By default, k9s starts with the standard directory that’s set as the context (in this case Minikube). To view all the current process press 0:

Image of the terminal UI

NOTE: In some circumstances you will be confronted with the need to inspect any services launched by your cluster at your ‘localhost’. For instance, if your cluster has problem with the network traffic tunnel configuration, it may limit or block the user’s access to destination resources over the connection.

k9s port-forward option shift + f allows you to access and interact with internal Kubernetes cluster processes from your localhost you can then use this method to investigate issues and adjust your services locally without the need to expose them beforehand.

Further Setup#

Theming#

JupyterHub Theme#

The QHub theme was originally based off the work of the pangeo team and is now located in github.com/Quansight/qhub-jupyterhub-theme. For simple modifications to the jupyterhub theme we suggest only editing infrastructure/jupyterhub.yaml and the value c.JupyterHub.template_vars. For most use cases this should provide enough flexibility.

hub:
  extraConfig:
    customPodHook: |
      c.JupyterHub.template_paths = ['/usr/local/share/jupyterhub/custom_templates/']
      c.JupyterHub.template_vars = {
          'pangeo_hub_title': 'QHub - $do-jupyterhub',
          'pangeo_hub_subtitle': 'Autoscaling Compute Environment on Digital Ocean',
          'pangeo_welcome': """Welcome to $. It is maintained by the <a href="http://quansight.com">Quansight staff</a>. The hub's configuration is stored in the github repository based on <a href="https://github.com/Quansight/qhub-kubernetes/">https://github.com/Quansight/qhub-kubernetes/</a>. To provide feedback and report any technical problems, please use the <a href="https://github.com/Quansight/qhub-kubernetes//issues">github issue tracker</a>."""
      }

For more serious modifications to the jupyterhub theme you will need to fork Quansight/qhub-jupyterhub-theme and edit the jupyterhub Dockerfile located at image/Dockerfile.jupyterhub. Modify the THEME_OWNER, THEME_REPO, and THEME_REV. This should change the Dockerfile to use your new theme. The Quansight/qhub-jupyterhub-theme has detailed documentation.

JupyterLab Theme#

Setting the JupyterLab theme is done via extensions. Edit the image/postBuild script to include the jupyterlab extension in the build. Within the image directory run the following to build JupyterLab.

docker build -f Docker.jupyterlab -t Quansight/qhub-jupyterlab:latest .

Finally, you can test the resulting image via the following docker command and open your web browser to localhost:8000.

docker run -p 8000:8000 -it Quansight/qhub-jupyterlab:latest jupyter lab --port 8000 --ip 0.0.0.0

Using a Private AWS ECR Container Registry#

By default, images such as the default JupyterLab image specified as quansight/qhub-jupyterhub:v2022.11.1 will be pulled from Docker Hub.

To specify a private AWS ECR (and this technique should work regardless of which cloud your QHub is deployed to), first provide details of the ECR and AWS access keys in qhub-config.yaml:

external_container_reg:
  enabled: true
  access_key_id: <AWS access key id>
  secret_access_key: <AWS secret key>
  extcr_account: 12345678
  extcr_region: us-west-1

This will mean you can specify private Docker images such as 12345678.dkr.ecr.us-west-1.amazonaws.com/quansight/qhub-jupyterlab:mytag in your qhub-config.yaml file. The AWS key and secret provided must have relevant ecr IAMS permissions to authenticate and read from the ECR container registry.

Cloud cost and capabilities

Upgrade