Troubleshooting and further setup#
This guide aims to provide useful information to developers in the detection and correction of issues within QHub.
General Troubleshooting#
To minimize the occurrence of errors on your QHub application, please follow the best practices described on the Installation, Setup and Usage sections.
Solutions for common problems#
Getting Kubernetes Context#
Digital Ocean#
To get the kubernetes context to interact with a Digital Ocean cluster use the following instructions.
Create Digital Ocean API Token likely already done
doctl access via api token
doctl auth init
doctl kubernetes cluster kubeconfig save "<project-name>-<namespace>"
After completing these steps. kubectl
should be able to access the cluster.
Google Cloud Platform#
To get the kubernetes context to interact with a GCP use the following instructions.
Login to GCP via
gcloud init
gcloud container clusters get-credentials <project-name>-<namespace> --region <region>
After completing these steps. kubectl
should be able to access the cluster.
Amazon Web Services#
To get the kubernetes context to interact with a AWS use the following instructions.
Download the aws command line
Create AWS Access Key and Secret Key likely already done
aws eks --region <region-code> update-kubeconfig --name <project-name>-<namespace>
After completing these steps. kubectl
should be able to access the cluster.
Debug your Kubernetes cluster#
k9s
is a terminal-based UI to manage Kubernetes clusters that aims to simplify navigating,
observing, and managing your applications in k8s
. k9s
continuously monitors Kubernetes clusters for changes and
provides shortcut commands to interact with the observed resources becoming a fast way to review and resolve day-to-day
issues in Kubernetes. It’s definitely a huge improvement to the general workflow, and a best-to-have tool for debugging
your Kubernetes cluster sessions.
Installation can be done on macOS, Windows, and Linux. Instructions for each operating system can be found here. Complete the installation to follow along.
By default, k9s
starts with the standard directory that’s set as the context (in this case Minikube). To view all the
current process press 0
:
NOTE: In some circumstances you will be confronted with the need to inspect any services launched by your cluster at your ‘localhost’. For instance, if your cluster has problem with the network traffic tunnel configuration, it may limit or block the user’s access to destination resources over the connection.
k9s
port-forward option shift + f allows you to access and interact with internal Kubernetes
cluster processes from your localhost you can then use this method to investigate issues and adjust your services
locally without the need to expose them beforehand.
Further Setup#
Theming#
JupyterHub Theme#
The QHub theme was originally based off the
work of the pangeo team and is now located in
github.com/Quansight/qhub-jupyterhub-theme. For simple
modifications to the jupyterhub theme we suggest only editing infrastructure/jupyterhub.yaml
and the value
c.JupyterHub.template_vars
. For most use cases this should provide enough flexibility.
hub:
extraConfig:
customPodHook: |
c.JupyterHub.template_paths = ['/usr/local/share/jupyterhub/custom_templates/']
c.JupyterHub.template_vars = {
'pangeo_hub_title': 'QHub - $do-jupyterhub',
'pangeo_hub_subtitle': 'Autoscaling Compute Environment on Digital Ocean',
'pangeo_welcome': """Welcome to $. It is maintained by the <a href="http://quansight.com">Quansight staff</a>. The hub's configuration is stored in the github repository based on <a href="https://github.com/Quansight/qhub-kubernetes/">https://github.com/Quansight/qhub-kubernetes/</a>. To provide feedback and report any technical problems, please use the <a href="https://github.com/Quansight/qhub-kubernetes//issues">github issue tracker</a>."""
}
For more serious modifications to the jupyterhub theme you will need to fork
Quansight/qhub-jupyterhub-theme and edit the jupyterhub Dockerfile
located at image/Dockerfile.jupyterhub
. Modify the THEME_OWNER
, THEME_REPO
, and THEME_REV
. This should change
the Dockerfile to use your new theme. The
Quansight/qhub-jupyterhub-theme has detailed documentation.
JupyterLab Theme#
Setting the JupyterLab theme is done via extensions. Edit the image/postBuild
script to include the jupyterlab
extension in the build. Within the image
directory run the following to build JupyterLab.
docker build -f Docker.jupyterlab -t Quansight/qhub-jupyterlab:latest .
Finally, you can test the resulting image via the following docker command and open your web browser to
localhost:8000
.
docker run -p 8000:8000 -it Quansight/qhub-jupyterlab:latest jupyter lab --port 8000 --ip 0.0.0.0
Using a Private AWS ECR Container Registry#
By default, images such as the default JupyterLab image specified as quansight/qhub-jupyterhub:v2022.11.1
will
be pulled from Docker Hub.
To specify a private AWS ECR (and this technique should work regardless of which cloud your QHub is deployed to), first
provide details of the ECR and AWS access keys in qhub-config.yaml
:
external_container_reg:
enabled: true
access_key_id: <AWS access key id>
secret_access_key: <AWS secret key>
extcr_account: 12345678
extcr_region: us-west-1
This will mean you can specify private Docker images such as
12345678.dkr.ecr.us-west-1.amazonaws.com/quansight/qhub-jupyterlab:mytag
in your qhub-config.yaml
file. The AWS key
and secret provided must have relevant ecr IAMS permissions to authenticate and read from the ECR container registry.