Tips for Testing

Using a development branch

To use qhub from a development branch such as main set the environment variable QHUB_GH_BRANCH before running qhub commands:

export QHUB_GH_BRANCH=main

Then qhub init will create a qhub-config.yaml containing, for example, quansight/qhub-jupyterlab:main which is the Docker image built based on the Dockerfiles specified in the main branch of the QHub repo (see below for more info on how these are specified). There is a GitHub workflow that will build these images and push to Docker Hub whenever a change is made to the relevant files on GitHub.

In addition, qhub deploy can use QHUB_GH_BRANCH to create GitHub/GitLab workflows which install the development branch of QHub for their own deploy steps.

If you want to use the development version of QHub for your init and deploy but want your resulting deployment to be based on a full release version, don’t set the QHUB_GH_BRANCH environment variable. In that case, Docker tags and workflow pip install qhub commands will be based on the qhub version specified in the qhub/version.py file, but these tags and releases may not yet exist, perhaps if the version has been updated to include a beta/dev component which has not been released. So you may need to manually modify the qhub-config.yaml to ‘downgrade’ the tags to a full release version.

Kubernetes Version Check for Cloud Providers

When qhub init <cloud provider> is called, it checks that the --kubernetes-version provided is supported by the preferred cloud provider. This flag is optional and if not provided, the kubernetes_version is set to the most recent kubernetes version available. This is achieved by using the cloud provider’s SDK which thus requires their appropriate credentials to be set. To get around this, simply set the QHUB_K8S_VERSION environment variable like so:

export QHUB_K8S_VERSION=1.20

Modifying Docker Images

All QHub docker images are located in qhub/template/image. You can build any image locally. Additionally, on Pull Requests each Docker-build will be tested.

docker build -f Dockerfile.<filename> .

Testing the JupyterLab Docker Image

Often times you would like to modify the JupyterLab default docker image and run the resulting configuration.

docker run -p 8888:8888 -it <image-sha> jupyter lab --ip=0.0.0.0

Then open the localhost (127.0.0.1) link that’s in the terminal

[I 2021-04-05 17:37:17.345 ServerApp] Jupyter Server 1.5.1 is running at:
...
[I 2021-04-05 17:37:17.346 ServerApp]  or http://127.0.0.1:8888/lab?token=8dbb7ff1dcabc5fab860996b6622ac24dc71d1efc34fcbed
...
[I 2021-04-05 17:37:17.346 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

Linting Dockerfiles

To lint Dockerfiles, developers use a tool called Hadolint. Hadolint is a Dockerfile linter that allows to discover issues with the Dockerfiles and recommends best practices to be followed. QHub CI automates Hadolint code reviews on every commit and pull request, reporting code style and error prone issues.

To run Hadolint locally you can either install it locally or use a container image. Instructions are available on the install documentation for HadoLint. The .hadolint.yml on the root directory defines the ignored rules. To run Hadolint on Dockerfiles run:

hadolint ./qhub/template/\{\{\ cookiecutter.repo_directory\ \}\}/image/Dockerfile.conda-store
hadolint ./qhub/template/\{\{\ cookiecutter.repo_directory\ \}\}/image/Dockerfile.dask-gateway
hadolint ./qhub/template/\{\{\ cookiecutter.repo_directory\ \}\}/image/Dockerfile.dask-worker
hadolint ./qhub/template/\{\{\ cookiecutter.repo_directory\ \}\}/image/Dockerfile.jupyterhub
hadolint ./qhub/template/\{\{\ cookiecutter.repo_directory\ \}\}/image/Dockerfile.jupyterlab

Hadolint will report error, warning, info and style while linting Dockerfiles. In case of an error, the CI fails.

Debug Kubernetes clusters

To debug Kubernetes clusters, checkout k9s, a terminal-based UI that aims to simplify navigation, observation, and management of applications in Kubernetes. k9s continuously monitors Kubernetes clusters for changes and provides shortcut commands to interact with the observed resources becoming a fast way to review and resolve day-to-day issues in deployed clusters.

Installation can be done on a macOS, in Windows, and Linux and instructions are located here. For more details on usage, review the Troubleshooting documentation.

Cypress Tests

Cypress automates testing within a web browser environment. It’s integrated into the GitHub Actions tests.yaml workflows in this repo, and you can also run it locally. To do so:

cd tests_e2e
npm install

export CYPRESS_BASE_URL=http://127.0.0.1:8000/
export QHUB_CONFIG_PATH=/Users/dan/qhub/data-mk/qhub-config.yaml
export CYPRESS_EXAMPLE_USER_PASSWORD=<password>

npm run cypress:open

The Base URL can point anywhere that should be accessible - it can be the URL of a QHub cloud deployment. The QHub Config Path should point to the associated yaml file for that site. Most importantly, the tests will inspect the yaml file to understand what tests are relevant. To start with, it checks security.authentication.type to determine what should be available on the login page, and how to test it. If the login type is ‘password’ then it uses the value in CYPRESS_EXAMPLE_USER_PASSWORD as the password (default username is example-user but this can be changed by setting CYPRESS_EXAMPLE_USER_NAME).

The final command, in the preceding code-snippet, opens the Cypress UI where you can run the tests manually and see the actions in the browser.

Note that tests are heavily state dependent, so any changes or use of the deployed QHub could affect the results.

Deployment and integration tests

Deployment and Integration testing makes it easier to test various features of deployed QHub on Minikube such as Dask Gateway, external integrations, state of the kubernetes cluster via simple Python code. You can run the integration and deployment tests via the following command:

pytest tests_deployment/ -v

Cloud Testing

Cloud testing on AWS, GCP, Azure, and Digital Ocean can be significantly more complicated and time consuming. But it’s the only way to truly test the cloud deployments, including infrastructure, of course. To test on cloud Kubernetes, just deploy QHub in the normal way on those clouds, but using the linked pip install of the QHub package.

Even with the dev install of the qhub package, you may find that the deployed cluster doesn’t actually reflect any development changes, for example to the Docker images for JupyterHub or JupyterLab. That will be because your qhub-config.yaml references fully released versions. See Using a development branch above for how to encourage the Docker images to be specified based on the latest development code.

You should always prefer the local testing when possible as it will be easier to debug, may be quicker to deploy, and is likely to be less expensive.

Docs testing

QHub uses Vale, a syntax-aware linter to lint documentation and recommend regions that needs improvement. Vale works with the Google developer documentation style guide, with specific configurations that are specific to Quansight. To test the documentation on the local machine, follow these steps:

  • Install Vale command-line tool.

  • Run Vale on the entire documentation source or a specific documentation piece.

    # Run Vale on the entire documentation source
$ vale docs/

# Run Vale on a specific file
$ vale README.md

  • Utilize the errors, warnings and the suggestions to make appropriate changes to the documentation.

  • In case of false positives, make appropriate changes to the Vale configurations hosted in the tests/vale/styles directory.

Vale runs on the GitHub Actions CI to automatically validate the documentation language. By default, Vale only checks the modified documentation to ensure that Vale doesn’t generate a lot of noise over the Pull Requests.

previous

Release Process for QHub

next

Minikube