This reference for version: 1.0.1

Boot is a solution to launch lightweight test and development environments in Kubernetes.

1. Getting Started

1.1. Quick Start Guide

This section shows how to install Boot limited to 1 CPU. Detailed information on installing a license key allowing to activate more CPUs is shown in Installing License section.

1.1.1. Installing to Kubernetes

Prerequisites

  1. Running Kubernetes cluster.

  2. kubectl client installed and pointing to the cluster

  3. A pair of SSH keys which is often stored in ~/.ssh/ directory as id_rsa (private key) and id_rsa.pub (public key) files. If you don’t have such files - use ssh-keygen command to generate them. To install Boot you will only need public key file which usually looks like this:

    $ cat ~/.ssh/id_rsa.pub
ssh-rsa AAAAB3N... some-comment

  4. If you are starting Boot in Kubernetes cluster deployed on workstation with minikube tool - use the following commands to start Kubernetes:

    Minikube startup under Linux
    $ minikube start --driver kvm2
    Minikube startup under MacOS
    $ minikube start --driver=hyperkit
    Minikube startup under Windows
    $ DISM /Online /Enable-Feature /All /FeatureName:Microsoft-Hyper-V # Enable Hyper-V
$ minikube start --driver=hyperv
Install Boot with Helm
Helm chart is the recommended Boot installation way. Steps below require Helm 3 and will not work with older releases.

We deliver already packed and published Helm charts, so installing Boot with Helm is straightforward:

  1. Prepare a values.yaml file with Helm chart settings:

    boot:
  jumphost:
    authorizedKeys:
      - ssh-rsa AAAAB3N... some-comment # Insert at least one public key contents here

  2. Add Aerokube charts repository:

    $ helm repo add aerokube https://charts.aerokube.com/
$ helm repo update

  3. To list available Boot versions type:

    $ helm search repo aerokube --versions

  4. Create a namespace:

    $ kubectl create namespace boot

  5. To install or upgrade Boot type:

    $ helm upgrade --install -f values.yaml -n boot boot aerokube/boot

  6. Boot chart has a lot of different configuration parameters that can be listed as follows:

    $ helm show values aerokube/boot

    To change one of these parameters - add an override to values.yaml file and reapply Helm chart again.

  7. Detect Boot load balancer IP address. Usually this address can be obtained from Boot service:

    Get Boot service IP address
    $ kubectl get svc boot -n boot
NAME   TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)    AGE
boot   LoadBalancer   10.107.117.163   192.168.64.5   2222/TCP   15d

    IP address is indicated in EXTERNAL-IP column. On Minikube you need to explicitly add IP address to Boot service with minikube ip command output:

    Patch Boot service with minikube IP address
    $ kubectl patch svc boot -n boot --patch "{\"spec\":{\"externalIPs\":[\"$(minikube ip)\"]}}"

    On Windows - you may need to insert minikube ip output manually, because $() expression expansion may not work.

  8. Make sure there is a domain name pointing to IP address. On Minikube - simply add an entry to hosts file:

    $ sudo echo "$(minikube ip) boot.aerokube.local" >> /etc/hosts

    On Windows you may need to update hosts file manually.

  9. Configure your SSH client to use Boot. To do that add the following lines to ~/.ssh/config file:

    $ cat ~/.ssh/config
# ... some lines here
Host bootjump
    IdentityFile ~/.ssh/id_rsa
    HostName boot.aerokube.local
    Port 2222
Host *.vm.boot.svc.cluster.local
    IdentityFile ~/.ssh/id_rsa
    ProxyJump jump@bootjump

  10. Create your first virtual machine. To do that create a YAML file with virtual machine definition:

    $ cat ~/vm.yaml
apiVersion: boot.aerokube.com/v1
kind: VirtualMachine
metadata:
  name: my-vm
  namespace: boot
spec:
  os: ubuntu
  authorizedKeys:
    - ssh-rsa AAAAB3N... some-comment # The same public key contents here too

    Then apply this file as follows:

    $ kubectl create -f vm.yaml

  11. Check SSH connectivity to virtual machine:

    $ ssh root@my-vm.vm.boot.svc.cluster.local
# Some welcome text goes here...
root@my-vm:~# # You now have a root access to created virtual machine

1.2. Architecture

1.2.1. Boot Components

Boot Components

boot components

Boot cluster consists of several components:

  1. One or more Boot application instances. Their main purpose is to start/stop virtual machines and to control access to created virtual machines via SSH. Boot is usually accessible on some SSH port, e.g. 2222.

  2. One or more Boot UI application instances (currently under development). Boot UI collects information from Boot and visualizes it. It is usually available on HTTP port 8080.

  3. Running virtual machine pods and respective virtual machine custom resources.

1.2.2. Boot Pod Contents

Every Boot pod contains several containers with different purpose.

Table 1. Boot Pod Containers
Name Purpose

boot

Starts and stops virtual machines

jumphost

Regular SSH server with secure configuration working as SSH jump host

reloader

During Boot updates waits for configured period of time for SSH connections to virtual machines to close or explicitly closes them after that time

1.2.3. Multiple Namespaces Support

Boot can launch virtual machines in any number of Kubernetes namespaces. By default, Boot itself and all launched virtual machines are running in the same Kubernetes namespace. This is suitable if only one team is using Boot or you don’t need to limit virtual machines consumption for different Boot users.

Boot with Multiple Namespaces Configured

multiple-namespaces-mode

It is easy to configure Boot to run in one namespace and to launch virtual machines in separate namespaces. Multiple namespaces are mainly needed when you want to control computing resources, virtual machine operating system versions or network access rules (network policies) available for every team. For example, user 1 virtual machines will be running in team 1 namespace and user 2 virtual machines will be running in team 2 namespace. How to configure multiple namespaces is described here.

1.3. Required Permissions

Boot requires a limited set of permissions and should work with default Kubernetes settings. Boot can launch virtual machines in any number of Kubernetes namespaces. All required roles are automatically created with Helm chart. The following table shows Boot permissions in the namespace where it is running:

Table 2. Required Permissions for the Namespace where Boot runs

Permission

Purpose

To get, watch and list and patch Boot custom resources in boot.aerokube.com API group

These custom resources store Boot configuration and are cluster-wide, so a ClusterRole is needed for these resources.

For every user namespace Boot needs the following permissions:

Table 3. Required Boot Permissions For User Namespace

Permission

Purpose

To get, watch, list, create, delete, update and patch pods

Used to manipulate pods with virtual machines

To get, watch, list, create, delete, update and patch config maps

Used to pass users and groups to virtual machines pods

2. Main Features

2.1. Virtual Machines

The main Boot feature is creating lightweight virtual machines. Such virtual machines can be used for:

  • Functional software testing, i.e. testing that implemented software features work as expected.

  • Penetration testing, i.e. testing that given software distribution does not contain security vulnerabilities.

  • Performance testing, i.e. testing how software performs in terms of responsiveness and stability under a particular workload.

  • Software development. This allows you to have more computing resources (CPUs, memory, disk) than a regular developer workstation allows.

Like traditional virtual machines, Boot virtual machines are accessed using SSH. That means that you can use any standard tools working on top of SSH to update virtual machine state:

2.1.1. Listing Virtual Machines

Boot virtual machines are native Kubernetes objects, so listing them can be done with standard kubectl commands:

Listing virtual machines
$ kubectl get vm -n boot
NAME    FQDN                              IP           OS            CPU         MEM     STATUS    UPTIME
my-vm   my-vm.vm.boot.svc.cluster.local   172.17.0.4   alpine:3.18   250m/128m   512Mi   Running   4s

You can see virtual machine name, fully qualified domain name and IP address to access it, used operating system image, CPU and memory limits and requests (limits/requests or simply limits if they have the same value), status and uptime.

2.1.2. Creating Virtual Machines

Option 1. Using kubectl and YAML manifest

Creating a virtual machine is as easy as creating any other Kubernetes resource.

  1. Create a file with virtual machine YAML definition:

    $ cat vm.yaml
apiVersion: boot.aerokube.com/v1
kind: VirtualMachine
metadata:
  name: my-vm
  namespace: boot
spec:
  os: ubuntu
  authorizedKeys: # Use one or more SSH public keys here
    - ssh-rsa AAAAB3N... some-key
    - ssh-rsa AAAAB3N... another-key

  2. Create virtual machine using YAML definition:

    $ kubectl create -f ~/vm.yaml
virtualmachine.boot.aerokube.com/my-vm created

Now you can list virtual machines and obtain fully qualified domain name to access this virtual machine. How to access virtual machines is described in Accessing Virtual Machines section.

Option 2. Using Boot plugin for kubectl

The second possible way of creating Boot virtual machines is using Boot plugin for kubectl.

The simplest way of installing Boot plugin is using Krew package manager:

Installing Boot plugin with Krew
$ kubectl krew index add aerokube https://github.com/aerokube/krew-index.git
$ kubectl krew install aerokube/vm

Another possibility is to download plugin distribution archive and unpack in to a directory in PATH:

Installing Boot plugin manually on Mac
$ curl -o /usr/local/bin/kubectl-vm https://github.com/aerokube/boot/releases/latest/download/kubectl-vm_darwin_amd64 # x86 version
$ curl -o /usr/local/bin/kubectl-vm https://github.com/aerokube/boot/releases/latest/download/kubectl-vm_darwin_arm64 # ARM version
$ chmod +x /usr/local/bin/kubectl-vm
Installing Boot plugin manually on Linux
$ curl -o /usr/local/bin/kubectl-vm https://github.com/aerokube/boot/releases/latest/download/kubectl-vm_linux_amd64
$ chmod +x /usr/local/bin/kubectl-vm
Installing Boot plugin manually on Windows
> mkdir %USERPROFILE%\bin # Add %USERPROFILE%\bin to PATH
> curl -o %USERPROFILE%\bin\kubectl-vm.exe https://github.com/aerokube/boot/releases/latest/download/kubectl-vm_windows_amd64.exe

To check that plugin was installed successfully:

$ kubectl vm --help

To create a virtual machine with plugin:

Creating a virtual machine with plugin
$ kubectl vm create my-vm -os ubuntu -n boot # Using latest operating system version
$ kubectl vm create my-vm -os ubuntu:22.04 -n boot # Using exact operating system version

By default, Boot plugin will use your public key from ~/.ssh/id_rsa.pub, but you can explicitly provide a list of authorized keys:

Providing explicit authorized keys file to plugin
$ echo "ssh-rsa AAAAB3N... some-key" >> authorized_keys # Create a list of SSH authorized keys
$ kubectl vm create my-vm -os ubuntu -auth-keys authorized_keys -n boot # Using latest operating system version

2.1.3. Deleting Virtual Machines

To delete a virtual machine use standard kubectl command:

Delete a virtual machine with kubectl
$ kubectl delete vm my-vm -n boot

To delete a virtual machine with kubectl plugin:

Delete a virtual machine with kubectl plugin
$ kubectl vm delete my-vm -n boot

2.1.4. Controlling Access

Boot supports two main SSH authentication technologies: authorized keys and user certificates. How they differ and work is described in Accessing Virtual Machines section. To add authorized public keys of allowed users - use authorizedKeys field in virtual machine specification:

Adding authorized keys
apiVersion: boot.aerokube.com/v1
kind: VirtualMachine
metadata:
  name: my-vm
  namespace: boot
spec:
  os: ubuntu
  authorizedKeys: # A list of SSH public keys allowed to access this virtual machine
    - ssh-rsa AAAAB3N... some-key
    - ssh-rsa AAAAB3N... another-key

To allow users to access virtual machine with SSH certificates - you have to add certification authorities used to sign these user certificates to virtual machine trustedUserCAKeys field:

Adding trusted certification authorities
apiVersion: boot.aerokube.com/v1
kind: VirtualMachine
metadata:
  name: my-vm
  namespace: boot
spec:
  os: ubuntu
  trustedUserCAKeys: # A list of trusted user certification authorities
    - ssh-ed25519 AAAAC3N... certification-authority-1
    - ssh-ed25519 AAAAC3N... certification-authority-2

How authorized keys and certificates differ is explained in detail in this section.

2.1.5. Adjusting Computing Resources

Boot is using exactly the same computing resources configuration YAML format as Kubernetes does.

By default, virtual machine is consuming computing resources configured in #operating-systems-computing-resources[operating system settings]. You can easily override this behavior by adding resources explicitly:

Setting computing resources explicitly
apiVersion: boot.aerokube.com/v1
kind: VirtualMachine
metadata:
  name: my-vm
  namespace: boot
spec:
  os: ubuntu
  resources: # Explicit computing resources
    limits:
      cpu: 1000m
      memory: 512Mi
    requests:
      cpu: 1000m
      memory: 512Mi

The same settings can be provided when creating virtual machine with kubectl plugin:

Providing explicit computing resource to a new virtual machine with a plugin
$ kubectl vm create my-vm -os ubuntu -cpu 1.0 -mem 512Mi -n boot
$ kubectl vm create my-vm -os ubuntu -cpu '1.0/0.5' -mem '512Mi/256Mi' -n boot # Different requests and limits

2.1.6. Adding Volumes

Boot is using exactly the same volumes configuration YAML format as Kubernetes does.

To share file between virtual machines or saving their state - use volumes. You can add a volume of any type supported by your Kubernetes cluster. To add a volume:

Setting computing resources explicitly
apiVersion: boot.aerokube.com/v1
kind: VirtualMachine
metadata:
  name: my-vm
  namespace: boot
spec:
  os: ubuntu
  volumes: # We declare a volume
  - name: my-volume
    emptyDir: {}
  volumeMounts: # We mount it to a path inside virtual machine
  - name: my-volume
    mountPath: /some/path

2.1.7. Adding Host Aliases

In some situations you may need to update /etc/hosts file inside virtual machine. To achieve this during virtual machine startup:

Updating /etc/hosts
apiVersion: boot.aerokube.com/v1
kind: VirtualMachine
metadata:
  name: my-vm
  namespace: boot
spec:
  os: ubuntu
  hostAliases:
    - ip: "127.0.0.1"
      hostnames:
      - "foo.local"
      - "bar.local"
    - ip: "10.1.2.3"
      hostnames:
      - "foo.remote"
      - "bar.remote"

2.2. Accessing Virtual Machines

We already noted in virtual machines section that Boot virtual machines are accessed using SSH. This section describes different options allowing to configure secure access to virtual machines.

2.2.1. Overview

As you probably know SSH consists of two components: an SSH server (also known as sshd) and an SSH client (usually command is called just ssh). SSH server in the majority of cases is listening on standard TCP port 22. Boot is running in Kubernetes and thus every network port that you wish to access from the outside, should be explicitly configured as Kubernetes service. Our goal is to be able to run an unlimited number of virtual machines, so doing this for every machine is too much work. Also, we want to be able to globally protect all virtual machines from unauthorized access. SSH provides a standard way of solving this task called jump host. Jump host is an intermediary host that SSH client should successfully connect in order to access target host.

Jump Host

jump host

For security purposes Boot is by default using non-standard port 2222 for jump host. This is because in Unix family operating systems port numbers up to 1024 require root privileges for the process, and it is recommended to use non-privileged users when possible. When you install Boot with default settings jump host is configured automatically and should work out of the box. If you wish to have full control over security, existing jump host configuration options are described here.

2.2.2. Authentication Technologies

SSH supports two main authentication technologies: keys (simpler and more popular) and certificates (more complicated but more powerful). Both technologies rely on public key cryptography, which is itself based on mathematical problems that are easy to do in one direction and very hard to do in the reverse direction for modern computers. For example, RSA system is using computational complexity of finding factors of large prime numbers. Similarly, ECDSA and EdDSA systems are using computational complexity of finding discrete logarithm of an elliptic curve over finite fields.

User and Host Keys

In public key cryptography every side of communication (user and remote host) has a key. A key belonging to a user is called user key. This key is used to access remote hosts with SSH instead of providing a password. Similarly, a key belonging to a remote host (e.g. a Boot virtual machine) is called host key. The main goal of this key from the user point of view is verifying authenticity of the host, i.e. making sure you are accessing the host you expected to access.

Using private and public keys for encrypted data exchange

keypair encryption

Both user and host keys are divided into two parts: public key part (or just public key) and private key part (or just private key). Because every key contains two parts, it is often called a keypair. Public key part is storing exact parameters of the mathematical problem used by exact public cryptography algorithm, e.g. large integers in RSA cryptography. Private key part is storing solution of the same mathematical problem, e.g. factors of the same large integer. Because of that a public key part can only be used to encrypt some message that an owner of the private key part will be able to decrypt.

Using private and public keys for digital signatures

keypair signature

A private key part in addition to decrypting messages can be also used to sign any piece of information (called message). Signing (or creating a digital signature) means generating a set of bytes allowing to verify that a concrete message was created by the owner of the private key and that it was not modified since that time. Generated signature is sent simultaneously with the message and every recipient having a sender public key is able to verify that signature is valid. Because of these properties, a public key part can be freely sent over insecure communication channels. Contrarily, a private key part should be accessible to its owner only.

Going back to SSH, let’s take a look at a typical usage of the private and public keys. A pair of user private and public keys can be generated with one command:

Generating a key pair
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/myuser/.ssh/id_rsa): my_key
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in my_key
Your public key has been saved in my_key.pub
The key fingerprint is:
SHA256:gHexXzD2EuQFcXgsnJVTpkTKfDMSI5uFUK3J58/rjf8 my_key
The key's randomart image is:
+---[RSA 3072]----+
|      .o+=%O*oo  |
|     .  .%*%=o   |
|    . o.=oBoB.   |
|     . o+..= o   |
|        So.      |
|          .      |
|           o     |
|            oo   |
|           .+oo.E|
+----[SHA256]-----+

In this output private key was saved to my_key file in current directory and public key - to my_key.pub. These files look like this:

How keys look like
# Private key
$ cat my_key
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn
NhAAAAAwEAAQAAAYEAoi/l2u3snJTbHYgcOSSq2WHmUw9Wy10ldZgXDCgR7HahtxvWB0n0
5XurIIeUmHpfLJoinJEK4ScI7uaagtwID7QoFHsok0zlkpSMHjJLbdYAAquotjmCzBuKtG
3YMcPX0CojSCkDtFAliqUTDe+RfJ0aHVyUeqLJhCOKMEeG7UNALt0lzpUSDeVNYiBnlyd8
B+iiBWtF7MGTiMTxk8fXbrO97XF8p1PjKhCppxjocilQIwhx5HB0rJjy5vDokpvrFc6kkc
Mjff2FzLm1BbMH/WOfqNpszTRV3lK8PCAQgmWM/vBINRdY7Z6eOJVeDSvy+J+A88AjnfEq
V0MHofSoYVrals26WGasFOCboJlTgsafFWYUxOi9BGPs5Ze1kqDZmbm6l3qWGC1TD0wyWw
6Bg0XU0m5zOsmZPG6M8XbIAXWIErmxssTZHx1sUzfg5f5W7WdoWNoYqkajZY01+IB0PhCr
BxRdio4B+pR4MVV0hgto8xtWr3x9qF+wa070SodfAAAFkI5E48yOROPMAAAAB3NzaC1yc2
EAAAGBAKIv5drt7JyU2x2IHDkkqtlh5lMPVstdJXWYFwwoEex2obcb1gdJ9OV7qyCHlJh6
XyyaIpyRCuEnCO7mmoLcCA+0KBR7KJNM5ZKUjB4yS23WAAKrqLY5gswbirRt2DHD19AqI0
gpA7RQJYqlEw3vkXydGh1clHqiyYQjijBHhu1DQC7dJc6VEg3lTWIgZ5cnfAfoogVrRezB
k4jE8ZPH126zve1xfKdT4yoQqacY6HIpUCMIceRwdKyY8ubw6JKb6xXOpJHDI339hcy5tQ
WzB/1jn6jabM00Vd5SvDwgEIJljP7wSDUXWO2enjiVXg0r8vifgPPAI53xKldDB6H0qGFa
2pbNulhmrBTgm6CZU4LGnxVmFMTovQRj7OWXtZKg2Zm5upd6lhgtUw9MMlsOgYNF1NJucz
rJmTxujPF2yAF1iBK5sbLE2R8dbFM34OX+Vu1naFjaGKpGo2WNNfiAdD4QqwcUXYqOAfqU
eDFVdIYLaPMbVq98fahfsGtO9EqHXwAAAAMBAAEAAAGAZlzastGlm7HrlXj5bytwVWEPsG
6m9hVk9hI2wapsnZTGPj5oWBNaaJgkCpTnuVDKzui9XZnBhxdO8RFEhcD/qYGoJj0Q/97x
qhDtWoWdy8XcHdNf2Rr1LYNYiMYnREl55V0jBYE1YFGRUC8dlpcUeNTizZNH9xrVGvwfVJ
dgVlEyqiFTok29pl2J+JvBJcp64rb1w3vQFzyZtCGw4venRaxV/A27ghRU9JCtstPqqVrf
xCypTWeYi/LAo/d6okWbGadc1y/RJZRob437+x2HuRg+bMStrOdBU3ry4OopHoHSF6UWHa
/u9uhwbDASvXN0DgnfQe83GRTunzKDyK3zGUyRh6DpDP/btS1Y899CTE2QXIgOnSk6Ah/R
zdY9Tw8yH02SvK4Ha4E+J1VUi3dWEt1izjv+qLbAgWJqn/i585vglp5NWE0BUGg56b6dme
MbwJJ7uG1VRWNBwbzirVHmeElNo6KxvUvUYok/wNxCfKRIkpKrF+Z1cwSK8lGnVF5JAAAA
wQCRweD9plo9rpk5vKggIHwGAsFVvgUBlfybJcCrza7fdGjw2fUnpdDDGN2nltLy3D+hn/
jFeVgXBaa0TNzUSuLyKXflzuoHHimjoBGBSXCh/TLbsbebdXWr3LEorErqeBDj5NUiWdyx
TEc0qXoL3k766ZtL67mYSM45kG76FKJxWpJ3JSB9NHG4FclQBGmZ1QphMEB4Nq1RiF5yH6
ZqoguKWxmSfZ2LfLTTyR9k4TLWrIEzfMRkMCHF3fIRrTsygbkAAADBANSkkcgeiL9uBdyP
C5W9zgLDPj4wCetdqlyP3Qoi1YvKBKwBNiW60UDQ1hda75roKmIr9rfFGZVYeYk0W/8JbS
QXG9t8vtyoWghRciIYokqQ5LJpfO6fn7NOJe2t44pcQ1COux4lIlPukfMU762cXVEio8qv
zAzlC09zbXtpxyQ9W/KT4PwI0o4AlYYANiSlPRjbXm8OtvBlD9Gp46Qp4Uzx1TBnWXEpKw
E1dgbmAYuJ8NIiK7cF4TYlhRCFhJagnQAAAMEAw0Gpx6b3df776ZEDlL+9wOtiEpgNJzzI
V/O9IzDXUIeJuduLnyPocuiPWm5bxUQD/m9ZHmZjmCPkRDszgTCc1wj32Yj2/ZqUidUkOd
IlYreuzNFcfpsDj8tN3LMYcA5c+LrjYt/+pyu5lj5URKo/nMKy/2WZQW3jqZzuSt+YKTRC
dxY2eKr+4J7p3UAJ6nU4YbTqfEvk6xZktrwfmJghW7Xsmjy2XTRU3UyzBM/K9REiuekry9
+jWazSoOMcZbErAAAAFXZhbmlhLXBvb2hAaTEwNzU5Nzk5OQECAwQF
-----END OPENSSH PRIVATE KEY-----

# Public key
$ cat my_key.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCiL+Xa7eyclNsdiBw5JKrZYeZTD1bLXSV1mBcMKBHsdqG3G9YHSfTle6sgh5SYel8smiKckQrhJwju5pqC3AgPtCgUeyiTTOWSlIweMktt1gACq6i2OYLMG4q0bdgxw9fQKiNIKQO0UCWKpRMN75F8nRodXJR6osmEI4owR4btQ0Au3SXOlRIN5U1iIGeXJ3wH6KIFa0XswZOIxPGTx9dus73tcXynU+MqEKmnGOhyKVAjCHHkcHSsmPLm8OiSm+sVzqSRwyN9/YXMubUFswf9Y5+o2mzNNFXeUrw8IBCCZYz+8Eg1F1jtnp44lV4NK/L4n4DzwCOd8SpXQweh9KhhWtqWzbpYZqwU4JugmVOCxp8VZhTE6L0EY+zll7WSoNmZubqXepYYLVMPTDJbDoGDRdTSbnM6yZk8bozxdsgBdYgSubGyxNkfHWxTN+Dl/lbtZ2hY2hiqRqNljTX4gHQ+EKsHFF2KjgH6lHgxVXSGC2jzG1avfH2oX7BrTvRKh18= my_key

Host keys have exactly the same format as user keys. Although you can generate these keys manually and point to them in SSH server configuration, in the majority of cases they are automatically generated during the first SSH server startup and usually reside in /etc/ssh/ssh_host_* files. To access a host in SSH you have to copy public keys of every authorized used to the list of authorized keys on this host. Exact location of the text configuration file(s) with a list of authorized user keys can be provided in SSH server settings. When working with virtual machines a typical workflow is to provide a list of authorized keys of a new virtual machine before virtual machine is created. Having this list, respective cloud platform automatically copies all authorized keys to correct configuration file inside virtual machine being started.

Boot works exactly the same as any other cloud platform and allows to provide a list of authorized public keys for every virtual machine. The only difference is that you have to indicate the same authorized public key in two locations:

  1. In jump host configuration. This is needed to allow authorized key owner to pass through jump host and to be able to connect (but not always access!) to the virtual machines created by Boot. A list of jump host authorized keys can be provided in values.yaml file when deploying Boot with Helm:

    Setting Boot jump host authorized keys in values.yaml
    boot:
  jumphost:
    authorizedKeys:
      - ssh-rsa AAAAB3NzaC...rTvRKh18= my_key # See public key contents above

    How to do this is explained in detail in jump host documentation.

  2. In every created virtual machine properties. This allows to access concrete virtual machine. When creating virtual machines with kubectl and YAML respective definition looks like this:

    Adding authorized keys to new virtual machine YAML definition
    apiVersion: boot.aerokube.com/v1
kind: VirtualMachine
metadata:
  name: my-vm
  namespace: boot
spec:
  os: ubuntu
  authorizedKeys: # A list of SSH public keys allowed to access this virtual machine
    - ssh-rsa AAAAB3NzaC...rTvRKh18= my_key # See public key contents above
    - ssh-rsa AAAAB3N... another-key # Another user public key contents

    If you are creating virtual machines with kubectl plugin - use -auth-keys flag:

    Adding authorized keys to new virtual machine with kubectl plugin
    $ kubectl vm create my-vm -os ubuntu -auth-keys /path/to/authorized_keys.file -n boot

    When this flag is missing, plugin will try to use your public key from ~/.ssh/id_rsa.pub file.

User and Host Certificates

Although user and host keys remain the most popular SSH authentication technology, they have several important drawbacks:

  1. Need to copy the same authorized user key to every virtual machine you need to access.

  2. Need to explicitly trust every host key before accessing this host.

  3. Keys do not store information about usernames allowed to authenticate with this key.

  4. Keys have no duration settings, i.e. no start and expiration date, so they are rarely \ never rotated.

  5. Keys have no granular permissions, i.e. they allow to use all features of SSH server that are not explicitly forbidden: interactive sessions, port forwarding, X11 forwarding, agent forwarding.

Recent SSH versions support another authentication technology called certificates. A certificate is a public user key with additional metadata (allowed usernames, duration settings, permissions and so on) signed by a certification authority (CA). A certification authority is any person or organization that you trust. This can be for example an information security department of your company or a specialized third-party organization.

Certificate is signed by a certification authority

certificate ca sign

Working with SSH certificates starts by creating a standard SSH key pair consisting of a private key and a public key. Then public key is sent to a certification authority, that adds a digital signature and required metadata to this public key. The resulting file is called an SSH certificate.

Certification authority should be trusted by user and host

certificate encryption

The main idea behind certificates is that instead of copying authorized keys of every user we have to trust all certificates signed by a certification authority that we trust. For example, we trust all SSH certificates created by our information security department. The only thing we need to provide in that case is certification authority certificate (CA certificate). Like any other certificate this certificate contains a public key and metadata: certification authority name, expiration information and other stuff. When we copy CA certificate to the list of trusted CA certificates a host, this host is then able to read public key from CA certificate and use it to check digital signature on every certificate that was created by this authority. When digital signature on user certificate is valid, then user SSH connection to the host is accepted.

Similarly to user certificate, every host can have a host certificate and users are able to check that this certificate was issued by a trusted certification authority. For connection encryption purposes certificates work exactly the same as authorized keys. To encrypt and send some data to a user, host during SSH connection receives certificate from user, extracts public key from this certificate and uses it for encryption. Only user having private key corresponding to certificate public key can decrypt the data.

  1. In the example below we create new certification authority keys. In real life your information security department or responsible third-party organization already has these keys. We demonstrate how to do this here just to show you the entire perspective.

  2. We use ed25519 encryption algorithm for certification authority because this is one of the most efficient algorithms for the moment.

Let’s now take a look at a typical usage of the user certificates. We assume that we already have a keypair consisting of my_key and my_key.pub files generated in previous section. Let’s start by creating a certification authority keys:

Creating certification authority
$ ssh-keygen -t ed25519 -f test_ca -C 'test-ca'
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in test_ca
Your public key has been saved in test_ca.pub
The key fingerprint is:
SHA256:uGmsLCSBD3Zwu05jIDMPKu+xMap+eBQTk5Tqf0enhFg test-ca
The key's randomart image is:
+--[ED25519 256]--+
|  ..o            |
| . *             |
|. + +            |
|O+ = E .         |
|=Oo * o S        |
|+.+B o = .       |
|.+@ . B o        |
| oo@ + o         |
|=o=.+ .          |
+----[SHA256]-----+

Like with a regular SSH key pair, this will create two files:

Certification authority files
# Private key, keep it secret!
$ cat test_ca
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
QyNTUxOQAAACDdXZr6c+pc0+IZskr+Hvgm8vU5HqSPQWfokHqaRLz1PwAAAJD4P2KI+D9i
iAAAAAtzc2gtZWQyNTUxOQAAACDdXZr6c+pc0+IZskr+Hvgm8vU5HqSPQWfokHqaRLz1Pw
AAAEAEVZJTuqanwlJTfVWJsX63gLv5W3ZoDXxQXLg+HHZnTd1dmvpz6lzT4hmySv4e+Cby
9TkepI9BZ+iQeppEvPU/AAAAB3Rlc3QtY2EBAgMEBQY=
-----END OPENSSH PRIVATE KEY-----

# Public key, we will use it in Boot settings.
$ cat test_ca.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN1dmvpz6lzT4hmySv4e+Cby9TkepI9BZ+iQeppEvPU/ test-ca

Having a certification authority, let’s now sign existing my_key.pub public key and create a user certificate:

Creating a user certificate
$ ssh-keygen -s test_ca -I 'my-test-cert' -z '0001' -n jump,root my_key.pub
Signed user key my_key-cert.pub: id "my-test-cert" serial 1 for jump,root valid forever

In this example we use CA private key file (test_ca) to sign user public key (my_key.pub) giving a resulting certificate file an identifier my-test-cert (mainly used for logging purposes on the server), serial number 0001 and allowed users jump and root. We need to allow jump user to authenticate on the jump host and root to authenticate on the virtual machine. The resulting certificate is saved to my_key-cert.pub file and looks like this:

A typical user certificate
$ cat ~/my_key-cert.pub
ssh-rsa-cert-v01@openssh.com AAAAHHNzaC1yc2EtY2VydC12MDFAb3BlbnNzaC5jb20AAAAgUmU82rtbl76TteeXo8oVJHhMR6m6//MG/KzG5O/3WGYAAAADAQABAAABgQCiL+Xa7eyclNsdiBw5JKrZYeZTD1bLXSV1mBcMKBHsdqG3G9YHSfTle6sgh5SYel8smiKckQrhJwju5pqC3AgPtCgUeyiTTOWSlIweMktt1gACq6i2OYLMG4q0bdgxw9fQKiNIKQO0UCWKpRMN75F8nRodXJR6osmEI4owR4btQ0Au3SXOlRIN5U1iIGeXJ3wH6KIFa0XswZOIxPGTx9dus73tcXynU+MqEKmnGOhyKVAjCHHkcHSsmPLm8OiSm+sVzqSRwyN9/YXMubUFswf9Y5+o2mzNNFXeUrw8IBCCZYz+8Eg1F1jtnp44lV4NK/L4n4DzwCOd8SpXQweh9KhhWtqWzbpYZqwU4JugmVOCxp8VZhTE6L0EY+zll7WSoNmZubqXepYYLVMPTDJbDoGDRdTSbnM6yZk8bozxdsgBdYgSubGyxNkfHWxTN+Dl/lbtZ2hY2hiqRqNljTX4gHQ+EKsHFF2KjgH6lHgxVXSGC2jzG1avfH2oX7BrTvRKh18AAAAAAAAAAQAAAAEAAAAMbXktdGVzdC1jZXJ0AAAAEAAAAARqdW1wAAAABHJvb3QAAAAAAAAAAP//////////AAAAAAAAAIIAAAAVcGVybWl0LVgxMS1mb3J3YXJkaW5nAAAAAAAAABdwZXJtaXQtYWdlbnQtZm9yd2FyZGluZwAAAAAAAAAWcGVybWl0LXBvcnQtZm9yd2FyZGluZwAAAAAAAAAKcGVybWl0LXB0eQAAAAAAAAAOcGVybWl0LXVzZXItcmMAAAAAAAAAAAAAADMAAAALc3NoLWVkMjU1MTkAAAAg3V2a+nPqXNPiGbJK/h74JvL1OR6kj0Fn6JB6mkS89T8AAABTAAAAC3NzaC1lZDI1NTE5AAAAQDoV3uwn0SBWrZcHtnzNuVVYPXe3orfxb3gpM6jhzszPaZ96G5Pu48lyQPN9tDiZE1PJJ7e/ovw07LQy4wqkSAo= some-comment-here

Now in order to work with created certificates in Boot - you need to add certification authority public key in two places:

  1. In jump host configuration. This is needed to allow SSH certificate owner to pass through jump host. A list of jump host trusted user certification authorities can be provided in values.yaml file when deploying Boot with Helm:

    Setting Boot jump host truster certification authorities in values.yaml
    boot:
  jumphost:
    trustedUserCAKeys:
      - ssh-ed25519 AAAAC3NzaC1....eppEvPU/ test-ca # See CA public key contents above

    How to do this is explained in detail in jump host documentation.

  2. In every created virtual machine properties. This allows to access concrete virtual machine. When creating virtual machines with kubectl and YAML respective definition looks like this:

    Adding authorized keys to new virtual machine YAML definition
    apiVersion: boot.aerokube.com/v1
kind: VirtualMachine
metadata:
  name: my-vm
  namespace: boot
spec:
  os: ubuntu
  trustedUserCAKeys: # A list of trusted certification authorities allowed to access this virtual machine
    - ssh-ed25519 AAAAC3NzaC1....eppEvPU/ test-ca # See CA public key contents above
    - ssh-ed25519 AAAAC3.... another-ca # Another CA public key contents

    If you are creating virtual machines with kubectl plugin - use `-user-ca-keys flag:

    Adding trusted certification authorities to new virtual machine with kubectl plugin
    $ kubectl vm create my-vm -os ubuntu -user-ca-keys /path/to/truster_user_ca.file -n boot

2.2.3. Accessing a Virtual Machine

Having a configured Boot jump host and a virtual machine called my-vm, let’s now access it.

A typical virtual machine
$ kubectl get vm -n boot
NAME    FQDN                              IP           OS             CPU    MEM   STATUS    UPTIME
my-vm   my-vm.vm.boot.svc.cluster.local   172.17.0.4   ubuntu:22.04   250m   1Gi   Running   10s

A full SSH command to access some host through jump host is as follows:

Accessing a virtual machine through jump host
$ ssh -i /path/to/my_key -J jump@boot.example.com:2222 root@my-vm.vm.boot.svc.cluster.local

In this command we use -i flag to provide a private key file. Then -J is needed to use a jump host and jump@boot.example.com:2222 notation means to access jump host boot.example.com on port 2222 using jump user (these are Boot defaults). Finally, you provide your destination host root@my-vm.vm.boot.svc.cluster.local, i.e. you access host my-vm.vm.boot.svc.cluster.local inside Kubernetes with root user. This command will work for both authorized keys and user certificates. In case of user certificates SSH client will automatically try all certificates corresponding to private key file (e.g. my_key-cert.pub). You can certainly provide an exact certificate file as follows:

Providing an explicit SSH user certificate file
$ ssh -o CertificateFile=/path/to/my_key-cert.pub -i /path/to/my_key -J jump@boot.example.com:2222 root@my-vm.vm.boot.svc.cluster.local

These commands look complicated, right? But most of these values except virtual machine host rarely change, so let’s move them to SSH client configuration file (usually resides in ~/.ssh/config):

Moving connection options to SSH client configuration
$ cat ~/.ssh/config
Host bootjump
    IdentityFile /path/to/my_key # Private key location
    HostName boot.example.com # Jump host name
    Port 2222 # Jump host port number
Host *.vm.boot.svc.cluster.local
    IdentityFile /path/to/my_key # Private key location
    ProxyJump jump@bootjump # Always use jump host declared above

Having this configuration file, SSH connection command can be simplified to:

Simplified SSH connection command
$ ssh root@my-vm.vm.boot.svc.cluster.local

But we can make this command even shorter. Virtual machine fully qualified domain name is my-vm.vm.boot.svc.cluster.local. In this name boot is namespace name and svc.cluster.local is standard Kubernetes DNS suffix. Kubernetes is able to automatically add this part of domain name, so you can add the following to SSH config:

How to support short domain name resolution
$ cat ~/.ssh/config
Host *.vm
    IdentityFile /path/to/my_key # Private key location
    ProxyJump jump@bootjump

With this configuration section the following command will work:

Even shorter SSH connection command
$ ssh root@my-vm.vm

2.2.4. Trusting Boot Host Certificate

In previous section we already mentioned that both user and host can have SSH certificates. User certificates are used by SSH server to identify users. Host certificates allow user to be sure that connection is made to the host that was expected to access. By default, when you try to access a virtual machine for the first time you will see the following:

Accessing a virtual machine for the first time
$ ssh root@my-vm.vm
The authenticity of host '[boot.example.com]:2222 ([XXX.XXX.XXX.XXX]:2222)' can't be established.
ED25519 key fingerprint is SHA256:F/qgcVr/35EPaGtxX4Y1YHR5H8oD28sUKs0IRFiRWfI.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[boot.aerokube.local]:2222' (ED25519) to the list of known hosts.
The authenticity of host 'my-vm.vm (<no hostip for proxy command>)' can't be established.
ED25519 key fingerprint is SHA256:mGSVISH1obExpV3YceqKuzVrIzbnv7vwiUjc9CccS1k.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'my-vm.vm' (ED25519) to the list of known hosts.
my-vm:~#

To stop seeing these warnings we need to add Boot virtual machines to the list of trusted hosts. During the installation Boot generates a certification authority that is used to create host certificates for all virtual machines. The pair of keys for this certification authority is stored in Kubernetes secret <boot-namespace>-vm-ssh-host-ca-keys, e.g. boot-vm-ssh-host-ca-keys when Boot is running in default boot namespace:

  1. Extract public key for certification authority from secret:

    $ kubectl get secret boot-vm-ssh-host-ca-keys -o jsonpath='{.data.ca_key\.pub}' -n boot | base64 -D
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFHZc6j06RCf+8/J1IuS8SY1/hasLHlik6XRnO87+Zhu user@boot-generate-ssh-keys-4j2dw

  2. Add this key to the list of SSH known hosts:

    $ cat ~/.ssh/known_hosts
# For fully qualified domain name
@cert-authority *.vm.boot.svc.cluster.local ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFHZc6j06RCf+8/J1IuS8SY1/hasLHlik6XRnO87+Zhu user@boot-generate-ssh-keys-4j2dw

# For short domain name
@cert-authority *.vm ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFHZc6j06RCf+8/J1IuS8SY1/hasLHlik6XRnO87+Zhu user@boot-generate-ssh-keys-4j2dw

  3. Now if you try to execute the same command - SSH will never ask you to confirm authenticity of virtual machines. You only need to confirm jump host (boot.example.com) authenticity once.

3. Configuration

3.1. Operating Systems

  1. Boot virtual machines are always running in privileged mode, so we intentionally did not add some existing Kubernetes features like security context.

  2. Default way of accessing Boot virtual machines is using [SSH] connection. An interactive SSH session never relies on server environment variables, so we did not add passing environment variables from operating system configuration. This simply makes no sense.

An operating system is the main Boot configuration unit. Every operating system object stores parameters used when you launch a virtual machine: container images, available computing resources, particular DNS configuration and so on. To list available operating systems:

Listing operating systems
$ kubectl get os -n boot
NAME     REPOSITORY            VERSION(S)   CPU    MEM   AGE
alpine   quay.io/boot/alpine   3.18         500m   1Gi   50m
ubi      quay.io/boot/ubi      9.2          500m   1Gi   50m
ubuntu   quay.io/boot/ubuntu   22.04        500m   1Gi   50m

A typical operating system definition looks like this:

Getting operating system definition in YAML format
$ kubectl get os ubuntu -n boot -o yaml
apiVersion: boot.aerokube.com/v1
kind: OperatingSystem
metadata:
  name: ubuntu
  namespace: boot
  # Other metadata fields...
spec:
  repository: quay.io/boot/ubuntu  (1)
  resources:                       (2)
    limits:
      cpu: "0.5"
      memory: 1Gi
    requests:
      cpu: "0.5"
      memory: 1Gi
  versions:                        (3)
  - "22.04"
status:
  # Fields used for listing
1 Repository with images for this operating system
2 Available computing resources
3 Available operating system versions

While such short configuration will work - more advanced settings exist:

More fields in operating system definition
$ kubectl get os ubuntu -n boot -o yaml
apiVersion: boot.aerokube.com/v1
kind: OperatingSystem
metadata:
  name: ubuntu
  namespace: boot
  # Other metadata fields...
spec:
  affinity:                            (1)
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-label-1
            operator: In
            values:
            - value-1
            - value-2
  dnsPolicy: None                      (2)
  dnsConfig:                           (3)
    nameservers:
      - 192.0.2.1
    searches:
      - ns1.svc.cluster-domain.example
      - my.dns.search.suffix
    options:
      # See Kubernetes documentation
  nodeSelector:                        (4)
      node-label-1: "label1-value"
      node-label-2: "label2-value"
  repository: quay.io/boot/ubuntu      (5)
  resources:                           (6)
    limits:
      cpu: "0.5"
      memory: 1Gi
    requests:
      cpu: "0.5"
      memory: 1Gi
  tolerations:                         (7)
  - key: "key1"
    operator: "Equal"
    value: "value1"
    effect: "NoSchedule"
  versions:                            (8)
  - "22.04"
status:
  # Fields used for listing
1 Applied affinity configuration
2 Applied DNS policy
3 Applied DNS configuration
4 Applied Kubernetes node selector
5 Repository with images for this operating system - this is the only required field
6 Available computing resources
7 Applied Kubernetes tolerations
8 Available operating system versions - limits allowed image tags that can be used for launching virtual machines

The next sections explain these settings in more detail.

To edit an operating system object:

Editing an operating system
$ kubectl edit os ubuntu -n boot # Do modifications in text editor, save and exit

Another possible option to edit an operating system definition that we recommend in the majority of cases is updating the same object with Helm. To do this - edit the following values.yaml section:

Updating operating systems with Helm
boot:
  # More fields go here
  os:
    ubuntu: # Helm chart fields for every operating system object are exactly the same as described above
      repository: quay.io/boot/ubuntu
      versions:
      - "22.04"
    nodeSelector: # Let's add node selector
      kubernetes.io/arch: "amd64"
  # Other fields go here

Now simply reapply Helm chart with modifications in values.yaml file, and you are all set.

3.1.1. Repository and Versions

The most important settings in operating system definition are the name of image repository and available versions. For example, let’s assume that you have the following operating system definition:

Repository and versions in operating system definition
$ kubectl get os ubuntu -n boot -o yaml
apiVersion: boot.aerokube.com/v1
kind: OperatingSystem
metadata:
  name: ubuntu
  namespace: boot
  # Other metadata fields...
spec:
  repository: quay.io/boot/ubuntu
  versions:
  - "20.04"
  - "22.04"
status:
  # Fields used for listing

Now you request to launch a virtual machine like this:

Requesting to launch an Ubuntu 20.04 virtual machine
apiVersion: boot.aerokube.com/v1
kind: VirtualMachine
metadata:
  name: my-ubuntu-vm
  namespace: boot
spec:
  os: ubuntu
  version: "20.04"

Boot will then create a pod using quay.io/boot/ubuntu:20.04 container image. Similarly, when you change version to 22.04, respective container image will be quay.io/boot/ubuntu:20.04. Versions list is mainly needed to show recommended operating system versions to user. Nothing will prevent user from creating a virtual machine with version 18.04 is respective image (quay.io/boot/ubuntu:18.04) exists.

3.1.2. Computing Resources

  1. Boot is using exactly the same computing resources configuration YAML format as Kubernetes does.

  2. We recommend to set memory requests and limits to the same value because otherwise sooner or later processes inside a virtual machines can be killed because of lack of memory (out of memory, OOM).

To limit computing resources available to every virtual machine, you need to add resources section to respective operating system definition.

Assigning computing resources to operating system definition
boot:
  # More fields go here
  os:
    ubuntu:
      repository: quay.io/boot/ubuntu
      resources: # Computing resources available to every virtual machine
        limits:
          cpu: "1.0"
          memory: 1Gi
        requests:
          cpu: "0.5"
          memory: 1Gi
  # Other fields go here

3.1.3. Node Selector

Boot is using exactly the same node selector YAML format as Kubernetes does.

Sometimes you may need to run virtual machines on particular Kubernetes nodes (i.e. hosts) only. Kubernetes allows to do this by specifying so-called node selectors. If you need to add the same node selector to an operating system definition:

Adding node selector to operating system definition
boot:
  # More fields go here
  os:
    ubuntu:
      repository: quay.io/boot/ubuntu
      nodeSelector: # Node selector definition
        node-label-1: "label1-value"
  # Other fields go here

3.1.4. Affinity

Boot is using exactly the same affinity configuration YAML format as Kubernetes does.

In addition to node selectors, you can also use all available node and pod affinity features available in Kubernetes. This allows you to have even more advanced pod scheduling settings like matching Kubernetes nodes against complex logical expressions, preventing some labeled pods to be running on the same node with another labeled pods and so on.

Adding affinity specification to operating system definition
boot:
  # More fields go here
  os:
    ubuntu:
      repository: quay.io/boot/ubuntu
      affinity: # Affinity settings have exactly the same YAML fields as Kubernetes pod affinity settings
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node-label-1
                operator: In
                values:
                - value-1
                - value-2
  # Other fields go here

3.1.5. Networking

Some scenarios require flexible networking configuration. For example, you may need to override used DNS server. This can be easily done as follows:

Adding dns settings to operating system definition
boot:
  # More fields go here
  os:
    ubuntu:
      repository: quay.io/boot/ubuntu
      dnsPolicy: None # DNS settings have exactly the same YAML fields (dnsPolicy and dnsConfig) as Kubernetes pod dns settings
      dnsConfig:
        nameservers:
          - 192.0.2.1
        searches:
          - ns1.svc.cluster-domain.example
          - my.dns.search.suffix
        options:
          # See Kubernetes documentation
  # Other fields go here

3.1.6. Tolerations

Boot is using exactly the same tolerations configuration YAML format as Kubernetes does.

In addition to node selector and affinity, Kubernetes has a concept of node taints. Taints allow nodes to repel some pods from being scheduled on them. If you wish to run virtual machines on tainted nodes - you have to add tolerations, that is to say a number of conditions to match against tainted nodes.

To add the tolerations to operating system definition:

Adding tolerations to operating system definition
boot:
  # More fields go here
  os:
    ubuntu:
      repository: quay.io/boot/ubuntu
      tolerations: # Tolerations settings have exactly the same YAML keys as Kubernetes pod tolerations settings
      - key: "key1"
        operator: "Equal"
        value: "value1"
        effect: "NoSchedule"
  # Other fields go here

3.2. Jump Host Configuration

Jump host is the main entrypoint to Boot virtual machines. Jump host is reconfigured by updating values.yaml file and reapplying Helm chart.

3.2.1. User

Jump host processes are always run by an unprivileged user (default name is jump). This gives additional security. Boot users can only access jump host by SSH using the name of this user. To set it:

Updating jump host user settings
$ cat values.yaml
boot:
  jumphost:
    user:
      name: my-user # Custom login user
      uid: 10000000 # User uid (you may need to change it in Openshift)
      gid: 10000000 # User gid (you may need to change it in Openshift)

3.2.2. Network Port

To change network port being used to access jump host by SSH:

Updating jump host user settings
$ cat values.yaml
boot:
  jumphost:
    port: 22222 # Setting custom network port

3.2.3. Allowed User Keys and Certificates

It is important to configure users allowed to access Boot cluster by SSH at all. This is done by adding authorized public SSH keys and trusted user CA certificates to jump host settings:

Updating jump host user authorized keys and trusted CA certificates
$ cat values.yaml
boot:
  jumphost:
    authorizedKeys: # User authorized keys
      - ssh-rsa AAAAB3N... some-key
      - ssh-rsa AAAAB3N... another-key
    trustedUserCAKeys: # A list of trusted user certification authorities
      - ssh-ed25519 AAAAC3N... certification-authority-1
      - ssh-ed25519 AAAAC3N... certification-authority-2

3.2.4. Computing Resources

If for some reason default jump host computing resources are not sufficient:

Updating jump host computing resources
$ cat values.yaml
boot:
  deployment:
    containers:
      jumphost:
        resources: # Custom jump host computing resources
          limits:
            cpu: 2000m
            memory: 1Gi
          requests:
            cpu: 200m
            memory: 1Gi

3.3. License Key

  1. According to license agreement you can use up to 1 CPU for free and for unlimited period of time. If you wish to have more CPUs - order a license key. This section describes how to list and install license keys. A limited duration trial license key with more CPUs can be generated on Boot website.

A typical license key is a text file with .key extension that looks like this:

$ cat license.key
Ulh3SnN6S3JYTVg5UmIzaHNiWUpOaE1rcGltQzJxRVZVbGdHMVliNlZDbnNjVkc5b1M1eGNEbTRZYkN6c2RaTmtaaGs0cDExQWRlOTA2YVNxK3NNV2JORHd0NkFEUEZTNk16UXVCcWhQMVovajhhdWlJZDJzdW9yVEFRTFppSnp2NHloMkdZYXNVVlNhRk05Q2ZOUk4rd1JCNHlXRlFwRmNwbVRFWk9hdXRQWjJvVUM0TldGdXR2OUtiangrT0hkRmJNK2xtQUhCYVArWDhlUTJoNnFzRlExdHl2Zm11QmtVWUNhRHBSTEhzVTdLQXVEZFZKWlhUSU9PRjNjUlFIODhyYmZKTkZVWm1sNG5UZnJHM2RFRTJmYkMyakNwVndLWmJaMkgrVi9zeGRXd0dDblZMNFAyYXVyNjQ4cDhnb0xvRGdZMGlnRmM1WXFPODVGK0U4TlRPWWpyMGtPRThnY1lRcU1JT1JWZEkwQ0ZNVkk3SkFpbHI0UzhHcHduY2Vwcks3ZERtbnVLNmRIeGVnZHhqSGNIN1laZlR6U2prZ001S2R5Q1RCSlF0RXB2VjkvUlF5MUV3M3RIcCtTRWcvTjl3eUF5VE4xUFl4Q0xtU2t0QjFNblZVeURZby9sWXlCYlQrSGNlSUExdktTVThDQlZFaFNRZmdRS1BmbUxFblBuSmxhVHZmWXhnVUF3b3B3dmFwaHFmaExRNTVEM2d6RzA4ZDlsNTVGVGE3Vlo4b2Vpd2FabUFDWHZRZ3NlMTUzT29SdDV1M1VsNGNVTmFUOGUxbWgva2JKajJ1Mjk2cysvalBBa3JVRnNVdWlNZHA1a2Zrc2hTQlhybDZyVlFJYytDcWE3MUFBdWpyT1lPNm1JZ3BNZXAvYUI4cXhRR29uTUVzVGRrRlVKR289O2V5SnNhV05sYm5ObFpTSTZJa1JsWm1GMWJIUWlMQ0p3Y205a2RXTjBJam9pUW05dmRDSXNJbTFoZUZObGMzTnBiMjV6SWpveGZRPT0=

In Boot license keys (or just licenses) are stored in custom Kubernetes resource.

3.3.1. Listing License Keys

Contrarily to other resources introduced by Boot, licenses are stored cluster-wide. Thus, you don’t need to provide namespace name in the following commands (-n boot is not needed).

To list available licenses:

Listing licenses (free license key output)
$ kubectl get licenses
NAME   LICENSEE   CPUS   USAGE   EXPIRES   STATUS   NAMESPACE   AGE
boot   Default    1      0%      Never     Ok       boot        2m42s

The output above is shown when the free license key is used. Columns meaning is as follows:

  • Name. License key object name.

  • Licensee. License key owner name. Usually equals to company name, e.g. Acme LLC. For free license key with 1 CPU equals to Default.

  • CPUs. Maximum number of CPUs available in this license key.

  • Expires. The number of days this license key expires in. Equals to Already when license key already expired and equals to Never if license key never expires.

  • Status. License key status. Can be one of: Ok - license key is active, Expired - license key has expired, Broken - invalid license key data was provided.

  • Namespace. Name of Kubernetes namespace where this license key is used.

You may have multiple custom Kubernetes resources named license. In that case in order to work with Boot licenses - simply use fully qualified resource name:

Listing licenses (longer and fully qualified resource name)
$ kubectl get licenses.boot
NAME   LICENSEE   CPUS   USAGE   EXPIRES   STATUS   NAMESPACE   AGE
boot   Default    1      0%      Never     Ok       boot        2m42s

$ kubectl get licenses.boot.aerokube.com
NAME   LICENSEE   CPUS   USAGE   EXPIRES   STATUS   NAMESPACE   AGE
boot   Default    1      0%      Never     Ok       boot        2m42s

To view a license key in YAML format:

Showing license in YAML format
$ kubectl get license boot -o yaml
apiVersion: boot.aerokube.com/v1
kind: License
metadata:
  name: boot                        (1)
  # Other Kubernetes metadata
spec:
  data: YkV0dmNsaGxTak51Y2....      (2)
status:
  # Other keys and values
1 License key name, should correspond to namespace name where this license key should be used
2 License key contents

3.3.2. Updating a License Key

To update an existing license key - simply update data field in respective license object:

Updating license key with kubectl
$ kubectl edit license boot # Replace data field with your new license key in text editor, save and exit

When you update a license key - all changes are applied immediately. This usually also leads to graceful Boot pods restart (does not interrupt running virtual machines).

3.3.3. Multiple License Keys

Boot allows to share the same license key among several Kubernetes namespaces and in the majority of cases a single license key should be enough. However, in some cases you may want to use a separate Boot instance and a separate license key for some teams. To achieve this:

  1. Deploy two independent Boot clusters to namespace ns1 and ns2

  2. Create two license objects with name field set to ns1 and ns2 and save them to file (e.g. license-keys.yaml):

    License keys to be created
    $ cat license-keys.yaml
apiVersion: boot.aerokube.com/v1
kind: License
metadata:
  name: ns1
spec:
  data: <license-key-1>
---
apiVersion: boot.aerokube.com/v1
kind: License
metadata:
  name: ns2
spec:
  data: <license-key-2>

  3. Apply resulting file:

    $ kubectl apply -f license-keys.yaml

    • If you try to create two license keys with the same data field value, then one of them will be considered as a duplicate and automatically deleted.

    • If you have two different license keys with the same namespace field, then Boot will always choose the most recently created one.

  4. License keys will be applied automatically, and you will see the following in licenses list:

    Two license keys are applied
    $ kubectl get licenses
NAME  LICENSEE   CPUS   USAGE   EXPIRES   STATUS   NAMESPACE   AGE
ns1   Acme Inc.  10     0%      20d       Ok       ns1         2m42s
ns2   Acme Inc.  20     0%      30d       Ok       ns2         2m42s

3.3.4. Deleting a License Key

To delete an existing license key - simply delete respective license object:

Deleting license key
$ kubectl delete license boot

When you delete the license key with name field set to some Boot namespace, Boot will automatically fall back to the free license key with 1 CPU included.

3.3.5. License Key Expiration

To check for expiring or expired license keys - just list them with kubectl:

Listing license keys to understand when they expire
$ kubectl get licenses
NAME  LICENSEE   CPUS   USAGE   EXPIRES   STATUS   NAMESPACE   AGE
ns1   Acme Inc.  10     0%      32d       Ok       ns1         2m42s
ns2   Acme Inc.  20     0%      today     Ok       ns2         2m42s

You can see that Expires column is showing the number of days remaining for every license key. When a license key expires the same command output will be:

When one license key has expired
$ kubectl get licenses
NAME  LICENSEE   CPUS   USAGE   EXPIRES   STATUS   NAMESPACE   AGE
ns1   Acme Inc.  10     0%      32d       Ok       ns1         2m42s
ns2   Acme Inc.  20     0%      Already   Expired  ns2         2m42s

For expired license key Expires column will be set to Already and license key status will be Expired.

You can also use Kubernetes API directly instead of kubectl to list license keys and find expiring or expired ones.

3.4. Using Private Container Registry

By default, Boot images (aerokube/boot, aerokube/jumphost and so on) are downloaded from public container images registry. If in your environment due to security restrictions container images can only be downloaded from private registry (e.g. my-registry.example.com) you need to configure Boot to work with this registry. To do this:

  1. Configure Kubernetes authentication to your private registry:

    $ kubectl create secret docker-registry my-registry.example.com --docker-server=my-registry.example.com --docker-username=some-user --docker-password=registry-password --docker-email=some-user@example.com -n boot
$ kubectl patch serviceaccount boot -p '{"imagePullSecrets": [{"name": "my-registry.example.com"}]}' -n boot # Use correct service account name here

  1. Copy all desired virtual machine images to your registry:

    quay.io/boot/ubuntu:22.04 => my-registry.example.com/boot/ubuntu:22.04

  2. Update operating system definition in Helm chart:

    Operating system definition in Helm chart
    $ cat values.yaml
boot:
  namespaces:
  os:
    ubuntu:
      repository: my-registry.example.com/boot/ubuntu
      versions:
      - "22.04"

  3. Copy desired version of the Boot images to your registry:

    aerokube/boot:1.0.0 => my-registry.example.com/aerokube/boot:1.0.0
aerokube/jumphost:1.0.0 => my-registry.example.com/aerokube/jumphost:1.0.0
aerokube/keygen:1.0.0 => my-registry.example.com/aerokube/keygen:1.0.0
aerokube/reloader:1.0.0 => my-registry.example.com/aerokube/reloader:1.0.0

  4. Use new Boot images from the previous step in Helm chart to start Boot:

    Custom registry for Boot images in Helm chart
    $ cat values.yaml
boot:
  deployment:
    containers:
      boot:
        image:
          repository: my-registry.example.com/aerokube/boot
      jumphost:
        image:
          repository: my-registry.example.com/aerokube/jumphost
      keygen:
        image:
          repository: my-registry.example.com/aerokube/keygen
      reloader:
        image:
          repository: my-registry.example.com/aerokube/reloader

3.5. Using Multiple Namespaces

This section shows how to configure Boot to launch virtual machines in multiple namespaces. To do this:

  1. Create one or more namespaces where virtual machines will be running:

    Creating team namespaces for virtual machines
    $ kubectl create namespace team-alpha
$ kubectl create namespace team-beta

    You can optionally limit computing resources available to every namespace by using Kubernetes resource quotas.

  2. Create a role for users in every namespace:

    Creating a namespace role
    $ cat team-alpha-roles.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: team-alpha-roles
  namespace: team-alpha # Use correct namespace name here
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - watch
  - list
  - create
  - update
  - patch
  - delete
- apiGroups:
  - boot.aerokube.com
  resources:
  - operatingsystems
  - operatingsystems/status
  verbs:
  - get
  - watch
  - list
- apiGroups:
  - boot.aerokube.com
  resources:
  - virtualmachines
  - virtualmachines/status
  verbs:
  - get
  - watch
  - list
  - create
  - update
  - patch
  - delete

$ kubectl apply -f team-alpha-roles.yaml # Create role

  3. Grant the role from previous step to every user that should be able to create virtual machines:

    Granting roles to users
    $ kubectl create rolebinding my-user-roles --role=team-alpha-roles --user=my-user -n team-alpha # Add role to user (use correct login here)

  4. Add namespaces to values.yaml file of Boot Helm chart:

    Adding namespaces to Boot
    $ cat values.yaml
boot:
  namespaces:
    - team-alpha
    - team-beta

  5. Reapply Helm chart:

    Applying namespace settings to Boot
    $ helm upgrade --install -f values.yaml -n boot boot aerokube/boot

  6. Now users should be able to create virtual machines in their namespaces:

    Creating a virtual machine from YAML definition
    $ cat vm.yaml
apiVersion: boot.aerokube.com/v1
kind: VirtualMachine
metadata:
  name: my-vm
  namespace: team-alpha # Note namespace name here
spec:
  os: ubuntu
  authorizedKeys:
    - ssh-rsa AAAAB3N... some-key

$ kubectl create -f ~/vm.yaml
virtualmachine.boot.aerokube.com/my-vm created
    Creating a virtual machine with plugin
    $ kubectl vm create my-vm -os ubuntu -n team-beta # Like in any kubectl command, -n flag sets namespace name

3.6. Log Files

Although Boot should just work out of the box, sometimes you may need the log output. Every Boot component is outputting logs to standard output (stdout), so you can use well-known kubectl commands to see the log. Everything related to starting virtual machines is being output by boot container:

$ kubectl logs -lapp=boot -c boot -n boot

To follow the logs while running the tests add -f flag:

$ kubectl logs -f -lapp=boot -c boot -n boot

You can also take a look at reloader and jumphost logs as follows:

$ kubectl logs -f -lapp=boot -c reloader -n boot
$ kubectl logs -f -lapp=boot -c jumphost -n boot

If you are encountering issues with virtual machines then take a look at respective pod:

$ kubectl logs virtual-machine-01 -n boot

Here virtual-machine-01 is the name of the virtual machine.

4. License Agreement

Last updated August 1st, 2023. Replaces the prior version in its entirety.

This is a legal agreement. By downloading, installing, copying, saving on Customer’s computer, or otherwise using Aerokube software, support or products Customer becomes a party to this Agreement and Customer consents to be bound by all the terms and conditions set forth below.

  1. Parties

    1. "Aerokube", "Licensor" or "We" means Aerokube Software OÜ, having its principal place of business at Harju maakond, Tallinn, Kesklinna linnaosa, Karu tn 14-8, 10120, Estonia, registered in the Commercial Register of Estonia, registry code: 14653208.

    2. "Customer", "Licensee" or "You" means the sole proprietor or legal entity specified in the Subscription Confirmation. For legal entities, "Customer" includes any entity which controls, is controlled by, or is under common control with Customer. For the purposes of this definition, "control" means one of the following:

      1. The power, directly or indirectly, to direct or manage such entity, whether by contract or otherwise.

      2. Ownership of fifty percent (50%) or more of the outstanding shares or beneficial ownership of such entity.

  2. Definitions

    1. "Agreement" means this License Agreement.

    2. "Product" means Boot software identified by Licensor as a software developer tool. For the avoidance of doubt, the Product is not produced to the specifications of Customer nor customized through modification or personalization, is intended for mass distribution, and no software code will be provided to Customer. Detailed description of the product is provided on Licensor’s web site: https://aerokube.com/boot/.

    3. "User" means any employee, independent contractor or other personnel obtaining access to the Product(s) from Customer.

    4. "Number of Central Processing Units" means maximum number of central processing units (also herein referred to as CPUs) being used by the Product in parallel.

    5. "License Key" means a unique key-code that enables a Licensee to use the Product by unlocking the fixed CPUs. Only Licensor and/or its representatives are permitted to produce License Keys for the Product.

    6. "Subscription" means an arrangement for making use of the Product of periodic nature on a prepayment plan. For the purpose of clarity, Subscription includes the subscription term, Products provided to Customer, subscription fees, payment schedules and fixed number of License Keys.

    7. "Product Evaluation" means using the Product without a valid License Key.

    8. "Subscription Confirmation" means an email confirming Customer’s rights to access and use Products, including total Number of Concurrent Sessions.

    9. "Product Installation" means a Product copy running on Customer’s computer device, hardware server or virtual machine.

    10. "Product Version" means a release, update, or upgrade of a particular Product that is not identified by Licensor as being made for the purpose of fixing software bugs.

    11. "Bug Fix Update" for a particular Product Version means a software update or release that is specifically identified by Licensor as a bug fix for that Product Version.

    12. "E-mail Support" means a form of customer support provided by the Licensor. At the time of writing, the corresponding e-mail address is support@aerokube.com; should the address be changed, the new address will be referred to on the Licensor’s web site.

    13. "Instant Messaging Support" means a form of customer support provided by the Licensor. At the time of writing, the corresponding address to support channel is https://t.me/aerokube_boot; should the address be changed, the new address will be referred to on the Licensor’s web site.

    14. "Affiliate" means any entity belonging to the same group as the Licensor.

  3. How this Agreement Works

    1. Entire Agreement. This Agreement, including the Third-Party Software license terms, constitutes the entire agreement between the parties concerning its subject matter and supersedes any prior agreements between Customer and Licensor regarding Customer’s use of any Products. No purchase order, other ordering document or any handwritten or typewritten text which purports to modify or supplement the printed text of this Agreement or any schedule will add to or vary the terms of this Agreement unless signed by both Customer and Licensor.

    2. Reservation of Rights. Aerokube reserves the right at any time to cease the support of the Product and to alter prices, features, specifications, capabilities, functions, terms of use, release dates, general availability or other characteristics of the Product.

    3. Changes to this Agreement. We may update or modify this Agreement from time to time, including any referenced policies and other documents. If a revision meaningfully reduces Customer’s rights, we will use reasonable efforts to notify Customer. If we modify this Agreement, the modified version of the Agreement will be effective from the start of the next Subscription term. In this case, if Customer objects to the updated Agreement terms, as Customer’s exclusive remedy, Customer may cancel the Subscription. Customer may be required to click through the updated Agreement to show its acceptance. For the avoidance of doubt, each Subscription Confirmation is subject to the version of the Agreement in effect on the Subscription Confirmation date.

    4. Opportunity to Review. Customer hereby declares that Customer has had sufficient opportunity to review this Agreement, understand the content of all of its clauses, negotiate its terms, and seek independent professional legal advice in that respect before entering into it. Consequently, any statutory "form contract" ("adhesion contract") regulations shall not be applicable to this Agreement.

    5. Severability. If a particular term of this Agreement is not enforceable, the unenforceability of that term will not affect any other terms of this Agreement.

    6. Headings. Headings and titles are for convenience only and do not affect the interpretation of this Agreement.

    7. No Waiver. Our failure to enforce or exercise any part of this Agreement is not a waiver of that section.

    8. Notice. Aerokube may deliver any notice to Customer via electronic mail to an email address provided by Customer, registered mail, personal delivery or renowned express courier (such as DHL, FedEx or UPS). Any such notice will be deemed to be effective:

      1. On the day the notice is sent to Customer via email.

      2. Upon personal delivery.

      3. One (1) day after deposit with an express courier or five (5) days after deposit in the mail, whichever occurs first.

    9. Governing Law. This Agreement will be governed by the laws of the Estonia, without reference to conflict of laws principles. Customer agrees that any litigation relating to this Agreement may only be brought in, and will be subject to the jurisdiction of, any competent court of the Estonia. The parties agree that the United Nations Convention on Contracts for the International Sale of Goods does not apply to this Agreement.

    10. Exceptions or Modifications. For exceptions or modifications to this Agreement, please contact Aerokube at: support@aerokube.com In case the terms of this Agreement are in conflict with the terms of any agreement individually negotiated and agreed between Aerokube and Customer, the terms of the latter shall prevail.

    11. Force Majeure. Except with respect to Customer’s payment obligations, neither party shall be liable to the other for any delay or failure to perform any obligation under this Agreement (except for a failure to pay fees) if the delay or failure is due to unforeseen events which occur after the signing of this Agreement and which are beyond the reasonable control of such party ("Force Majeure Event"), such as a strike, blockade, war, act of terrorism, riot, natural disaster, failure or diminishment of power or telecommunications or data networks or services, or refusal of a license by a government agency. In the event of a Force Majeure Event that prevents one part from substantially performing its obligations hereunder for a period of ten (10) days or more, either party may terminate this Agreement on five (5) days written notice.

  4. Grant of Rights

    1. The Product include code and libraries licensed to Licensor by third parties, including open source software.

    2. The Product is provided basing on the CPUs. If Customer complies with the terms of this Agreement, Customer has the rights stipulated hereunder for each Subscription that Customer acquires. Customer’s rights acquired in relation to the Product are limited to those necessary to enable Customer and its Users to effectively operate the Product(s). All other rights remain reserved to Licensor.

    3. Unless the Subscription has expired or this Agreement is terminated in accordance with respective section, and subject to the terms and conditions specified herein, Licensor grants Customer a non-exclusive and non-transferable right to use each Product covered by the Subscription as stipulated below.

    4. Customer may:

      1. For each License Key included to Subscription have one Product Installation of any version covered by the Subscription on any operating system supported by the Product.

      2. Do Product Evaluation on one Product Installation of any version on any operating system supported by the Product.

      3. Make one backup copy of the Product solely for archival/security backup purposes.

    5. Customer may not:

      1. Allow the same Product Installation to be used concurrently by more than the Number of CPUs specified for used License Key in Subscription Confirmation.

      2. Rent, lease, reproduce, modify, adapt, create derivative works of, distribute, sell, or transfer the Product.

      3. Provide access to the Product or the right to use the Product to a third party.

      4. Reverse engineer, decompile, disassemble, modify, translate, make any attempt to discover the source code of the Product.

      5. Remove or obscure any proprietary or other notices contained in the Product.

    6. Customer acknowledges that no ownership right is conveyed to Customer under this Agreement, irrespective of the use of terms such as "purchase" or "sale". Licensor has and retains all rights, title and interest, including all intellectual property rights, in and to the Products and any and all related or underlying technology, and any modifications or derivative works thereof, including without limitation as they may incorporate Feedback (as defined below).

    7. This Agreement applies whether Customer purchases a Subscription directly from Licensor or through resellers. If Customer purchases through a reseller, the Subscription details shall be as stated in the Subscription Confirmation issued by the reseller to Customer, and the reseller is responsible for the accuracy of any such Subscription Confirmation. Resellers are not authorized to make any promises or commitments on Licensor behalf, and Customer understands and agrees that Licensor is not bound by any obligations to Customer other than as specified in this Agreement.

  5. Access to Products

    1. All deliveries under this Agreement will be electronic. Customer and its Users must have an Internet connection in order to receive any deliveries. For the avoidance of doubt, Customer is responsible for downloading and installing the Products. Download instructions are made available on Licensor website at https://aerokube.com/boot/.

    2. Customer enables full access to Product Installation by specifying a License Key from Subscription Confirmation.

    3. Subject to the terms of this Agreement, Customer is granted a right to install and use the Product for evaluation purposes without charge for unlimited amount of time. The Product contains a feature that will automatically limit allowed Number of CPUs. Licensor reserves the right at any time to change that limit in new Product versions.

  6. Fees

    1. Customer shall pay its Subscription fees in accordance with Licensor Terms of Purchase or the reseller’s terms of purchase, whichever are applicable.

    2. The Subscription fees shall be paid in full, and any levies, duties and/or taxes imposed by Customer’s jurisdiction (including, but not limited to, value added tax, sales tax and withholding tax), shall be borne solely by Customer.

    3. Customer may not deduct any amounts from fees payable to Licensor or the reseller, unless otherwise specified in the applicable terms of purchase.

  7. Feedback

    1. Customer has no obligation to provide Licensor with ideas, suggestions, or proposals ("Feedback").

    2. If Customer or Users submit Feedback to Licensor, then Customer grants Licensor a non-exclusive, worldwide, royalty-free license that is sub-licensable and transferable, to make, use, sell, have made, offer to sell, import, reproduce, publicly display, distribute, modify, or publicly perform the Feedback in any manner without any obligation, royalty or restriction based on intellectual property rights or otherwise.

  8. LIMITED WARRANTY

    ALL PRODUCTS ARE PROVIDED TO CUSTOMER ON AN "AS IS" AND "AS AVAILABLE" BASIS WITHOUT WARRANTIES. USE OF THE PRODUCTS IS AT YOUR OWN RISK. AEROKUBE MAKES NO WARRANTY AS TO THEIR USE OR PERFORMANCE. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, AEROKUBE, AND ITS SUPPLIERS (WHICH SHALL INCLUDE THE PROVIDERS OF THE THIRD PARTY SOFTWARE) AND RESELLERS, DISCLAIM ALL WARRANTIES AND CONDITIONS, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, AND NON-INFRINGEMENT, WITH REGARD TO THE PRODUCTS, AND THE PROVISION OF OR FAILURE TO PROVIDE SUPPORT SERVICES. THIS LIMITED WARRANTY GIVES CUSTOMER SPECIFIC LEGAL RIGHTS. CUSTOMER MAY HAVE OTHER RIGHTS, WHICH VARY FROM STATE/JURISDICTION TO STATE/JURISDICTION. AEROKUBE (AND ITS AFFILIATES, AGENTS, DIRECTORS AND EMPLOYEES) DOES NOT WARRANT:

    1. THAT THE PRODUCTS ARE ACCURATE, RELIABLE OR CORRECT

    2. THAT THE PRODUCTS WILL MEET YOUR REQUIREMENTS

    3. THAT THE PRODUCTS WILL BE AVAILABLE AT ANY PARTICULAR TIME OR LOCATION, UNINTERRUPTED OR SECURE

    4. THAT ANY DEFECTS OR ERRORS WILL BE CORRECTED

    5. THAT THE PRODUCTS ARE FREE OF VIRUSES OR OTHER HARMFUL COMPONENTS

    ANY CONTENT OR DATA DOWNLOADED OR OTHERWISE OBTAINED THROUGH THE USE OF THE PRODUCTS ARE DOWNLOADED AT YOUR OWN RISK AND YOU WILL BE SOLELY RESPONSIBLE FOR ANY DAMAGE TO YOUR PROPERTY OR LOSS OF DATA THAT RESULTS FROM SUCH DOWNLOAD. NO WARRANTY OR LIABILITY AT ALL IS GIVEN TO PRODUCTS UNDER EVALUATION.

  9. DISCLAIMER OF DAMAGES

    1. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT WILL AEROKUBE (OR ITS AFFILIATES, AGENTS, DIRECTORS, OR EMPLOYEES), OR AEROKUBE LICENSORS, SUPPLIERS OR RESELLERS BE LIABLE TO CUSTOMER OR ANYONE ELSE FOR:

      1. ANY LOSS OF USE, DATA, GOODWILL, OR PROFITS, WHETHER OR NOT FORESEEABLE

      2. ANY LOSS OR DAMAGES IN CONNECTION WITH TERMINATION OR SUSPENSION OF CUSTOMER’S ACCESS TO OUR PRODUCTS IN ACCORDANCE WITH THIS AGREEMENT

      3. ANY SPECIAL, INCIDENTAL, INDIRECT, CONSEQUENTIAL, EXEMPLARY OR PUNITIVE DAMAGES WHATSOEVER (EVEN IF WE HAVE BEEN ADVISED OF THE POSSIBILITY OF THESE DAMAGES), INCLUDING THOSE:

        1. RESULTING FROM LOSS OF USE, DATA, OR PROFITS, WHETHER OR NOT FORESEEABLE

        2. BASED ON ANY THEORY OF LIABILITY, INCLUDING BREACH OF CONTRACT OR WARRANTY, STRICT LIABILITY, NEGLIGENCE OR OTHER TORTIOUS ACTION

        3. ARISING FROM ANY OTHER CLAIM ARISING OUT OF OR IN CONNECTION WITH CUSTOMER’S USE OF OR ACCESS TO THE PRODUCTS OR SUPPORT.

    2. THE FOREGOING LIMITATION OF LIABILITY SHALL APPLY TO THE FULLEST EXTENT PERMITTED BY LAW IN THE APPLICABLE JURISDICTION.

    3. THE TOTAL LIABILITY IN ANY MATTER ARISING OUT OF OR IN RELATION TO THIS AGREEMENT IS LIMITED TO ONE HUNDRED (100) US DOLLARS OR THE AGGREGATE AMOUNT PAID OR PAYABLE BY THE CUSTOMER FOR PRODUCTS DURING THE THREE-MONTH PERIOD PRECEDING THE EVENT GIVING RISE TO THE LIABILITY, WHICHEVER IS GREATER. THIS LIMITATION WILL APPLY EVEN IF WE OR YOU HAVE BEEN ADVISED OF THE POSSIBILITY OF LIABILITY EXCEEDING SUCH AMOUNT AND NOTWITHSTANDING ANY FAILURE OF THE ESSENTIAL PURPOSE OF ANY LIMITED REMEDY.

  10. Term and Termination

    1. The term of this Agreement will commence upon acceptance of this Agreement by Customer as set forth in the preamble above, and will continue for each Product through the end of the applicable subscription period specified in the respective Subscription Confirmation. This Agreement will automatically renew with respect to each Product for a successive subscription term, unless terminated as set forth herein.

    2. Customer may terminate this Agreement at any time by cancelling its Product subscription. If such termination occurs during a then-current subscription period, this Agreement will continue to be effective until the end of that subscription period. Such termination does not relieve Customer of the obligation to pay any outstanding subscription fees owed to Licensor, and no credits or refunds will be issued to Customer for prepaid subscription fees (except as specified in the Licensor Terms of Purchase, if applicable).

    3. Licensor may terminate this agreement if:

      1. Customer has materially breached this Agreement and fails to cure such breach within thirty (30) days of written notice thereof.

      2. Customer fails to make the timely payment of subscription fees in accordance with "Fees" Section of this Agreement.

      3. Licensor is required to do so by law (for example, where the provision of the Product to Customer is, or becomes, unlawful).

      4. Licensor elects to discontinue providing the Product, in whole or in part.

    4. Licensor will make reasonable efforts to notify Customer via email as follows:

      1. Thirty (30) days prior to termination of the Agreement when required to terminate by law or because of discontinued Product. In such events Customer will be entitled to a refund of the unused portion of prepaid subscription fees, if applicable.

      2. Three (3) days prior to termination of the Agreement in other cases. In such events Customer will not be entitled to any refund of the unused portion of prepaid subscription fees.

  11. Temporary Suspension for Non-payment

    1. Licensor reserves the right to suspend or limit Customer’s access to Aerokube Products if Customer fails to pay subscription fees on time.

    2. If Licensor suspends or limits Customer’s access to Aerokube Products for non-payment according, Customer must pay all past due amounts in order to restore full access to Aerokube Products.

    3. Customer hereby agrees that Licensor is entitled to charge Customer for the time period during which Customer has access to Aerokube Products until Customer or Licensor terminates or suspends Customer’s subscription in accordance with this Agreement.

  12. Export Regulations

    Customer shall comply with all applicable laws and regulations with regards to economic sanctions, export controls, import regulations, and trade embargoes (all herein referred to as "Sanctions"), including those of the European Union and United States (specifically the Export Administration Regulations (EAR)). Customer declares that it is not a person targeted by Sanctions nor is it otherwise owned or controlled by or acting on behalf of any person targeted by Sanctions. Further, Customer warrants that it will not download or otherwise export or re-export the Product or any related technical data directly or indirectly to any person targeted by Sanctions or download or otherwise use the Product for any end-use prohibited or restricted by Sanctions.

  13. Customer Support

    1. Licensor provides Email Support as well as Instant Messaging Support. The response time will be reasonable, but no specific response time guarantees are given.

    2. Customer may request additional paid support from Licensor which is subject of a supplementary individually negotiated Agreement between Customer and Licensor.

    3. Any guarantees of support availability only apply to the latest version of Licensed Software available in Customer Subscription.

  14. Customer Data

    1. Use of Name and Logo. Customer agrees that Licensor may identify it as a customer of Aerokube and may refer to it by name, trade name and trademark, if applicable. Licensor may also briefly describe Customer’s business in Licensor marketing materials, on the Aerokube website and/or in public or legal documents. Customer hereby grants Licensor a worldwide, non-exclusive and royalty-free license to use Customer’s name and any of Customer’s trade names and trademarks solely pursuant to this marketing section. Notwithstanding anything to the contrary herein, Licensor acknowledges that in some cases Customer licenses and does not own marks or logos (for example, marks or logos of the Affiliates) and cannot permit Licensor to use such marks.

    2. Gathering of Usage Statistics. Customer acknowledges and agrees that the Product may contain a feature that reports the usage statistics, diagnostics information and usage meta-information of the Product back to the Licensor. Customer may opt out of the gathering of usage statistics by turning off this feature in the Product settings.

5. Pricing

Last updated August 1st, 2023. Replaces the prior version in its entirety.

  1. Boot price is based on the total number of CPUs you can use for your virtual machines. We control this by limiting total number of CPUs all running browser pods can consume in all namespaces.

  2. When no license key is provided 1 (one) CPU maximum is allowed. If such limit is sufficient for you - you are allowed use Boot without license key for unlimited period of time.

  3. If free limit is insufficient - you need a paid license. Such license can include any desired number of CPUs.

  4. Every CPU has a fixed cost - $5 USD (five United States dollars). If you are an EU-based company - then we convert the price to euro (€).

    An example price calculation
    42 CPUs * $5/month = $210/month

  5. For simplicity, we calculated monthly prices for some frequent cases:

    Table 4. Boot License Pricing
    Number of CPUs Price per Month, USD

    1

    free

    5

    $25

    10

    $50

    15

    $75

    20

    $100

    25

    $125

    30

    $150

    40

    $200

    50

    $250

    75

    $375

    100

    $500

    150

    $750

    200

    $1000

    250

    $1250

    500

    $2500

    750

    $3750

    1000

    $5000