This reference for version: latest

Moon is a browser automation solution compatible with Selenium, Cypress, Playwright and Puppeteer using Kubernetes or Openshift to launch browsers.

1. Getting Started

1.1. Quick Start Guide

This section shows how to install Moon limited to 4 parallel browser sessions. Detailed information on installing a license key allowing to activate more parallel sessions is shown in Installing License section.

1.1.1. Installing to Kubernetes

Option 1: you have a running Kubernetes cluster
  1. Prerequisites:

    • Running Kubernetes cluster

    • kubectl client installed and pointing to the cluster

      1. If you are running Kubernetes cluster on virtual machines, we usually recommend having bigger VMs instead of smaller ones. This allows to avoid available CPUs and memory fragmentation issues. For example having 24 CPU cores overall it is better to start 3 x 8 CPU core VMs instead of 12 x 2 CPU core.

      2. If you are starting Moon in Kubernetes cluster deployed on workstation with minikube tool - see Option 3: you have Minikube.

  2. Clone an example configuration repository:

    $ git clone https://github.com/aerokube/moon-deploy.git
    $ cd moon-deploy
  3. Run one command to start Moon and Moon API:

    $ kubectl apply -f moon.yaml

    If you have only several CPUs available in your cluster - use moon-local.yaml file instead. By default Moon is started in a separate namespace called moon so we append -n moon to the next commands.

  4. Wait for LoadBalancer IP to allocate:

    $ kubectl get svc -n moon
    NAME      TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
    browsers  ClusterIP      None            <none>        <none>                          51s
    moon      LoadBalancer   10.63.242.109   <pending>     4444:31894/TCP,8080:30625/TCP   17s

    It will look like this when finished:

    $ kubectl get svc -n moon
    NAME      TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                         AGE
    browsers  ClusterIP      None            <none>           <none>                          51s
    moon      LoadBalancer   10.63.242.109   104.154.161.58   4444:31894/TCP,8080:30625/TCP   1m

    Now you can optionally point a domain name to this IP address:

    $ host moon.example.com
    moon.example.com has address 104.154.161.58
  5. Run your tests against moon like you do with regular Selenium:

    http://104.154.161.58:4444/wd/hub
    When running your tests for the first time for every browser version Kubernetes will download and cache requested browser image. This can take several minutes depending on your network connection.

    Same address using domain name:

    http://moon.example.com:4444/wd/hub
  6. Moon web interface is available at:

    http://104.154.161.58:8080/
Option 2: you are using Helm
Steps below require Helm 3 and will not work with older releases.

We deliver already packed and published Helm charts, so installing Moon with Helm is straightforward:

  1. Add Aerokube charts repository:

    $ helm repo add aerokube https://charts.aerokube.com/
    $ helm repo update
  2. To list available Moon versions type:

    $ helm search repo aerokube --versions
  3. Create a namespace:

    $ kubectl create namespace moon
  4. To install or upgrade Moon type:

    $ helm upgrade --install -n moon moon aerokube/moon
  5. Moon chart has a lot of various configuration parameters that can be listed as follows:

    $ helm show values aerokube/moon

    To change one of these parameters - use --set flag:

    $ helm upgrade --install --set=moon.enabled.resources=false -n moon moon aerokube/moon
Option 3: you have Minikube
  1. Clone an example configuration repository:

    $ git clone https://github.com/aerokube/moon-deploy.git
    $ cd moon-deploy
  2. Run one command to start Moon and Moon API:

    $ minikube kubectl -- apply -f moon-local.yaml
  3. Minikube does not automatically assign IP address to LoadBalancer, so let’s do this manually:

    $ minikube kubectl -- patch svc moon -n moon --patch "{\"spec\":{\"externalIPs\":[\"$(minikube ip)\"]}}"
  4. The last step is to get Selenium URL:

    $ minikube ip
    192.168.99.100

    Use the following as Selenium URL:

    http://192.168.99.100:4444/wd/hub

    Moon web interface is available at:

    http://192.168.99.100:8080/

    When Minikube is using Docker driver to access Moon you can use minikube tunnel command instead:

    1. In a separate terminal run command to start Minikube tunnel:

      $ minikube tunnel
    2. Use 127.0.0.1 to access Moon:

      http://127.0.0.1:4444/wd/hub
      http://127.0.0.1:8080/

1.1.2. Installing to Openshift

  1. Prerequisites:

    • Running Openshift cluster

    • oc client installed and pointing to the cluster

  2. Clone an example configuration repository:

    $ git clone https://github.com/aerokube/moon-deploy.git
    $ cd moon-deploy
  3. Add edit policy to default account so Moon could read project pod quota:

    $ oc policy add-role-to-user edit system:serviceaccount:moon:default

    Here we assume that Openshift project for Moon is called moon.

  4. Start Moon with Openshift UI or oc client using moon-openshift.yaml file:

    $ oc create -f moon-openshift.yaml -n moon
  5. Run Selenium tests against moon service listening on port 4444:

    http://<moon-ip-or-hostname>:4444/wd/hub

1.1.3. Moon Checklist

Although Moon installation is as simple as deploying one YAML manifest - there are some well-known issues you may encounter while running your tests. The following checklist will help you to have everything running smoothly from the beginning:

  1. Make sure you turned off DNS cache for Moon namespace as described in Browser pods are freezing section.

  2. For Moon 1.4.0+ make sure you have a required Kubernetes service described in Sessions are not starting section.

  3. While using Moon for the first time - make sure you are not suffering from various Kubernetes issues described in No space left on device and DNS lookup timeouts sections as well as some Github issues: kubernetes/kops#5916.

  4. In case of customized Moon installation (e.g. when changing Moon service name from default moon to something else) make sure you are setting -moon-url parameter correctly as described in Browser session timeouts do not work section.

  • Use the biggest possible cluster node sizes. For example having 100 CPUs overall it is better to launch 5 nodes with 20 CPUs each than 50 nodes with 2 CPUs each. Browser pods can in some cases require more than 2 CPUs and this can lead to preliminary cluster fragmentation.

  • Avoid cluster nodes with RedHat \ CentOS if possible. Nodes using these distributions are known to suffer from issues related to firewall \ SeLinux and can be more complicated to configure correctly.

  • Use Calico container network interface instead of Flannel if possible. Calico has better performance than Flannel especially on big clusters.

  • Turn off DNS cache at least for Moon namespace. Being a completely stateless solution, Moon significantly depends on cluster DNS stability. One of the known Kubernetes issues is occasional DNS inconsistency - when somew browser pods are not accessible using domain name (it does not resolve). Such inconsistency can lead to frozen browser pods and cluster exhaustion. Possible way to turn off DNS cache is described in Browser pods are freezing section.

  • Use more than 1 Kubernetes API replica if needed. Moon is using Kubernetes API to create and delete browser pods. If you plan to run hundreds of browsers in parallel - take a look at Kubernetes API (Kubernetes master) host system metrics. Overloaded master can stop responding to requests properly and this can lead to frozen browser pods.

1.3. Required Permissions

Moon requires very limited set of permissions and should work with default Kubernetes settings. The following table summarizes what needs to be accessible.

Table 1. Required Moon Permissions

Permission

Purpose

To start and stop pods

Used to start pods with browsers

To start and stop services

Used to provide network connectivity to browser pods

To read data from config maps

Used to share various configuration data among Moon replicas: S3 configuration, custom images configuration and so on

To read data from secrets

Used to share S3 credentials among Moon replicas

To read resource quota information

Used to verify that Moon can run no more than total number of browser pods allowed by license

To create and update resource quota information

Optional. Used only in case when no resource quota was configured

To create and delete secrets

Optional. Used only in case when no resource quota was configured

To create and delete configmaps

Optional. Used only for Cypress tests.

All these permissions are normally enabled by default when running Moon and its browser pods in the same Kubernetes namespace. To configure a namespace for browser pods - use -namespace flag for Moon and Moon API. If you are starting Moon in one namespace and specify -namespace flag pointing to another namespace, then you may require additional Kubernetes permissions to be enabled.

1.4. Cluster Architecture

1.4.1. Moon Components

Moon Cluster Architecture

architecture

Moon cluster consists of several important components:

  1. One or more Moon application instances. Their main purpose is to start and stop browser containers. These replicas are usually exposed as Kubernetes service available on standard Selenium port 4444. You should run all the tests against this service.

  2. One or more Moon API application instances. This API collects and returns various data about running browsers. Moon API can be optionally exposed as Kubernetes service available on HTTP port 8888. By default it is only accessible inside Kubernetes cluster.

  3. One or more Moon UI application instances. Moon UI collects information from Moon API and visualizes it. It is usually available on HTTP port 8080.

  4. Running browser pods. Moon is using exactly the same browser images as Selenoid.

1.4.2. Why Moon is Stateless

Moon is completely stateless and does not require any external database to run browser sessions.

Why Moon is Stateless

moon-stateless

When user requests a new browser session - a Kubernetes pod containing desired browser version is created. Moon also creates a Kubernetes service (with generated name like chrome-71-0-<UUID>) to deliver network connectivity to browser pod. Having a running pod and service - a standard new Selenium session request is sent to it. After receiving unique session identifier (so called Selenium session ID) Moon replaces it with the generated Kubernetes service name and returns the response to user. Subsequent Selenium requests are just proxied to respective service by using Selenium session ID as service name.

1.4.3. Browser Pod Contents

In addition to container with browser every pod created by Moon contains one or more service images.

Table 2. Service Images
Name Purpose Started

defender

Allows only one browser session to be created in the pod, handles session timeouts

Always

logger

Uploads sessions logs to S3 storage

When S3 storage is configured

video-recorder

Captures running browser session video and uploads it to S3 storage

When S3 storage is configured and video recording is requested by test

vnc-server

Delivers VNC connectivity to browser images

x-server

1.5. Moon vs Selenoid

Moon takes all the best practices and features from Selenoid and adds many more:

  1. Unlimited automatic scalability. You always have enough browsers of any desired version available in the cluster. When running the cluster in cloud platforms such as Google Cloud you can adjust settings to automatically scale depending on current load. This allows to combine efficiency with competitive cost.

  2. Completely stateless. Selenoid stores in memory information about currently running browser sessions. If for some reason its process crashes - then all running sessions are lost. Moon contrarily has no internal state and can be replicated across datacenters. Browser sessions remain alive even if one or more replicas go down.

  3. Uniform load distribution. Load balancers like Ggr are using random load distribution across available browser hosts. This makes them inefficient when overall load is above 80% of cluster capacity. Moon delivers exactly uniform distribution working good even under 100% load.

  4. Fully graceful. Any maintenance operations with the cluster do not interrupt running browser sessions. Every cluster component shuts down gracefully.

2. Configuration

2.1. Users List

Moon is a multi-user application allowing different users access different browser versions. A single htpasswd file is used to store users information. This file is saved to Kubernetes Secret and mounted to Moon container as a volume. A typical users file is a text file with user names and their encrypted passwords separated by colon:

$ cat users.htpasswd
test:$apr1$.dZyHlKN$jdoZkin/kPviFNArx/cVL1 # User is test, password is encrypted
alice:$apr1$mLYJAC4y$VYeJstWjWP/4iVlH/TNcD.
bob:$apr1$gyqzbSpt$RBNcxrsQaolPZCQZW0VQW1

2.1.1. Updating Users List

To add or remove users:

  1. Modify users.htpasswd with htpasswd command:

    $ htpasswd -Bbn new-user new-user-password >> users.htpasswd # Adding new user
    $ htpasswd -Bb users.htpasswd some-user new-password # Updating password
    $ htpasswd -D users.htpasswd test-user # Deleting existing user
  2. Update users secret:

    $ kubectl replace secret users --filename=./users.htpasswd -n moon
  3. Changes are applied immediately without Moon restart.

2.2. Browsers List

Moon is using a simple JSON-based configuration to store available browsers list. A typical browsers list file looks like the following:

Typical Browsers List File
{
  "firefox": {                                      (1)
    "default": "62.0",                              (2)
    "versions": {                                   (3)
      "62.0": {                                     (4)
        "image": "selenoid/firefox:62.0",           (5)
        "port": "4444",                             (6)
        "path": "/wd/hub",                          (7)
        "resources": {                              (8)
          "limits": {
            "cpu": "2",
            "memory": "2Gi"
          },
          "requests": {
            "cpu": "200m",
            "memory": "1Gi"
          }
        },
        "privileged": true,                         (9)
        "nodeSelector": {                           (10)
          "node-type": "hardware"
        },
        "env": ["TZ=Europe/Moscow", "LANG=ru"],     (11)
        "hosts": ["example.com:192.168.0.1"],       (12)
      },
      "60.0": {
        //...
      }
    }
  },
  "chrome": {
    //...
  },
  "opera": {
    "default": "56.0",
    "versions": {
        //...
    }
  }
}
1 Browser name
2 Default browser version
3 A list of available browser versions
4 Version name
5 Image name
6 Port to proxy connections to. In the majority of cases should be 4444.
7 Path relative to / where we request a new session, see below
8 Resource configuration for browser container (CPU and memory)
9 Whether to run browser pod in privileged mode (default is false)
10 Kubernetes node selector (allows to run pods on particular hosts only)
11 Environment variables passed to browser container
12 Custom /etc/hosts entries to be passed to browser container in hostname:ip format

2.2.1. Per-user Browser Lists

As you already know Moon is a multi-user application. For every user you need to create one file - <username>.json. For example for user alice from users.htpasswd you should create alice.json. All JSON files should be stored in the same directory specified by -quota-dir flag:

Per-user browser lists
\---quota
    |---- alice.json
    |---- browsers.json
    |---- bob.json
    |---- test.json

Such directory is then uploaded to Kubernetes ConfigMap and mounted to Moon container as a volume.

2.2.2. Guest Quota

In some cases you may need anonymous Selenium access - without username and password. In terms of Moon this is called guest quota and is configured with -guest-user flag (default value is browsers). Any browser versions specified in <guest-user>.json file will be available without username and password:

Guest quota file name
-guest-user browsers ====> browsers.json # This is the default
-guest-user guest-user ====> guest-user.json

2.2.3. Updating Browsers List

To add or remove browsers:

  1. Having configuration files stored in quota directory apply desired modifications.

  2. Update quota ConfigMap using contents of quota directory:

    $ kubectl replace configmap quota --filename=quota -n moon
  3. Changes are applied immediately without Moon restart. All running user sessions will continue to work without any interruption.

2.2.4. Using External Selenium

Moon expects to run the majority of browsers in pods inside Kubernetes or Openshift cluster. However sometimes you may need to run Selenium tests on some external hosts: hardware servers or virtual machines. Mainly this could be needed in two situations:

  1. Running Selenium tests on some complicated platforms such as MacOS or iOS. According to license agreement these platforms require Apple hardware devices and it is complicated to run Kubernetes on top of these devices.

  2. Using Selenium online platforms (e.g. Aerokube Browsers) for some browsers. In that case you can run the majority of browsers (e.g. Firefox, Chrome, Opera) in Moon and complicated browsers (don’t work on standard virtual machines) such as Chrome Mobile in external Selenium platform.

Using External Hosts

For this case you should have the following:

  1. A set of hosts with Selenium-compatible solution (Selenoid, Appium, Selenium Grid, etc.): host1.example.com:4444, host1.example.com:4444 and so on.

  2. A VNC server listening on every such host on standard port 5900.

    Every VNC server should be password protected with the same password having 8+ characters.

For every browser version supported on the hosts you need to add the following to browsers list file:

Typical Browsers List File
"safari": {
  "default": "13.0",
  "versions": {
    "13.0": {
      "image": "aerokube/moon-external-host:1.0.0",                                         (1)
      "port": "4444",
      "path": "/",
      "env": [
        "URLS=[\\\"http://host1.example.com:4444/\\\", \\\"http://host2.example.com:4444/\\\"]",    (2)
        "VNC_PASSWORD=myvncpassword", // At least 8 symbols                                 (3)
        "SCREEN_RESOLUTION=1921x1080x24" // This is optional                                (4)
      ]
    }
  }
}
1 Special proxy image for external hosts
2 A list of external hosts
3 VNC password value for the hosts
4 Optional. Screen resolution of remote host to use for VNC.

With such configuration Selenium session requests with be randomly load-balanced across the hosts specified in URLS environment variable. VNC feature should also work - you should be seeing remote host screen in Moon UI.

2.3. Configuration File

Moon stores all advanced configuration options (e.g. S3 settings) in a special JSON file called service.json:

Typical service.json file
{
  "s3": {                                          (1)
    "endpoint": "https://storage.googleapis.com",  (2)
    "bucketName": "moon-test",                     (3)
    "version": "S3v2",                             (4)
    "keyPattern": "$quota/$date"                   (5)
  },
  "images": {                                      (6)
    "logger": {                                    (7)
      "image": "my-reg.com/moon/logger:1.2.0",     (8)
      "resources": {                               (9)
        "limits": {                                (10)
          "cpu": "0.3",                            (11)
          "memory": "1024Mi"                       (12)
        },
        "requests": {                              (13)
          "cpu": "0.3",                            (14)
          "memory": "1024Mi"                       (15)
        }
      }
    }
  },
  "annotations": {                                 (16)
    "key1": "value1",                              (17)
    "key2": "value2"
  },
  "labels": {                                      (18)
    "key1": "value1",                              (19)
    "key2": "value2"
  },
  "runAs": {                                       (20)
    "user": 1000,                                  (21)
    "group": 2000                                  (22)
  }
}
1 S3 configuration section
2 S3 endpoint URL
3 S3 bucket name
4 S3 signature version (should be S3v2 or S3v4)
5 S3 key pattern
6 Custom Moon system images section (to use Moon with private Docker registries)
7 Custom Moon image definition (one of logger, defender, videoRecorder)
8 Custom Moon image reference
9 Custom Moon image resources definition
10 Custom Moon image limits section
11 Custom Moon image CPU limit
12 Custom Moon image memory limit
13 Custom Moon image requests section
14 Custom Moon image CPU request
15 Custom Moon image memory request
16 Custom pod annotations section (applies globally to all browser versions)
17 Custom pod annotation key and value
18 Custom pod labels section (applies globally to all browser versions)
19 Custom pod label key and value
20 Custom pod user and group id section
21 Custom pod user id
22 Custom pod group id

service.json is mounted to Moon container as a regular file using Kubernetes config map. Path to configuration file is specified with -config-file Moon flag. Any changes to configuration file are applied immediately - no need to restart Moon.

Adding service.json file to Moon
apiVersion: v1
kind: ConfigMap
metadata:
  name: config
  namespace: moon
data:
  service.json: |
    {
      "s3": {
        "endpoint": "https://storage.googleapis.com",
        "bucketName": "moon-test",
        "version": "S3v2"
      }
    }
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: moon
  namespace: moon
spec:
  template:
    metadata:
      labels:
        app: moon
    spec:
      containers:
      - name: moon
        image: aerokube/moon:latest-release
        args: ["-config-file", "/config/service.json"]
        volumeMounts:
        - name: config
          mountPath: /config
          readOnly: true
    volumes:
      - name: config
        configMap:
          name: config

2.4. Devices File

Moon load information about available mobile devices for Mobile Emulation from a special file usually called devices.json:

Typical devices.json file
{
    "Google Nexus 4": { (1)
      "width": 384, (2)
      "height": 640, (3)
      "pixelRatio": 2, (4)
      "userAgent": "user-agent-string-for-chrome-%s", (5)
      "printVersion": true (6)
    }
}
1 Mobile device name
2 Device screen width
3 Device screen height
4 Device pixel ratio
5 Device user agent
6 Whether to substitute Chrome version to user agent string (%s placeholder is replaced by Chrome version)

devices.json similarly to service.json is mounted to Moon container as a regular file using Kubernetes config map. Path to configuration file is specified with -device-file Moon flag. Any changes to configuration file are applied immediately - no need to restart Moon.

2.5. Credentials Secret

Moon reads all credentials (e.g. S3 keys) from an optional Kubernetes secret called credentials. This secret is mounted to Moon container as a regular directory with distinct files corresponding to different keys. Path to credentials directory is specified with -credentials-dir Moon flag. Any changes to credentials secret are applied immediately - no need to restart Moon.

Adding credentials secret to Moon
apiVersion: v1
kind: Secret
metadata:
  name: credentials
  namespace: moon
stringData:
  s3.accessKey: "access-key-value"
  s3.secretKey: "secret-key-value"
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: moon
  namespace: moon
spec:
  template:
    metadata:
      labels:
        app: moon
    spec:
      containers:
      - name: moon
        image: aerokube/moon:latest-release
        args: ["-credentials-dir", "/credentials"]
        volumeMounts:
        - name: credentials
          mountPath: /credentials
          readOnly: true
    volumes:
    - name: credentials
      secret:
        secretName: credentials

The following keys are supported:

Table 3. Supported Credentials Secret Keys
Key Meaning

s3.accessKey

S3 access key value

s3.secretKey

S3 secret key value

2.6. Installing License

  1. According to license agreement you can use up to 4 parallel sessions without a license. If you wish to have more parallel sessions - order a license and follow this section steps to install the license. A trial license key with more parallel sessions can be generated on Moon website.

  2. We recommend installing Moon to a clean namespace because pods already running in an existing namespace will be by default considered as browser pods and thus will consume your license.

To install a license you should have a license key file. A typical license file looks like this:

$ cat license.key
Ti90a3ljSWpvV3hicU5ZY3U3ZVNMUzdzalVjU29nSVU2a3lmWDdENmhOSEs4SFJVVE5LNEpKL2oxRGhWTnRhZVBEQmx5cm1SM0dFWlVpbGIwOWhBSUpvWmlQMnNtYmtoZVRDbnJ3UTlIV3grMUMzaXFJalU3L0NIZDJuSzN3QUpIS0lyV2VoVjlRVUh5NzNCMFg0NEV5KzFSVXg1elhvaC9RUkNjTlBIOHR4WWtrRFBmTFhxYTFESlI0aXNObGtobWpXY2VJNVdna1lsWHFFOUp0OUxValZPN05hb2pCcHJzY0pvZjBrZys2YkhWUDR3cUJvYVFoSFJTYzMzZzNRSTkrWVd4dlZCeExmdTVxM2k2ZnluMDRHSEhGc21Fd2owVHdVa1Z3MmI5Z2FyVXlaVFF4RDdCRjVSVjBmSUVrU1pEQ0ZQdmIvMHZoKythY1V4OVRrR2FBREZzbWR3dExxRlU4NHF3MzBVdE5nTGhnNWYxRXRCQ215R1BHR0wyK2NpR0RDZDBTSXg4K1RsRGNWVzUyVVY4MUt6bkdOanJhTklOeXVtYjVPVXF6aVVpdmJZeVJNVVBSMWF3NUJRVGJDaHVycWVVdGtMMWFRZDgwOCtoWGRxODFOakxRNEJlVENNQTdPL2FneHBTUE1nL0J2QmVQKy84ZGhJeXd5Y0lxQVJhVWxocUdmUko2K05tbkFYV0hzZ2RjTng1dUVHU0w5VWF5TU1rR3RkZEZKQ1FiWmF0RnJMaDFHbHhKTndBRHFWcXIvQ0oxSzBQL2p6K2NFSWpsSG82TktvM3pudVFlanhYT2tGU1p1bWZrS1krZmF3VkVRZWlvcFlmZFk0TU9tc3U5TThsbng3T2VXQXZmOTFUQ2w1NUhBK1ZsQTMzN3VFOG1WV3ZlS0E1Tlk9O2V5SnNhV05sYm5ObFpTSTZJa0ZqYldVZ1RFeERJaXdpY0hKdlpIVmpkQ0k2SWsxdmIyNGlMQ0p0WVhoVFpYTnphVzl1Y3lJNk1qQXNJblJ2SWpveE5UTTRNelV5TURBd2ZRPT0=

2.6.1. Option 1: you are using official YAML manifests

Our official YAML manifests already contain everything needed to install a license key and by default have the following definition:

License key secret definition in YAML manifest
apiVersion: v1
kind: Secret
metadata:
  name: licensekey
  namespace: moon
stringData:
  license.key: MG1RSVdpc2Z6YjdQQVZjd2lpei9KMkd1T3dzMTFuL1dlRjVSc3NOMUcxZk9QaUxWa3Q5SnBIakIxa09wWm0vVFJqQ0tsa21xVG1OODVRZnlQbjBjVmRHVWFLampTOFF1a3VLRXRPcEUwbnEySG16QWFQWHRDYTVjMm9jZzZFaUJqeFd5ODE4UFBHZzNCNWpCYXlha3oweFBscFl1RnB0V0U1Q3FwOGl5VDdKTk9abG5aSmlPdnRmZDFvSG1nNnVwVXBLV2E4RmYwWHcreERIR29ZTE1XTldPb1hvT2ZCUnZpcDhPWW05a1FqN0hBWWVOYUtLT1lPWlVJa1dsb1gxdjNOT1htTFpZalhsQ3h1Q3V6NWhiQjIwSjVIY0JTYnZybm9zYm14RXFkSFpQWVBKWUlKTzZvVlBnODhQeFErZ1EyTk5sWG82TC9XeXU3aisrNU0rSEdPcXlOSEdlNGx4Zm1nNVhjMWlnNkN1OCtNSVVYRzNqUllqOUY4ZHdReWpSbFNMNmFpL2dRQnc3TzY0U0lwdVF2d29jYi9kVzFSYWFRVkd3ZXYrOVdING8zRWRrYkVONUhRTmQ2MUxsUnFNdmtKeWVHV21tVlVUZ2dsMDRsTFFLTmZNVG81L2JVakNBMGhNeER5VHNJdmVRRGFMMklvTWpvcFk4VERlK1U2bUJvUDVxNVYrcCtDQVhjbjYxQlRaUVp0bmNqL0JBVkdNOEZ4NW9rWHRYSVAxUkY0a1VCckZVTDFyTWF1VkZqSk5xU1pLT293dUpMTTg2SEZ0Sld0eUlRK3ZZZm1pZU0xM292MnVleDBoRlhRdFkvMkt1dUhhN3dKV2pFT0pqaEVzTjhXSy82ZlFFbi9EQzcrNkw3NzhlbmVVZ2lLZ3VFbjlMMXZMYVZ5VWtQaWc9O2V5SnNhV05sYm5ObFpTSTZJa1JsWm1GMWJIUWlMQ0p3Y205a2RXTjBJam9pVFc5dmJpSXNJbTFoZUZObGMzTnBiMjV6SWpvMGZRPT0=

You have to replace value of license.key with yours and reapply the manifest with kubectl apply.

2.6.2. Option 2: you are using Helm

In our Helm charts license key is exposed as moon.licenseKey parameter that could be set as follows:

$ helm install --upgrade --set-file moon.licenseKey=/path/to/license.key -n moon moon aerokube/moon

The same command using shell expansion:

$ helm install --upgrade --set moon.licenseKey=$(</path/to/license.key) -n moon moon aerokube/moon

2.6.3. Option 3: doing everything manually

Moon container is expecting to find license key file in the path specified by -license-file parameter. You need to mount license key file as a volume to every Moon container.

  1. Having a license key file - create a Kubernetes secret:

    $ kubectl create secret generic license-key --filename /path/to/license.key -n moon

    If you prefer YAML - this will look like:

    Creating a secret for license key
    apiVersion: v1
    kind: Secret
    metadata:
      name: license-key
      namespace: moon
    stringData:
      license.key: Ti90a3ljSWpvV3hicU5ZY3U3ZVNMUzdzalVjU29nSVU2a3lmWDdENmhOSEs4SFJVVE5LNEpKL2oxRGhWTnRhZVBEQmx5cm1SM0dFWlVpbGIwOWhBSUpvWmlQMnNtYmtoZVRDbnJ3UTlIV3grMUMzaXFJalU3L0NIZDJuSzN3QUpIS0lyV2VoVjlRVUh5NzNCMFg0NEV5KzFSVXg1elhvaC9RUkNjTlBIOHR4WWtrRFBmTFhxYTFESlI0aXNObGtobWpXY2VJNVdna1lsWHFFOUp0OUxValZPN05hb2pCcHJzY0pvZjBrZys2YkhWUDR3cUJvYVFoSFJTYzMzZzNRSTkrWVd4dlZCeExmdTVxM2k2ZnluMDRHSEhGc21Fd2owVHdVa1Z3MmI5Z2FyVXlaVFF4RDdCRjVSVjBmSUVrU1pEQ0ZQdmIvMHZoKythY1V4OVRrR2FBREZzbWR3dExxRlU4NHF3MzBVdE5nTGhnNWYxRXRCQ215R1BHR0wyK2NpR0RDZDBTSXg4K1RsRGNWVzUyVVY4MUt6bkdOanJhTklOeXVtYjVPVXF6aVVpdmJZeVJNVVBSMWF3NUJRVGJDaHVycWVVdGtMMWFRZDgwOCtoWGRxODFOakxRNEJlVENNQTdPL2FneHBTUE1nL0J2QmVQKy84ZGhJeXd5Y0lxQVJhVWxocUdmUko2K05tbkFYV0hzZ2RjTng1dUVHU0w5VWF5TU1rR3RkZEZKQ1FiWmF0RnJMaDFHbHhKTndBRHFWcXIvQ0oxSzBQL2p6K2NFSWpsSG82TktvM3pudVFlanhYT2tGU1p1bWZrS1krZmF3VkVRZWlvcFlmZFk0TU9tc3U5TThsbng3T2VXQXZmOTFUQ2w1NUhBK1ZsQTMzN3VFOG1WV3ZlS0E1Tlk9O2V5SnNhV05sYm5ObFpTSTZJa0ZqYldVZ1RFeERJaXdpY0hKdlpIVmpkQ0k2SWsxdmIyNGlMQ0p0WVhoVFpYTnphVzl1Y3lJNk1qQXNJblJ2SWpveE5UTTRNelV5TURBd2ZRPT0=
    1. License key file contains raw (NOT Base64-encoded) license key data. In some cases (e.g. when using data field in YAML) you may need to encode key data contents yourself.

    2. When using YAML with stringData field - you don’t need to Base64-encode license key one more time.

  2. When starting Moon - mount this secret as a volume and point -license-file to mounted file:

    Starting Moon with license key
    apiVersion: apps/v1beta1
    kind: Deployment
    metadata:
      name: moon
      namespace: moon
    spec:
      replicas: 3
      template:
        metadata:
          labels:
            app: moon
        spec:
          containers:
          - name: moon
            image: aerokube/moon:latest-release
            args: ["-license-file", "/license/license.key"] # Pointing to license key file
            resources:
                # Resources here
            ports:
                # Ports here
            volumeMounts:
            # Other mounts here
            - name: license-key # Mounting volume with license key
              mountPath: /license
              readOnly: true
        volumes:
        # Other volumes here
        - name: license-key # Creating volume from secret
          secret:
            secretName: license-key
  3. If you update an existing secret with license key (already present in default deployment manifests) then settings are applied immediately without Moon restart. How fast Moon pods will "see" an updated license key depends on Kubernetes cache propagation delay (default value is 1 minute).

2.6.4. How Licenses Work

Every license key gives you an ability to run fixed number of parallel browser sessions. To limit maximum number of browser sessions Moon is using built-in Kubernetes resource quota feature. Every browser session in Moon is a separate Kubernetes pod. When started Moon checks that respective pod resource quota exists (that is to say total number of pods is limited). If not found such resource quota is automatically created. You can have any desired number of Moon, Moon API and Moon UI replicas (i.e. pods) in your cluster. These pods are added to total number of parallel browser sessions from license key. For example:

  1. Your license key covers 100 parallel sessions

  2. You have 5 pods with Moon, Moon API and Moon UI (Moon pods)

  3. Correct pods quota is no more than 100 + 5 = 105 pods

Having such pods quota and 5 running Moon pods, Kubernetes will allow to start 100 browser pods maximum. New browser session requests exceeding the quota will wait for previous sessions to finish, i.e. previous pods to be removed by Moon. To determine that a running pod is a Moon pod its labels are analysed. Any pod is considered a Moon pod if the following label selector matches:

app in (moon, moon-api, moon-ui)

So when starting Moon - you should add a label app with value: moon, moon-api or moon-ui to respective pod definitions.

2.6.5. Installing License to Restricted Cluster

In some environments Moon has no permission to automatically create resource quota for its namespace. In that case you need to ask your Kubernetes administrator to create resource quota manually:

Required ResourceQuota object definition
apiVersion: v1
kind: ResourceQuota
metadata:
    name: max-moon-sessions
    namespace: moon
spec:
    hard:
      pods: "105" # 100 pods with browsers + 5 pods with Moon
    scopes: ["NotTerminating"]

2.7. Monitoring

2.7.1. Browsers Consumption

Moon has a dedicated microservice moon-monitor allowing to easily visualize browsers consumption with Prometheus and Grafana.

An example Grafana dashboard

grafana

moon-monitor (distributed as aerokube/moon-monitor image) exposes browsers consumption information in Prometheus text format via /metrics HTTP API. One of the simplest ways of deploying Prometheus in Kubernetes is using Prometheus Operator. When using it monitoring solution could work like the following:

  1. Moon should be already running (e.g. in moon namespace).

  2. Deploy Prometheus and Grafana using Prometheus Operator (e.g. to monitoring namespace).

    An example installation command using Helm 3 is:

    $ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
    $ helm repo update
    $ helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --create-namespace --namespace monitoring
  3. Deploy moon-monitor to the same monitoring namespace (optionally using a service account with permissions to access moon namespace):

    Deploying moon-monitor with Kubernetes Service and Deployment
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: moon-monitor
      namespace: monitoring
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: moon-monitor
      namespace: moon
    rules:
    - apiGroups:
      - ""
      resources:
      - pods
      - resourcequotas
      verbs:
      - get
      - watch
      - list
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: moon-monitor
      namespace: moon
    roleRef:
      kind: Role
      name: moon-monitor
      apiGroup: rbac.authorization.k8s.io
    subjects:
    - kind: ServiceAccount
      name: moon-monitor
      namespace: monitoring
    ---
    kind: Service
    apiVersion: v1
    metadata:
      name: moon-monitor
      namespace: monitoring
      labels:
        app: moon-monitor
    spec:
      selector:
        app: moon-monitor
      ports:
      - name: metrics
        port: 8080
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: moon-monitor
      namespace: monitoring
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: moon-monitor
      template:
        metadata:
          labels:
            app: moon-monitor
        spec:
          serviceAccountName: moon-monitor
          containers:
          - name: moon-monitor
            image: aerokube/moon-monitor:latest
            args:
            - -namespace
            - moon
            imagePullPolicy: IfNotPresent
            ports:
            - name: metrics
              containerPort: 8080
  4. Create a ServiceMonitor object to start fetching Moon metrics:

    Creating ServiceMonitor for Moon
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: moon-monitor
      namespace: monitoring
      labels:
        release: kube-prometheus-stack
    spec:
      selector:
        matchLabels:
          app: moon-monitor
      endpoints:
      - port: metrics
        interval: 15s
  5. To fetch browser consumption information from Prometheus, query it like the following:

    sum(moon_browser_count{browserName="chrome"}) by (version)

    To create a Grafana dashboard you can import and customize our example configuration. An example YAML manifest for deploying moon-monitor could be found here.

2.7.2. Filter by Labels

A common use case is getting browser consumption information for an exact project or product. This can be for example achieved as follows:

  1. Add labels to browser pods using browsers list file or capabilities.

  2. Configure Prometheus as shown above. This will also install kube-state-metrics component.

  3. Use the following Prometheus query to fetch information about browser pods with some labels set:

    kube_pod_labels{label_moon="browser", label_project="MyCoolProject", label_browserName="chrome"}

    Full list of available expressions can be found here.

2.8. Advanced Configuration

2.8.1. Uploading Files to S3

How to Configure S3 Support

You can configure Moon to send sessions logs and recorded video files to S3-compatible storage. Such type of storage is supported by AWS, Google Cloud, Microsoft Azure, Digital Ocean and many other cloud providers. To deploy a private S3-compatible storage you can use Minio. To enable S3 support in Moon:

  1. Create an S3 bucket. In this example bucket name is moon-test. You will get an access key and a secret key for the bucket. Also you need to know supported S3 protocol version (usually specified in documentation).

  2. Create a service.json file with the following contents:

    $ cat service.json
    {
      "s3": {
        "endpoint": "https://storage.googleapis.com",
        "bucketName": "moon-test",
        "version": "S3v2"
      }
    }

    You can create an S3-compatible bucket in the majority of public cloud platforms. How to configure Moon with these platforms in shown in the table below:

    Table 4. S3 settings for popular cloud platforms
    Platform Name Service Name Endpoint Signature Version

    AWS

    AWS S3

    Depends on region, e.g. https://s3.us-east-2.amazonaws.com. See AWS documentation for detailed list of endpoints.

    S3v4

    DigitalOcean

    DigitalOcean Spaces

    Depends on region, e.g. https://nyc3.digitaloceanspaces.com. See documentation for more details.

    S3v4

    Google Cloud

    Google Cloud Storage

    https://storage.googleapis.com

    S3v2

    Microsoft Azure

    Azure Blob Storage

    No built-in S3 support. Need to deploy additional software like Minio.

    S3v4

  3. Save service.json to a Kubernetes config map, mount it as a file to Moon container and then specify path to the file using -config-file flag as shown in Configuration File section.

  4. Create a Kubernetes credentials secret, mount it as a directory to Moon container and then specify path to this directory using -credentials-dir flag as shown in Credentials Secret section.

    S3 credentials secret
    apiVersion: v1
    kind: Secret
    metadata:
      name: credentials
      namespace: moon
    stringData:
      s3.accessKey: "access-key-value"
      s3.secretKey: "secret-key-value"
  5. When you update already mounted config map with service.json or a credentials secret - settings are applied immediately without Moon restart.

Custom S3 Layout

By default files are uploaded to S3 bucket as follows:

Default S3 bucket layout
\---my-bucket
    \---- <session-id>
        |---- video.mp4
        |---- session.log

Moon allows to organize any custom S3 keys layout using S3 key pattern with placeholders. A typical S3 key pattern looks like the following:

Typical S3 key pattern
$quota/$browserName/$browserVersion/$platformName/$sessionId

Here every placeholder such as $quota, $browserName, $browserVersion and so on will be replaced by corresponding information: user name, browser name, browser version. The resulting S3 key will be used as a directory to save log and video files. A list of supported placeholders is shown in the table below:

Table 5. S3 Key Placeholders
Placeholder Meaning

$sessionId

Replaced by Selenium session ID

$browserName

Replaced by Selenium browser name capability value

$browserVersion

Replaced by Selenium browser version capability value

$platformName

Replaced by Selenium platform capability value

$date

Replaced by current date, e.g. 2018-11-01

$quota

Replaced by quota name (i.e. user name provided in Selenium URL)

Default S3 key pattern is just $sessionId:

Default video and log paths
my-bucket/chrome-71-0-686efb96-eabe-4435-af31-21a33c8a4c8b/video.mp4
my-bucket/chrome-71-0-686efb96-eabe-4435-af31-21a33c8a4c8b/session.log

You change S3 key pattern in service.json file as follows:

Setting custom S3 key pattern in service.json
{
  "s3": {
    // The rest of S3 settings go here...
    "keyPattern": "$quota/$browserName/$browserVersion/$platformName/$sessionId"  (1)
  },
  "images": {
    //...
  },
  "annotations": {
    //...
  }
}
1 Custom S3 key pattern

To define S3 key pattern for every browser session independently - use s3KeyPattern capability described in Special Capabilities section.

2.8.2. Using Custom Docker Registry

By default Moon downloads all images (aerokube/defender, aerokube/logger and so on) from public Docker registry. If in your environment due to security restrictions Docker images can only be downloaded from private registry you need to configure Moon to work with this registry. To do this:

  1. Configure Kubernetes authentication to your private registry:

    $ kubectl create secret docker-registry my-registry.example.com --docker-server=my-registry.example.com --docker-username=some-user --docker-password=registry-password --docker-email=some-user@example.com
    $ kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "my-registry.example.com"}]}' # Use correct service account name here

    In case of Openshift the following commands will work:

    $ oc create secret docker-registry my-registry.example.com --docker-server=my-registry.example.com --docker-username=some-user --docker-password=registry-password --docker-email=some-user@example.com
    $ oc secrets link default my-registry.example.com --for=pull
  2. Copy all desired browser images to your registry:

    selenoid/chrome:73.0 => my-registry.example.com/moon/chrome:73.0
  3. Update Browsers List file (browsers.json) to use new browser images:

    Browsers List File with Custom Images
    {
      "chrome": {
        "default": "73.0",
        "versions": {
            "73.0": {
                "image": "my-registry.example.com/moon/chrome:73.0",
                "port": "4444"
            },
            "72.0": {
                "image": "my-registry.example.com/moon/chrome:72.0",
                "port": "4444"
            }
        }
      }
    }
  4. Copy desired version of the Moon service images to your registry:

    aerokube/moon-video-recorder:1.3.4 => my-registry.example.com/moon/moon-video-recorder:1.3.4
    aerokube/defender:1.3.4 => my-registry.example.com/moon/defender:1.3.4
    aerokube/logger => my-registry.example.com/moon/logger:1.3.4
  5. Override Moon service images, create a service.json file with the following contents:

    $ cat service.json
    {
      "images": {
        "videoRecorder": {
            "image": "my-registry.example.com/moon/moon-video-recorder:latest-release"
        },
        "defender": {
            "image": "my-registry.example.com/moon/defender:latest-release"
        },
        "logger": {
            "image": "my-registry.example.com/moon/logger:latest-release"
        }
      }
    }

    If you already have S3 configuration in service.json file - just append new images key to it.

  6. Save service.json to a Kubernetes config map, mount it as a file to Moon container and then specify path to the file using -config-file flag as shown in Configuration File section. When you update a config map with service.json - settings are applied immediately without Moon restart.

  7. Copy desired version of Moon main images to your registry:

    aerokube/moon:1.3.4 => my-registry.example.com/moon/moon:1.3.4
    aerokube/moon-api:1.3.4 => my-registry.example.com/moon/moon-api:1.3.4
    aerokube/selenoid-ui:1.6.5 => my-registry.example.com/moon/selenoid-ui:1.6.5
  8. Use new main Moon images from the previous step in Kubernetes YAML definitions to start Moon, Moon API and Moon UI.

  9. If you are using Playwright, Cypress or Chrome Developer Tools feature - to override repository you have to set the following flags to moon container:

    -chrome-devtools-repository my-registry.example.com/cdtp
    -cypress-repository my-registry.example.com/browsers
    -playwright-repository my-registry.example.com/playwright

2.8.3. Adjusting Timeouts

Adjusting Moon Timeouts

Sometimes things go wrong: user can unexpectedly disconnect or browser session starts longer than needed. This can lead to overall cluster degradation because of broken browser pods occupying all available hardware. To prevent such cases Moon automatically detects and closes suspicious browser sessions. Several flags allow to adjust timeout settings:

Table 6. Available Timeout Flags
Flag Default Value Meaning Notes

-timeout

1 minute

Maximum browser session idle time - measured as maximum time between separate HTTP requests corresponding to a running session.

When this timeout expires - session is automatically closed. You may need to increase this timeout when tested application pages load too slowly.

-session-attempt-timeout

10 minutes

Maximum time to start browser pod.

This time includes Kubernetes scheduling time and browser image download duration.

Adjusting Other Timeouts

Not only Moon timeout settings can cause your tests to break. A typical Moon installation looks like the following:

Possible Timeout Sources

timeouts

In addition to Moon timeouts other possible sources of timeouts exist:

  1. Client-side Timeout. Every Selenium library is internally using an HTTP client having default request timeout settings. If you are frequently seeing CLIENT_DISCONNECTED messages (meaning that client disconnected before request handling completed) in Moon log - this could be a sign to increase HTTP client timeouts in your code.

  2. Load Balancer Timeout. Usually Moon is running behind load balancer (LoadBalancer, Ingress or Router) and it also has a default request proxy timeout. A frequent value is 60 seconds, so if you are often seeing test fails with 502 Bad Gateway or 504 Gateway Timeout errors - this could be a sign to increase load balancer timeout. How to do this depends on your cloud platform and load balancer type being used. So refer to their documentation for more details. An example of doing this for AWS cloud is shown in Connection was closed unexpectedly section.

  3. Cluster Capacity Reached. When you have already started the maximum number of browser pods allowed by your license, all new browser requests on top of this number will be waiting in queue for already running pods to stop. Waiting too much in queue will lead to user timeout described above. So if you are seeing a lot of FORBIDDEN_TO_CREATE_POD messages in the log that can signalize that you used all available browsers or all available computing resources (CPUs and memory) assigned to Moon namespace.

  4. Cluster Fragmentation. Similarly to the previous one, in some cases you can have sufficient number of cores and not all browsers are exhausted. However sometimes for example you can have 4 CPUs available distributed among 4 Kubernetes nodes (1 available CPU per node) and a new browser pod requiring at least 2 CPUs to start (all pod containers always run on the same node). In that case although total number of available CPUs is sufficient to start a pod, there is no node where pod will be able to start. If you are seeing too many browser pods in Pending state - check with kubectl describe command why these pods are not starting.

2.8.4. Adjusting Resources Consumption

Browser Resources Consumption

Moon has reasonable defaults for resources consumed by every browser pod. Sometimes you may need to override these settings. To override resource settings globally for every browser image use one of the following flags:

Table 7. Resources Consumption Flags
Flag Default Value Meaning

-cpu-limit

1

Hard CPU limit for browser container (no more than this limit is given)

-cpu-request

1

Soft (guaranteed) CPU limit for browser container

-memory-limit

1Gi

Hard memory limit for browser container

-memory-request

512Mi

Soft memory limit for browser container

You can also override the same values for every browser image in browsers list file:

Overriding Resources in Browsers List File
{
  "firefox": {
    "default": "62.0",
    "versions": {
      "62.0": {
        "image": "selenoid/firefox:62.0",
        "port": "4444",
        "path": "/wd/hub",
        "resources": {                              (1)
          "limits": {                               (2)
            "cpu": "2",                             (3)
            "memory": "2Gi"                         (4)
          },
          "requests": {                             (5)
            "cpu": "200m",                          (6)
            "memory": "1Gi"                         (7)
          }
        }
      }
    }
  }
}
1 Resources definition section
2 Limits definition section
3 CPU limit field
4 Memory limit field
5 Requests definition section
6 CPU request field
7 Memory request field

In some installations you may also want to completely disable CPU or memory limits for all browsers pods. This can be achieved with one of the following flags:

Table 8. Disabling Memory and CPU limits
Flag Meaning

-disable-cpu-limits

Completely disables CPU limits for all browser pods

-disable-memory-limits

Completely disables memory limits for all browser pods

Service Images Resources Consumption

By default service images have the following resources requirements:

Table 9. Service Images Default Resources Limits
Name CPU Limit Memory Limit

defender

0.1

64Mi

logger

0.1

64Mi

videoRecorder

1.0

1024Mi

vnc-server

0.25

128Mi

x-server

0.25

128Mi

You can optionally adjust CPU and memory consumption for each service image in service.json configuration file similarly to browser images:

Adjusting memory and CPU limits for service images
{
  "images": {
    "logger": {
      "image": "my-registry.example.com/moon/logger:latest-release",
      "resources": {
        "limits": {
          "cpu": "0.3",
          "memory": "1024Mi"
        },
        "requests": {
          "cpu": "0.3",
          "memory": "1024Mi"
        }
      }
    }
  }
}
Pods Quality of Service

Browser automation stability and speed highly depends on how many computing resources are actually available to browser pods. Kubernetes has so-called Quality of Service (QoS) defining how many resources are assigned to pods being started. For stable browser automation we recommend always setting Guaranteed QoS class to Moon browser pods. To deliver this you have to make sure that requests and limits values for CPU and memory have equal values:

  1. Moon by default sets requests equal to limits for service images like defender, logger and videoRecorder. But in recent releases you can override them independently if you wish.

  2. For browser containers you can override requests and limits independently as shown above. Anyway we also recommend setting them to equal values. Only this way you will be sure that browsers are always getting the same computing resources. Otherwise you may encounter randomly failing browser tests caused by insufficient computing resources assigned to some browser pods.

2.8.5. Using Volumes

This was initially created for mounting really big files cached on Kubernetes nodes to browser containers. We do not recommend using the feature for uploading files to browser. Instead use standard Selenium file upload capability.

Currently Moon supports only mounting volumes from the Kubernetes node (hostPath volumes). Volumes are mounted only to browser container within browser pod. To add such volume - modify browsers list file as follows:

Mounting HostPath Volumes
{
  "firefox": {
    "default": "62.0",
    "versions": {
      "62.0": {
        "image": "selenoid/firefox:62.0",
        "port": "4444",
        "path": "/wd/hub",
        "volumes": ["/host/path:/mount/path/inside/container"]   (1)
      }
    }
  }
}
1 A list of hostPath volumes to mount

2.8.6. Running Browser Pods in Privileged Mode

In some cases like running Android emulators browser container should be run in privileged mode. This setting can be applied separately to each browser version in browsers list file as follows:

Starting Container in Privileged Mode
{
  "firefox": {
    "default": "62.0",
    "versions": {
      "62.0": {
        "image": "selenoid/firefox:62.0",
        "port": "4444",
        "path": "/wd/hub",
        "privileged": true                  (1)
      }
    }
  }
}
1 Launch container in privileged mode

2.8.7. Using Node Selectors

Sometimes you may need to run browser pods on particular Kubernetes nodes (i.e. hardware hosts) only. Kubernetes allows to do this by specifying so called node selectors. To provide such selector to Moon browser pods update browsers list as shown below:

Adding Node Selectors
{
  "firefox": {
    "default": "62.0",
    "versions": {
      "62.0": {
        "image": "selenoid/firefox:62.0",
        "port": "4444",
        "path": "/wd/hub",
        "nodeSelector": {                    (1)
          "node-type": "hardware"
        }
      }
    }
  }
}
1 Node selector for this browser version

2.8.8. Using Tolerations

Kubernetes has a cool feature called Taints allowing to fine tune pods scheduling across nodes. If you wish to run browser pods on tainted nodes - you have to adjust so-called tolerations, that is to say a number of conditions to match against tainted nodes. Moon is using exactly the same format as Kubernetes to define tolerations:

Adding Tolerations
{
  "firefox": {
    "default": "62.0",
    "versions": {
      "62.0": {
        "image": "selenoid/firefox:62.0",
        "port": "4444",
        "path": "/wd/hub",
        "tolerations": [                    (1)
            {
                "key": "key1",
                "operator": "Exists",
                "value": "value1",
                "effect": "NoExecute",
                "tolerationSeconds": 3600
            }
        ]
      }
    }
  }
}
1 A list of tolerations for this browser version

2.8.9. Using Custom Kubernetes Annotations

In some cases you may need to add custom Kubernetes annotations to started browser pods. Moon allows to configure custom annotations globally or individually for every browser version. Global annotations are configured in Configuration File (aka service.json) as follows:

Adding global annotations to service.json file
{
  "s3": {
    //...
  },
  "images": {
    //...
  },
  "annotations": {                                 (1)
    "key1": "value1",
    "key2": "value2"
  }
}
1 A list of global Kubernetes annotations

To configure Kubernetes annotations for single desired browser version - just add the same annotations section to Browsers List file as follows:

Adding annotations to browser version
{
  "firefox": {
    "default": "62.0",
    "versions": {
      "62.0": {
        "image": "selenoid/firefox:62.0",
        "port": "4444",
        "path": "/wd/hub",
        "annotations": {                           (1)
          "key1": "value1",
          "key2": "value2"
        }
      }
    }
  }
}
1 A list of Kubernetes annotations for this browser version

Moon adds some annotations by default to browser pods and their names are reserved:

Table 10. Moon Reserved Annotation Names
Key Meaning

name

Custom session label passed in name capability

timeZone

Custom time zone value passed in timeZone capability

2.8.10. Using Custom Kubernetes Labels

In some cases you may need to add custom Kubernetes labels to started browser pods. Moon allows to configure custom labels globally or individually for every browser version. Similarly to annotations global labels are configured in Configuration File (aka service.json) as follows:

Adding global labels to service.json file
{
  "s3": {
    //...
  },
  "images": {
    //...
  },
  "labels": {           (1)
    "key1": "value1",
    "key2": "value2"
  }
}
1 A list of global Kubernetes labels

To configure Kubernetes labels for single desired browser version - just add the same labels section to Browsers List file as follows:

Adding labels to browser version
{
  "chrome": {
    "default": "77.0",
    "versions": {
      "77.0": {
        "image": "selenoid/chrome:77.0",
        "port": "4444",
        "labels": {                           (1)
          "key1": "value1",
          "key2": "value2"
        }
      }
    }
  }
}
1 A list of Kubernetes labels for this browser version

Moon adds some labels by default to browser pods and their names are reserved:

Table 11. Moon Reserved Label Names
Key Meaning

app

Stores unique name for every pod

browserName

Stores browserName capability value

enableVideo

Stores enableVideo capability value

enableVNC

Stores enableVNC capability value

moon

System label, always equal to browsers

quota

Stores user quota name

screenResolution

Stores screen resolution requested by user

version

Stores version or browserVersion capability value

2.8.11. Using Custom Linux Kernel Capabilities

By default Moon starts all browser pods without any explicit Linux kernel capabilities. In some cases you may need to set custom Linux capabilities to started browser pods (e.g. SYS_ADMIN). Full list of such capabilities is available in Linux kernel source. Linux capabilities are configured in Configuration File (aka service.json) as follows:

Adding global Linux capabilities to service.json file
{
  "s3": {
    //...
  },
  "images": {
    //...
  },
  "kernelCaps": [ "NET_ADMIN", "KILL" ]   (1)
}
1 A list of global Linux capabilities

2.8.12. Using Custom Service Account

By default Moon is using service account named default to access Kubernetes API. In some cases you only have permissions for another service account, e.g. named moon. To use this service account with Moon you need to change deployment as follows:

  1. Optionally create a service account as follows:

    Creating custom service account
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: custom-moon-service-account # <== Insert desired service account name
      namespace: moon
  2. Make sure you have respective Kubernetes Role and RoleBinding for desired service account:

    Assigning roles to custom service account
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      namespace: moon
      name: moon
    rules:
    - apiGroups:
      - "*"
      resources:
      - "*"
      verbs:
      - "*"
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: moon
      namespace: moon
    roleRef:
      kind: Role
      name: moon
      apiGroup: rbac.authorization.k8s.io
    subjects:
    - kind: ServiceAccount
      namespace: moon
      name: custom-moon-service-account # <== Insert name of service account to assign roles
  3. Use your custom service account when deploying Moon pods:

    Using custom service account for Mood pods
    apiVersion: apps/v1
    kind: Deployment
    metadata:
        # ... deployment metadata definition
    spec:
      replicas: 2
      selector:
        # ... selector definition
      template:
        metadata:
            # ... spec metadata definition
        spec:
          serviceAccountName: custom-moon-service-account # <== Insert name of service account here
          containers:
          - name: moon
            image: aerokube/moon:1.4.1
        # ... the rest of spec

2.8.13. Setting Custom User and Group Identifier to Browser Pods

Starting from release 1.4.2 by default Moon starts all browser pods with fixed UID 106 (selenium) and GID 65534 (nogroup). This allows to run browser pods even in restricted clusters without root permissions allowed. In order to change UID and GID values being set - use the following syntax in Configuration File:

Setting custom UID and GID globally in service.json
{
    "runAs": {
      "user": 1000,                  (1)
      "group": 2000                  (2)
    }
}
1 Custom user identifier (UID)
2 Custom group identifier (GID)

To set custom UID and GID values for selected browser versions - use Browsers List file:

Setting custom UID and GID for selected browser version
{
  "chrome": {
    "versions": {
    "default": "80.0",
      "80.0": {
        "image": "selenoid/chrome:80.0",
        "port": "4444",
        "runAs": {
          "user": 1000,                  (1)
          "group": 2000                  (2)
        }
      }
    }
  }
}
1 Custom user identifier (UID)
2 Custom group identifier (GID)

Similarly to

2.9. Log Files

Although Moon should just work out of the box, sometimes you may need the log output. Every Moon component is outputting logs to standard output (stdout) so you can use well-known kubectl commands to see the log. Everything related to Selenium sessions is being output by moon container:

$ kubectl logs -l app=moon -c moon -n moon
2019/05/03 11:20:37 [INIT] [Loading quota files from: quota]
2019/05/03 11:20:37 [INIT] [Quota configuration updated]
2019/05/03 11:20:37 [INIT] [Loading users from: users/users.htpasswd]
...

To follow the logs while running the tests add -f flag:

$ kubectl logs -f -l app=moon -c moon -n moon

Every log line contains:

Table 12. Log entry contents
Field Example Notes

Time

2017/11/01 19:12:42

-

Status

[SESSION_ATTEMPTED]

See table below for complete list of statuses.

Additional fields

[my-user]

One or more sections showing additional information such as browser name, user name, IP address or error message

Duration

[4.15s]

For some log entries this field shows how much time operation took

The following statuses are available:

Table 13. Log entry statuses
Status Description

AWAITING_DRIVER

Waiting for webdriver binary to start

BAD_LABEL

Wrong pod label is requested

BAD_REQUEST

Failed to process user request

BAD_RESOURCES

User incorrectly configured browser container resources

BAD_SCREEN_RESOLUTION

User requested to set wrong custom screen resolution

BAD_TIMEZONE

User requested to set wrong custom time zone inside container

BAD_TIMEOUT

User requested to set wrong session timeout

BAD_VIDEO_FRAME_RATE

User requested to capture video with wrong frame rate

BAD_VIDEO_SCREEN_SIZE

User requested to capture video with wrong screen size

CLIENT_DISCONNECTED

User disconnected and session was interrupted

CREATING_POD

Starting to create pod with browser

DELETING_POD

Starting to delete pod with browser

DELETING_SESSION

Received request to delete browser session

EMPTY_REQUEST

Received empty request from user

FAILED_TO_CREATE_POD

Failed to create browser pod

FAILED_TO_DELETE_POD

Failed to delete browser pod

FAILED_TO_DELETE_SESSION

Failed to delete browser session

FAILED_TO_GET_LOGS

Failed to get session logs

FORBIDDEN_TO_CREATE_POD

Kubernetes return forbidden status when creating pod

INIT

Service initialization messages

LOGS

Requested session logs

LOGS_CLIENT_DISCONNECTED

Client disconnected while streaming session logs

MALFORMED_VOLUME

Moon skipped malformed volume specification from browsers.json

POD_CREATED

Browser pod created

POD_DELETED

Browser pod deleted

PROXYING

Proxying requests to specified URL

PROXY_ERROR

Failed to proxy requests to specified URL

SERVER_ERROR

Failed to create pod or service due to internal server error (probably a bug)

DRIVER_IS_UP

Webdriver binary successfully started

SESSION_CREATED

Browser session successfully created

SESSION_DELETED

Session successfully deleted

SESSION_FAILED

Failed to create new session

SESSION_TIMED_OUT

Existing session timed out

SESSION_NOT_FOUND

Received request with empty session ID

SHUTTING_DOWN

Shutting down the service

STATUS_ERROR

Failed to refresh Moon status

STATUS_REQUEST

Received Moon status request

STATUS_REQUEST_ERROR

Failed to return Moon status

STOP_AWAITING_DRIVER

Stopped waiting for webdriver binary to start

UNSUPPORTED_BROWSER

User requested unsupported browser

USER_NOT_FOUND

Trying to request session with unknown user

VNC_CLIENT_DISCONNECTED

User disconnected while proxying VNC traffic

VNC_ERROR

An error occurred while proxying VNC traffic

VNC_SESSION

User requested VNC session

VNC_SESSION_CLOSED

User closed VNC session

You can also take a look at moon-api and moon-ui logs as follows:

$ kubectl logs -f moon-58f8b57966-s648q -c moon-api -n moon
$ kubectl logs -f moon-58f8b57966-s648q -c moon-ui -n moon

If you are encountering browser pods not being deleted - then take a look at defender container logs for every frozen browser pod:

$ kubectl logs chrome-73-0-ac15ffaa-e641-4c7f-a54c-f25b5be1f135 -c defender -n moon # Here chrome-73-0-ac15ffaa-e641-4c7f-a54c-f25b5be1f135 is Selenium session ID equal to browser pod name

2.10. CLI Flags

These flags should be specified in Kubernetes YAML files when starting the cluster.

2.10.1. Moon Container Flags

The following flags are supported:

-browser-timeout duration
    awaiting browser timeout (default 30s)
-chrome-devtools-repository string
    repository to pull chrome devtools images (default "cdtp")
-config-file string
    optional configuration file (default "config/service.json")
-cpu-limit string
    browser container cpu limit (default "1")
-cpu-request string
    browser container cpu request (default "0.5")
-credentials-dir string
    directory where credentials are mounted (default "credentials")
-cypress-repository string
    repository to pull cypress images (default "browsers")
-device-file string
    mobile devices configuration file (default "config/devices.json")
-disable-cpu-limits
    disable cpu limits for pods
-disable-memory-limits
    disable memory limits for pods
-domain-name string
    browsers service domain name (default "browsers")
-grace-period duration
    graceful shutdown (default 30s)
-guest-user string
    guest quota user name (default "browsers")
-license-file string
    path to license file (default "license/license.key")
-listen string
    address to bind (default ":4444")
-memory-limit string
    browser container memory limit (default "1Gi")
-memory-request string
    browser container memory request (default "512Mi")
-moon-url string
    moon service url (default "http://moon:4444/wd/hub")
-namespace string
    namespace
-playwright-repository string
    repository to pull playwright images (default "playwright")
-quota-dir string
    quota directory (default "quota")
-service-account-name string
    service account name (default "default")
-session-attempt-timeout duration
    new session attempt timeout (default 30m0s)
-timeout duration
    override session timeout (default 1m0s)
-users-file string
    path to users file (default "users/users.htpasswd")
-version
    show version and exit

2.10.2. Moon API Container Flags

The following flags are supported:

-domain-name string
    browsers service domain name (default "browsers")
-grace-period duration
    graceful shutdown period (default 30s)
-guest-user string
    guest quota user name (default "browsers")
-license-file string
    path to license file (default "license/license.key")
-listen string
    address to bind (default ":8080")
-namespace string
    namespace
-quota-dir string
    quota directory (default "quota")
-sse-period duration
    sse refresh period (default 5s)
-version
    Show version and exit

2.10.3. Moon Auth Container Flags

The following flags are supported:

-ca-cert string
    ca certificate to verify discovery cert (optional)
-client-id string
    client id (required)
-client-secret string
    client secret (required)
-discovery-url value
    oidc discovery url (required)
-fail-login-timeout duration
    request timeout (default 30s)
-grace-period duration
    graceful shutdown period (default 30s)
-group value
    allowed user groups (optional)
-listen string
    address to bind (default ":4545")
-request-timeout duration
    request timeout (default 30s)
-upstream-url value
    upstream url (default http://127.0.0.1:4444/)
-version
    show version and exit

3. Main Features

3.1. Special Capabilities

Moon supports a set of custom capabilities. You can pass them in tests to enable or disable some features.

3.1.1. Live Browser Screen: enableVNC

Moon supports showing browser screen during test execution. To see browser screen add capability:

Type: boolean
enableVNC: true

Browser screen will be shown in Selenoid UI.

3.1.2. Custom Screen Resolution: screenResolution

Moon allows you to set custom screen resolution in containers being run:

Type: string, format: <width>x<height>
screenResolution: "1280x1024"

You can optionally add colors depth:

Type: string, format: <width>x<height>x<colors-depth>
screenResolution: "1280x1024x24"

This capability sets only screen resolution - not browser window size. Most of browsers have some default window size value this is why your screenshot size can be smaller than screen resolution specified in capability. You should manually resize window to desired width and height or use Selenium maximize operation.

3.1.3. Android Skin: skin

When testing for Android platform you can select emulator skin with capabilities. List of available skins:

Table 14. Available Android Skins
Skin Screen Resolution DPI

QVGA

240x320

120

WQVGA400

240x400

120

WQVGA432

240x432

120

HVGA

320x480

160

WVGA800

480x800

240

WVGA854

480x854

240

WSVGA

1024x600

160

WXGA720

720x1280

320

WXGA800

1280x800

160

WXGA800-7in

800x1280

213

To select a skin - set skin capability:

Type: string, format: <desired-skin>
skin: "WXGA720"

You can also pass desired screen resolution as follows:

Type: string, format: <desired-screen-resolution>
skin: "720x1280"

3.1.4. Video Recording: enableVideo, videoName, videoScreenSize, videoFrameRate, videoCodec

To enable video recording for session, add:

Type: boolean
enableVideo: true
  • By default saved video files are named video.mp4. To provide custom video name specify:

    Type: string
    videoName: "my-cool-video.mp4"
    It is important to add mp4 file extension.
  • By default the entire screen picture is being recorded. Specifying screenResolution capability changes recorded video size (width and height) accordingly. You can override video screen size by passing a capability. In case of videoScreenSize resolution is less than actual, screen on video will be trimmed starting from top-left corner:

    Type: string
    videoScreenSize: "1024x768"
  • Default video frame rate is 12 frames per second. Specifying videoFrameRate capability changes this value:

    Type: int
    videoFrameRate: 24
  • By default Moon is using libx264 codec for video output. If this codec is consuming too much CPU, you can change it using videoCodec capability:

    Type: string
    videoCodec: "mpeg4"

3.1.5. Custom Log Name: logName

By default saved log files are named session.log. To provide custom log name specify:

Type: string
logName: "session.log"
It is important to add log file extension.

3.1.6. Custom Test Name: name

For debugging purposes it is often useful to give a distinct name to every test case. You can set test case name by passing the following capability:

Type: string
name: "myCoolTestName"

The main application of this capability - is debugging tests in the UI which is showing specified name for every running session.

3.1.7. Per-session Time Zone: timeZone

Some tests require particular time zone to be set in operating system.

Type: string
timeZone: "Europe/Moscow"

You can find most of available time zones here. Without this capability launched browser containers will have Moon timezone.

3.1.8. Per-session Environment Variables: env

Sometimes you may want to set some environment variables for every test case (for example to test with different default locales). To achieve this pass one more capability:

Type: array, format: <key>=<value>
env: ["LANG=ru_RU.UTF-8", "LANGUAGE=ru:en", "LC_ALL=ru_RU.UTF-8"]

Environment variables from this capability are appended to variables from configuration file.

3.1.9. Hosts Entries: hostsEntries

Although you can configure a separate list of /etc/hosts entries for every browser image in Browsers List sometimes you may need to add more entries for particular test cases. This can be easily achieved with:

Type: array, format: <hostname>:<ip-address>
hostsEntries: ["example.com:192.168.0.1", "test.com:192.168.0.2"]

Entries from this capability will be override /etc/hosts entries from browsers list file.

3.1.10. Custom DNS Servers: nameservers

By default browser pods are using global Kubernetes DNS settings. Sometimes you may need to override used DNS servers list for particular test cases. This can be easily achieved with:

Type: array, format: <dns-ip-address>
nameservers: ["192.168.0.1", "192.168.0.2"]

3.1.11. Custom Session Timeout: sessionTimeout

Sometimes you may want to change idle timeout for selected browser session. To achieve this - pass the following capability:

Type: string
sessionTimeout: "1m30s"

Timeout is always specified in Golang duration format, e.g. 30s or 2m or 1h2m30s and so on.

3.1.12. S3 Key Pattern: s3KeyPattern

This capability allows to override S3 key pattern (specified by s3 > keyPattern key in service.json) used when uploading files to S3.

Type: string
s3KeyPattern: "$quota/$browserName/$sessionId"

The same key placeholders are supported. Please refer to Uploading Files To S3 section for more details.

3.1.13. Mobile Emulation: mobileDevice

This capability configures desired mobile device Mobile Emulation:

Type: object
"mobileDevice": {
    "deviceName": "Apple iPhone XR",
    "orientation": "landscape"
}

To select which device to emulate use deviceName key:

Type: string
deviceName: "Apple iPhone XR"

To explicitly specify device screen orientation (portait or landscape) use orientation key:

Type: string
orientation: "landscape"

Possible orientation values are: portait, vertical (alias for portrait), landscape, horizontal (alias for landscape).

3.1.14. Pod Labels: labels

Sometimes you may want to pass additional metadata to every browser session: environment, VCS revision, build number, project name and so on. These labels can be then used to get various browser usage statistics.

Type: map, format: "<key>": "<value>"
labels: {"project": "MyCoolProject", "build-number": "14353"}

Labels from this capability override labels from browsers configuration file. More information about labels is described in Using Custom Kubernetes Labels section.

3.1.15. Specifying Capabilities via Protocol Extensions

Some Selenium clients allow passing only a limited number of capabilities specified in WebDriver specification. For such cases Moon supports reading capabilities using WebDriver protocol extensions feature. The following two examples deliver the same result. Usually capabilities are passed like this:

Passing Capabilities as Usually
{"browserName": "firefox", "version": "62.0", "screenResolution": "1280x1024x24"}

Moon is using moon:options key to read protocol extension capabilities:

Passing Capabilities using Protocol Extensions
{"browserName": "firefox", "version": "62.0", "moon:options": {"screenResolution": "1280x1024x24"}}

3.2. Accessing Files Downloaded with Browser

Files are accessible only when browser session is running.

Your tests may need to download files with browsers. To analyze these files a common requirement is then to somehow extract downloaded files from browser containers. Moon provides a /download API dramatically simplifying downloading such files. To work with it:

  1. Start a new session, for example with ID firefox-61-0-f2bcd32b-d932-4cdc-a639-687ab8e4f840.

  2. In tests code save all files to ~/Downloads directory.

  3. Access all downloaded files using an URL:

    http://moon-host.example.com:4444/download/firefox-61-0-f2bcd32b-d932-4cdc-a639-687ab8e4f840/myfile.txt
  4. Close the session

3.3. Accessing Clipboard

Clipboard is accessible only when browser session is running.

Sometimes you may need to interact with the clipboard to check that your application copy-paste feature works. Moon has a dedicated API to interact with the clipboard. To use it:

  1. Start a new session, for example with ID firefox-61-0-f2bcd32b-d932-4cdc-a639-687ab8e4f840.

  2. To get clipboard value send the following HTTP request:

    $ curl http://moon-host.example.com:4444/clipboard/firefox-61-0-f2bcd32b-d932-4cdc-a639-687ab8e4f840
    
    some-clipboard-value
  3. To update clipboard value:

    $ curl -X POST --data 'some-clipboard-value' http://moon-host.example.com:4444/clipboard/firefox-61-0-f2bcd32b-d932-4cdc-a639-687ab8e4f840

3.4. Using Playwright

  1. This feature is supported in Moon 1.6.0 and above.

  2. An example project demonstrating this feature can be found here.

Moon is able to run browser images for Playwright framework out of the box. An example Playwright test that will work with Moon looks like the following:

An example Playwright test working with Moon
const { firefox } = require('playwright');

(async () => {
  const browser = await firefox.connect({ timeout: 0, wsEndpoint: 'ws://moon.example.com:4444/playwright/firefox' });
  const page = await browser.newPage();
  await page.goto('https://aerokube.com/moon/');
  await page.screenshot({ path: `screenshot.png` });
  await browser.close();
})();

You can see that the only difference from standard Playwright example is a web socket endpoint URL. Playwright compared to Selenium has no capabilities concept. The only way to request an exact browser version or e.g. enable video recording is passing all these requirements in websocket endpoint URL. Next section describe supported URL naming conventions.

If your Moon instance is accessible over HTTPS connection (e.g. https://moon.example.com/ instead of http://moon.example.com/) then endpoint URL should start with wss:// instead of ws:// (e.g. wss://moon.example.com/).

3.4.1. Selecting Requested Browser

You can request one of browsers supported by Playwright (firefox, chromium or webkit) by specifying its name in URL. By default Moon will use playwright/<browser-name>:latest public image.

Requesting Chromium (playwright/chromium:latest image)
ws://moon.example.com:4444/playwright/chromium
Requesting Firefox (playwright/firefox:latest image)
ws://moon.example.com:4444/playwright/firefox
Requesting Webkit (playwright/webkit:latest image)
ws://moon.example.com:4444/playwright/webkit

3.4.2. Selecting Exact Browser Version

Every browser version is distributed as a separate image tagged playwright/<browser-name>:<browser-version>. To use an exact browser version - add this version as follows:

Selecting Firefox 76.0 (playwright/firefox:76.0 image)
ws://moon.example.com:4444/playwright/firefox/76.0

Full list of images can be found here.

3.4.3. Selecting Exact Playwright Version

Currently Playwright API can change from version to version. Because of that it is recommended to make sure that Playwright client version being used in your code corresponds to Playwright server version in browser image. To use an image compatible with exact Playwright version - add this version as follows:

Selecting an image compatible to Playwright 1.5.1 (playwright/firefox:playwright-1.5.1 image)
ws://moon.example.com:4444/playwright/firefox/playwright-1.5.1

3.4.4. Enabling Additional Features

In addition to selecting browser and its version - you can enable additional features like using headless browser versions, enabling video recording and so on. All these optional features are set by adding parameters to URL:

Adding parameters to enable additional features
ws://moon.example.com:4444/playwright/chrome?headless=false&enableVideo=true

Full list of supported parameters and their meaning is shown below.

Table 15. Supported parameters for enabling additional features
Parameter name Possible values Default value Description

arg

Browser command-line arguments

Not set

One or more additional command-line arguments to be passed to browser. This parameter can be passed multiple times: arg=--use-fake-ui-for-media-stream&arg=--use-gl.

devtools

true or false

false

Whether to show Chrome Developer Toolbar (only applicable to chromium browser).

enableVideo

true or false

false

Whether to record video of running browser session (same meaning as enableVideo Selenium capability). Only works in conjunction with headless=false, i.e. headless=false&enableVideo=true. You can see browser screen in Moon UI only when this is set to false.

env

Environment variables

Not set

One or more environment variables that will be visible to the browser. This parameter can be passed multiple times: env=LANG%3Dde_AT.UTF-8&env=LANGUAGE%3Dat:de.

headless

true or false

true

Whether to run browser in headless mode.

name

Any human-readable string

Not set

Allows to set custom test name (same meaning as name Selenium capability).

screenResolution

1280x1024 or 1280x1024x24

1920x1080x24

Sets resolution of the desktop where browser is running. Use Playwright methods to set browser window size.

timeZone

Europe/Moscow

Time zone set to Moon

Sets time zone being used by browser.

videoCodec

Supported video codec name

libx264

Allows to customize video codec name being used by video recorder (same meaning as videoCodec Selenium capability).

videoFrameRate

24

12

Allows to customize frame rate of the video being recorded (same meaning as videoFrameRate Selenium capability).

videoName

<your-name>.mp4

video.mp4

Allows to customize video file name being uploaded to S3 (same meaning as videoName Selenium capability).

videoScreenSize

1280x1024

1920x1080

Sets resolution of the video being recorded (same meaning as videoScreenSize Selenium capability).

3.5. Accessing Chrome Developer Tools

  1. This is feature is supported in Moon 1.7.0 and above.

  2. This feature will work only for Chrome 63+.

  3. We recommend using the most recent Chrome version possible.

  4. An example project demonstrating this feature can be found here.

Moon can automate browsers using Chrome Developer Tools Protocol. This allows you to run tests in parallel using libraries like Puppeteer or Taiko. In order to start a new browser with these tools - simply use the following URL:

ws://moon.example.com:4444/cdtp/chrome

If your Moon instance is accessible over HTTPS connection (e.g. https://moon.example.com/ instead of http://moon.example.com/) then URL should start with wss:// instead of ws:// (e.g. wss://moon.example.com/).

An example Puppeteer test is shown below:

Accessing Developer Tools API with Puppeteer
const puppeteer = require('puppeteer-core');
const host = 'moon.example.com';
(async () => {
    const devtools = await puppeteer.connect(
        { timeout: 0, browserWSEndpoint: `ws://${host}:4444/cdtp/chrome` }
    ); // For every call of this method a new browser is started
    const page = await devtools.newPage();
    await page.goto('https://aerokube.com');
    await page.screenshot({path: 'screenshot.png'});
    const title = await page.title();

    console.log(title);

    await devtools.close();
})();

Similarly to Playwright you can choose browser version and enable additional features by changing connection URL.

Selecting Chrome version (cdtp/chrome:85.0 image)
ws://moon.example.com:4444/cdtp/chrome/85.0
Adding parameters to enable additional features
ws://moon.example.com:4444/cdtp/chrome?headless=false&enableVideo=true
Table 16. Supported parameters for enabling additional features
Parameter name Possible values Default value Description

devtools

true or false

false

Whether to show Chrome Developer Toolbar.

enableVideo

true or false

false

Whether to record video of running browser session.

headless

true or false

true

Whether to run browser in headless mode.

name

Any human-readable string

Not set

Allows to set custom test name.

screenResolution

1280x1024 or 1280x1024x24

1920x1080x24

Sets resolution of the desktop where browser is running.

timeZone

Europe/Moscow

Time zone set to Moon

Sets time zone being used by browser.

videoCodec

Supported video codec name

libx264

Allows to customize video codec name being used by video recorder.

videoFrameRate

24

12

Allows to customize frame rate of the video being recorded.

videoName

<your-name>.mp4

video.mp4

Allows to customize video file name being uploaded to S3.

videoScreenSize

1280x1024

1920x1080

Sets resolution of the video being recorded.

3.6. Using Cypress

  1. This feature is supported in Moon 1.9.0 and above.

  2. No changes in Cypress project are required.

  3. An example project demonstrating this feature can be found here.

Moon is able to run Cypress tests out of the box. To do this:

  1. Install a tool allowing to execute Cypress tests remotely:

    $ npm install @aerokube/cypress-moon
  2. Run your tests against Moon cluster:

    $ cd /path/to/my-test-project
    my-test-project$ cypress-moon http://moon.example.com:4444/cypress/chrome

    Each call of cypress-moon command will start a new browser in Moon.

  1. If your Moon instance is running behind Ingress then correct URL would usually be: https://moon.example.com/cypress/chrome.

  2. Running Cypress tests requires sending compressed project to Moon. If your Cypress project is big and Moon is running behind Ingress you may need to increase maximum HTTP request body size. For example Nginx Ingress Controller requires the following annotation to be added:

    nginx.ingress.kubernetes.io/proxy-body-size: 128m

Cypress compared to Selenium has no capabilities concept. The only way to request an exact browser type or additional features is passing all these requirements in HTTP endpoint URL. Next section describes supported URL naming conventions.

3.6.1. Selecting Requested Browser

You can request one of browsers supported by Cypress (chrome, chromium, edge, electron or firefox) by specifying its name in URL. By default Moon will use browsers/cypress-<browser-name>:latest public image.

Requesting Chrome (browsers/cypress-chrome:latest image)
$ cypress-moon http://moon.example.com:4444/cypress/chrome
Requesting Chromium (browsers/cypress-chromium:latest image)
$ cypress-moon http://moon.example.com:4444/cypress/chromium
Requesting Electron (browsers/cypress-electron:latest image)
$ cypress-moon http://moon.example.com:4444/cypress/electron
Requesting Microsoft Edge (browsers/cypress-edge:latest image)
$ cypress-moon http://moon.example.com:4444/cypress/edge
Requesting Firefox (browsers/cypress-firefox:latest image)
$ cypress-moon http://moon.example.com:4444/cypress/firefox

3.6.2. Selecting Exact Cypress Version

Cypress API can change from version to version. Because of that it is recommended to make sure that Cypress version being used in your project corresponds to Cypress version in browser image. To use an image compatible with exact Cypress version - add this version as follows:

Selecting an image compatible to Cypress 7.3.0 (browsers/cypress-electron:cypress-7.3.0 image)
$ cypress-moon http://moon.example.com:4444/cypress/electron/cypress-7.3.0

3.6.3. Enabling Additional Features

In addition to selecting Cypress version - you can enable additional features like changing screen resolution, passing custom test name and so on. All these optional features are set by adding parameters to URL:

Adding parameters to enable additional features
$ cypress-moon http://moon.example.com:4444/cypress/electron/cypress-7.3.0?noExit=true&headless=false&env=LANG%3Dde_AT.UTF-8&env=LANGUAGE%3Dat:de

Full list of supported parameters and their meaning is shown below.

Table 17. Supported parameters for enabling additional features
Parameter name Possible values Default value Description

env

Environment variables

Not set

One or more environment variables that will be visible to the browser. This parameter can be passed multiple times: env=LANG%3Dde_AT.UTF-8&env=LANGUAGE%3Dat:de.

headless

true or false

true

Whether to run browser in headless mode.

name

Any human-readable string

Not set

Allows to set custom test name (same meaning as name Selenium capability).

noExit

true or false

false

Whether to leave container running after executing all tests. Mainly needed for debugging purposes.

screenResolution

1280x1024 or 1280x1024x24

1920x1080x24

Sets resolution of the desktop where browser is running. Use Cypress methods to set browser window size.

3.7. Mobile Emulation

  1. This feature is supported in Moon 1.8.0 and above.

  2. This feature works with Chrome browser only.

  3. Testing mobile applications is not possible.

Running automated tests in mobile platforms is nowadays very important. Using a set of real devices connected to server via USB requires too much work to deploy and maintain. Running Android Emulators requires hardware server or virtual machines with nested virtualization enabled. Running iOS Simulators requires to have Apple hardware. Even with correct computing resources tests are slower than on desktop platforms and consume slightly more CPUs and memory per browser.

Your goal however is catching bugs and not deploying complicated browser automation infrastructure. There are a lot of cases when a bug related to mobile version of tested web application can be reproduced simply by having exactly the same screen size and User-Agent HTTP header being sent by browser. This feature is already available in Chromium-based browsers and is called Mobile Emulation.

Moon provides an improved version of mobile emulation. In order to start a new browser with mobile emulation you have to use the following URL as Selenium URL:

http://moon.example.com:4444/mobile

Example tests using this functionality are shown below:

Mobile Emulation in Java
ChromeOptions options = new ChromeOptions();
options.setCapability("browserVersion", "87.0");
options.setCapability("moon:options", new HashMap<String, Object>() {{
    put("mobileDevice", new HashMap<String, Object>(){
        {
            put("deviceName", "Apple iPhone XR");
            put("orientation", "landscape");
        }
    });
}});
WebDriver driver = new RemoteWebDriver(new URL("http://moon.example.com:4444/mobile"), options);
Mobile Emulation in Python
import selenium.webdriver

capabilities = {
    "browserName": "chrome",
    "version": "87.0",
    "moon:options": {
        "mobileDevice": {
            "deviceName": "Apple iPhone XR",
            "orientation": "landscape"
        }
    }
}

driver = selenium.webdriver.remote.webdriver.WebDriver(
    command_executor='http://moon.example.com:4444/mobile',
    desired_capabilities = capabilities,
    keep_alive = False,
)

try:
    driver.get('https://aerokube.com/moon/')
    driver.get_screenshot_as_file('screenshot.png')
finally:
    driver.quit()

3.7.1. Supported Mobile Devices

Moon comes with a preconfigured list of supported devices stored in Devices File. In order to add your own mobile devices definitions - simply update this file and wait for Moon to reload it.

Table 18. Supported Mobile Devices
deviceName capability Notes

Apple iPad

Apple iPad 10.2 (2019)

Apple iPad Mini

Apple iPad Mini 4

Apple iPad Pro

Apple iPad Pro (10.5)

iPad Pro 10.5"

Apple iPad Pro (12.9)

iPad Pro 12.9"

Apple iPhone 11

Apple iPhone 11 Pro

Apple iPhone 11 Pro Max

Apple iPhone 4

Apple iPhone 5/SE

Apple iPhone 6/7/8

Apple iPhone 6/7/8 Plus

Apple iPhone 7

Apple iPhone 7 Plus

Apple iPhone 8

Apple iPhone 8 Plus

Apple iPhone SE

Apple iPhone X

Apple iPhone XR

Apple iPhone Xs

Apple iPhone Xs Max

Blackberry PlayBook

BlackBerry Z30

Google Nexus 4

Google Nexus 5

Google Nexus 5X

Google Nexus 6

Google Nexus 6P

Google Nexus 7

Google Nexus 10

Google Pixel 2

Google Pixel 2 XL

Google Pixel 3

Google Pixel 3 XL

Google Pixel 4

Google Pixel 4 XL

JioPhone 2

Kindle Fire HDX

Laptop with HiDPI screen

Laptop with MDPI screen

Laptop with touch

LG Optimus L70

Microsoft Lumia 550

Microsoft Lumia 950

Microsoft Surface Duo

Moto G4

Nokia Lumia 520

Nokia N9

Palm PVG100

Red Hydrogen One

Samsung Galaxy A20

Samsung Galaxy Fold

Samsung Galaxy Note 2

Samsung Galaxy Note 3

Samsung Galaxy Note 8

Samsung Galaxy Note 9

Samsung Galaxy Note 10

Samsung Galaxy Note 10+

Samsung Galaxy S3

Samsung Galaxy S5

Samsung Galaxy S7

Samsung Galaxy S8

Samsung Galaxy S8+

Samsung Galaxy S9

Samsung Galaxy S9+

Samsung Galaxy S10

Samsung Galaxy S10+

Samsung Galaxy S10e

Samsung Galaxy Tab S3

Samsung Galaxy Tab S4

3.8. Changing Browser Locale

In some test cases you may need to override preferred browser locale. You can do this with standard Selenium capabilities. How to override locale depends on browser.

3.8.1. Firefox

Overriding Browser Locale in Firefox
FirefoxOptions options = new FirefoxOptions();
options.setCapability("browserVersion", "75.0");
options.addPreference("intl.accept_languages", "de");
WebDriver driver = new RemoteWebDriver(new URL("http://moon.example.com:4444/wd/hub"), options);

3.8.2. Chromium-based Browsers

Overriding Browser Locale in Chromium-base Browsers
ChromeOptions options = new ChromeOptions();
options.setCapability("browserVersion", "81.0");
options.setCapability("env", Arrays.asList("LANG=de_AT.UTF-8", "LANGUAGE=at:de", "LC_ALL=de_AT.UTF-8"));
WebDriver driver = new RemoteWebDriver(new URL("http://moon.example.com:4444/wd/hub"), options);

3.9. Using self-signed TLS certificates

In corporate networks tested environments are often using self-signed TLS certificates. Such certificates are issued by a root certification authority not known to browsers. When trying to open an HTTPS web-page using a self-signed certificate, your browser by default will refuse to do this saying that "Your connection is not private" or "This connection is untrusted". In Selenium tests you can use a standard capability (acceptInsecureCerts = true) to ignore such certificate errors but this will not work when your web-page is using HSTS.

In order to work properly with self-signed certificates, you have to add your root certification authority certificate to a list of trusted certificates:

  1. Find root certificate for certification authority being used to secure your tested environment. Usually such certificates are being issued by IT security team or systems administrators and are publicly available. For example your root certificate can look like this:

    $ cat rootCA.crt
    -----BEGIN CERTIFICATE-----
    MIIGjzCCBHegAwIBAgIJAK1lW/5z8ZSoMA0GCSqGSIb3DQEBCwUAMIGLMQswCQYD
    VQQGEwJFRTEQMA4GA1UECBMHRXN0b25pYTEQMA4GA1UEBxMHVGFsbGlubjEeMBwG
    A1UEChQVQWVyb2t1YmUgU29mdHdhcmUgT8OcMRUwEwYDVQQDEwxhZXJva3ViZS5j
    b20xITAfBgkqhkiG9w0BCQEWEmFkbWluQGFlcm9rdWJlLmNvbTAeFw0yMTAyMTcw
    NjQ5NDJaFw0yMzEyMDgwNjQ5NDJaMIGLMQswCQYDVQQGEwJFRTEQMA4GA1UECBMH
    RXN0b25pYTEQMA4GA1UEBxMHVGFsbGlubjEeMBwGA1UEChQVQWVyb2t1YmUgU29m
    dHdhcmUgT8OcMRUwEwYDVQQDEwxhZXJva3ViZS5jb20xITAfBgkqhkiG9w0BCQEW
    EmFkbWluQGFlcm9rdWJlLmNvbTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoC
    ggIBAKdh54x9WZsSxIMfz1rFEHuJ8+3meUua0Q8cpgC/70F0G6X6BXOki0Cu7iET
    6ETfirWuUdRKKGKXHLF8Fdv6WTqnLlDqgzy1Wp9DuPIgeJ+ztKZt+uJFkWjfQb9R
    mn7Qs4vp/F9HTwqlTZl5jMQ+/nrcNAQeNEZ1H1AfZWAuSvrqp3rW33wl6IBZcqfD
    VsMBknBKm/Zc8GpggY8NYxkfj7Jo2izwn/tV+DFgwF0pJkUrDZPPTiNW7q8Se2Vb
    7tC6Iy9ZVgkH8hkrWrPzwW4zxz/d/Si7/cnn9A9+bF+pKrsHktnQ0ScDEAR5+52J
    XAXkES/4pINpBcxvNUHGO6KXKH4rJVf3QvXXany0ugwVQ+QXirA6yOoY3XFgBxgU
    P7Qd5pyQdVf/SwJ5Uk5Z9b2HXk8k/6jNxe1A6WiojTOnn1fD/VzOTn4xiobqNIpE
    w5dUhlj/TiN+g3uGBH4BPo6IYHCmfsXFEcSZW75k7dRlZ3ZMI4k0utUVm3Y8B+TC
    sj4WmwnXetFP2EMnRft7BnR13oLyzrFB8tkFafstcVoE6oR20pIBtAFxrSDWJ5dA
    XdX2NGPNUCnd1RqJxu2SGA/xHHsyPT06iJeIZGUyRXmv6vBvyCkyeLtMEdq2Gzfi
    MT0GtDkG5R+al/A+Ot3w3CMbMgUFrxvEhlxM1sEitclXJc4tAgMBAAGjgfMwgfAw
    HQYDVR0OBBYEFBb9mCFAqV/JgmMxtwQ6UKzoLIQQMIHABgNVHSMEgbgwgbWAFBb9
    mCFAqV/JgmMxtwQ6UKzoLIQQoYGRpIGOMIGLMQswCQYDVQQGEwJFRTEQMA4GA1UE
    CBMHRXN0b25pYTEQMA4GA1UEBxMHVGFsbGlubjEeMBwGA1UEChQVQWVyb2t1YmUg
    U29mdHdhcmUgT8OcMRUwEwYDVQQDEwxhZXJva3ViZS5jb20xITAfBgkqhkiG9w0B
    CQEWEmFkbWluQGFlcm9rdWJlLmNvbYIJAK1lW/5z8ZSoMAwGA1UdEwQFMAMBAf8w
    DQYJKoZIhvcNAQELBQADggIBAIUmJsxdrT8AN2yZqzI69qQKjLnDhuojdgM3XGL3
    gJTldXR5OIMnw/na8WcIC3onHjgijUeEfslTIIHmNcqOd3hTfOq4Qq2/Qmpp+h1d
    5dCzScrLFiDgjnzkX0VczOj/BtnZMgxx5x8YO80MMUWVEmVCk+i2bFVTypV9e4qw
    1EJLmGTnKoo7l2jPHLUB5lL2LvSO4KHDhmWG5wtFg7/nd097yG5uBHda5ytbc6S8
    CIS8IBJzd7TA4fr3qOhC298LMD96nJdccHqKYtlFvf9YZZ500nrA+pH6Kpo8PD67
    8WiIW/CMtO0X9pxw+KRlmaDmCGGgRhvPyHoYqbX4svrca8uvErePtXIQILe/IISJ
    TXLkiVsej8k3UDu77q/wX3ZdzknWakZyPj+CtYkkZL4vqkIDIFSUcXfynyDZNZEo
    2d+npABzPB42+4xGZGGnFIsfuTMAgpbK8TAgPQNMIawfWTq2KhZ8MYHfPdkU3FBo
    MaExr684sviAImqOotcoNQV2iMOKdwzA097jRBrfa43LhpdoWM0v7RVxB8s+kG0P
    8nHOGmp6r6cIAk5hjHYAwQYiZjXuzvnFTtD9Ily63i+yVh8nRSY9NSLhpFpl4ezo
    hn+savO4nm/HueAATnGR1iPlKnfXNVqQYdl+wwzqK1/3iHjzUUjyQkk0oTBk4Bez
    ejbh
    -----END CERTIFICATE-----
  2. Encode certificate data with Base64 algorithm:

    $ cat rootCA.crt | base64
    LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUdqekNDQkhlZ0F3SUJBZ0lKQUsxbFcvNXo4WlNvTUEwR0NTcUdTSWIzRFFFQkN3VUFNSUdMTVFzd0NRWUQKVlFRR0V3SkZSVEVRTUE0R0ExVUVDQk1IUlhOMGIyNXBZVEVRTUE0R0ExVUVCeE1IVkdGc2JHbHViakVlTUJ3RwpBMVVFQ2hRVlFXVnliMnQxWW1VZ1UyOW1kSGRoY21VZ1Q4T2NNUlV3RXdZRFZRUURFd3hoWlhKdmEzVmlaUzVqCmIyMHhJVEFmQmdrcWhraUc5dzBCQ1FFV0VtRmtiV2x1UUdGbGNtOXJkV0psTG1OdmJUQWVGdzB5TVRBeU1UY3cKTmpRNU5ESmFGdzB5TXpFeU1EZ3dOalE1TkRKYU1JR0xNUXN3Q1FZRFZRUUdFd0pGUlRFUU1BNEdBMVVFQ0JNSApSWE4wYjI1cFlURVFNQTRHQTFVRUJ4TUhWR0ZzYkdsdWJqRWVNQndHQTFVRUNoUVZRV1Z5YjJ0MVltVWdVMjltCmRIZGhjbVVnVDhPY01SVXdFd1lEVlFRREV3eGhaWEp2YTNWaVpTNWpiMjB4SVRBZkJna3Foa2lHOXcwQkNRRVcKRW1Ga2JXbHVRR0ZsY205cmRXSmxMbU52YlRDQ0FpSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnSVBBRENDQWdvQwpnZ0lCQUtkaDU0eDlXWnNTeElNZnoxckZFSHVKOCszbWVVdWEwUThjcGdDLzcwRjBHNlg2QlhPa2kwQ3U3aUVUCjZFVGZpcld1VWRSS0tHS1hITEY4RmR2NldUcW5MbERxZ3p5MVdwOUR1UElnZUorenRLWnQrdUpGa1dqZlFiOVIKbW43UXM0dnAvRjlIVHdxbFRabDVqTVErL25yY05BUWVORVoxSDFBZlpXQXVTdnJxcDNyVzMzd2w2SUJaY3FmRApWc01Ca25CS20vWmM4R3BnZ1k4Tll4a2ZqN0pvMml6d24vdFYrREZnd0YwcEprVXJEWlBQVGlOVzdxOFNlMlZiCjd0QzZJeTlaVmdrSDhoa3JXclB6d1c0enh6L2QvU2k3L2NubjlBOStiRitwS3JzSGt0blEwU2NERUFSNSs1MkoKWEFYa0VTLzRwSU5wQmN4dk5VSEdPNktYS0g0ckpWZjNRdlhYYW55MHVnd1ZRK1FYaXJBNnlPb1kzWEZnQnhnVQpQN1FkNXB5UWRWZi9Td0o1VWs1WjliMkhYazhrLzZqTnhlMUE2V2lvalRPbm4xZkQvVnpPVG40eGlvYnFOSXBFCnc1ZFVobGovVGlOK2czdUdCSDRCUG82SVlIQ21mc1hGRWNTWlc3NWs3ZFJsWjNaTUk0azB1dFVWbTNZOEIrVEMKc2o0V213blhldEZQMkVNblJmdDdCblIxM29MeXpyRkI4dGtGYWZzdGNWb0U2b1IyMHBJQnRBRnhyU0RXSjVkQQpYZFgyTkdQTlVDbmQxUnFKeHUyU0dBL3hISHN5UFQwNmlKZUlaR1V5UlhtdjZ2QnZ5Q2t5ZUx0TUVkcTJHemZpCk1UMEd0RGtHNVIrYWwvQStPdDN3M0NNYk1nVUZyeHZFaGx4TTFzRWl0Y2xYSmM0dEFnTUJBQUdqZ2ZNd2dmQXcKSFFZRFZSME9CQllFRkJiOW1DRkFxVi9KZ21NeHR3UTZVS3pvTElRUU1JSEFCZ05WSFNNRWdiZ3dnYldBRkJiOQptQ0ZBcVYvSmdtTXh0d1E2VUt6b0xJUVFvWUdScElHT01JR0xNUXN3Q1FZRFZRUUdFd0pGUlRFUU1BNEdBMVVFCkNCTUhSWE4wYjI1cFlURVFNQTRHQTFVRUJ4TUhWR0ZzYkdsdWJqRWVNQndHQTFVRUNoUVZRV1Z5YjJ0MVltVWcKVTI5bWRIZGhjbVVnVDhPY01SVXdFd1lEVlFRREV3eGhaWEp2YTNWaVpTNWpiMjB4SVRBZkJna3Foa2lHOXcwQgpDUUVXRW1Ga2JXbHVRR0ZsY205cmRXSmxMbU52YllJSkFLMWxXLzV6OFpTb01Bd0dBMVVkRXdRRk1BTUJBZjh3CkRRWUpLb1pJaHZjTkFRRUxCUUFEZ2dJQkFJVW1Kc3hkclQ4QU4yeVpxekk2OXFRS2pMbkRodW9qZGdNM1hHTDMKZ0pUbGRYUjVPSU1udy9uYThXY0lDM29uSGpnaWpVZUVmc2xUSUlIbU5jcU9kM2hUZk9xNFFxMi9RbXBwK2gxZAo1ZEN6U2NyTEZpRGdqbnprWDBWY3pPai9CdG5aTWd4eDV4OFlPODBNTVVXVkVtVkNrK2kyYkZWVHlwVjllNHF3CjFFSkxtR1RuS29vN2wyalBITFVCNWxMMkx2U080S0hEaG1XRzV3dEZnNy9uZDA5N3lHNXVCSGRhNXl0YmM2UzgKQ0lTOElCSnpkN1RBNGZyM3FPaEMyOThMTUQ5Nm5KZGNjSHFLWXRsRnZmOVlaWjUwMG5yQStwSDZLcG84UEQ2Nwo4V2lJVy9DTXRPMFg5cHh3K0tSbG1hRG1DR0dnUmh2UHlIb1lxYlg0c3ZyY2E4dXZFcmVQdFhJUUlMZS9JSVNKClRYTGtpVnNlajhrM1VEdTc3cS93WDNaZHprbldha1p5UGorQ3RZa2taTDR2cWtJRElGU1VjWGZ5bnlEWk5aRW8KMmQrbnBBQnpQQjQyKzR4R1pHR25GSXNmdVRNQWdwYks4VEFnUFFOTUlhd2ZXVHEyS2haOE1ZSGZQZGtVM0ZCbwpNYUV4cjY4NHN2aUFJbXFPb3Rjb05RVjJpTU9LZHd6QTA5N2pSQnJmYTQzTGhwZG9XTTB2N1JWeEI4cytrRzBQCjhuSE9HbXA2cjZjSUFrNWhqSFlBd1FZaVpqWHV6dm5GVHREOUlseTYzaSt5Vmg4blJTWTlOU0xocEZwbDRlem8KaG4rc2F2TzRubS9IdWVBQVRuR1IxaVBsS25mWE5WcVFZZGwrd3d6cUsxLzNpSGp6VVVqeVFrazBvVEJrNEJlegplamJoCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    1. Output should be a single string, so under Linux correct command would be:

      $ cat rootCA.crt | base64 -w0
    2. On Windows there is no built-in command, so you can use an online Base64 encoder.

  3. Set an environment variable starting with ROOT_CA_ to browser image. How to do this depends on used automation technology. When using Selenium you have to set env capability:

    Adding root certification authority
    import selenium.webdriver
    
    capabilities = {
        "browserName": "chrome",
        "version": "87.0",
        "moon:options": {
            "env": [
                # You can add as many certification authorities as needed
                "ROOT_CA_MY_CA_1=LS0tLS1CRUdJTiBDRVJUSU.....",
                "ROOT_CA_MY_CA_2=...",
            ]
        }
    }
    
    driver = selenium.webdriver.remote.webdriver.WebDriver(
        command_executor='http://moon.example.com:4444/wd/hub',
        desired_capabilities = capabilities,
    )

    When using Playwright and Puppeteer - add the same environment variables in connection URL using URL encoding:

    Adding root certification authority to Playwright or Puppeteer
    ws://moon.example.com:4444/playwright/chrome?headless=false&enableVideo=true&env=ROOT_CA_MY_CA_1%3DLS0tLS1CRUdJ...&env=ROOT_CA_MY_CA_2%3DLS0tLS1CRUdJ...

4. Frequently Asked Questions

4.1. Where are Moon logs?

See Log Files section.

4.2. Where are recorded videos stored?

Moon automatically saves session logs and recorded video files to S3 compatible storage. If S3 storage is not configured - then video recording will not work.

4.3. How to update configuration of a running Moon cluster?

Just update respective config maps (config, quota) or secrets (users, credentials) with standard Kubernetes commands (kubectl edit or kubectl replace). For example:

$ kubectl edit configmap config -n moon # Updating global Moon configuration
$ kubectl edit configmap quota -n moon # Updating Moon quota
$ kubectl edit secret users -n moon # Updating Moon users
$ kubectl edit secret credentials -n moon # Updating Moon credentials (e.g. S3 access keypair)

These commands will open your preferred editor with respective data: do any desired modifications, save and exit. Changes are applied immediately but how fast Moon will see them depends on Kubernetes caching settings (default is 1 minute).

4.4. Is it possible to configure Kubernetes service account for Moon?

Yes, Moon has -service-account-name flag described in Moon Container Flags section.

4.5. Is it possible to assign custom firewall rules to browser pods?

Yes, using built-in Kubernetes Network Policies feature. Moon already can assign custom labels to running browser pods. How to do this is described in Using Custom Kubernetes Labels section. To apply a firewall rule to browser pods you need to assign a set of custom labels to these pods and then create a NetworkPolicy matching pods with podSelector using these labels.

4.6. Sessions are not starting

In recent Moon versions (1.4.0+) you may encounter an issue when browser sessions are not starting at all. Moon logs will show something like this:

2019/12/26 13:33:16 [POD_CREATED] [browsers] [172.26.213.8] [chrome-78-0-77632dac-b46c-4e6e-af24-8a160251e931] [0.03s]
2019/12/26 13:33:16 [AWAITING_DRIVER] [browsers] [172.26.213.8] [chrome-78-0-77632dac-b46c-4e6e-af24-8a160251e931]
2019/12/26 13:49:14 [CLIENT_DISCONNECTED] [browsers] [172.26.213.8] [chrome-78-0-77632dac-b46c-4e6e-af24-8a160251e931] [] [957.78s]

In the majority of cases this is because you did not create a special Kubernetes service like the following:

apiVersion: v1
kind: Service
metadata:
  name: browsers
  namespace: moon
spec:
  selector:
    moon: browser
  clusterIP: None
  publishNotReadyAddresses: true

This service is required for successful network communication between Moon and created browser pods and is included to our official deployment manifests by default.

One more possible reason could be related to DNS configuration in your cluster. Moon relies on resolving short domain names like chrome-85-0-xxxxxx to communicate with browser pods. If in your cluster only fully qualified domain names (such as chrome-85-0-xxxxxx.moon.svc.cluster-domain.example) are allowed you need to additionally configure DNS settings of Moon pod:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: moon
  namespace: moon
spec:
  replicas: 2
  selector:
    matchLabels:
      app: moon
  template:
    metadata:
      labels:
        app: moon
    spec:
      containers:
      - name: moon
        # ...
      dnsConfig:
        nameservers:
        - 1.2.3.4
        searches:
          - ns1.svc.cluster-domain.example
          - my.dns.search.suffix

4.7. Connection was closed unexpectedly

If your HTTP requests are randomly hanging - this can mean that you can have too low HTTP request timeout value on your network load balancer (LoadBalancer, Ingress, Openshift Route). Very often default value is about 30 seconds and this can lead to closed connections when a lot of new Selenium session requests are being sent to Moon. How to set timeout setting usually depends on cloud platform you are using. For example in AWS this can look like:

kind: Service
apiVersion: v1
metadata:
  name: moon
  namespace: moon
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60" # AWS load balancer timeout setting
spec:
  type: LoadBalancer
  # The rest of spec goes here...

4.8. Browser pods are freezing

This is a known issue in Kubernetes CoreDNS server settings. Current fix is to turn off DNS caching for Moon namespace. To do this:

  1. Open DNS configuration map for editing (this will open an editor):

    $ kubectl edit cm coredns -n kube-system
  2. You will see YAML content like the following:

    apiVersion: v1
    data:
      Corefile: |
        .:53 {
            errors
            health
            kubernetes cluster.local in-addr.arpa ip6.arpa {
               pods insecure
               upstream
               fallthrough in-addr.arpa ip6.arpa
            }
            prometheus :9153
            proxy . /etc/resolv.conf
            cache 30
            loop
            reload
            loadbalance
        }
    kind: ConfigMap
    metadata:
      creationTimestamp: 2019-01-17T10:32:38Z
      name: coredns
      namespace: kube-system
      resourceVersion: "206"
      selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
      uid: 3668d0fe-1a43-11e9-ad0b-025000000001
  3. Update Corefile section as follows:

    apiVersion: v1
    data:
      Corefile: |
        moon.svc.cluster.local:53 {
            errors
            kubernetes cluster.local {
              namespaces moon
            }
        }
        .:53 {
            errors
            health
            ready
            kubernetes cluster.local in-addr.arpa ip6.arpa {
               pods insecure
               fallthrough in-addr.arpa ip6.arpa
               ttl 30
            }
            cache 30
            prometheus :9153
            forward . /etc/resolv.conf
            loop
            reload
            loadbalance
        }
    # ... the rest of the file
  4. Save changes and exit the editor.

4.9. No space left on device

If you are getting the following error…​

2019/01/23 13:47:11 [INIT] [Failed to init quota reloader: unable to watch directory: no space left on device]

…​and have sufficient disk space then it can be caused by insufficient number of inotify watchers. Moon is using inotify to automatically reload configuration files when they change. To fix this execute one command on Kubernetes hosts:

$ echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

4.10. DNS lookup timeouts

If you are using Flannel as Kubernetes networking engine and receiving DNS lookup timeouts like the following…​

2019/02/21 08:37:32 [VNC_ERROR] [10.244.1.1] [dial tcp: lookup chrome-71-0-686efb96-eabe-4435-af31-21a33c8a4c8b on 10.96.0.10:53: read udp 10.244.1.11:40603->10.96.0.10:53: i/o timeout]

…​then you may need to set the following kernel property on Kubernetes nodes:

$ sysctl net.bridge.bridge-nf-call-iptables=1

4.11. No such host messages

If you are seeing messages like this…​

[PROXY_ERROR] [unknown] [10.240.6.76] [POST http://chrome-60-0-1ae4b9f3-5b5c-441a-af0b-b1f24b39c252.browsers:4545/session] [dial tcp: lookup chrome-60-0-1ae4b9f3-5b5c-441a-af0b-b1f24b39c252.browsers on 10.0.0.10:53: no such host]
  1. then check that you disabled DNS cache for Moon namespace as shown in Browser pods are freezing section.

4.12. Browser session timeouts do not work

This could because of incorrectly set -moon-url flag value. By default Moon is being exposed using Kubernetes service named moon and is available on port 4444. In that case everything works out of the box with default -moon-url flag value. In customized deployment your Moon service name can differ and you have to set -moon-url value flag explicitly. For example, having Moon being exposed with service named my-custom-moon-service on port 3333, you have to explicitly add flag -moon-url http://my-custom-moon-service:3333/wd/hub to your deployment manifests.

4.13. JSON processing errors in tests

In some environments your Selenium tests could from time to time start getting JSON processing errors like this:

Json exception: Expected to read a START_MAP but instead have: END. Last 0 characters read

The main reason of such behavior usually is incorrectly configured request proxy timeout on LoadBalancer or Ingress time. Very frequently default timeout value is 60 seconds and in cases when some Selenium operation takes more time, load balancer will abort request and send 502 error with no body or with HTML body. However Selenium clients always expect to have JSON in Selenium response body and anything else leads to JSON processing exceptions. To solve this - increase request timeout on load balancer side. How to do this should be described in load balancer documentation.

5. License Agreement

Last updated December 11th, 2020. Replaces the prior version in its entirety.

This is a legal agreement. By downloading, installing, copying, saving on Customer’s computer, or otherwise using Aerokube software, support or products Customer becomes a party to this Agreement and Customer consents to be bound by all the terms and conditions set forth below.

  1. Parties

    1. "Aerokube", "Licensor" or "We" means Aerokube Software OÜ, having its principal place of business at Harju maakond, Tallinn, Kesklinna linnaosa, Estonia pst 5-309B, 10143, Estonia, registered in the Commercial Register of Estonia, registry code: 14653208.

    2. "Customer", "Licensee" or "You" means the sole proprietor or legal entity specified in the Subscription Confirmation. For legal entities, "Customer" includes any entity which controls, is controlled by, or is under common control with Customer. For the purposes of this definition, "control" means one of the following:

      1. The power, directly or indirectly, to direct or manage such entity, whether by contract or otherwise.

      2. Ownership of fifty percent (50%) or more of the outstanding shares or beneficial ownership of such entity.

  2. Definitions

    1. "Agreement" means this License Agreement.

    2. "Product" means any generally available Licensor’s software product identified by Licensor as a software developer tool. For the avoidance of doubt, the Product is not produced to the specifications of Customer nor customized through modification or personalization, is intended for mass distribution, and no software code will be provided to Customer.

    3. "User" means any employee, independent contractor or other personnel obtaining access to the Product(s) from Customer.

    4. "Number of Concurrent Sessions" means maximum number of software testing processes being run using the Product in parallel. This can be for example browsers executing User’s tests.

    5. "License Key" means a unique key-code that enables a Licensee to use the Product by unlocking the fixed Number of Concurrent Sessions. Only Licensor and/or its representatives are permitted to produce License Keys for the Product.

    6. "Subscription" means an arrangement for making use of the Product of periodic nature on a prepayment plan. For the purpose of clarity, Subscription includes the subscription term, Products provided to Customer, subscription fees, payment schedules and fixed number of License Keys.

    7. "Product Evaluation" means using the Product without a valid License Key.

    8. "Subscription Confirmation" means an email confirming Customer’s rights to access and use Products, including total Number of Concurrent Sessions.

    9. "Product Installation" means a Product copy running on Customer’s computer device, hardware server or virtual machine.

    10. "Product Version" means a release, update, or upgrade of a particular Product that is not identified by Licensor as being made for the purpose of fixing software bugs.

    11. "Bug Fix Update" for a particular Product Version means a software update or release that is specifically identified by Licensor as a bug fix for that Product Version.

    12. "Fallback Date" means the date that was 12 months prior to the date of expiration of the Subscription.

    13. "Fallback Version" means the most recent Product Version that Licensor made available for public purchase prior to the Fallback Date, along with any Bug Fix Updates for that Product Version. For the purpose of clarity, Fallback Version does not include any Product updates or upgrades other than Bug Fix Updates that Customer may have used in the period between the Fallback Date and the date of expiration of the Subscription.

    14. "E-mail Support" means a form of customer support provided by the Licensor. At the time of writing, the corresponding e-mail address is support@aerokube.com; should the address be changed, the new address will be referred to on the Licensor’s web site.

    15. "Instant Messaging Support" means a form of customer support provided by the Licensor. At the time of writing, the corresponding address to support channel is https://t.me/aerokube_moon; should the address be changed, the new address will be referred to on the Licensor’s web site.

    16. "Affiliate" means any entity belonging to the same group as the Licensor.

  3. How this Agreement Works

    1. Entire Agreement. This Agreement, including the Third-Party Software license terms, constitutes the entire agreement between the parties concerning its subject matter and supersedes any prior agreements between Customer and Licensor regarding Customer’s use of any Products. No purchase order, other ordering document or any handwritten or typewritten text which purports to modify or supplement the printed text of this Agreement or any schedule will add to or vary the terms of this Agreement unless signed by both Customer and Licensor.

    2. Reservation of Rights. Aerokube reserves the right at any time to cease the support of the Product and to alter prices, features, specifications, capabilities, functions, terms of use, release dates, general availability or other characteristics of the Product.

    3. Changes to this Agreement. We may update or modify this Agreement from time to time, including any referenced policies and other documents. If a revision meaningfully reduces Customer’s rights, we will use reasonable efforts to notify Customer. If we modify this Agreement, the modified version of the Agreement will be effective from the start of the next Subscription term. In this case, if Customer objects to the updated Agreement terms, as Customer’s exclusive remedy, Customer may cancel the Subscription. Customer may be required to click through the updated Agreement to show its acceptance. For the avoidance of doubt, each Subscription Confirmation is subject to the version of the Agreement in effect on the Subscription Confirmation date.

    4. Opportunity to Review. Customer hereby declares that Customer has had sufficient opportunity to review this Agreement, understand the content of all of its clauses, negotiate its terms, and seek independent professional legal advice in that respect before entering into it. Consequently, any statutory "form contract" ("adhesion contract") regulations shall not be applicable to this Agreement.

    5. Severability. If a particular term of this Agreement is not enforceable, the unenforceability of that term will not affect any other terms of this Agreement.

    6. Headings. Headings and titles are for convenience only and do not affect the interpretation of this Agreement.

    7. No Waiver. Our failure to enforce or exercise any part of this Agreement is not a waiver of that section.

    8. Notice. Aerokube may deliver any notice to Customer via electronic mail to an email address provided by Customer, registered mail, personal delivery or renowned express courier (such as DHL, FedEx or UPS). Any such notice will be deemed to be effective:

      1. On the day the notice is sent to Customer via email.

      2. Upon personal delivery.

      3. One (1) day after deposit with an express courier or five (5) days after deposit in the mail, whichever occurs first.

    9. Governing Law. This Agreement will be governed by the laws of the Estonia, without reference to conflict of laws principles. Customer agrees that any litigation relating to this Agreement may only be brought in, and will be subject to the jurisdiction of, any competent court of the Estonia. The parties agree that the United Nations Convention on Contracts for the International Sale of Goods does not apply to this Agreement.

    10. Exceptions or Modifications. For exceptions or modifications to this Agreement, please contact Aerokube at: support@aerokube.com In case the terms of this Agreement are in conflict with the terms of any agreement individually negotiated and agreed between Aerokube and Customer, the terms of the latter shall prevail.

    11. Force Majeure. Except with respect to Customer’s payment obligations, neither party shall be liable to the other for any delay or failure to perform any obligation under this Agreement (except for a failure to pay fees) if the delay or failure is due to unforeseen events which occur after the signing of this Agreement and which are beyond the reasonable control of such party ("Force Majeure Event"), such as a strike, blockade, war, act of terrorism, riot, natural disaster, failure or diminishment of power or telecommunications or data networks or services, or refusal of a license by a government agency. In the event of a Force Majeure Event that prevents one part from substantially performing its obligations hereunder for a period of ten (10) days or more, either party may terminate this Agreement on five (5) days written notice.

  4. Grant of Rights

    1. The Product include code and libraries licensed to Licensor by third parties, including open source software.

    2. The Product is provided basing on the Number of Concurrent Sessions. If Customer complies with the terms of this Agreement, Customer has the rights stipulated hereunder for each Subscription that Customer acquires. Customer’s rights acquired in relation to the Product are limited to those necessary to enable Customer and its Users to effectively operate the Product(s). All other rights remain reserved to Licensor.

    3. Unless the Subscription has expired or this Agreement is terminated in accordance with respective section, and subject to the terms and conditions specified herein, Licensor grants Customer a non-exclusive and non-transferable right to use each Product covered by the Subscription as stipulated below.

    4. Customer may:

      1. For each License Key included to Subscription have one Product Installation of any version covered by the Subscription on any operating system supported by the Product.

      2. Do Product Evaluation on one Product Installation of any version on any operating system supported by the Product.

      3. Make one backup copy of the Product solely for archival/security backup purposes.

    5. Customer may not:

      1. Allow the same Product Installation to be used concurrently by more than the Number of Concurrent Sessions specified for used License Key in Subscription Confirmation.

      2. Rent, lease, reproduce, modify, adapt, create derivative works of, distribute, sell, or transfer the Product.

      3. Provide access to the Product or the right to use the Product to a third party.

      4. Reverse engineer, decompile, disassemble, modify, translate, make any attempt to discover the source code of the Product.

      5. Remove or obscure any proprietary or other notices contained in the Product.

    6. Following the expiration of this Agreement, the rights stipulated in "Grant of Rights" section shall continue on a perpetual, royalty-free, non-exclusive, and non-transferable basis for the continued use of a Fallback Version of each Product covered by the Subscription. The limitations set forth in this section apply to the usage of the Fallback Version. The rights granted in this section are expressly contingent upon Customer not being in breach of this Agreement, including having paid in full the applicable Subscription fees for the preceding 12 months or longer without interruption.

    7. Customer acknowledges that no ownership right is conveyed to Customer under this Agreement, irrespective of the use of terms such as "purchase" or "sale". Licensor has and retains all rights, title and interest, including all intellectual property rights, in and to the Products and any and all related or underlying technology, and any modifications or derivative works thereof, including without limitation as they may incorporate Feedback (as defined below).

    8. This Agreement applies whether Customer purchases a Subscription directly from Licensor or through resellers. If Customer purchases through a reseller, the Subscription details shall be as stated in the Subscription Confirmation issued by the reseller to Customer, and the reseller is responsible for the accuracy of any such Subscription Confirmation. Resellers are not authorized to make any promises or commitments on Licensor behalf, and Customer understands and agrees that Licensor is not bound by any obligations to Customer other than as specified in this Agreement.

  5. Access to Products

    1. All deliveries under this Agreement will be electronic. Customer and its Users must have an Internet connection in order to receive any deliveries. For the avoidance of doubt, Customer is responsible for downloading and installing the Products. Download instructions are made available on Licensor website at https://aerokube.com.

    2. Customer enables full access to Product Installation by specifying a License Key from Subscription Confirmation.

    3. Subject to the terms of this Agreement, Customer is granted a right to install and use the Product for evaluation purposes without charge for unlimited amount of time. The Product contains a feature that will automatically limit allowed Number of Concurrent Sessions. Licensor reserves the right at any time to change that limit in new Product versions.

  6. Fees

    1. Customer shall pay its Subscription fees in accordance with Licensor Terms of Purchase or the reseller’s terms of purchase, whichever are applicable.

    2. The Subscription fees shall be paid in full, and any levies, duties and/or taxes imposed by Customer’s jurisdiction (including, but not limited to, value added tax, sales tax and withholding tax), shall be borne solely by Customer.

    3. Customer may not deduct any amounts from fees payable to Licensor or the reseller, unless otherwise specified in the applicable terms of purchase.

  7. Feedback

    1. Customer has no obligation to provide Licensor with ideas, suggestions, or proposals ("Feedback").

    2. If Customer or Users submit Feedback to Licensor, then Customer grants Licensor a non-exclusive, worldwide, royalty-free license that is sub-licensable and transferable, to make, use, sell, have made, offer to sell, import, reproduce, publicly display, distribute, modify, or publicly perform the Feedback in any manner without any obligation, royalty or restriction based on intellectual property rights or otherwise.

  8. LIMITED WARRANTY

    ALL PRODUCTS ARE PROVIDED TO CUSTOMER ON AN "AS IS" AND "AS AVAILABLE" BASIS WITHOUT WARRANTIES. USE OF THE PRODUCTS IS AT YOUR OWN RISK. AEROKUBE MAKES NO WARRANTY AS TO THEIR USE OR PERFORMANCE. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, AEROKUBE, AND ITS SUPPLIERS (WHICH SHALL INCLUDE THE PROVIDERS OF THE THIRD PARTY SOFTWARE) AND RESELLERS, DISCLAIM ALL WARRANTIES AND CONDITIONS, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, AND NON-INFRINGEMENT, WITH REGARD TO THE PRODUCTS, AND THE PROVISION OF OR FAILURE TO PROVIDE SUPPORT SERVICES. THIS LIMITED WARRANTY GIVES CUSTOMER SPECIFIC LEGAL RIGHTS. CUSTOMER MAY HAVE OTHER RIGHTS, WHICH VARY FROM STATE/JURISDICTION TO STATE/JURISDICTION. AEROKUBE (AND ITS AFFILIATES, AGENTS, DIRECTORS AND EMPLOYEES) DOES NOT WARRANT:

    1. THAT THE PRODUCTS ARE ACCURATE, RELIABLE OR CORRECT

    2. THAT THE PRODUCTS WILL MEET YOUR REQUIREMENTS

    3. THAT THE PRODUCTS WILL BE AVAILABLE AT ANY PARTICULAR TIME OR LOCATION, UNINTERRUPTED OR SECURE

    4. THAT ANY DEFECTS OR ERRORS WILL BE CORRECTED

    5. THAT THE PRODUCTS ARE FREE OF VIRUSES OR OTHER HARMFUL COMPONENTS

    ANY CONTENT OR DATA DOWNLOADED OR OTHERWISE OBTAINED THROUGH THE USE OF THE PRODUCTS ARE DOWNLOADED AT YOUR OWN RISK AND YOU WILL BE SOLELY RESPONSIBLE FOR ANY DAMAGE TO YOUR PROPERTY OR LOSS OF DATA THAT RESULTS FROM SUCH DOWNLOAD. NO WARRANTY OR LIABILITY AT ALL IS GIVEN TO PRODUCTS UNDER EVALUATION.

  9. DISCLAIMER OF DAMAGES

    1. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT WILL AEROKUBE (OR ITS AFFILIATES, AGENTS, DIRECTORS, OR EMPLOYEES), OR AEROKUBE LICENSORS, SUPPLIERS OR RESELLERS BE LIABLE TO CUSTOMER OR ANYONE ELSE FOR:

      1. ANY LOSS OF USE, DATA, GOODWILL, OR PROFITS, WHETHER OR NOT FORESEEABLE

      2. ANY LOSS OR DAMAGES IN CONNECTION WITH TERMINATION OR SUSPENSION OF CUSTOMER’S ACCESS TO OUR PRODUCTS IN ACCORDANCE WITH THIS AGREEMENT

      3. ANY SPECIAL, INCIDENTAL, INDIRECT, CONSEQUENTIAL, EXEMPLARY OR PUNITIVE DAMAGES WHATSOEVER (EVEN IF WE HAVE BEEN ADVISED OF THE POSSIBILITY OF THESE DAMAGES), INCLUDING THOSE:

        1. RESULTING FROM LOSS OF USE, DATA, OR PROFITS, WHETHER OR NOT FORESEEABLE

        2. BASED ON ANY THEORY OF LIABILITY, INCLUDING BREACH OF CONTRACT OR WARRANTY, STRICT LIABILITY, NEGLIGENCE OR OTHER TORTIOUS ACTION

        3. ARISING FROM ANY OTHER CLAIM ARISING OUT OF OR IN CONNECTION WITH CUSTOMER’S USE OF OR ACCESS TO THE PRODUCTS OR SUPPORT.

    2. THE FOREGOING LIMITATION OF LIABILITY SHALL APPLY TO THE FULLEST EXTENT PERMITTED BY LAW IN THE APPLICABLE JURISDICTION.

    3. THE TOTAL LIABILITY IN ANY MATTER ARISING OUT OF OR IN RELATION TO THIS AGREEMENT IS LIMITED TO ONE HUNDRED (100) US DOLLARS OR THE AGGREGATE AMOUNT PAID OR PAYABLE BY THE CUSTOMER FOR PRODUCTS DURING THE THREE-MONTH PERIOD PRECEDING THE EVENT GIVING RISE TO THE LIABILITY, WHICHEVER IS GREATER. THIS LIMITATION WILL APPLY EVEN IF WE OR YOU HAVE BEEN ADVISED OF THE POSSIBILITY OF LIABILITY EXCEEDING SUCH AMOUNT AND NOTWITHSTANDING ANY FAILURE OF THE ESSENTIAL PURPOSE OF ANY LIMITED REMEDY.

  10. Term and Termination

    1. The term of this Agreement will commence upon acceptance of this Agreement by Customer as set forth in the preamble above, and will continue for each Product through the end of the applicable subscription period specified in the respective Subscription Confirmation. This Agreement will automatically renew with respect to each Product for a successive subscription term, unless terminated as set forth herein.

    2. Customer may terminate this Agreement at any time by cancelling its Product subscription. If such termination occurs during a then-current subscription period, this Agreement will continue to be effective until the end of that subscription period. Such termination does not relieve Customer of the obligation to pay any outstanding subscription fees owed to Licensor, and no credits or refunds will be issued to Customer for prepaid subscription fees (except as specified in the Licensor Terms of Purchase, if applicable).

    3. Licensor may terminate this agreement if:

      1. Customer has materially breached this Agreement and fails to cure such breach within thirty (30) days of written notice thereof.

      2. Customer fails to make the timely payment of subscription fees in accordance with "Fees" Section of this Agreement.

      3. Licensor is required to do so by law (for example, where the provision of the Product to Customer is, or becomes, unlawful).

      4. Licensor elects to discontinue providing the Product, in whole or in part.

    4. Licensor will make reasonable efforts to notify Customer via email as follows:

      1. Thirty (30) days prior to termination of the Agreement when required to terminate by law or because of discontinued Product. In such events Customer will be entitled to a refund of the unused portion of prepaid subscription fees, if applicable.

      2. Three (3) days prior to termination of the Agreement in other cases. In such events Customer will not be entitled to any refund of the unused portion of prepaid subscription fees.

    5. Survival. Upon the expiration or termination of this Agreement by Customer and if Customer elects to use the Fallback Version of the Product this Agreement statements will also survive with respect to said Fallback Version.

  11. Temporary Suspension for Non-payment

    1. Licensor reserves the right to suspend or limit Customer’s access to Aerokube Products if Customer fails to pay subscription fees on time.

    2. If Licensor suspends or limits Customer’s access to Aerokube Products for non-payment according, Customer must pay all past due amounts in order to restore full access to Aerokube Products.

    3. Customer hereby agrees that Licensor is entitled to charge Customer for the time period during which Customer has access to Aerokube Products until Customer or Licensor terminates or suspends Customer’s subscription in accordance with this Agreement.

  12. Export Regulations

    Customer shall comply with all applicable laws and regulations with regards to economic sanctions, export controls, import regulations, and trade embargoes (all herein referred to as "Sanctions"), including those of the European Union and United States (specifically the Export Administration Regulations (EAR)). Customer declares that it is not a person targeted by Sanctions nor is it otherwise owned or controlled by or acting on behalf of any person targeted by Sanctions. Further, Customer warrants that it will not download or otherwise export or re-export the Product or any related technical data directly or indirectly to any person targeted by Sanctions or download or otherwise use the Product for any end-use prohibited or restricted by Sanctions.

  13. Customer Support

    1. Licensor provides Email Support as well as Instant Messaging Support. The response time will be reasonable, but no specific response time guarantees are given.

    2. Customer may request additional paid support from Licensor which is subject of a supplementary individually negotiated Agreement between Customer and Licensor.

    3. Any guarantees of support availability only apply to the latest version of Licensed Software available in Customer Subscription.

  14. Customer Data

    1. Use of Name and Logo. Customer agrees that Licensor may identify it as a customer of Aerokube and may refer to it by name, trade name and trademark, if applicable. Licensor may also briefly describe Customer’s business in Licensor marketing materials, on the Aerokube website and/or in public or legal documents. Customer hereby grants Licensor a worldwide, non-exclusive and royalty-free license to use Customer’s name and any of Customer’s trade names and trademarks solely pursuant to this marketing section. Notwithstanding anything to the contrary herein, Licensor acknowledges that in some cases Customer licenses and does not own marks or logos (for example, marks or logos of the Affiliates) and cannot permit Licensor to use such marks.

    2. Gathering of Usage Statistics. Customer acknowledges and agrees that the Product may contain a feature that reports the usage statistics, diagnostics information and usage meta-information of the Product back to the Licensor. Customer may opt out of the gathering of usage statistics by turning off this feature in the Product settings.

6. Privacy Policy

This privacy policy is applicable to Moon Cloud solution only.

Last updated: December, 15th, 2020. Replaces the prior version in its entirety.

This Privacy Policy describes how Aerokube Software OÜ and its affiliates ("Aerokube", "we," "our" or "us") collect, use, and share information in connection with your use of Moon Cloud service (collectively, the "Services"). An affiliated company (referred to in this Privacy Notice as “Affiliated” or an “Affiliate”) means any entity that directly or indirectly controls, is controlled by, or is under common control with us, and “control” for purposes of this definition, means direct or indirect ownership or control of more than 50% of the voting interests of the subject entity. This Privacy Policy (the "Privacy Policy") does not apply to information our customers may process when using our Services.

In this Privacy Policy, the term "Personal Data" means data relating to a living individual who is or can be identified either from the data or from the data in conjunction with other information that is in, or is likely to come into, our possession, and includes personal data as described in Data Protection Legislation. Any Personal Data that we collect from you for these purposes is hereinafter referred to together as "Your Data".

We may collect and receive information about users of our Services ("users," "you," or "your") from various sources, including:

  1. Information you provide through your user account on the Services (your "Account") if you register for the Services

  2. Your use of the Services

  3. From third-party websites, services, and partners

We recommend that you read this Privacy Policy in full to ensure you are fully informed. If you have any questions about this Privacy Policy or Aerokube’s data collection, use, and disclosure practices, please contact us at privacy@aerokube.com. The data controller of your personal information is Aerokube Software OÜ.

6.1. Information We Collect

6.1.1. Information You Provide

  1. Account Registration. There is no dedicated account registration procedure. We create and provide one or several pairs of access credentials (user name and password) to Services ourselves.

  2. Communications. If you contact us directly, we may receive additional information about you such as your name, email address, phone number, the contents of the message with attachments you may send us, and any other information you may choose to provide. We may also receive a confirmation when you open an email from us.

6.1.2. Information We Collect When You Use Our Services

  1. Log Files. We gather certain information automatically and store it in log files. When you use our Services, we may collect certain information automatically from your device: IP addresses, internet service provider, browser type, operating system, date or time stamp, landing page and referring URL. In some countries, including countries in the European Economic Area, the information referenced above in this paragraph may be considered personal information under applicable data protection laws.

  2. Cookies. Cookie is a small text file that is placed on your device by a web server and allows a website to recognize your browser and to provide features based on who you are. We store information that we collect through cookies or log files to record your preferences. We may also automatically collect information about your use of features of our Services, about the functionality of our Services, frequency of visits, and other information related to your interactions with the Services.

  3. Usage of our Services. When you use our Services, we may collect information about your utilization of our Services. We collect and store system-level metrics such as processor and memory usage, storage capacity. We use this data to operate the Services, maintain and improve the performance and utilization of the Services, develop new features, protect the security and safety of our Services and our customers, and provide customer support. We also use this data to develop aggregate analysis and business intelligence that enable us to operate, protect, make informed decisions, and report on the performance of our business.

6.1.3. Information We Receive from Third Parties

  1. Third-Party Partners. We may receive publicly available information about you from our third-party partners and combine it with data that we have about you.

6.2. How We Use Information

We use the information we collect in various ways:

  • To provide, operate, maintain, improve, personalize, and expand our Services

  • To understand and analyze how you use our Services

  • To develop new products, services, features, and functionality

  • To communicate with you for customer service

  • To provide you with updates relating to the Service

  • For marketing and promotional purposes

  • To process your transactions

  • To find and prevent fraud

  • To enforce our Terms of Service or other legal rights

  • To comply to applicable laws and regulations when requested by any judicial process or governmental agency

6.3. How We Share Information

We may share the information we collect in various ways:

  • Vendors and Service Providers. We may share information with third-party vendors and service providers that provide services on our behalf, such as helping to provide our Services, for promotional or marketing purposes, to provide you with information relevant to you such as product announcements, software updates, special offers, or other information.

  • Aggregate Information. Where legally permissible, we may use and share information about users with our partners in aggregated or de-identified form that can’t reasonably be used to identify you.

  • Third-Party Partners. We also share information about users with third-party partners in order to receive additional publicly available information about you.

  • Analytics. We use analytics providers such as Google Analytics and Yandex Metrica.

  • Business Transfers. Information may be disclosed and otherwise transferred to any potential acquirer, successor, or assignee as part of any proposed merger, acquisition, debt financing, sale of assets, or similar transaction, or in the event of insolvency, bankruptcy, or receivership in which information is transferred to one or more third parties as one of our business assets.

  • When Required By Law. We may also share information to:

    1. Satisfy any applicable law, regulation, legal process, or governmental request

    2. Enforce this Privacy Policy and our Terms of Service, including investigation of potential violations hereof

    3. Detect, prevent, or otherwise address fraud, security, or technical issues

    4. Respond to your requests

    5. Protect our rights, property or safety, our users and the public

    This includes exchanging information with other companies and organizations for fraud protection and spam or malware prevention.

Where we wish to use Your Data in any other way, we will ensure that we notify you and get your consent first. You will be given the opportunity to withhold or withdraw your consent for the use of Your Data for purposes other than those listed in this Privacy Policy.

Aerokube is a global business. We may transfer personal information to countries other than the country in which the data was originally collected. These countries may not have the same data protection laws as the country in which you initially provided the information. When we transfer your personal information to other countries, we will protect that information as described in this Privacy Policy.

6.4. Access

  1. If you are a registered user, you may access certain information associated with your Account by logging into our Services or emailing privacy@aerokube.com.

  2. If you terminate your Account, any public activity on your Account prior to deletion may remain stored on our servers and may remain accessible to the public.

  3. To protect your privacy and security, we may also take reasonable steps to verify your identity before updating or removing your information.

  4. The information you provide us may be archived or stored periodically by us according to backup processes for disaster recovery purposes.

  5. Your ability to access and correct your information may be temporarily limited where access and correction could:

    • Inhibit our ability to comply with a legal obligation, to investigate, make or defend legal claims

    • Result in disclosure of personal information about a third party

    • Result in breach of a contract or disclosure of trade secrets or other proprietary business information belonging to Aerokube or a third party.

6.5. Data Retention

  1. We retain personal information we collect from you where we have an ongoing legitimate business need to do so. For example, to provide you with a service you have requested or to comply with applicable legal, tax, or accounting requirements.

  2. When we have no ongoing legitimate business need to process your personal information, we will either delete or anonymize it. If this is not possible (for example, because your personal information has been stored in backup archives), then we will securely store your personal information and isolate it from any further processing until deletion is possible.

  3. In general personal information that you provide to us, and any logs created by us relating to User Uploaded Data, will be kept and stored for 90 days from the date of upload/creation, after which point they will then be automatically deleted by us.

6.6. How We Protect Information

  1. We do our utmost to protect user privacy through the appropriate use of security technology. We restrict access to Your Data to employees, contractors and agents who need to know such Your Data in order to operate, develop or improve the services that we provide. We ensure that we have appropriate physical and technological security measures to protect your information; and we ensure that when we outsource any processes that the service provider has appropriate security measures in place.

  2. The Service may contain hyperlinks to websites owned and operated by third parties. These third party websites have their own privacy policies, including cookies. We do not accept any responsibility or liability for the privacy practices of such third party websites and your use of such websites is at your own risk.

  3. We will implement appropriate technical and organisational measures to ensure a level of security appropriate to the risks that are presented by the processing of Your Data. In particular, we will consider the risks presented by accidental or unlawful destruction, loss, alteration, unauthorised disclosure of, or access to Your Data transmitted, stored or otherwise processed.

  4. We use strong secure data transmission techniques such as TLS (HTTPs) on all parts of its service/website. You Data is always being trasmitted in encrypted form.

  5. Passwords and security tokens are stripped before they are logged into our system.

  6. Unfortunately, the transmission of information via the Internet is not completely secure. Although we will do our best to protect Your Data, we cannot guarantee the security of any data transmitted us and any such transmission is at your own risk. Once we have received your information, we will use strict procedures and security features to try to prevent unauthorised access. To the extent permitted by law, we are not responsible for any delays, delivery failures, or any other loss or damage resulting from the transfer of data over communications networks and facilities, including the internet, or any delay or delivery failure on the part of any other service provider not contracted by us, and you acknowledge that the Service may be subject to limitations, delays and other problems inherent in the use of such communications facilities. You will appreciate that we cannot guarantee the absolute prevention of cyber-attacks such as hacking, spyware and viruses. Accordingly, you will not hold us liable for any unauthorized disclosure, loss or destruction of Your Data arising from such risks.

6.7. Children’s Privacy

We do not knowingly collect information from children under the age of 13. Children under 13 are prohibited from using our Services. If you learn that a child has provided us with personal information in violation of this Privacy Policy, you can alert us at privacy@aerokube.com.

6.8. General Data Protection Regulation (GDPR)

If you are a resident of the European Economic Area (EEA), you have the following data protection rights:

  • To access, correct, update, or request deletion of your personal information. You can do so at any time by emailing privacy@aerokube.com.

  • In addition, you can object to the processing of your personal information, ask us to restrict the processing of your personal information, or request portability of your personal information. Again, you can exercise these rights by emailing privacy@aerokube.com.

  • You have the right to opt-out of marketing communications we send you at any time. You can exercise this right by clicking on the "unsubscribe" or "opt-out" link in the marketing emails we send you. To opt-out of other forms of marketing, please contact us by emailing privacy@aerokube.com.

  • Similarly, if we have collected and process your personal information with your consent, then you can withdraw your consent at any time. Withdrawing your consent will not affect the lawfulness of any processing we conducted prior to your withdrawal, nor will it affect the processing of your personal information conducted in reliance on lawful processing grounds other than consent.

  • You have the right to complain to a data protection authority about our collection and use of your personal information. For more information, please contact your local data protection authority.

We respond to all requests we receive from individuals wishing to exercise their data protection rights in accordance with applicable data protection laws.

6.9. Changes To This Privacy Policy

This Privacy Policy may be modified from time to time, so please review it frequently. Changes to this Privacy Policy will be posted on our websites. If we materially change the ways in which we use or share personal information previously collected from you through our Services, we will notify you through our Services, by email, or other communication.

6.10. How to Contact Us

If you need to contact us with regard to any of your rights as set out in this Policy, all such requests should be made in writing by email to privacy@aerokube.com.

7. Pricing

Last updated August 7th, 2019. Replaces the prior version in its entirety.

  1. Moon price is calculated using so-called Number of Concurrent Sessions that is to say total number of browser sessions being run in parallel. We control this by limiting total number of simultaneously running browser pods to the value you are purchasing.

  2. When no license key is provided 4 (four) parallel browser sessions maximum are allowed. If such limit is sufficient for you - you are allowed use Moon without license key for unlimited period of time.

  3. If free limit is insufficient - you need a paid license. Such license can include any desired number of parallel browser sessions (yes, even 42).

  4. Every parallel session has a fixed cost - $5 USD (five United States dollars). If you are a EU-based company - then we convert the price to euro (€) and an additional VAT 20% applies.

    An example price calculation
    42 sessions * $5/month = $210/month
  5. For simplicity we calculated monthly prices for some frequent cases:

    Table 19. Moon License Pricing
    Number of Parallel Sessions Price per Month, USD

    0-4

    free

    5

    $25

    10

    $50

    15

    $75

    20

    $100

    25

    $125

    30

    $150

    40

    $200

    50

    $250

    75

    $375

    100

    $500

    150

    $750

    200

    $1000

    250

    $1250

    500

    $2500

    750

    $3750

    1000

    $5000

Appendix A: Browser Image information