This reference covers version: latest.

1. Getting Started

1.1. Quick Start Guide

To use Go Grid Router do the following:

  1. Install Docker to host

  2. Create configuration directory:

    $ mkdir -p /etc/grid-router/quota
  3. Create users.htpasswd file:

    $ htpasswd -bc /etc/grid-router/users.htpasswd test test-password
  4. Start Selenoid on host selenoid.example.com and port 4444.

  5. Create quota file (use correct browser name and version):

    $ cat /etc/grid-router/quota/test.xml
    <qa:browsers xmlns:qa="urn:config.gridrouter.qatools.ru">
    <browser name="firefox" defaultVersion="88.0">
        <version number="88.0">
            <region name="1">
                <host name="selenoid.example.com" port="4444" count="1"/>
            </region>
        </version>
    </browser>
    </qa:browsers>
    File name should correspond to username you added to htpasswd file. For user test we added on previous steps you should create test.xml.
  6. Start Ggr container:

    # docker run -d --name \
        ggr -v /etc/grid-router/:/etc/grid-router:ro \
        --net host aerokube/ggr:latest-release
  7. Access Ggr on port 4444 in the same way you do for Selenium Hub but using the following url:

    http://test:test-password@localhost:4444/wd/hub

2. Configuration

Ggr is using two types of configuration files:

  1. A single file to store user credentials - users file.

  2. One username.xml file (quota file) for each user storing information about available browsers.

2.1. Creating Users File

Ggr is using htpasswd files to store authentication data. Passwords are stored in encrypted form. To create such file type:

  1. Ensure you have htpasswd utility installed (e.g. from apache2-utils package on Ubuntu).

  2. Create a new users file…​

    $ htpasswd -bc /path/to/new.htpasswd username password

    ... or update an existing one:

    $ htpasswd -b /path/to/existing.htpasswd username password

2.2. Creating Quota Files

  1. Quota files define available browsers for each user. All quota files should be placed to the same directory.

  2. For user username quota file should be named username.xml.

  3. Each quota file contains the following XML:

    A typical quota file
    <qa:browsers xmlns:qa="urn:config.gridrouter.qatools.ru">
    <browser name="firefox" defaultVersion="61.0">
        <version number="61.0">
            <region name="1">
                <host name="host1.example.com" port="4444" count="1"/>
                <host name="host2.example.com" port="4444" count="1"/>
                ...
            </region>
            <region name="2">
                ...
            </region>
        </version>
        <version number="60.0">
            ...
        </version>
    </browser>
    <browser name="chrome" defaultVersion="67.0">
        ...
    </browser>
    ...
    </qa:browsers>

    Here we define a list of browser names, their versions and default version for each browser. Each version has one or more regions (in cloud term, i.e. data centers). Every region contains one or more hosts. Each host defined in XML should have Selenium listening on specified port. The XML namespace is needed to be fully compatible with original Java GridRouter implementation.

    A frequent question being asked is the meaning of count attribute because the name for historical questions is a bit confusing. You can have hosts with different number of CPU and memory allowing to run different number of browsers simultaneously. So count is the relative host weight allowing to adjust the load to every host depending on its capacity. For example if your quota contains two hosts with count = 1 and count = 3 then new session requests will be distributed as 1:3 between these hosts. The easiest way to deliver uniform load distribution is to set count equal to total number of browsers available on the host. This is what we always recommend to do by default.
  4. Browser name is matched against browserName capability. Values are compared as strings and should be exactly equal. When testing mobile applications (e.g. with Appium) browserName capability makes no sense and can be replaced by deviceName or appium:deviceName capability. Ggr will try to match both against browser name specified in XML.

  5. Version number is matched against version or browserVersion capability by prefix. For example both 61 and 61.0 in version capability (i.e. in your code) will match version number 61.0.

  6. Similarly, version platform attribute is matched against platform or platformName capability by prefix. When platform from capabilities equals to ANY - default platform will be chosen.

  7. Sometimes you may need to have the same browser name and version on different platforms, e.g. Firefox on both Linux and Windows. To achieve this you need to add defaultPlatform and platform attributes to quota file as follows:

    Adding platform information to quota file
    <qa:browsers xmlns:qa="urn:config.gridrouter.qatools.ru">
    <browser name="firefox" defaultVersion="61.0" defaultPlatform="LINUX">
        <version number="61.0" platform="LINUX">
            ...
        </version>
        <version number="61.0" platform="WINDOWS">
            ...
        </version>
    </browser>
    ...
    </qa:browsers>

2.2.1. Proxying VNC Traffic

Sometimes for debugging purposes you need to see the screen of the browser where your test is being executed. The simplest way to do that is to install a VNC server to browser host. Default TCP port for VNC protocol is 5900. Ggr can proxy the VNC traffic from multiple hosts and makes it accessible on a WebSocket. For example having the running session ID you can get the VNC traffic by accessing the following endpoint:

Typical VNC traffic URL
ws://ggr-host.example.com:4444/vnc/<session-id>

Browser VNC clients like noVNC work with such URLs out of the box. By default, having an XML host entry like this…​

<host name="host1.example.com" port="4444" count="1"/>

…​Ggr proxies all traffic from host1.example.com:5900. You can customize this by adding an optional vnc attribute to a host:

<host name="host1.example.com" port="4444" count="1" vnc="vnc://host1.example.com:6900"/>

Two notations are supported in vnc attribute:

  1. If the remote server has a VNC server listening on dedicated TCP port you should use vnc:// protocol and specify only host and port:

    vnc://my-vnc-host.example.com:5900
  2. If the remote server also returns VNC traffic on a web socket (which is a case for Selenoid) - then you should specify a ws:// URL without trailing <session-id> value:

    ws://my-selenoid-host.example.com:4444/vnc

    Having this URL Ggr will append session ID and proxy VNC traffic from:

    ws://my-selenoid-host.example.com:4444/vnc/<session-id>

2.2.2. Working with External Selenium Services

Although Ggr is mainly used for creating your own Selenium cluster you can also configure it to obtain some browsers in external Selenium services such as Saucelabs, BrowserStack or TestingBot. These services always require username and password to be specified. Credentials should be set for each browser version in respective quota file:

Providing username and password for external Selenium service
<qa:browsers xmlns:qa="urn:config.gridrouter.qatools.ru">
<browser name="firefox" defaultVersion="45.0">
    <version number="45.0">
        <region name="1">
            <host name="hub.browserstack.com" port="443" count="1" username="test-user" password="my-password" scheme="https" />
        </region>
    </version>
</browser>
</qa:browsers>

2.2.3. Guest Quota

By default, every quota file corresponds to a user. In some cases you may need to give anonymous (guest) access to some users. This is very useful to maintain a sandbox where users can debug their tests without disturbing other tests. Guest quota is enabled using Ggr flags:

$ ./ggr -guests-allowed -guests-quota test <the-rest-of-the-flags...>

With these flags specified any browsers declared in test.xml file will be accessible without password.

2.2.4. Fetching Quota Information

You may want to show lists of available browsers in the user interface. To fetch this information from Ggr just use the following request:

$ curl -s http://test:test-password@example.com:4444/quota

This request returns quota file contents as JSON for specified user.

2.3. Quota Reloading

  • To reload quota files just send SIGHUP to process or Docker container:

    $ kill -HUP <pid>
    $ docker kill -s HUP <container-id-or-name>
    Use only one of these commands depending on whether you have Docker installed.

2.4. Proxying Session Video

Ggr gives you an ability to view Selenium session videos captured directly from browser screen.

  1. Video files are expected to be stored on the hub hosts specified in quota files and accessible via the following URL:

    http://hub-host.example.com:4444/video/<real-session-id>.mp4

    This API for example is supported by Selenoid.

  2. To download video file via Ggr just use the same request but with the session ID returned to test:

    $ curl -o video-file.mp4 http://ggr-host.example.com:4444/video/<test-session-id>
    Notice that session ID for hub host (aka real-session-id) and for Ggr (aka test-session-id) differ - the latter is longer because it contains information about hub host. Trying to use shorter session ID will not work.
  3. To delete video file via Ggr just change HTTP method to DELETE:

    $ curl -X DELETE http://ggr-host.example.com:4444/video/<test-session-id>

2.5. Proxying Session Logs

Similarly to video files Ggr gives you an ability to view Selenium session logs.

  1. To download a log file via Ggr send the following request:

    $ curl -o log-file.log http://ggr-host.example.com:4444/logs/<ggr-test-session-id>
  2. To delete log file via Ggr just change HTTP method to DELETE:

    $ curl -X DELETE http://ggr-host.example.com:4444/logs/<ggr-test-session-id>

2.6. Proxying Downloaded Files

Similarly to proxying video files Ggr is able to return any files downloaded by browser in running Selenium session.

  1. Downloaded files are expected to be stored on the hub hosts and accessible via the following URL:

    http://hub-host.example.com:4444/download/<real-session-id>/filename.txt

    Such notation for example is supported by Selenoid.

  2. To get downloaded file via Ggr just use the same request but with the session ID returned to test:

    $ curl http://ggr-host.example.com:4444/download/<test-session-id>/filename.txt

2.7. Proxying Browser Developer Tools

Similarly to proxying VNC traffic Ggr is able to proxy browser developer tools traffic in running Selenium session. To access developer tools - just use the following web socket URL:

ws://ggr-host.example.com:4444/devtools/<session-id>

2.8. Encrypting Connection to Ggr

Ggr itself does not support any modern encryption technologies such as TLS or WebSocket Secure. In order to use them you are expected to set up a reverse proxy having such capabilities. A typical Nginx configuration looks like the following:

Nginx’s configuration for encrypted connection
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

upstream ggr {
    server ggr1.example.com:4444 weight=10  max_fails=30 fail_timeout=180s;
    server ggr2.example.com:4444 weight=10  max_fails=30 fail_timeout=180s;
    server ggr3.example.com:4444 weight=10  max_fails=30 fail_timeout=180s;
}

server {
  server_name selenium.example.com;

  listen 4444 ssl;
  listen [::]:4444 ssl;

  ssl_prefer_server_ciphers on;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers kEECDH+AESGCM+AES128:kEECDH+AES128:kRSA+AESGCM+AES128:kRSA+AES128:DES-CBC3-SHA:!RC4:!aNULL:!eNULL:!MD5:!EXPORT:!LOW:!SEED:!CAMELLIA:!IDEA:!PSK:!SRP:!SSLv2;
  ssl_session_cache    shared:SSL:64m;
  ssl_session_timeout  28h;

  # These two files are private key and certificate from SSL certificate provider
  ssl_certificate /etc/ssl/selenium.pem;
  ssl_certificate_key /etc/ssl/selenium.key;

  access_log /var/log/nginx/selenium_access.log;
  error_log /var/log/nginx/selenium_error.log;

  location / {
    proxy_pass              http://ggr;
    proxy_http_version      1.1;
    proxy_set_header        Connection "";
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_connect_timeout   10;
    proxy_send_timeout      300;
    proxy_read_timeout      300;
    proxy_buffers           32 64m;
    proxy_buffer_size       64k;
    proxy_next_upstream    error timeout http_502 http_503 http_504;
    client_max_body_size    64m;
    client_body_buffer_size 64m;
    add_header Access-Control-Allow-Methods "GET,PUT,OPTIONS,POST,DELETE";
    add_header Access-Control-Allow-Origin "*";
    add_header Access-Control-Allow-Headers "Content-Type,Authorization";
    add_header Access-Control-Allow-Credentials "true";
  }

  location ~ ^/(vnc|devtools)/ {
    proxy_pass http://ggr;
    proxy_http_version 1.1;
    proxy_read_timeout 950s;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
  }

}

2.9. How it Works

See the full story in our article - Selenium testing: a new hope:

2.10. Using Multiple Ggr Instances

In order to obtain reliable Selenium cluster you need to install multiple Ggr instances behind reliable load balancer such as AWS Elastic Load Balancing or Keepalived. To check Ggr instance health use /ping:

Request
$ curl -s http://example.com:4444/ping
Result
{"uptime":"2m46.854829503s","lastReloadTime":"2017-05-12 12:33:06.322038542 +0300 MSK","numRequests":42, "numSessions":19, "version": "1.6.3"}

It returns 200 OK when Ggr operates normally. Additionally, server uptime, last quota reload time and overall number of session requests from service startup are returned in JSON format.

2.10.1. Why Ggr is Stateless

Selenium uses an HTTP-based protocol. That means every action in Selenium, e.g. launching browser or taking screenshot is a separate HTTP request. When multiple instances of Ggr are handling requests behind load balancer every request can be routed to any of these instances. Here’s how it works.

Multiple Ggr Instances Working Together

ui

On the picture above there is one SLB, two instances of Ggr and one Selenium hub. Every Selenium session consists of two stages: creating new session and other requests. When you request a new session you get its identifier called session ID. You must always use this session ID in consequent requests so Selenium applies these requests to correct browser process.

For example let’s assume that new session request is routed to Ggr 1 (black arrows). This Ggr randomly chooses a hub hub1.example.com and creates a new session on it. This hub returns some session identifier marked as ID on the picture. Ggr 1 in its turn knows that session was created on hub1.example.com and extends the session ID by adding an MD5 sum S of hub hostname. This is why it returns to user new longer session with identifier S+ID just by concatenating two strings. Every Ggr instance during startup creates an in-memory map - storing host names and their MD5 sums. When consequent request for the same session S+ID arrives e.g. to Ggr 2 (red arrows) it extracts S from extended session ID and finds hub hostname in its map. Then it removes S from request session identifier and simply proxies this request to hub corresponding to S.

Because of stateless Ggr architecture you can use an unlimited number of Ggr instances behind load balancer. In order to work properly it is very important to have exactly the same XML quota files on every Ggr instance. Otherwise, some Ggr instances will return 404 error when a request with unknown host S arrives.

2.10.2. Getting Host by Session ID

Sometimes you may want to get real hostname behind Ggr that is executing browser session. This can be done using session ID:

Request
$ curl -s http://test:test-password@ggr.example.com:4444/host/4355afe3f54e61eb320993066fb425133b79d8c3-e725-4f90-9e57-548b8ec3f160
Result
{"Name":"my-host.example.com","Port":4444,"Count":5,"Username":"","Password":""}

2.11. Log Files

A typical log file looks like the following:

2017/04/18 03:52:36 [12413389] [SESSION_ATTEMPTED] [my_quota] [192.168.2.3] [firefox-42.0] [firefox42-1.example.com:4444] [1]
2017/04/18 03:52:40 [12413389] [SESSION_FAILED] [my_quota] [192.168.2.3] [firefox-42.0] [firefox42-1.example.com:4444] Error forwarding the new session Request timed out waiting for a node to become available.
2017/04/18 03:52:40 [12413390] [SESSION_ATTEMPTED] [my_quota] [192.168.2.3] [firefox-42.0] [firefox42-5.example.com:4444] [2]
2017/04/18 03:52:45 [12413390] [5.86s] [SESSION_CREATED] [my_quota] [192.168.2.3] [firefox-42.0] [firefox42-5.example.com:4444] [0c500a6f-98d2-4871-acb7-637d85e1416a] [2]
....
2017/04/18 03:53:05 [SESSION_DELETED] [192.168.2.3] [firefox42-5.example.com:4444] [0c500a6f-98d2-4871-acb7-637d85e1416a]

Every line contains:

Table 1. Log entry contents
Field Example Notes

Time

2017/04/18 03:52:36

-

Request counter

[12413389]

So far as session ID is unknown when doing attempts this counter is used to find all session attempts for each new session request.

Status

[SESSION_ATTEMPTED]

See table below for complete list of statuses.

Quota name

[my_quota]

Extracted from basic HTTP auth headers.

User IP

[192.168.2.3]

IPv4 or IPv6 address

Browser

[firefox-42.0]

Name and version. Only present for new session requests.

Hub host

[firefox42-1.example.com:4444]

Host from quota XML file

Attempt number

[1]

For SESSION_ATTEMPTED entries means current attempt number. For SESSION_CREATED entries means total number of attempts to create this session.

Session ID

[0c500a6f-98d2-4871-acb7-637d85e1416a]

As arrived from hub

Session start time

[5.86s]

-

Error

Error forwarding the new session Request timed out waiting for a node to become available.

Only present for SESSION_FAILED

The following statuses are available:

Table 2. Log entry statuses
Status Description

BAD_CAPABILITY

User is passing non-string value for capability that should be string

BAD_JSON

User request does not contain valid Selenium data

BAD_RESPONSE

Upstream server returned data using unknown JSON protocol

BROWSER_NOT_SET

Browser name is not present or empty string

CLIENT_DISCONNECTED

User disconnected and doing session attempts was interrupted

INIT

Server initialization messages

INVALID_HOST_VNC_URL

Failed to parse VNC host URL specified in quota configuration

INVALID_DOWNLOAD_REQUEST_URL

Download request URL do not contain enough information to determine upstream host

INVALID_LOG_REQUEST_URL

Log request URL do not contain enough information to determine upstream host

INVALID_VNC_REQUEST_URL

VNC request URL do not contain enough information to determine upstream host

INVALID_VIDEO_REQUEST_URL

Video request URL do not contain enough information to determine upstream host

INVALID_URL

Session ID does not contain information about host where it was created

PROXY_ERROR

An error occurred while proxying request

PROXYING

Proxying Selenium request (shown in verbose mode only)

PROXYING_DOWNLOAD

Starting to proxy downloaded file from upstream host

PROXYING_LOG

Starting to proxy log from upstream host

PROXYING_TO_VNC

Starting to proxy VNC traffic

PROXYING_VIDEO

Starting to proxy video from upstream host

QUOTA_INFO_REQUESTED

Quota information request arrived

ROUTE_NOT_FOUND

Trying to proxy session to unknown host. Usually means quota files inconsistency between multiple Ggr instances.

SESSION_ATTEMPTED

New user request for session arrived

SESSION_CREATED

A new session was created and returned to user

SESSION_DELETED

Existing session was deleted by user request

SESSION_FAILED

Session attempt on specified host failed

SESSION_NOT_CREATED

Attempts to create a new session on all hosts failed. An error was returned to user.

SHUTTING_DOWN

Server is shutting down and waiting graceful shutdown timeout for currently proxied requests to finish

UNKNOWN_DEVTOOLS_HOST

Requested to proxy devtools to host not present in quota

UNKNOWN_DOWNLOAD_HOST

Requested to proxy downloaded file to host not present in quota

UNKNOWN_LOG_HOST

Requested to proxy log to host not present in quota

UNKNOWN_VNC_HOST

Requested to proxy VNC to host not present in quota

UNKNOWN_VIDEO_HOST

Requested to proxy video to host not present in quota

UNSUPPORTED_BROWSER

Requested browser name and version is not present in quota

UNSUPPORTED_HOST_VNC_SCHEME

Invalid URL protocol specified for host in quota VNC configuration (should be vnc:// or ws://)

WS_CLIENT_DISCONNECTED

Client disconnected from websocket API

WS_ERROR

An error occurred when trying to proxy websocket traffic

WS_SESSION_CLOSED

Client closed websocket session

2.11.1. Custom Labels in Log File

Sometimes you may need to add some custom metadata like build number or branch name to Selenium logs. Just add the following capability to your tests under ggr:options:

Type: map, format: "<key>": "<value>"
labels: {"buildNumber": 122, "branch": "feature-XXX"}

With this capability browser column will look like this:

[firefox-42.0 buildNumber=122 branch=feature-XXX]

Such additional metadata in logs allows to better analyze respective Selenium sessions.

2.12. Ggr CLI Flags

The following flags are supported by ggr command:

  -graceful-period duration
            graceful shutdown period in time.Duration format, e.g. 300s or 500ms (default 5m0s)
  -guests-allowed
            Allow guest (unauthenticated) users to access the grid
  -guests-quota string
            Which quota file to use for guests (default "guest")
  -listen string
            host and port to listen to (default ":4444")
  -quotaDir string
            quota directory (default "quota")
  -timeout duration
            session creation timeout in time.Duration format, e.g. 300s or 500ms (default 5m0s)
  -users string
            htpasswd auth file path (default ".htpasswd")
  -verbose
            enable verbose mode
  -version
            show version and exit

For example:

$ ./ggr -quotaDir /my/custom/quota/dir

When using Ggr inside Docker container these flags are passed like the following:

# docker run -d --name \
    ggr -v /etc/grid-router/:/etc/grid-router:ro \
    --net host aerokube/ggr:latest-release
    -quotaDir /my/custom/quota/dir

3. Contributing & Development

To build Ggr:

  1. Install Golang 1.12 and above

  2. Clone Ggr source:

    $ git clone https://github.com/aerokube/ggr.git
  3. Go to project directory:

    $ cd ggr
  4. Build source:

    $ go build
  5. Run Ggr:

    $ ./ggr --help

To build Docker container type:

$ GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build
$ docker build -t ggr:latest .

3.1. Documentation

Locally can be generated with:

$ docker run --rm -v `pwd`/docs/:/documents/      \
    asciidoctor/docker-asciidoctor                \
    asciidoctor -D /documents/output/ index.adoc