Ggr - A lightweight active load balancer used to create scalable and highly-available Selenium clusters

This reference covers version: latest.

1. Getting Started

1.1. Quick Start Guide

To use Go Grid Router do the following:

Install Docker to host
Create configuration directory:
```
$ mkdir -p /etc/grid-router/quota
```

Create users.htpasswd file:

$ htpasswd -bc /etc/grid-router/users.htpasswd test test-password

Start Selenoid on host selenoid.example.com and port 4444.

Create quota file (use correct browser name and version):

$ cat /etc/grid-router/quota/test.xml
<qa:browsers xmlns:qa="urn:config.gridrouter.qatools.ru">
<browser name="firefox" defaultVersion="88.0">
    <version number="88.0">
        <region name="1">
            <host name="selenoid.example.com" port="4444" count="1"/>
        </region>
    </version>
</browser>
</qa:browsers>

File name should correspond to username you added to htpasswd file. For user test we added on previous steps you should create test.xml.

Start Ggr container:

# docker run -d --name \
    ggr -v /etc/grid-router/:/etc/grid-router:ro \
    --net host aerokube/ggr:latest-release

Access Ggr on port 4444 in the same way you do for Selenium Hub but using the following url:
```
http://test:test-password@localhost:4444/wd/hub
```

2. Configuration

Ggr is using two types of configuration files:

A single file to store user credentials - users file.
One username.xml file (quota file) for each user storing information about available browsers.

2.1. Creating Users File

Ggr is using htpasswd files to store authentication data. Passwords are stored in encrypted form. To create such file type:

Ensure you have htpasswd utility installed (e.g. from apache2-utils package on Ubuntu).

Create a new users file…

$ htpasswd -bc /path/to/new.htpasswd username password

... or update an existing one:

$ htpasswd -b /path/to/existing.htpasswd username password

2.2. Creating Quota Files

Quota files define available browsers for each user. All quota files should be placed to the same directory.
For user username quota file should be named username.xml.

Each quota file contains the following XML:

A typical quota file

<qa:browsers xmlns:qa="urn:config.gridrouter.qatools.ru">
<browser name="firefox" defaultVersion="61.0">
    <version number="61.0">
        <region name="1">
            <host name="host1.example.com" port="4444" count="1"/>
            <host name="host2.example.com" port="4444" count="1"/>
            ...
        </region>
        <region name="2">
            ...
        </region>
    </version>
    <version number="60.0">
        ...
    </version>
</browser>
<browser name="chrome" defaultVersion="67.0">
    ...
</browser>
...
</qa:browsers>

Here we define a list of browser names, their versions and default version for each browser. Each version has one or more regions (in cloud term, i.e. data centers). Every region contains one or more hosts. Each host defined in XML should have Selenium listening on specified port. The XML namespace is needed to be fully compatible with original Java GridRouter implementation.

A frequent question being asked is the meaning of count attribute because the name for historical questions is a bit confusing. You can have hosts with different number of CPU and memory allowing to run different number of browsers simultaneously. So count is the relative host weight allowing to adjust the load to every host depending on its capacity. For example if your quota contains two hosts with count = 1 and count = 3 then new session requests will be distributed as 1:3 between these hosts. The easiest way to deliver uniform load distribution is to set count equal to total number of browsers available on the host. This is what we always recommend to do by default.

Browser name is matched against browserName capability. Values are compared as strings and should be exactly equal. When testing mobile applications (e.g. with Appium) browserName capability makes no sense and can be replaced by deviceName or appium:deviceName capability. Ggr will try to match both against browser name specified in XML.
Version number is matched against version or browserVersion capability by prefix. For example both 61 and 61.0 in version capability (i.e. in your code) will match version number 61.0.
Similarly, version platform attribute is matched against platform or platformName capability by prefix. When platform from capabilities equals to ANY - default platform will be chosen.

Sometimes you may need to have the same browser name and version on different platforms, e.g. Firefox on both Linux and Windows. To achieve this you need to add defaultPlatform and platform attributes to quota file as follows:

Adding platform information to quota file

<qa:browsers xmlns:qa="urn:config.gridrouter.qatools.ru">
<browser name="firefox" defaultVersion="61.0" defaultPlatform="LINUX">
    <version number="61.0" platform="LINUX">
        ...
    </version>
    <version number="61.0" platform="WINDOWS">
        ...
    </version>
</browser>
...
</qa:browsers>

2.2.1. Proxying VNC Traffic

Sometimes for debugging purposes you need to see the screen of the browser where your test is being executed. The simplest way to do that is to install a VNC server to browser host. Default TCP port for VNC protocol is 5900. Ggr can proxy the VNC traffic from multiple hosts and makes it accessible on a WebSocket. For example having the running session ID you can get the VNC traffic by accessing the following endpoint:

Typical VNC traffic URL

ws://ggr-host.example.com:4444/vnc/<session-id>

Browser VNC clients like noVNC work with such URLs out of the box. By default, having an XML host entry like this…

<host name="host1.example.com" port="4444" count="1"/>

…Ggr proxies all traffic from host1.example.com:5900. You can customize this by adding an optional vnc attribute to a host:

<host name="host1.example.com" port="4444" count="1" vnc="vnc://host1.example.com:6900"/>

Two notations are supported in vnc attribute:

If the remote server has a VNC server listening on dedicated TCP port you should use vnc:// protocol and specify only host and port:
```
vnc://my-vnc-host.example.com:5900
```
If the remote server also returns VNC traffic on a web socket (which is a case for Selenoid) - then you should specify a ws:// URL without trailing <session-id> value:
```
ws://my-selenoid-host.example.com:4444/vnc
```
Having this URL Ggr will append session ID and proxy VNC traffic from:
```
ws://my-selenoid-host.example.com:4444/vnc/<session-id>
```

2.2.2. Working with External Selenium Services

Although Ggr is mainly used for creating your own Selenium cluster you can also configure it to obtain some browsers in external Selenium services such as Saucelabs, BrowserStack or TestingBot. These services always require username and password to be specified. Credentials should be set for each browser version in respective quota file:

Providing username and password for external Selenium service

<qa:browsers xmlns:qa="urn:config.gridrouter.qatools.ru">
<browser name="firefox" defaultVersion="45.0">
    <version number="45.0">
        <region name="1">
            <host name="hub.browserstack.com" port="443" count="1" username="test-user" password="my-password" scheme="https" />
        </region>
    </version>
</browser>
</qa:browsers>

2.2.3. Guest Quota

By default, every quota file corresponds to a user. In some cases you may need to give anonymous (guest) access to some users. This is very useful to maintain a sandbox where users can debug their tests without disturbing other tests. Guest quota is enabled using Ggr flags:

$ ./ggr -guests-allowed -guests-quota test <the-rest-of-the-flags...>

With these flags specified any browsers declared in test.xml file will be accessible without password.

2.2.4. Fetching Quota Information

You may want to show lists of available browsers in the user interface. To fetch this information from Ggr just use the following request:

$ curl -s http://test:test-password@example.com:4444/quota

This request returns quota file contents as JSON for specified user.

2.3. Quota Reloading

To reload quota files just send SIGHUP to process or Docker container:
```
$ kill -HUP <pid>
$ docker kill -s HUP <container-id-or-name>
```
Use only one of these commands depending on whether you have Docker installed.

2.4. Proxying Session Video

Ggr gives you an ability to view Selenium session videos captured directly from browser screen.

Video files are expected to be stored on the hub hosts specified in quota files and accessible via the following URL:
```
http://hub-host.example.com:4444/video/<real-session-id>.mp4
```
This API for example is supported by Selenoid.

To download video file via Ggr just use the same request but with the session ID returned to test:

$ curl -o video-file.mp4 http://ggr-host.example.com:4444/video/<test-session-id>

Notice that session ID for hub host (aka real-session-id) and for Ggr (aka test-session-id) differ - the latter is longer because it contains information about hub host. Trying to use shorter session ID will not work.

To delete video file via Ggr just change HTTP method to DELETE:

$ curl -X DELETE http://ggr-host.example.com:4444/video/<test-session-id>

2.5. Proxying Session Logs

Similarly to video files Ggr gives you an ability to view Selenium session logs.

To download a log file via Ggr send the following request:

$ curl -o log-file.log http://ggr-host.example.com:4444/logs/<ggr-test-session-id>

To delete log file via Ggr just change HTTP method to DELETE:

$ curl -X DELETE http://ggr-host.example.com:4444/logs/<ggr-test-session-id>

2.6. Proxying Downloaded Files

Similarly to proxying video files Ggr is able to return any files downloaded by browser in running Selenium session.

Downloaded files are expected to be stored on the hub hosts and accessible via the following URL:
```
http://hub-host.example.com:4444/download/<real-session-id>/filename.txt
```
Such notation for example is supported by Selenoid.
To get downloaded file via Ggr just use the same request but with the session ID returned to test:
```
$ curl http://ggr-host.example.com:4444/download/<test-session-id>/filename.txt
```

2.7. Proxying Browser Developer Tools

Similarly to proxying VNC traffic Ggr is able to proxy browser developer tools traffic in running Selenium session. To access developer tools - just use the following web socket URL:

ws://ggr-host.example.com:4444/devtools/<session-id>

2.8. Encrypting Connection to Ggr

Ggr itself does not support any modern encryption technologies such as TLS or WebSocket Secure. In order to use them you are expected to set up a reverse proxy having such capabilities. A typical Nginx configuration looks like the following:

Nginx’s configuration for encrypted connection

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

upstream ggr {
    server ggr1.example.com:4444 weight=10  max_fails=30 fail_timeout=180s;
    server ggr2.example.com:4444 weight=10  max_fails=30 fail_timeout=180s;
    server ggr3.example.com:4444 weight=10  max_fails=30 fail_timeout=180s;
}

server {
  server_name selenium.example.com;

  listen 4444 ssl;
  listen [::]:4444 ssl;

  ssl_prefer_server_ciphers on;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers kEECDH+AESGCM+AES128:kEECDH+AES128:kRSA+AESGCM+AES128:kRSA+AES128:DES-CBC3-SHA:!RC4:!aNULL:!eNULL:!MD5:!EXPORT:!LOW:!SEED:!CAMELLIA:!IDEA:!PSK:!SRP:!SSLv2;
  ssl_session_cache    shared:SSL:64m;
  ssl_session_timeout  28h;

  # These two files are private key and certificate from SSL certificate provider
  ssl_certificate /etc/ssl/selenium.pem;
  ssl_certificate_key /etc/ssl/selenium.key;

  access_log /var/log/nginx/selenium_access.log;
  error_log /var/log/nginx/selenium_error.log;

  location / {
    proxy_pass              http://ggr;
    proxy_http_version      1.1;
    proxy_set_header        Connection "";
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_connect_timeout   10;
    proxy_send_timeout      300;
    proxy_read_timeout      300;
    proxy_buffers           32 64m;
    proxy_buffer_size       64k;
    proxy_next_upstream    error timeout http_502 http_503 http_504;
    client_max_body_size    64m;
    client_body_buffer_size 64m;
    add_header Access-Control-Allow-Methods "GET,PUT,OPTIONS,POST,DELETE";
    add_header Access-Control-Allow-Origin "*";
    add_header Access-Control-Allow-Headers "Content-Type,Authorization";
    add_header Access-Control-Allow-Credentials "true";
  }

  location ~ ^/(vnc|devtools)/ {
    proxy_pass http://ggr;
    proxy_http_version 1.1;
    proxy_read_timeout 950s;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
  }

}

2.9. How it Works

See the full story in our article - Selenium testing: a new hope:

2.10. Using Multiple Ggr Instances

In order to obtain reliable Selenium cluster you need to install multiple Ggr instances behind reliable load balancer such as AWS Elastic Load Balancing or Keepalived. To check Ggr instance health use /ping:

Request

$ curl -s http://example.com:4444/ping

Result

{"uptime":"2m46.854829503s","lastReloadTime":"2017-05-12 12:33:06.322038542 +0300 MSK","numRequests":42, "numSessions":19, "version": "1.6.3"}

It returns 200 OK when Ggr operates normally. Additionally, server uptime, last quota reload time and overall number of session requests from service startup are returned in JSON format.

2.10.1. Why Ggr is Stateless

Selenium uses an HTTP-based protocol. That means every action in Selenium, e.g. launching browser or taking screenshot is a separate HTTP request. When multiple instances of Ggr are handling requests behind load balancer every request can be routed to any of these instances. Here’s how it works.

Multiple Ggr Instances Working Together

On the picture above there is one SLB, two instances of Ggr and one Selenium hub. Every Selenium session consists of two stages: creating new session and other requests. When you request a new session you get its identifier called session ID. You must always use this session ID in consequent requests so Selenium applies these requests to correct browser process.

For example let’s assume that new session request is routed to Ggr 1 (black arrows). This Ggr randomly chooses a hub hub1.example.com and creates a new session on it. This hub returns some session identifier marked as ID on the picture. Ggr 1 in its turn knows that session was created on hub1.example.com and extends the session ID by adding an MD5 sum S of hub hostname. This is why it returns to user new longer session with identifier S+ID just by concatenating two strings. Every Ggr instance during startup creates an in-memory map - storing host names and their MD5 sums. When consequent request for the same session S+ID arrives e.g. to Ggr 2 (red arrows) it extracts S from extended session ID and finds hub hostname in its map. Then it removes S from request session identifier and simply proxies this request to hub corresponding to S.

Because of stateless Ggr architecture you can use an unlimited number of Ggr instances behind load balancer. In order to work properly it is very important to have exactly the same XML quota files on every Ggr instance. Otherwise, some Ggr instances will return 404 error when a request with unknown host S arrives.

2.10.2. Getting Host by Session ID

Sometimes you may want to get real hostname behind Ggr that is executing browser session. This can be done using session ID:

Request

$ curl -s http://test:test-password@ggr.example.com:4444/host/4355afe3f54e61eb320993066fb425133b79d8c3-e725-4f90-9e57-548b8ec3f160

Result

{"Name":"my-host.example.com","Port":4444,"Count":5,"Username":"","Password":""}

2.11. Log Files

A typical log file looks like the following:

2017/04/18 03:52:36 [12413389] [SESSION_ATTEMPTED] [my_quota] [192.168.2.3] [firefox-42.0] [firefox42-1.example.com:4444] [1]
2017/04/18 03:52:40 [12413389] [SESSION_FAILED] [my_quota] [192.168.2.3] [firefox-42.0] [firefox42-1.example.com:4444] Error forwarding the new session Request timed out waiting for a node to become available.
2017/04/18 03:52:40 [12413390] [SESSION_ATTEMPTED] [my_quota] [192.168.2.3] [firefox-42.0] [firefox42-5.example.com:4444] [2]
2017/04/18 03:52:45 [12413390] [5.86s] [SESSION_CREATED] [my_quota] [192.168.2.3] [firefox-42.0] [firefox42-5.example.com:4444] [0c500a6f-98d2-4871-acb7-637d85e1416a] [2]
....
2017/04/18 03:53:05 [SESSION_DELETED] [192.168.2.3] [firefox42-5.example.com:4444] [0c500a6f-98d2-4871-acb7-637d85e1416a]

Every line contains:

Table 1. Log entry contents
Field	Example	Notes
Time	2017/04/18 03:52:36	-
Request counter	[12413389]	So far as session ID is unknown when doing attempts this counter is used to find all session attempts for each new session request.
Status	[SESSION_ATTEMPTED]	See table below for complete list of statuses.
Quota name	[my_quota]	Extracted from basic HTTP auth headers.
User IP	[192.168.2.3]	IPv4 or IPv6 address
Browser	[firefox-42.0]	Name and version. Only present for new session requests.
Hub host	[firefox42-1.example.com:4444]	Host from quota XML file
Attempt number	[1]	For SESSION_ATTEMPTED entries means current attempt number. For SESSION_CREATED entries means total number of attempts to create this session.
Session ID	[0c500a6f-98d2-4871-acb7-637d85e1416a]	As arrived from hub
Session start time	[5.86s]	-
Error	Error forwarding the new session Request timed out waiting for a node to become available.	Only present for SESSION_FAILED

The following statuses are available:

Table 2. Log entry statuses
Status	Description
BAD_CAPABILITY	User is passing non-string value for capability that should be string
BAD_JSON	User request does not contain valid Selenium data
BAD_RESPONSE	Upstream server returned data using unknown JSON protocol
BROWSER_NOT_SET	Browser name is not present or empty string
CLIENT_DISCONNECTED	User disconnected and doing session attempts was interrupted
INIT	Server initialization messages
INVALID_HOST_VNC_URL	Failed to parse VNC host URL specified in quota configuration
INVALID_DOWNLOAD_REQUEST_URL	Download request URL do not contain enough information to determine upstream host
INVALID_LOG_REQUEST_URL	Log request URL do not contain enough information to determine upstream host
INVALID_VNC_REQUEST_URL	VNC request URL do not contain enough information to determine upstream host
INVALID_VIDEO_REQUEST_URL	Video request URL do not contain enough information to determine upstream host
INVALID_URL	Session ID does not contain information about host where it was created
PROXY_ERROR	An error occurred while proxying request
PROXYING	Proxying Selenium request (shown in verbose mode only)
PROXYING_DOWNLOAD	Starting to proxy downloaded file from upstream host
PROXYING_LOG	Starting to proxy log from upstream host
PROXYING_TO_VNC	Starting to proxy VNC traffic
PROXYING_VIDEO	Starting to proxy video from upstream host
QUOTA_INFO_REQUESTED	Quota information request arrived
ROUTE_NOT_FOUND	Trying to proxy session to unknown host. Usually means quota files inconsistency between multiple Ggr instances.
SESSION_ATTEMPTED	New user request for session arrived
SESSION_CREATED	A new session was created and returned to user
SESSION_DELETED	Existing session was deleted by user request
SESSION_FAILED	Session attempt on specified host failed
SESSION_NOT_CREATED	Attempts to create a new session on all hosts failed. An error was returned to user.
SHUTTING_DOWN	Server is shutting down and waiting graceful shutdown timeout for currently proxied requests to finish
UNKNOWN_DEVTOOLS_HOST	Requested to proxy devtools to host not present in quota
UNKNOWN_DOWNLOAD_HOST	Requested to proxy downloaded file to host not present in quota
UNKNOWN_LOG_HOST	Requested to proxy log to host not present in quota
UNKNOWN_VNC_HOST	Requested to proxy VNC to host not present in quota
UNKNOWN_VIDEO_HOST	Requested to proxy video to host not present in quota
UNSUPPORTED_BROWSER	Requested browser name and version is not present in quota
UNSUPPORTED_HOST_VNC_SCHEME	Invalid URL protocol specified for host in quota VNC configuration (should be vnc:// or ws://)
WS_CLIENT_DISCONNECTED	Client disconnected from websocket API
WS_ERROR	An error occurred when trying to proxy websocket traffic
WS_SESSION_CLOSED	Client closed websocket session

2.11.1. Custom Labels in Log File

Sometimes you may need to add some custom metadata like build number or branch name to Selenium logs. Just add the following capability to your tests under ggr:options:

Type: map, format: "<key>": "<value>"

labels: {"buildNumber": 122, "branch": "feature-XXX"}

With this capability browser column will look like this:

[firefox-42.0 buildNumber=122 branch=feature-XXX]

Such additional metadata in logs allows to better analyze respective Selenium sessions.

2.12. Ggr CLI Flags

The following flags are supported by ggr command:

  -graceful-period duration
            graceful shutdown period in time.Duration format, e.g. 300s or 500ms (default 5m0s)
  -guests-allowed
            Allow guest (unauthenticated) users to access the grid
  -guests-quota string
            Which quota file to use for guests (default "guest")
  -listen string
            host and port to listen to (default ":4444")
  -quotaDir string
            quota directory (default "quota")
  -timeout duration
            session creation timeout in time.Duration format, e.g. 300s or 500ms (default 5m0s)
  -users string
            htpasswd auth file path (default ".htpasswd")
  -verbose
            enable verbose mode
  -version
            show version and exit

For example:

$ ./ggr -quotaDir /my/custom/quota/dir

When using Ggr inside Docker container these flags are passed like the following:

# docker run -d --name \
    ggr -v /etc/grid-router/:/etc/grid-router:ro \
    --net host aerokube/ggr:latest-release
    -quotaDir /my/custom/quota/dir

3. Contributing & Development

To build Ggr:

Install Golang 1.12 and above

Clone Ggr source:

$ git clone https://github.com/aerokube/ggr.git

Go to project directory:
```
$ cd ggr
```
Build source:
```
$ go build
```
Run Ggr:
```
$ ./ggr --help
```

To build Docker container type:

$ GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build
$ docker build -t ggr:latest .

3.1. Documentation

Locally can be generated with:

$ docker run --rm -v `pwd`/docs/:/documents/      \
    asciidoctor/docker-asciidoctor                \
    asciidoctor -D /documents/output/ index.adoc