Installation and Upgrade

Requirements

The resource availability monitor is delivered as a self-contained docker image using the Docker Content Trust (DCT) system to ensure the integrity and publisher of all the data downloaded. Installation of the image can be undertaken on any host running docker¹.

In this document the host running the monitor container is referred to as the docker host. This host must have network access to any cluster resources to be monitored so it in turn can make those resources available to the container running the monitor itself.

For example, to monitor a clustered NFS share then the docker host must be able to mount and access that share, as the docker container will require access to successfully monitor that resources availability. Also note, the process running the monitor docker image must belong to the docker group on the host OS (this is a requirement of docker).

Running under docker, the monitor tool requires a persistent location to store its data, logs and ancillary files. This location is provided by the host OS and mapped to the directory /tmp/hamonitor when the container is started (see section 3 Starting the Monitor). This way existing configurations, logs and other components are preserved during upgrades, migrations etc.

Installation

The monitor is distributed from a docker content trust server and is installed as follows:

Download and install the docker application package from www.docker.com onto the docker host machine. Start the docker daemon on the host machine.
Enable docker trust; docker uses environment variables to modify its behaviour, therefore, to enable trust, set the following:
Unix command shell
```
export DOCKER_CONTENT_TRUST=1
```
Windows power shell
```
$Env: DOCKER_CONTENT_TRUST=1
```

Set the high-availability notary server for the trust system, again using an environment variable:

Unix command shell

export DOCKER_CONTENT_TRUST_SERVER=https://notary-server.high-availability.com:4443

Windows power shell

$Env:DOCKER_CONTENT_TRUST_SERVER="https://notary-server.high-availability.com:4443"

In order to use the docker registry for trusted downloads it is necessary to have a username, password combination - this should be requested using the email address docker-trust@high-availability.com.
Using the user/password combination retrieved in the previous step, login to the docker framework:
```
# docker login dkr.high-availability.com
```
The docker login subcommand will prompt for a username and password. Note that once you have successfully logged into the server, docker saves a login token locally in the users’ home directory in the file .docker/config.json thereby avoiding the need for this user to login again. The token can be cleared using the docker logout subcommand.

Inspect the list of signed monitor images available from the registry:

# docker trust inspect --pretty dkr.high-availability.com/hamonitor

here is some example output showing two signed versions of the monitor:

Signatures for dkr.high-availability.com/hamonitor

SIGNED_TAG DIGEST                                                  SIGNERS
v1.0 46d706ebead9e7746b3c1ffcbc2247562d035a5eed85410dc54eebe5c1aed hacsigner
v1.1 eaea423478652348753463487563487563324856473285763434876538475 hacsigner

List of signers and their keys for dkr.high-availability.com/hamonitor

SIGNER    KEYS
hacsigner 346c3155f96c

Administrative keys for dkr.high-availability.com/hamonitor

Repository Key: 2058e5cfcc725b7b00607c60941e6105d5709ab20d56278ac3a4e1dd386c0
Root Key: a6ae8b3cc1ac73aa34d4237d6c0562fb5d69c5eaa3ab9a9e78b2ac3ecce93c39

Download the desired version:

# docker pull dkr.high-availability.com/hamonitor:v1.0

Note that with the trust framework enabled, the docker command line tool takes care of validating the digest of the images downloaded and checks against the official signatures held by the notary server. Any image that has not been signed and verified will be blocked from download.
The output from the docker pull should look similar to the following:

Pull (1 of 1): dkr.high-availability.com/hamonitor:v1.0@sha256:46d706ebead9e7746b3c1ffcbc2247562d038865a5eed85410dc54eebe5c1aed
sha256:46d706ebead9e7746b3c1ffcbc2247562d038865a5eed85410dc54eebe5c1aed:

Pulling from hamonitor

ab3acf868d91: Pull complete
bf8f3d9e8100: Pull complete
4cf71b2b4422: Pull complete
668c80dc67a6: Pull complete
1b527012fdfd: Pull complete
ade8b6ab4354: Pull complete
4849fab77f68: Pull complete
ccedab781a09: Pull complete

Status: Downloaded newer image for dkr.high-availability.com/hamonitor@sha256:46d706ebead9e7746b3c1ffcbc2247562d038865a5eed85410dc54eebe5c1aed

Tagging dkr.high-availability.com/hamonitor@sha256:46d706ebead9e7746b3c1ffcbc2247562d038865a5eed85410dc54eebe5c1aed as dkr.high-availability.com/hamonitor:v1.0
dkr.high-availability.com/hamonitor:v1.0 docker pull dkr.high-availability.com/hamonitor:v1.0

Confirmation of the downloaded image can be performed by comparing the digest of the local images with those held by the remote trust server (use the trust inspect docker subcommand as detailed earlier to retrieve the remote image digests). To list the digest of locally installed images use the command:
```
# docker images --digests
```
The output will look similar to the following - the third digest fields should correspond to the digest listed by the trust server:
```
REPOSITORY                          TAG  DIGEST                                                                  IMAGE ID     CREATED     SIZE
dkr.high-availability.com/hamonitor v1.0 sha256:46d706ebead9e7746b3c1ffcbc2247562d038865a5eed85410dc54eebe5c1aed fc254100c107 4 weeks ago 768MB
```

Installation and upgrade on dark sites

To install the monitor on hosts that have no external internet connection (and thus cannot make a connection to the docker trust server) necessitates a two-step approach, the result of which creates an offline image which can then be used to install the monitor on any hosts, regardless of whether they have external connectivity or not.

The first step in creating the image is to designate a download host (with internet connectivity) to download the monitor as described in the previous section. The next step is to create an image which can be shipped to the non-connected hosts and installed locally. Creating an image is accomplished as follows:

On the host where the monitor has been downloaded create an image of the monitor:
```
# docker image save -o hamonitor_v1.0.tar dkr.high-availability.com/hamonitor:v1.0
```
The newly created tar file (in this example hamonitor_v1.0.tar) can then be copied to any host and installed using the following command:
```
# docker load -i hamonitor_v1.0.tar
```

Finally, on the host where the image was loaded, check the image ID is the same, output will be similar to the following:

# docker images --digests
REPOSITORY                          TAG  DIGEST IMAGE ID     CREATED  SIZE
dkr.high-availability.com/hamonitor v1.0 <none> fc254100c107 3 months 768MB

The monitor is now installed and ready to run. To upgrade a dark site installation simply create a new image using docker image save from the updated local image and add it to the dark site host using docker load.

The <none> digest column for loaded images

When an image is loaded from an image file (created using the docker image save command), the digest field will always be shown as . To understand why this is first requires some understanding of where the digest field originates from.

When an image is pushed to a docker registry, the layers that go to make up that image are transferred over in an uncompressed format. When docker saves those layers, it saves them in a compressed format. Once all the layers that make up an image have been received and stored in compressed format, docker then creates an image manifest listing all the layers and a SHA256 checksum for each compressed layer. Once the manifest has been created a digest is calculated and the image tag is signed. The signatures and digest can be seen by inspecting the trust information for the image using the command:

# docker trust inspect --pretty dkr.high-availability.com/hamonitor:v1.0

The resulting output will look similar to the following:

# Signatures for dkr.high-availability.com/hamonitor:v1.0
SIGNED TAG   DIGEST                                                       SIGNERS
v1.0   46d706ebead9e7746b3c1ffcbc2247562d038865a5eed85410dc54eebe5c1aed   hacsigner

List of signers and their keys for dkr.high-availability.com/hamonitor:v1.0

SIGNER      KEYS
hacsigner   96d8fb669c3e

Administrative keys for dkr.high-availability.com/hamonitor:v1.0

  Repository Key: 619faa1dc970583f7b366fe68ecfa48b5e6cd5b07ccea2647d5c0ab7bb50191e
  Root Key:    4dd62ed17d61204196017fff4c1e9f2ded9508d9bfbdf5321a10f401b555a414

The digest listed for the signed tag v1.0 is the same as the one shown when requesting digests for locally installed images using docker images --digests.

The process described so far details how an image is uploaded and verified to a docker registry using the trust framework. When that image is downloaded using docker pull, the trust chain is maintained because a signed digest for the manifest is available, meaning the manifest can be trusted (after login/key exchange etc.), and therefore its contents, and therefore the checksums for the compressed layers and so on.

However, when an image file is created from a locally installed image, the original manifest cannot be used because, firstly, it contains checksums for the compressed layers and locally generated image files contain uncompressed layers, but more importantly, any manifest shipped with an image has no way to establish trust of that manifest as it is not derived from the original docker registry and thus there is no way to verify its contents (most importantly the layer checksums).

Because of this, verification of an image file is done using the image ID field from the image once it has been installed. The image ID should correspond to the original image ID from the host where the image file was originally created. This is because the image ID is calculated by applying the algorithm (SHA256) to a layer's content and so long as that content has not changed from the originating host, then the ID’s will match, and therefore the image can be trusted.

Upgrading the monitor image

As the configuration data, logs etc. for the hamonitor are kept in a shared directory on the host machine (specified with the --volume parameter when creating the container), upgrading to a newer version is simply a matter of stopping the monitor, deleting the container with the older version, then restarting with the updated version.

To check to see if there is an updated version of the monitor available firstly list the locally installed version(s):

# docker images
REPOSITORY                            TAG       IMAGE ID       CREATED        SIZE
dkr.high-availability.com/hamonitor   v1.0      fc254100c107   6 months ago   768MB

This shows a single instance of the monitor with the tag indicating release 1.0. Next check on the remote docker server for any available updates:

# export DOCKER_CONTENT_TRUST=1
# export DOCKER_CONTENT_TRUST_SERVER=https://notary-server.high-availability.com:4443
# docker trust inspect --pretty dkr.high-availability.com/hamonitor

Signatures for dkr.high-availability.com/hamonitor

SIGNED TAG   DIGEST                                                       SIGNERS
v1.0   46d706ebead9e7746b3c1ffcbc2247562d038865a5eed85410dc54eebe5c1aed   hacsigner
v2.0   4ab6fce6603377e42afde815612215913580689e4d700271c8d607a82c683235   hacsigner

List of signers and their keys for dkr.high-availability.com/hamonitor

SIGNER      KEYS
hacsigner   96d8fb669c3e

Administrative keys for dkr.high-availability.com/hamonitor

  Repository Key:   619faa1dc970583f7b366fe68ecfa48b5e6cd5b07ccea2647d5c0ab7bb50191e
  Root Key: 4dd62ed17d61204196017fff4c1e9f2ded9508d9bfbdf5321a10f401b555a414

This shows there is an updated v2.0 of the monitor container available on the trust server. The next step is to download the update using the docker pull command. Note that it is not necessary to stop a running monitor in order to pull a newer version:

# docker pull dkr.high-availability.com/hamonitor:v2.0

Pull (1 of 1): dkr.high-availability.com/hamonitor:v2.0@sha256:4ab6fce6603377e42afde815612215913580689e4d700271c8d607a82c683235
dkr.high-availability.com/hamonitor@sha256:4ab6fce6603377e42afde815612215913580689e4d700271c8d607a82c683235: Pulling from hamonitor
e22122b926a1: Pull complete
d52c8f0f3e69: Pull complete
…
abbb7acd7b5e: Pull complete
e37b0c1f324f: Pull complete

Digest: sha256:4ab6fce6603377e42afde815612215913580689e4d700271c8d607a82c683235
Status: Downloaded newer image for dkr.high-availability.com/hamonitor@sha256:4ab6fce6603377e42afde815612215913580689e4d700271c8d607a82c683235
Tagging dkr.high-availability.com/hamonitor@sha256:4ab6fce6603377e42afde815612215913580689e4d700271c8d607a82c683235 as dkr.high-availability.com/hamonitor:v2.0
dkr.high-availability.com/hamonitor:v2.0

Listing the locally available images should now show the update in the list of available images:

# docker images --digests
REPOSITORY                            TAG       DIGEST                                                                    IMAGE ID       CREATED        SIZE
dkr.high-availability.com/hamonitor   v2.0      sha256:4ab6fce6603377e42afde815612215913580689e4d700271c8d607a82c683235   08fdb3bdf6a4   8 days ago     687MB

dkr.high-availability.com/hamonitor   v1.0      sha256:46d706ebead9e7746b3c1ffcbc2247562d038865a5eed85410dc54eebe5c1aed   fc254100c107   6 months ago   768MB

Note that the local and remote image digest should agree. Now the updated image is available all that remains is to stop and remove the v1.0 monitors container and restart using the updated version of the container:

# docker stop hamonitor
# docker rm hamonitor
# docker run \
    --detach \
    --name hamonitor \
    --net host \
    --privileged \
    --restart unless-stopped \
    --volume <host directory>:/tmp/hamonitor \
    dkr.high-availability.com/hamonitor:v2.0 \
    --publish 13514

Finally, check the running monitor:

# docker ps
CONTAINER ID   IMAGE                                 COMMAND                  CREATED         STATUS         PORTS     NAMES
c41d6c875459   dkr.high-availability.com/hamonitor   "python3 /root/api/h…"   5 minutes ago   Up 5 minutes             hamonitor

Note that if you pass the --no-trunc parameter to docker ps then the running image ID digest will be displayed – this can be compared to the one derived from the previous docker trust inspect command.

Note that at present docker only supports IPv6 on hosts running Linux, therefore if IPv6 monitoring is required (i.e. monitoring of shared resources over IPv6) then a Linux derivative should be chosen as the docker host. ↩