Installation and Upgrade
Requirements
The resource availability monitor is delivered as a self-contained docker image using the Docker Content Trust (DCT) system to ensure the integrity and publisher of all the data downloaded. Installation of the image can be undertaken on any host running docker1.
In this document the host running the monitor container is referred to as the docker host. This host must have network access to any cluster resources to be monitored so it in turn can make those resources available to the container running the monitor itself.
For example, to monitor a clustered NFS share then the docker host must be able to mount and access that share, as the docker container will require access to successfully monitor that resources availability. Also note, the process running the monitor docker image must belong to the docker group on the host OS (this is a requirement of docker).
Running under docker, the monitor tool requires a persistent location
to store its data, logs and ancillary files. This location is provided
by the host OS and mapped to the directory /tmp/hamonitor
when the
container is started (see section 3 Starting the Monitor). This way
existing configurations, logs and other components are preserved
during upgrades, migrations etc.
Installation
The monitor is distributed from a docker content trust server and is installed as follows:
-
Download and install the docker application package from www.docker.com onto the docker host machine. Start the docker daemon on the host machine.
-
Enable docker trust; docker uses environment variables to modify its behaviour, therefore, to enable trust, set the following:
export DOCKER_CONTENT_TRUST=1
$Env: DOCKER_CONTENT_TRUST=1
-
Set the high-availability notary server for the trust system, again using an environment variable:
export DOCKER_CONTENT_TRUST_SERVER=https://notary-server.high-availability.com:4443
$Env:DOCKER_CONTENT_TRUST_SERVER="https://notary-server.high-availability.com:4443"
-
In order to use the docker registry for trusted downloads it is necessary to have a username, password combination - this should be requested using the email address docker-trust@high-availability.com.
-
Using the user/password combination retrieved in the previous step, login to the docker framework:
The docker login subcommand will prompt for a username and password. Note that once you have successfully logged into the server, docker saves a login token locally in the users’ home directory in the file# docker login dkr.high-availability.com
.docker/config.json
thereby avoiding the need for this user to login again. The token can be cleared using the docker logout subcommand. -
Inspect the list of signed monitor images available from the registry:
here is some example output showing two signed versions of the monitor:# docker trust inspect --pretty dkr.high-availability.com/hamonitor
Signatures for dkr.high-availability.com/hamonitor SIGNED_TAG DIGEST SIGNERS v1.0 46d706ebead9e7746b3c1ffcbc2247562d035a5eed85410dc54eebe5c1aed hacsigner v1.1 eaea423478652348753463487563487563324856473285763434876538475 hacsigner List of signers and their keys for dkr.high-availability.com/hamonitor SIGNER KEYS hacsigner 346c3155f96c Administrative keys for dkr.high-availability.com/hamonitor Repository Key: 2058e5cfcc725b7b00607c60941e6105d5709ab20d56278ac3a4e1dd386c0 Root Key: a6ae8b3cc1ac73aa34d4237d6c0562fb5d69c5eaa3ab9a9e78b2ac3ecce93c39
-
Download the desired version:
Note that with the trust framework enabled, the docker command line tool takes care of validating the digest of the images downloaded and checks against the official signatures held by the notary server. Any image that has not been signed and verified will be blocked from download. The output from the docker pull should look similar to the following:# docker pull dkr.high-availability.com/hamonitor:v1.0
Pull (1 of 1): dkr.high-availability.com/hamonitor:v1.0@sha256:46d706ebead9e7746b3c1ffcbc2247562d038865a5eed85410dc54eebe5c1aed sha256:46d706ebead9e7746b3c1ffcbc2247562d038865a5eed85410dc54eebe5c1aed: Pulling from hamonitor ab3acf868d91: Pull complete bf8f3d9e8100: Pull complete 4cf71b2b4422: Pull complete 668c80dc67a6: Pull complete 1b527012fdfd: Pull complete ade8b6ab4354: Pull complete 4849fab77f68: Pull complete ccedab781a09: Pull complete Status: Downloaded newer image for dkr.high-availability.com/hamonitor@sha256:46d706ebead9e7746b3c1ffcbc2247562d038865a5eed85410dc54eebe5c1aed Tagging dkr.high-availability.com/hamonitor@sha256:46d706ebead9e7746b3c1ffcbc2247562d038865a5eed85410dc54eebe5c1aed as dkr.high-availability.com/hamonitor:v1.0 dkr.high-availability.com/hamonitor:v1.0 docker pull dkr.high-availability.com/hamonitor:v1.0
-
Confirmation of the downloaded image can be performed by comparing the digest of the local images with those held by the remote trust server (use the trust inspect docker subcommand as detailed earlier to retrieve the remote image digests). To list the digest of locally installed images use the command:
The output will look similar to the following - the third digest fields should correspond to the digest listed by the trust server:# docker images --digests
REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE dkr.high-availability.com/hamonitor v1.0 sha256:46d706ebead9e7746b3c1ffcbc2247562d038865a5eed85410dc54eebe5c1aed fc254100c107 4 weeks ago 768MB
Installation and upgrade on dark sites
To install the monitor on hosts that have no external internet connection (and thus cannot make a connection to the docker trust server) necessitates a two-step approach, the result of which creates an offline image which can then be used to install the monitor on any hosts, regardless of whether they have external connectivity or not.
The first step in creating the image is to designate a download host (with internet connectivity) to download the monitor as described in the previous section. The next step is to create an image which can be shipped to the non-connected hosts and installed locally. Creating an image is accomplished as follows:
-
On the host where the monitor has been downloaded create an image of the monitor:
# docker image save -o hamonitor_v1.0.tar dkr.high-availability.com/hamonitor:v1.0
-
The newly created tar file (in this example
hamonitor_v1.0.tar
) can then be copied to any host and installed using the following command:# docker load -i hamonitor_v1.0.tar
-
Finally, on the host where the image was loaded, check the image ID is the same, output will be similar to the following:
# docker images --digests REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE dkr.high-availability.com/hamonitor v1.0 <none> fc254100c107 3 months 768MB
The monitor is now installed and ready to run. To upgrade a dark site
installation simply create a new image using docker image save
from
the updated local image and add it to the dark site host using docker
load
.
The <none> digest column for loaded images
When an image is loaded from an image file (created using the docker
image save command), the digest field will always be shown as
When an image is pushed to a docker registry, the layers that go to make up that image are transferred over in an uncompressed format. When docker saves those layers, it saves them in a compressed format. Once all the layers that make up an image have been received and stored in compressed format, docker then creates an image manifest listing all the layers and a SHA256 checksum for each compressed layer. Once the manifest has been created a digest is calculated and the image tag is signed. The signatures and digest can be seen by inspecting the trust information for the image using the command:
# docker trust inspect --pretty dkr.high-availability.com/hamonitor:v1.0
The resulting output will look similar to the following:
# Signatures for dkr.high-availability.com/hamonitor:v1.0
SIGNED TAG DIGEST SIGNERS
v1.0 46d706ebead9e7746b3c1ffcbc2247562d038865a5eed85410dc54eebe5c1aed hacsigner
List of signers and their keys for dkr.high-availability.com/hamonitor:v1.0
SIGNER KEYS
hacsigner 96d8fb669c3e
Administrative keys for dkr.high-availability.com/hamonitor:v1.0
Repository Key: 619faa1dc970583f7b366fe68ecfa48b5e6cd5b07ccea2647d5c0ab7bb50191e
Root Key: 4dd62ed17d61204196017fff4c1e9f2ded9508d9bfbdf5321a10f401b555a414
The digest listed for the signed tag v1.0 is the same as the one shown
when requesting digests for locally installed images using docker
images --digests
.
The process described so far details how an image is uploaded and verified to a docker registry using the trust framework. When that image is downloaded using docker pull, the trust chain is maintained because a signed digest for the manifest is available, meaning the manifest can be trusted (after login/key exchange etc.), and therefore its contents, and therefore the checksums for the compressed layers and so on.
However, when an image file is created from a locally installed image, the original manifest cannot be used because, firstly, it contains checksums for the compressed layers and locally generated image files contain uncompressed layers, but more importantly, any manifest shipped with an image has no way to establish trust of that manifest as it is not derived from the original docker registry and thus there is no way to verify its contents (most importantly the layer checksums).
Because of this, verification of an image file is done using the image ID field from the image once it has been installed. The image ID should correspond to the original image ID from the host where the image file was originally created. This is because the image ID is calculated by applying the algorithm (SHA256) to a layer's content and so long as that content has not changed from the originating host, then the ID’s will match, and therefore the image can be trusted.
Upgrading the monitor image
As the configuration data, logs etc. for the hamonitor are kept in a
shared directory on the host machine (specified with the --volume
parameter when creating the container), upgrading to a newer version
is simply a matter of stopping the monitor, deleting the container
with the older version, then restarting with the updated version.
To check to see if there is an updated version of the monitor available firstly list the locally installed version(s):
# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
dkr.high-availability.com/hamonitor v1.0 fc254100c107 6 months ago 768MB
This shows a single instance of the monitor with the tag indicating release 1.0. Next check on the remote docker server for any available updates:
# export DOCKER_CONTENT_TRUST=1
# export DOCKER_CONTENT_TRUST_SERVER=https://notary-server.high-availability.com:4443
# docker trust inspect --pretty dkr.high-availability.com/hamonitor
Signatures for dkr.high-availability.com/hamonitor
SIGNED TAG DIGEST SIGNERS
v1.0 46d706ebead9e7746b3c1ffcbc2247562d038865a5eed85410dc54eebe5c1aed hacsigner
v2.0 4ab6fce6603377e42afde815612215913580689e4d700271c8d607a82c683235 hacsigner
List of signers and their keys for dkr.high-availability.com/hamonitor
SIGNER KEYS
hacsigner 96d8fb669c3e
Administrative keys for dkr.high-availability.com/hamonitor
Repository Key: 619faa1dc970583f7b366fe68ecfa48b5e6cd5b07ccea2647d5c0ab7bb50191e
Root Key: 4dd62ed17d61204196017fff4c1e9f2ded9508d9bfbdf5321a10f401b555a414
This shows there is an updated v2.0 of the monitor container available on the trust server. The next step is to download the update using the docker pull command. Note that it is not necessary to stop a running monitor in order to pull a newer version:
# docker pull dkr.high-availability.com/hamonitor:v2.0
Pull (1 of 1): dkr.high-availability.com/hamonitor:v2.0@sha256:4ab6fce6603377e42afde815612215913580689e4d700271c8d607a82c683235
dkr.high-availability.com/hamonitor@sha256:4ab6fce6603377e42afde815612215913580689e4d700271c8d607a82c683235: Pulling from hamonitor
e22122b926a1: Pull complete
d52c8f0f3e69: Pull complete
…
abbb7acd7b5e: Pull complete
e37b0c1f324f: Pull complete
Digest: sha256:4ab6fce6603377e42afde815612215913580689e4d700271c8d607a82c683235
Status: Downloaded newer image for dkr.high-availability.com/hamonitor@sha256:4ab6fce6603377e42afde815612215913580689e4d700271c8d607a82c683235
Tagging dkr.high-availability.com/hamonitor@sha256:4ab6fce6603377e42afde815612215913580689e4d700271c8d607a82c683235 as dkr.high-availability.com/hamonitor:v2.0
dkr.high-availability.com/hamonitor:v2.0
Listing the locally available images should now show the update in the list of available images:
# docker images --digests
REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE
dkr.high-availability.com/hamonitor v2.0 sha256:4ab6fce6603377e42afde815612215913580689e4d700271c8d607a82c683235 08fdb3bdf6a4 8 days ago 687MB
dkr.high-availability.com/hamonitor v1.0 sha256:46d706ebead9e7746b3c1ffcbc2247562d038865a5eed85410dc54eebe5c1aed fc254100c107 6 months ago 768MB
Note that the local and remote image digest should agree. Now the updated image is available all that remains is to stop and remove the v1.0 monitors container and restart using the updated version of the container:
# docker stop hamonitor
# docker rm hamonitor
# docker run \
--detach \
--name hamonitor \
--net host \
--privileged \
--restart unless-stopped \
--volume <host directory>:/tmp/hamonitor \
dkr.high-availability.com/hamonitor:v2.0 \
--publish 13514
Finally, check the running monitor:
# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c41d6c875459 dkr.high-availability.com/hamonitor "python3 /root/api/h…" 5 minutes ago Up 5 minutes hamonitor
Note that if you pass the --no-trunc
parameter to docker ps
then the
running image ID digest will be displayed – this can be compared to
the one derived from the previous docker trust inspect
command.
-
Note that at present docker only supports IPv6 on hosts running Linux, therefore if IPv6 monitoring is required (i.e. monitoring of shared resources over IPv6) then a Linux derivative should be chosen as the docker host. ↩