Skip to content

CLI reference guide

Introduction

The utility hamonitor is used to perform all monitor actions. It is supplied as part of the docker image. To run the utility first gain shell access to the running docker instance:

# docker exec -it hamonitor bash

The hamonitor utility is self-documented; typing any command or subcommand with no arguments produces a help summary:

# hamonitor
NAME:
hamonitor - RSF-1 shared resources monitor

USAGE:
hamonitor [global options] command [command options] [arguments...]

VERSION:
2.1

COMMANDS:
    alert     Alert management
    iscsi     iSCSI initiator
    login     Log in to RSF-1 shared resources monitor
    logout    Sign out of RSF-1 shared resources monitor
    resource  Resource management
    user      User management
    help, h   Shows a list of commands or help for one command

GLOBAL OPTIONS:
--help, -h     show help
--version, -v  print the version

User management

Administration of users is done using the user subcommand. It allows for user creation, deletion and listing of existing users (modification is not supported in this release).

User addition

To create a new user, enter the subcommand:

# hamonitor user create
Enter username: oper
Enter password: *********
Verify password: *********
Enter real name [None]: Operator
Enter email address [None]: ops@example.com
Available roles: 0 (view only), 1 (operator), 2 (admin)
Enter role [0]: 1
User oper successfully created

Listing users

To show a list of configured users enter the subcommand:

# hamonitor user list
(JSON output...)

Note that the resulting list is provided in JavaScript Object Notation (JSON) format; to make it more readable pipe the output through jq (a utility to print JSON into a more human readable format, shipped with the docker image). Piping the above command through jq results in:

# hamonitor user list | jg
[
    {
        "userid": 1,
        "creation_time": "1600692753.915523",
        "username": "admin",
        "realname": "",
        "password": "<*** HIDDEN ***>",
        "role": 2,
        "enabled": "True"
    },
    {
        "userid": 2,
        "creation_time": "1600701250.404133",
        "username": "user",
        "realname": "",
        "password": "<*** HIDDEN ***>",
        "role": 1,
        "enabled": "True"
    }
]

Deleting a user

To delete a user enter the subcommand:

# hamonitor user delete
Enter userid: 2
Do you really want to remove user 2? [y/n]: y
User successfully deleted

Note that users are referenced by their userid - these can be shown using the user list subcommand.

Logging in and out

Before any changes are made to the monitor, it is necessary to authenticate using the login subcommand. The connection URL uses localhost and port 13514 by default - an alternative can be entered when logging in:

# hamonitor login
Enter URL [https://localhost:13514 if empty]:
Enter Username: admin
Enter Password:
Welcome admin
To logout issue the logout subcommand:
# hamonitor logout

Resource management

Introduction

Resources are managed using the hamonitor resource subcommand. Note that the monitor does not create the monitored resource and does not modify the access rights of a monitored resource, rather it monitors the availability of an existing resource.

Adding a monitored resource

To configure a monitored resource, use the resource add subcommand:

# hamonitor resource add
Enter protocol [NFS, SMB, iSCSI, LDAP]: SMB
IP address of the SMB server: 192.168.4.1
SMB share name: Scratch
Mount options [return for None]:
User name [return for None]: system
Password [return for None]:
Do you want to test the connection now? [y/n]y
Test successful.
201: Created

Authentication for a monitored resource

When adding a monitored resource, the username and password fields are not required when the resource is publicly shared, i.e. where no authentication is required to access the resource.

In the case where authentication is required to access a resource, then credentials should be supplied using the username and password fields during addition of the monitored resource.

Removing a monitored resource

To remove a resource from the monitor use the resource remove subcommand along with the resource ID to identify the resource to be removed:

# hamonitor resource remove --id 1
200: OK
Any alerts associated with the resource are also removed.

Enabling and Disabling a monitored resource

The active monitoring state of a resource can be toggled between enabled or disabled. When a resource is first added, its monitor state is enabled. To suspend monitoring without removing the resource entirely (say for example the resource is going offline for maintenance) its state can be set to disabled in the monitor. Reinstate monitoring for a resource by enabling it.

To disable a resource use the resource disable subcommand:

# hamonitor resource disable --id 1
200: OK
To enable a resource use the resource enable subcommand:
# hamonitor resource enable --id 1
200: OK

Listing resources being monitored

To list all resources configured use the resource list subcommand:

# hamonitor resource list
...
The list is reported back as a JSON object; pipe it through jq for a more human readable form.

Status of an individual resource

To check the status of individual resources use the resource status subcommand:

# hamonitor resource status --id 1 | jq
{
    "path": "Scratch",
    "ip": “192.168.4.1”,
    "protocol": "SMB",
    "enabled": true,
    "creationDate": "2020-09-22T12:16:54+00:00",
    "notifications": {
        "slack": false,
        "teams": false,
        "email": false,
        "snmp": false
    }
}

iSCSI resource monitoring

Introduction

As of release 2.1 the ability to perform iSCSI resource monitoring has been added. The monitor acts as an initiator and is connected to the remote target to be monitored.

For Linux hosts to take advantage of this feature it is necessary to map the hosts /lib/modules directory to the container using the docker --volume option when the container is started:

# docker ...\
--volume /lib/modules:/lib/modules\
...
This mapping is required so the iSCSI kernel module used for monitoring (iscsi_tcp) can be loaded by the hamonitor running in the container, a process which in turn triggers docker to load the module in the docker host, thereby providing iSCSI kernel features to the process running in the docker container - this is a feature of docker.

Note the iSCSI module is only loaded in the docker host, it does not run in the container. The process of running the iSCSI module in the host exposes its facilities to clients in the docker container, thereby enabling iSCSI monitoring. If you do not intend to run any iSCSI monitoring, it is not necessary to map the modules library

The iSCSI initiator

To monitor an iSCSI resource, it is first necessary to define a local initiator within the scope of the monitor so connections can be made to remote targets. By default, the monitor generates a suitable initiator to use in iSCSI connections. To view and manipulate the default iSCSI initiator the hamonitor iscsi sub-command is used:

# hamonitor iscsi
NAME:
   hamonitor iscsi – iSCSI initiator

USAGE:
   hamonitor iscsi command [command options] [arguments…]

COMMANDS:
     modify, m   Modify system’s iSCSI initiator name
     restart, r. Restart open-iscsi daemon
     show, s.    Show current iSCSI initiator name

OPTIONS:
--help, -h    show help

To display the current iSCSI initiator name:

# hamonitor iscsi show

iqn.1993-08.org.debian:01:c9159016459f

Changing the initiator name

It may be necessary to change the default initiator name, for example when more than one instance of the monitor is being run on the same host or where iSCSI authentication is based on the initiator name1 and a specific format/name is required. The initiator name is changed using the iscsi modify command:

# hamonitor iscsi modify
Enter the new iSCSI initiator name: iqn.1993-08.org.debian:01:c91590164000
New iSCSI initiator name set to iqn.1993-08.org.debian:01:c91590164000
After changing the iSCSI initiator name, it is necessary to restart the hamonitor iSCSI daemon instance for the change to take effect:
# hamonitor iscsi restart
iSCSI module restarted
The change can be validated using the iscsi show command:
# hamonitor iscsi show
iqn.1993-08.org.debian:01:c91590164000

Monitoring an iSCSI resource

In order to monitor an iSCSI target it is added as a resource to the hamonitor. The following example adds monitoring for an iSCSI target using CHAP authentication2:

# hamonitor resource add
Enter protocol [NFS, SMB, iSCSI, LDAP]: iSCSI
IP:PORT of the iSCSI Target Portal: 192.168.5.2:3263
Target IQN: iqn.2021-05.com.vmlab-nef:target0
Mount options [return for None]:
Username [return for None]: hac
Password [return for None]: <hidden>
Do you want to test the connection now? [y/n] y
Test successful
201: Created

Use the resource list command to display the newly added resource:

# hamonitor resource list | jq
{
    "path": "iqn.2021-05.com.vmlab-nef:target0",
    "ip": "192.168.5.2:3263",
    "protocol": "iSCSI",
    "enabled": true,
    "creationDate": "2021-05-19T19:18:00+00:00",
    "notifications": {
        "slack": false,
        "teams": false,
        "email": false,
        "snmp": false
    },
    "activeIssues": false,
    "status": "online",
    "onlineSince": "2021-05-19T19:18:09+00:00",
    "uptime": 78,
    "issues": {
        "total": 1,
        "lastHour": 1,
        "lastSixHours": 1,
        "last24Hours": 1,
        "lastWeek": 1,
        "totalDowntime": 6,
        "averageDowntimePerIssue": 6,
        "lastIssueOpen": "2021-05-19T19:18:03+00:00",
        "lastIssueClose": "2021-05-19T19:18:09+00:00"
    },
    "availability": 0.9323,
    "resourceid": 15
}
Finally, to add an alert for the iSCSI resource (see the alert management section for details on the different types of alerts):
# hamonitor alert teams create -id 15
Enter Teams hook URL: https://highavailabilitycom.webhook.office.com/webhookb2/...

LDAP resource monitoring

Introduction

As of release 2.1 the ability to monitor LDAP servers has been added. Monitoring of LDAP servers is autonomous and does not require login information on the remote server.

Monitoring an LDAP resource

In order to monitor an LDAP server it is added as a resource to the hamonitor:

# hamonitor resource add
Enter protocol [NFS, SMB, iSCSI, LDAP]: LDAP
IP address of the LDAP server: 10.5.4.33
Distinguished name (DN): ou=group,dc=mydomain,dc=com
Do you want to test the connection now? [y/n]y
Test successful
201: Created
Use the resource list command to display the newly added resource:
# hamonitor resource list | jq
{
    "path": "ou=group,dc=mydomain,dc=com",
    "ip": "10.5.4.33",
    "protocol": "LDAP",
    "enabled": true,
    "creationDate": "2021-05-19T19:18:00+00:00",
    "notifications": {
      "slack": false,
      "teams": false,
      "email": false,
      "snmp": false
    },
    "activeIssues": false,
    "status": "online",
    "onlineSince": "2021-05-19T19:18:09+00:00",
    "uptime": 78,
    "issues": {
      "total": 1,
      "lastHour": 1,
      "lastSixHours": 1,
      "last24Hours": 1,
      "lastWeek": 1,
      "totalDowntime": 6,
      "averageDowntimePerIssue": 6,
      "lastIssueOpen": "2021-05-19T19:18:03+00:00",
      "lastIssueClose": "2021-05-19T19:18:09+00:00"
    },
    "availability": 0.9323,
    "resourceid": 15
}

Alert management

The monitor supports several types of alerts. Alerts are configured on a per-resource basis so each resource has its own alert schema. Alerts are bound to resources using the mandatory --id argument when adding an alert.

Adding an email alert

To configure an email alert, a valid SMTP server is required along with a user name and password. The monitor only supports email delivery over an encrypted TLS connection.

The (optional) TLS port on the email server is specified when the server address is entered:

```
# hamonitor alert email create --id 5
This alert method only supports authenticated email delivery over TLS.

Enter SMTP server address (MX:[PORT]): mx1.yourcomany.com:587
Enter SMTP server user name: realuseraccount@yourcompany.com
Enter SMTP server user password:
Verify password:
Enter email FROM address: alerts_sender_alias@yourcompany.com
Enter email TO address: alerts_manager_alias@yourcompany.com

201: Created
```

Adding a Slack alert

A slack alert will update a slack channel with events for which the resource has been configured. Before creating an alert, a slack webhook is required. Webhook's are created using the slack application itself, please see the slack documentation for how to create a suitable webhook.

Once a webhook link has been generated it is then used as part of the URL when creating the alert:

# hamonitor alert slack create --id 5
Enter Slack hook URL: https://hooks.slack.com/services/<link>
201: Created
An alert published to slack will be similar to:
Resource OFFLINE: nfs 192.168.22.6 /pool/nfs-share

Adding a Microsoft Teams alert

A teams alert will update a teams channel with events for which the resource has been configured. Before creating an alert a teams webhook is required. Webhook's are created from the teams application itself, please see the teams documentation for how to create a suitable webhook.

Once a webhook link has been generated it is then used as part of the URL when creating the alert:

# hamonitor alert teams create --id 5
Enter Teams hook URL: https://outlook.office.com/webhook/<link>
201: Created
An alert published to teams will look similar to this example:
Resource OFFLINE: smb 10.6.11.12 /pool/smb-share

Adding an SNMP alert / MIB location

To add an SNMP alert the IP address of an SNMP manager is added to the resource being monitored:

hamonitor alert snmp create --id 5
Enter SNMP manager address: 10.5.14.22
201: Created
A copy of the SNMP MIB file for the monitor is shipped with the docker image in /root/RSF-MIB.txt.

Changing the HTTPS authentication certificate

The CLI communicates with the monitor using its REST API over HTTPS (TLS version 1.3). A self signed certificate is shipped in the docker image as /root/cert.pem.

A site specific certificate can be used instead by installing it in the shared host directory as cert.pem. It is necessary to restart the docker container for it to pick up the new certificate; to restart the container from the docker host run the command:

# docker restart hamonitor

Changing the frequency of the resource availability check

As delivered the monitor checks the availability of resources at a regular interval – this interval is known in the monitor as the frequency; its default value is 3 seconds.

In high latency or low speed networks, monitoring of resources could trigger false alarms due to higher-than-normal times the monitor needs to wait for responses. In cases such as these the monitoring frequency can be adjusted using the frequency subcommand.

To show the current frequency setting:

# hamonitor frequency show
To set a different frequency:
# hamonitor frequency modify
Enter a new frequency in seconds (min 3s): 10
Frequency updated to 10 seconds
Changing the frequency setting is effective immediately (there is no need to restart the monitor).


  1. For systems where the authentication is based on CHAP user/password combinations, the authentication information is provided during the resource creation process. 

  2. If CHAP authentication is not required, then press Enter when prompted for Username and Password.