FreeBSD High Availability Resource Page

FreeBSD HA Cluster FAQs

Why do serial heartbeats stop working?

There is a known problem with serial heartbeats in some Opensolaris derived operating systems. The problem is that after serial heartbeats have been used for some time, the serial interface stops working and the heartbeats go down. When this happens, … Continue reading

How to configure a three node cluster

RSF-1 can be configured to create a cluster of any number of nodes. HA services can be allowed to run on any node, or a subset of nodes, with simple changes to the config file. The most important consideration when … Continue reading

Understanding ZFS Failover times with RSF-1 High Availability Cluster

RSF-1 allow for manual and automatic failovers. An automatic failover happens when RSF-1 detects a system failure and automatically moves the service from the failed active node to an alternative node. A manual failover is an operator-initiated command to move … Continue reading

RSF-1 Heartbeat Sequencing

The way old heartbeats are detected is to include a “serial number” in each one. This number starts as a low value, and then increments with each new heartbeat. When a heartbeat is received the number is checked, and if … Continue reading

zilstat – Will a separate ZIL log device help your ZFS performance?

Contribution from Richard Elling ( Richard.Elling@RichardElling.com ): zilstat is useful for determining how much write activity is generated by your workload before you add a separate log to a pool (which can later not easily be removed). The general recommendation … Continue reading

Where can I find documentation for the RSF-1 C/C++ API

There are now two C/C++ APIs available for use by developers. The first API contains functions for reading and writing to the RSF-1 config. Full documentation for this API can be found here. The second API contains functions for interrogating … Continue reading

How do I enable ALUA from the command line

This page describes how to manually enable ALUA support for a two node cluster, using the command line. Firstly both nodes need to create the config file at path /opt/HAC/RSF-1/etc/.stmf_proxy This file needs three lines: PROXYHOST=<node1> NODEID=0 REMOTE=<node2> PROXYHOST should … Continue reading

How do I increase the level of debugging from RSF-1

To increase the level of RSF-1 logging, debugging can be turned on. RSF-1 has 10 debugging levels – zero (off) to nine (full) – and defaults to level zero. To increase debugging by one level, run the command: /opt/HAC/RSF-1/bin/rsfcli debug … Continue reading

How to Integrate NexentaStor shares with Active Directory

This guide assumes the user already has Windows Server 2003 set up to run an Active Directory server and DNS server. From the ‘Active Directory users and computers console, add computer objects for both nodes in the cluster. Then, do … Continue reading

Change in the use of SCSI reservations

When a clustered service is created, a number of drives are selected as heartbeat drives. In versions of RSF-1 prior to 3.6.5, these drives have also been used as SCSI-2 reservation drives. Before a pool is imported by a node, … Continue reading

How to use an auto-snap service with a clustered volume

Currently, the auto-snap service is local to the node that it was created on, so when the volume fails over to the other node, snapshots will no longer be taken. As a workaround, the following steps should be taken: Create … Continue reading

Creating Cluster-Wide Target Groups for ALUA

NMV is currently unable to create the cluster-wide target groups necessary for allowing client-side multipathing software to take full advantage of COMSTAR Asymmetric Logical Unit Access support provided by the new HA Cluster STMF Proxy service. To add this functionality … Continue reading

What is mapmgr’s COMSTAR manager functionality?

Overview Traditionally the mapmgr program’s basic functionality is to move the state of the views and logical units between nodes by zpool. Versions of mapmgr 3.6.0 and above will now also manage all the other COMSTAR objects. Any object that … Continue reading

Creating iSCSI Target and Logical Unit configuration on a clustered pool

Assuming a two node cluster with members nodea and nodeb, create an iSCSI Target to receive SCSI I/O commands for the Logical Unit on the first cluster node: root@nodea:~# itadm create-target Target iqn.1986-03.com.sun:02:31ac217b-bed5-e8c2-919b-eafd4f1bb413 successfully created Create the same iSCSI Target … Continue reading

Status page

This page provides information about the current state of volumes and heartbeats in the cluster. The volume status page also provides a popup menu to control volume services in the cluster. Cluster status tab This tab gives you a live … Continue reading

What do the mapmgr events mean?

STMF_API_IMPORT_LU_FAIL ARGS: zvol= stmfResult=0x<code_in_hex> stmfResultConverted= This event signals that the stmf API call stmfImportLu has failed for the LU corresponding to .  The hex value <code_in_hex> or the human readable string may help determine why this call has failed.  <code_in_hex> is … Continue reading

How should I test mapmgr?

1 TESTING PLAN AND ADVANCED FUNCTIONALITY OVERVIEW FOR MAPMGR V3.4.6 1.1 Set up an enviroment that uses the full capabilities of stmf. The enviroment should include multiple pools, multiple zvols with multiple depths, multiple target/host groups and multiple views.  An … Continue reading

What is COMSTAR/stmf and what is the Interchangeble Naming Scheme (INS)

It may be helpful to use this FAQ in conjunction with the diagram of the help FAQ. 1 Formal Definitions 1.0 Mapping A mapping, map or function is a pair of sets (A x B, f), where f is a … Continue reading

What does mapmgr (V3.4.6.2) do and how do I use it?

Help Documentation for mapmgr V3.4.6.2 Overview The mapmgr program’s basic functionality is to move COMSTAR state between nodes, in particular the state of the views and logical units.  The mapmgr has extensive logging and can also handle synchronization under certain … Continue reading

How to use the License Server

Overview The RSF-1 License Server is a tool that automatically licenses RSF-1 on cluster pairs based on their IP addresses. The user can enter keys into the License Server to unlock licenses.  Once the licenses have been unlocked the user … Continue reading

Manually adding the cluster licence .deb file

The Nexenta HA plugin is licensed on a per machine basis. Licences come in two forms, a temporary one that expires after 45 days, and a permanent one that never expires. Licences are delivered in a packaged .deb file. Normally … Continue reading

Initializing cluster

The initialize menu is used to start configuring a NexentaStor cluster. The cluster must be initialised before any volumes can be added to the cluster using this form: The cluster name is used to identify this cluster instance. If desired … Continue reading

When exporting pool fails –> Check ‘canmount=on’ is set

To troubleshoot a Export pool failure during an Auto or Manual RSF Failover in NexentaStor v3.0.4 implementation. 1) Get to the rsfcli status by this command, verify the state # /opt/HAC/RSF-1/bin/rsfcli -i0 status 2) Now to verify  if the mount … Continue reading

Creating a VLAN for clustering.

For a cluster a vlan interface is treated the same way as a normal interface, the only extra step required is to create the VLAN device without assigning an IP address to it. In NMV navigate to the Network summary … Continue reading

Advanced Operations

The Advanced Operations menu has a number of features accessed via a set of tabbed panels. The functionality of each of these panels is described in the following sections. Cluster Operations Provides a set of operations that are performed on … Continue reading

Adding and Removing volumes.

Adding a volume To add a volume to the cluster first select the Add a volume tab from the volume operations section. This presents a configuration similar to the one shown below. Note that volumes presented in the Volume drop … Continue reading

Server warning … machine X is not configured to run service Y

ISSUE: In releases of the ha-cluster plugin prior to 3.3.7 this issus relates to the use of upper case in either host names or service names. VERSION: The current ha-cluster plugin version can be obtained from the header of the … Continue reading

How do I distribute a new configuration file.

You can distribute a configuration file to nodes in a cluster using the config_dist command, the format is: /opt/HAC/RSF-1/bin/config_dist –hot config_file nodea nodeb node… The config_file parameter points to the configuration file to be distributed amongst the cluster nodes; the … Continue reading

Client Multi-path configuration for COMSTAR ALUA

RedHat 6.0: devices { device { vendor “NEXENTA” product “COMSTAR” prio alua path_checker tur path_grouping_policy group_by_prio features “1 queue_if_no_path” failback immediate hardware_handler “1 alua” } }   RedHat 5.7: devices { device { vendor “NEXENTA” product “COMSTAR” path_checker tur prio_callout … Continue reading

COMSTAR Fibre Channel and Client Multipathing

Introduction This guide describes how to configure COMSTAR fibre channel targets for failover, eliminating single points of failure from the client to the target using redundant fabrics and native multi-pathing functionality. The following diagram depicts the hardware configuration for this … Continue reading

Drive installation for 53c1000 / 53c1010-66 / LSI20160

To enable this chipset/card the symhisl driver is used. This does not come with OpenSolaris/NexentaStor so a copy needs to be acquired from an existing Solaris installation (located as /kernel/drv/symhisl & /kernel/drv/symhisl.conf). Copy these two files into the /kernel/drv directory … Continue reading

How do I configure COMSTAR Fibre Channel targets for fail over?

As a prerequisite for configuring Fibre Channel targets, the HBA port modes of both appliances need to be changed from Initiator mode to Target mode. To change HBA port mode, select SCSI Target Plus from the Data Management tab in … Continue reading

Cable configuration

UTP ethernet cable (10 base T & 100 base T, twisted pair) ——————————————————— A normal cable is connected pin to pin, with RJ-45 Male Plug pin numbers twisted pairs on pins 1/2, 3/6, 4/5 and 7/8. as seen from the … Continue reading

RSF-1 QuickStart Guide

  1 RSF-1 Quick Start Guide   1.1 Introduction   This RSF-1 Quick Start Guide is designed to help install and configure RSF-1 on a simple 2-node asymmetric cluster without direct-attached shared-SCSI storage. In addition, this guide assumes both nodes … Continue reading

Is there a Quick Start Guide for RSF-1 generic installations

The quick start guide for standard RSF-1 installations can be found on our website. Please click on the following link or copy the address into your browser. http://www.high-availability.com/downloads/Support/RSF_1_QuickStart_Guide.pdf  

How do I set up a build environment on 3.0.0

First of all /etc/apt/sources.list should be updated to point to the build package repository by appending the following lines: deb http://apt.nexentastor.org hardy-unstable main contrib non-free deb-src http://apt.nexentastor.org hardy-unstable main contrib non-free Next run apt-get update – update is used to … Continue reading

Using the NMV HA cluster plug-in

To view the status of the NetentaStor HA cluster via NMV, navigate from the main page to Status->HA cluster. You will then be presented with an cluster overview pane similar to the one shown below. The status pane is divided … Continue reading

How can I fail over more than one interface/address per service.

To add further VIPs to an existing interface, or to add additional interfaces to a service (say for example when you want to direct traffic down specific networks), a per service addition is made to the cluster configuration file. These … Continue reading

How do I configure COMSTAR iSCSI targets for fail over?

When adding a shared volume to a cluster which will have zvols created as backing storage for iSCSI targets, it is vital all client iSCSI initiators, irrespective of operating system, are configured to access those targets using the shared logical … Continue reading

When use S30interface script to configure an additional VIP, does RSF-1 monitor this network interface? And if this interface fails, does this trigger a fail over?

RSF-1 monitors the physical interface state (if configured to do so) and as a by product any VIP’s bound to that interface are therefore also monitored. When interface monitoring is configured, any services that use that physical interface are considered … Continue reading

What is Simple-HA

In the context of RSF-1, “Simple-HA” means a restricted mode of operation in which a cluster is configured as normal, but services can not be left in automatic mode, and so will not failover automatically. The machines in the cluster … Continue reading

How do I run RSF-1 with realtime scheduling priority inside a Solaris 10 Zone?

By default, Solaris 10 zones run without the proc_priocntl privilege which is a pre-requisite for running RSF-1 inside a zone in real-time scheduling mode. To see which system privileges a zone has been granted, run the command ppriv –lv zone … Continue reading

Do all client systems that access the cluster have to be in the same network?

No they do not – as long as routing is configured correctly on the client systems to reach the address ranges used in the cluster for its services.

When an application/service error is detected in and RSF-1 Cluster, is it possible to restart components locally or does failover always have to happen?

Application and service errors are detected by the RSF-1 agent framework. An agent is started as part of a service and can utilise a number of ways to test the health of the applications in that service (for instance writing … Continue reading

How do I modify the GRUB boot menu on Nexenta OS? Editing /boot/grub/menu.lst does not seem to work.

You need to edit the menu.lst located in the syspool filesystem which is not mounted at boot time: mount -F zfs syspool /mnt vi /mnt/boot/grub/menu.lst umount /mnt init 6 For instance, to change from 64 bit to 32 bit booting … Continue reading

I have several applications on a server, each listening on different IP ports. How many agents do we need to monitor these applications?

You only need one agent per RSF-1 service; the agent itself is threaded so can be configured to monitor multiple applications from a single thread. The type of monitoring performed by a single agent can be freely intermixed – i.e. … Continue reading

Can we use more than one network heartbeat.

RSF-1 does not place any restrictions on the number of heartbeats it supports via network, disk or serial. For network heartbeats a typical configuration is to have one private heartbeat over a ethernet crossover cable, and as many public heartbeats … Continue reading

According to the RSF-1 Admin guide the port numbers for rsfnet and rsfreq are both 1195, is this a typo or are rsfnet and rsfreq really on the same port?

RSF-1 uses port 1195 for both TCP and UDP requests. The UDP port is used for heartbeat and cluster discovery packets, with the TCP port being used for the GUI and command line interface. Port 1195 is officially assigned for … Continue reading

Are there any guidelines for characteristics of the quorum drive, size, speed etc?

RSF-1 only requires a single small partition per zfs pool for heartbeating, 1Mb is more than sufficient (in fact it’s generally configured to write to blocks 34, 36 and 38 only). There is also no speed requirement, 5400rpm drives are more … Continue reading

Do all client systems that access the cluster have to be in the same network?

Clients access services in the cluster using the floating IP address for each service. This IP address is a normal, routable, IP address, and acts like any other such address. If the service is accessible when run as a simple … Continue reading

Do you have a hardware compatibility matrix for RSF-1 Cluster?

No, we do not maintain an HCL because there is nothing in RSF-1 Cluster that is hardware dependent. There are no special kernel drivers etc., no special/bespoke hardwrae interconnects; we merely utilise what is available in the vanilla operating system … Continue reading

Where can I find the RSF-1 Cluster Documentation?

You can find all the RSF-1 documentation on the high-availability.com website at http://www.high-availability.com/links/4-0-support.php. This will include the RSF-1 Administration Guide, The RSF-1 Quickstart Guide and the RSF-1 Agent Framework Documentation

Can RSF-1 be configured to fail over services if network connectivity is lost.

RSF-1 is configurable to monitor the state of network connections and perform a fail over of any services bound to that interface. Enable IP failure detection in RSF-1 by adding the following line (IPDEVICE_MONITOR) at the top of the RSF-1 … Continue reading

How do I check, if our RSF-1 services are in Auto or Manual failover mode?

To check on the current settings for RSF, run the following command as root (on one of the boxes RSF is running on): /opt/HAC/RSF-1/bin/rsfcli -v list This results in output similar to: nodea: internal     stopped  manual   unblocked    storm        bge0:1   60  … Continue reading

How do I associate more than one ZFS pool with a service

To add aditional ZFS volumes to a NexentaStor volume service extra options are added to the RSF-1 configuration file  located at: /opt/HAC/RSF-1/etc/config For each additional volume to be imported/exported with a volume service, first locate the volume service in the … Continue reading

What does broken_safe and broken_unsafe mean and how do I fix it.

Broken_safe and broken_unsafe refer to a state of an RSF-1 service that has either failed to start up or shut down correctly. As a service is started or stopped RSF-1 executes the scripts in the directory /opt/HAC/RSF-1/etc/rc.<service>.d/* where <service> is … Continue reading

How do I configure RSF-1 with Solaris IPMP

Multiipathing functionality is often best delivered using the inbuilt Solaris support for link aggregation with RSF-1 happily (and invisibly) using the specified aggregated interfaces in its configuration. However where link aggregation is not possible (older switches/fabric etc) or where a niche network toplogy mandates … Continue reading

How do I Install the RSF-1 HA Plugin for NexentaStor

To install this plugin, run: nmc$ setup plugin install rsf-cluster This pluggable module must be installed on each appliance that is a member of a HA Cluster of appliances. You can use management console ‘show’ command to view already installed … Continue reading