Installing a VIRL OpenStack Cluster

Why Use a VIRL OpenStack Cluster?

Most users will install VIRL onto a single bare-metal or virtual machine node. Every node (router, switch, server, or Linux Container (LXC)) used within a given VIRL simulation runs on this single machine.

Depending on the number and type of nodes used, it is possible for a simulation to require more compute and memory resources than can supported by a single node.

VIRL on Openstack Clusters enables you to combine multiple nodes (up to five) into a cluster, and to distibute the nodes in large, resource-intenstive simulations across this cluster so they can take advantage of the additional compute and memory resources.

At a minimum, a cluster must be composed of one 'controller' and one 'compute' node. Today, VIRL clusters can be scaled to a maximum of one controller and four compute nodes.

VIRL OpenStack Cluster Terminology

Term Description
Controller The primary VIRL node that includes a complete installation of the VIRL server software, including full compute, storage, and network functionality and all of the node and container images.
Compute node A node that includes a partial installation of the VIRL server software that enables it to provide additional compute and networking resources for use by a VIRL simulation.
Cluster A collection of nodes operating in concert. At a minimum, a cluster can be composed of one 'controller and one compute node.
VIRL Server Image A standard VIRL installation source (OVA or ISO) that contains the full compliment of VIRL software.
VIRL Compute Image A VIRL installation source (OVA or ISO) that contains only the VIRL software necessary to provide compute and networking services.

When using VIRL Clusters, the IP address used to reach the User Workspace Manager and by VM Maestro will be that of the Controller.

Installation Requirements

Just so you know.

The instructions that follow were developed for use with Cisco UCS C-series servers - either bare-metal or running vSphere ESXi.  You may encounter installation issues and need to adapt them to suit other environments.

Cluster-Member Resources

To successfully deploy a VIRL OpenStack Cluster please ensure that the following minimum requirements are met:

Software

Please observe the following minimum requirements for VIRL software when deploying VIRL OpenStack Clusters:
Node-Type VIRL Software Release
Controller 1.0.26 or later
Compute Node 1.0.26 or later

Network Time Protocol (NTP)

Every bare-metal node, ESXi-node, and cluster-member must be configured to properly synchronize with a valid NTP clock source.

If you are in a lab or other environment where special requirements apply to the use of NTP, please work with your network administrators to ensure that NTP is propery and successfully configured.

Networking

The VIRL networks are named 'Management', 'Flat', 'Flat1', 'SNAT', and 'INT'.  These are used for management, Layer-2 and Layer-3 connectivity, and cluster control-plane functions, respectively.

Each of the five required interfaces on a cluster member are connected to these networks in order, as shown below for both bare-metal and virtualized environments.

Bare-Metal Interface Mapping

In bare-metal deployments multiple LAN switches or VLANs must be used to provide seamless, isolated connectivity for each of the VIRL networks. The five phyiscal network interfaces on each node are connected as illustrated below:

Interface Switch or VLAN
eth0 Management
eth1 Flat
eth2 Flat1
eth3 SNAT
eth4 INT
vSphere ESXi Interface Mapping

In vSphere ESXi deployments multiple Port-Groups should used to provide seamless, isolated connectivity for each of the VIRL networks. The five virtual network interfaces (vNIC) on each virtual machine are connected as illustrated below:

vNIC Port-Group
eth0 VM Network (default)
eth1 Flat
eth2 Flat1
eth3 SNAT
eth4 INT

The default vSphere ESXi port-group used for the 'Management' network is 'VM Network' but any port-group may be used.  Please adapt as-needed to conform to site-specific configurations.

The 'Flat' and 'Flat1' Port-Groups must be configured in 'Promiscuous-Mode' in order to allow communications between nodes running in different simulations.  Please refer to the vSphere Client or Web-Client installation sections for detailed steps.

In deployments where cluster-members are deployed across multiple vSphere ESXi hosts care must be taken to ensure that seamless connectivity is maintained for each VIRL network.  This can be done in one of two ways:


Regardless of the method used, the logical connectivity between ESXi hosts and within the VIRL OpenStack Cluster must be as illustrated below:

Interface Addressing

The default interface addressing convention for VIRL on OpenStack Clusters is described below.  The adresses for the 'Management', 'Flat', 'Flat1', and 'SNAT' networks can and should be adjusted to suit your exact deployment requirements when necessary.

Controller Compute-1 Compute-2 Compute-3 Compute-4
eth0 DHCP or Static DHCP or Static DHCP or Static DHCP or Static DHCP or Static
eth1 172.16.1.254 172.16.1.241 172.16.1.242 172.16.1.243 172.16.1.244
eth2 172.16.2.254 172.16.2.241 172.16.2.242 172.16.2.243 172.16.2.244
eth3 172.16.3.254 172.16.3.241 172.16.3.242 172.16.3.243 172.16.3.244
eth4 172.16.10.250 172.16.10.241 172.16.10.242 172.16.10.243 172.16.10.244

You must not change the subnet used for the 'INT' network.  This must remain on the 172.16.10.0/24 subnet, and the Controller must be assigned 172.16.10.250 on interface 'eth4'.

If you are installing a VIRL OpenStack Cluster alongside an existing standalone VIRL deployment you must ensure that they remain isolated using distinct switches, VLANs, or port-groups.  Otherwise, conflicts will occur on one or more of the Controller interfaces.

Controller Deployment

Installing the Controller Software

The Controller in a VIRL OpenStack Cluster is adapted from a VIRL standalone node.  As such, start by using menu at the left to select and follow the full installation process appropriate to your target environment.

Do not proceed until you have fully installed, configured, licensed, and verified your VIRL node using the installation process described for your environment.  A validated connection to the VIRL Salt infrastructure is required for successful cluster deployment.

Configuring the Controller

The first series of steps inform the new VIRL standalone node that it will be operating in a cluster:

  1. Login to the controller at the IP address recording during installation using the username 'virl' and password 'VIRL':

    ssh virl@<controller-ip-address>

  2. Make a copy of the VIRL configuration file 'virl.ini':

    sudo cp /etc/virl.ini /etc/virl.ini.orig

  3. Open 'virl.ini' using the 'nano' editor:

    sudo nano /etc/virl.ini

  4. Locate the configuration element 'virl_cluster: FALSE' and change its value to 'TRUE'.

Next you must identify how many compute nodes - from 1 to 4 - will be present in the cluster:

  1. Locate the configuration element for the first compute node - 'compute1_active: FALSE' - and change its value to 'TRUE'.


  2. Repeat for each additional compute node to be included in the cluster, from 2 to 4.

Save and apply the configuration changes:

  1. Enter 'Control-X', 'Y' and 'Enter' to save the file and exit.


  2. Apply the changes and update the controller's Salt configuration using the following commands:

    vinstall vinstall
    vinstall salt
    sudo salt-call state.sls common.salt-master.cluster
    sudo salt-call state.sls virl.hostname.cluster
    vinstall salt

Continue configuring the controller using the User Workspace Manager (UWM):

  1. Open a browser and navigate to the the controller's UWM interface - 'http://<controller-ip-address>:19400'.


  2. Login using username 'uwmadmin' and password 'password'


  3. Select 'VIRL Server' from the menu.


  4. Select 'System Configuration'.


The 'System Configuration' page will include tabs for each of the enabled compute nodes.

  1. For each compute node in your cluster, select the appropriate tab and adjust its configuration to match your environment.

    The 'Cluster Configuration and Defaults' section below includes a description of each cluster configuration element and the default values associated with each of the four possible compute nodes.


  2. Once you have made all of the necessary changes select the 'Apply Changes' button.


Loading the VIRL Salt-States

Can you Get to GitHub?

If your controller is installed in a location that prevents access to GitHub except through a proxy you must perform the following steps.  If access to GitHub is not an issue skip ahead to the 'Restart the Controller' section below.

  1. Remove the pre-installed VIRL salt-states directory:

    sudo rm -r /srv/salt

  2. Change to the '/srv' (service data) directory:

    cd /srv

  3. Configure the GitHub proxy:

    git config --global http.proxy http://<proxy-fqdn>:<port>

  4. Clone the VIRL salt-states from GitHub into the 'salt' directory:

    sudo git clone https://github.com/Snergster/virl-salt salt

  5. Confirm that the VIRL salt-states were properly cloned and the file 'LICENSE' exists:

    ls salt

Restarting the Controller

You must now restart the controller in order to finalize the configuration and services:

  1. Restart the controller:

    sudo reboot now

Compute Node Deployment

Compute nodes are installed using specialized OVAs or ISOs named 'compute.n.n.n.ova' or 'compute.n.n.n.iso', respectively.  Please refer to your license confirmation email for information on how to download these installation sources.

Installing Compute Node Software

The installation of a compute node starts as an abbreviated version of a standalone installation:

  1. Download the compute node package appropriate to your deployment - OVA or ISO.

    Trade time for key-strokes.

    The download site will include pre-configured versions of the software for each of the four possible compute nodes (1 through 4) named 'computeN.n.n.n.ova' or 'computeN.n.n.n.iso'.  You can take the time to download and deploy these if you have no plans to deviate from the default configuration settings.  Once you've deployed, move on to the 'Restart the Controller' section below.


  2. Follow the standalone installation instructions appropriate to your deployment target (Bare-Metal, vSphere Client, vSphere Web...).

    Before you go.

    Before heading off to install the compute node software - remember to:

    • Use 'computeN' (as defined on the controller) if prompted for a host name.  Don't use 'virl'.

    • Carefully configure proxy settings if needed for your deployment location.

    • Stop when you reach the activation step.  Activation and the steps beyond are only needed for controller and standalone nodes.

    • Reboot the compute node before continuing with the customization steps below.

Customizing the Compute Node

Once you've deployed and configured the software for each compute node, proceed with the customization steps below.  Start by ensuring that the compute node name is represented in the key configuration files ('computeN' for each compute node in the cluster):

  1. Login to the compute node at the IP address recording during installation using the username 'virl' and password 'VIRL':

    ssh virl@<compute-node-ip-address>

  2. Open the node-specific name-resolution file for editing:

    sudo nano /etc/hosts

  3. Ensure that IP address '127.0.1.1' maps to the appropriate compute node name, as defined on the controller.  In this example 'compute1':


  4. Enter 'Control-X', 'Y' and 'Enter' to save the file and exit.


  5. Open the host-name configuration file for editing:

    sudo nano /etc/hostname

  6. Ensure that the node name is set to the appropriate compute node name, as defined on the controller, and save the file.  In this example 'compute1':


  7. Open the Salt Minion configuration file for editing:

    sudo nano /etc/salt/minion.d/extra.conf

  8. Ensure that the 'id' is set to the appropriate compute node name, as defined on the controller. and save the file.  In this example 'compute1':

Now configure the IP addresses for the key VIRL network interfaces to reflect those defined for the compute node on the controller (in 'virl.ini')

  1. Open the network interface configuration file for editing:

    sudo nano /etc/network/interfaces

  2. Ensure that the IP addresses for 'eth1', 'eth2', 'eth3', and 'eth4' match those defined in 'virl.ini' on the controller and save the file.  In this example for 'compute1' these would be as indicated below by default:

If installed in a location that requires an Internet Proxy, configure APT so that repositories can be accessed.

  1. Open the APT configuration file for editing:

    sudo nano /etc/apt/apt.conf

  2. Enter the proxy configuration information for use by APT and save the file:

    Acquire::http::proxy "http://proxy.domain.tld:port/";

    Enter the above statement using the exact format but substitute your proxy information.  For example 'Acquire::http::proxy "http://proxy.foobar.com:80/";'

For each of these changes to take affect the compute node must be restarted.

  1. Reboot the compute node to apply the configuration changes:

    sudo reboot now

Enabling Compute Node Operations

Once the compute node has rebooted continue by validating connectivity to the controller and enabling the VIRL compute node software using the steps below:

  1. Login to the compute node at the IP address recording during installation using the username 'virl' and password 'VIRL':

    ssh virl@<compute-node-ip-address>

  2. Ensure that connectivity to the controller exists:

    ping 172.16.10.250
  3. Connectivity to the controller must be confirmed.  Explore and resolve any issues before proceeding.

    Skip ahead?

    If no changes to any configuration files have been made up to this point you can skip the next two steps and move on to 'Restarting the Controller'.

  4. Enable VIRL compute node functionality using the following Salt commands:

    sudo salt-call state.sls virl.basics
    sudo salt-call state.sls common.virl
    sudo salt-call state.sls openstack.compute
    sudo salt-call state.sls openstack.setup

    Each command will end with a summary of the number of successful and failed steps.  Do not be alarmed by a small number of failures.


  5. Restart the compute node to finalize the installation:

    sudo reboot now

For each compute node you wish to deploy repeate this 'Compute Node Deployment' section.

Restarting the Controller

You must now restart the controller so that it registers the pre-built or custom compute nodes that have been deployed.

  1. Login to the controller at the IP address recording during installation using the username 'virl' and password 'VIRL':

    ssh virl@<controller-ip-address>

  2. Reboot the controller:

    sudo reboot now

Cluster Validation

Once the controller and each compute node has been deployed you should validate that the cluster is properly configured and operational:

  1. Login to the controller at the IP address recording during installation using the username 'virl' and password 'VIRL':

    ssh virl@<controller-ip-address>

  2. Verify that each compute node is registered with OpenStack Nova:

    nova service-list

    In the example below there are five 'nova-compute' services registered - one on the controller and another for each compute node that has beeen deployed:


  3. Verify that each compute node is registered with OpenStack Neutron

    neutron agent-list

    In the example below there are five 'Linux bridge agents' registered - one on the controller and another for each compute node that has beeen deployed:

At this point your VIRL OpenStack Cluster should be fully operational and you should be able to start a VIRL simulation and observe all of the nodes become 'Active'.

Cluster Troubleshooting

In situations where communications is lost between the controller and a compute node, or if a compute node becomes inoperable, you can determine the state of each compute node from the controller using the 'nova service-list'' and 'neutron agent-list' commands you learned above.

For example, in the illustration below communications has been lost with 'compute4'.  Observe that Nova shows the node as 'down', and Neutron shows the agent as 'xxx' (dead):


In this circumstances proper operation may be restored by restarting the affected node.

If restarting the compute node does not restore proper operation you may also want to check that the node has associated with a valid NTP clock source:

sudo ntpq -p

Valid peer associations are indicated by a '*' alongside the clock-source name, as illustrated:

Cluster Maintenance

Maintaining the VIRL Salt States When GitHub is Blocked

If you have deployed behind an Internet proxy it will be necessary to routinely (perhaps once per week) refresh the VIRL Salt states on the controller.  These are used to update and maintain the software on the compute nodes:

  1. Login to the controller at the IP address recording during installation using the username 'virl' and password 'VIRL'.

  2. Change to the '/srv/salt' (VIRL Salt) directory.

  3. Pull the latest VIRL Salt states from GitHub:

    sudo git pull

Adding Additional Compute Nodes

To add additional compute nodes to an existing VIRL cluster:

  1. Login to the controller at the IP address recording during installation using the username 'virl' and password 'VIRL'.

  2. Edit '/etc/virl.ini' and set the configuration element for the new compute node to 'True'.

  3. Apply the changes and update the controller's cluster configuration:

    vinstall salt

  4. Complete the instructions for 'Compute Node Deployment' for the new compute node.

  5. Complete the instructions in 'Cluster Validation' to ensure that the new compute node was properly deployed.

Cluster Configuration and Defaults

The following configuration elements defiend in '/etc/virl.ini' or via the UWM are used to define VIRL OpenStack Cluster configurations:

Parameter Default Description Notes
computeN_active False Specifies the absense or presence of the compute node 'N' in the cluster. Set to 'True' for each available compute node (1 through 4).
computeN_nodename computeN Specifies the hosname associated with the compute node. This field must match the nodename defined on the compute node.
computeN_public_port eth0 Specifies the name of the port used to reach the Internet on the compute node. This field must match exactly the public port name and format specified on the compute node.
computeN_using_dhcp_on_the_public_port True Specifies the addressing method used for the public port on the compute node. Set to 'False' if using Static IP addressing.
computeN_static_ip 10.10.10.10 The Static IP address assigned to the public port. Not used if DHCP is enabled.  Review and modify to match deployment requirements.
computeN_public_netmask 255.255.255.0 The network mask assigned to the public port. Not used if DHCP is enabled.  Review and modify to match deployment requirements.
computeN_public_gateway 10.10.10.1 The IP address of the default gateway assigned to the public port. Not used if DHCP is enabled.  Review and modify to match deployment requirements.
computeN_first_nameserver 8.8.8.8 The IP address of the first name-server assigned to the public port. Not used if DHCP is enabled.  Review and modify to match deployment requirements.
computeN_second_nameserver 8.8.4.4 The IP address of the second name-server assigned to the public port. Not used if DHCP is enabled.  Review and modify to match deployment requirements.
computeN_l2_port eth1 The name of the first layer-2 network port ('Flat') on the compute node. This field must match exactly the name and format specified on the compute node.
computeN_l2_address 172.16.1.24N The IP address assigned to the first layer-2 network port ('Flat') on the compute node. This field must match exactly the IP address specified on the compute node.  'N must match the nodename / position in the cluster.
computeN_l2_port2 eth2 The name of the second layer-2 network port ('Flat1') on the compute node. This field must match exactly the name and format specified on the compute node.
computeN_l2_address2 172.16.2.24N The IP address assigned to the second layer-2 network port ('Flat1') on the compute node. This field must match exactly the IP address specified on the compute node.  'N must match the nodename / position in the cluster.
computeN_l3_port eth2 The name of the layer-3 network port ('SNAT') on the compute node. This field must match exactly the name and format specified on the compute node.
computeN_l3_address 172.16.3.24N The IP address assigned to layer-3 network port ('SNAT') on the compute node. This field must match exactly the IP address specified on the compute node.  'N must match the nodename / position in the cluster.
computeN_internalnet_ip 172.16.10.24N The IP address assigned to internal / cluster network interface ('eth4'). This field must match exactly the IP address specified on the compute node.  'N must match the nodename / position in the cluster.

The default configuration elements for each compute node in a VIRL OpenStack Cluster are as follows:

The configuration element defaults that are identical across compute nodes have been greyed-out in the table below.

Compute Node 1 Compute Node 2 Compute Node 3 Compute Node 4
computeN_active False False False False
computeN_nodename compute1 compute2 compute3 compute4
computeN_public_port eth0 eth0 eth0 eth0
computeN_using_dhcp_on_public_port True True True True
computeN_static_ip 10.10.10.10 10.10.10.10 10.10.10.10 10.10.10.10
computeN_public_netmask 255.255.255.0 255.255.255.0 255.255.255.0 255.255.255.0
computeN_public_gateway 10.10.10.1 10.10.10.1 10.10.10.1 10.10.10.1
computeN_first_nameserver 8.8.8.8 8.8.8.8 8.8.8.8 8.8.8.8
computeN_second_nameserver 8.8.4.4 8.8.4.4 8.8.4.4 8.8.4.4
computeN_l2_port eth1 eth1 eth1 eth1
computeN_l2_address 172.16.1.241 172.16.1.242 172.16.1.243 172.16.1.244
computeN_l2_port2 eth2 eth2 eth2 eth2
computeN_l2_address2 172.16.2.241 172.16.2.242 172.16.2.243 172.16.2.244
computeN_l3_port eth3 eth3 eth3 eth3
computeN_l2_address2 172.16.3.241 172.16.3.242 172.16.3.243 172.16.3.244
computeN_internalnet_port eth4 eth4 eth4 eth4
computeN_internalnet_ip 172.16.10.241 172.16.10.242 172.16.10.243 172.16.10.244

End of Installation.