Managing vSphere with pyVmomi - Part 3 - Adding Hosts to a Cluster

This post is part 3 of a small series of articles about how to use pyVMomi to build, configure and manage vmware vsphere environments.

For more information on the earlier posts please refer to Part 1 or Part 2

Updating our YAML file for Clusters with member Hosts

You will recall that the strategy for our sample program is to encapsulate the configuration we want to achieve in a simple YAML file.

In this section we will extend that YAML file structure such that Clusters with member Hosts can be represented.

YAML configuration file

Previously we had the following definition :


We will now extend this by adding member ESXi hosts to the Cluster.

 1cat labconfig.yaml
 2lab:
 3    vsphere:
 4        host:
 5        -
 6                   ip: 192.168.10.101
 7                   hostname: "beast1.example.com"
 8                   user: root
 9                   pw: vmware123
10        -
11                   ip: 192.168.10.102
12                   hostname: "beast2.example.com"
13                   user: root
14                   pw: vmware123
15        vcenter:
16                   ip: 192.168.10.11
17                   user: administrator@vsphere.local
18                   pw: vmware
19        topology:
20        -          dc:
21                        name: LABDC
22                        clusters:
23                        -    name: LABC
24                             members:
25                             -  ip: 192.168.10.101
26                             -  ip: 192.168.10.102

As you can see, a simple change has been made to add a members and ip section to the YAML file.

Additionally, a section has been added to contain host details. This section is cross-referenced with the member ip’s so that we can access the required userid and password details for the new hosts. The structure exists this way as it’s simpler to change the topology without having to shuffle around all the host details and keeps that information in one place.

Now, let’s look at the code side of things. The YAML parser will take care of capturing our configuration from the labconfig.yaml file. We need to use that information to make the correct calls to the vSphere SDK via pyVmomi.

A quick browse shows us that we have documentation on Clusters and that there is a method on that class called AddHost. AddHost in turn utilises a vim.host.ConnectSpec to define the characteristics of the host we’re adding. The important thing to note here is that the method returns a vim.Task

Managing Tasks

vSphere allows us to submit multiple long running tasks. Each task is represented by a vim.Task that can be queried to determine the status, success or otherwise of the requested action.

To manage this a simple helper method is added to out Vcenter class to allow us to track a task till completion. In our application we don’t run tasks in parallel so it’s a simple case of waiting for a launched task to either succeed or fail.

 1    def wait_for_task(self, task):
 2
 3        while task.info.state == (self.pyVmomi.vim.TaskInfo.State.running or self.pyVmomi.vim.TaskInfo.State.queued):
 4            time.sleep(2)
 5
 6        if task.info.state == self.pyVmomi.vim.TaskInfo.State.success:
 7            if task.info.result is not None:
 8                out = 'Task completed successfully, result: %s' % (task.info.result,)
 9                print out
10        elif task.info.state == self.pyVmomi.vim.TaskInfo.State.error:
11            out = 'Error - Task did not complete successfully: %s' % (task.info.error,)
12            raise ValueError(out)
13        return task.info.result

As you can see we basically spin in a sleep(2) loop until the task succeeds or fails.

Adding a Host

 1    def add_host(self, cluster_name, hostname,  username, password):
 2        host = self.get_obj([self.pyVmomi.vim.HostSystem], hostname)
 3        if host is not None:
 4            print("host already exists")
 5            return host
 6        else:
 7            if hostname is None:
 8                raise ValueError("Missing value for name.")
 9            cluster = self.get_obj([self.pyVmomi.vim.ClusterComputeResource], cluster_name)
10            if cluster is None:
11                error = 'Error - Cluster %s not found. Unable to add host %s' % (cluster_name, hostname)
12                raise ValueError(error)
13
14            try:
15                hostspec = self.pyVmomi.vim.host.ConnectSpec(hostName=hostname,userName=username, password=password, force=True)
16                task=cluster.AddHost(spec=hostspec,asConnected=True)
17            except self.pyVmomi.vmodl.MethodFault as error:
18                print "Caught vmodl fault : " + error.msg
19                return -1
20            self.wait_for_task(task, self.service_instance)
21            host = self.get_obj([self.pyVmomi.vim.HostSystem], hostname)
22            return host

Firstly, our add_host() method checks to see if there is already a host object with the name we’re trying to add. If it does exist we simply return with the pointer to that Host object. In theory we could move the host to the cluster selected but that is left as an exercise to the reader. ie. The use case is that Hosts have already been added to the vCenter but the task is to move them into an existing Cluster definition.

If the host doesn’t exist we get a pointer the the desired cluster and add the Host to the cluster using a HostSpec that contains the credentials for the Host.

Trying it out

 1$ ./buildLabConfig.py
 2Connecting to 192.168.10.11 using username administrator@vsphere.local
 3/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
 4  InsecureRequestWarning)
 5Connected to 192.168.10.11, VMware vCenter Server 5.5.0 build-2183111
 6Creating Datacenter LABDC
 7Creating Cluster LABC
 8Traceback (most recent call last):
 9  File "./buildLabConfig.py", line 159, in module
10    main()
11  File "./buildLabConfig.py", line 152, in main
12    vc.add_host(j['name'],iptoadd,user,pw)
13  File "./buildLabConfig.py", line 78, in add_host
14    self.wait_for_task(task)
15  File "./buildLabConfig.py", line 105, in wait_for_task
16    raise ValueError(out)
17ValueError: Error - Task did not complete successfully: (vim.fault.SSLVerifyFault) {
18   dynamicType = unset,
19   dynamicProperty = (vmodl.DynamicProperty) [],
20   msg = "Authenticity of the host's SSL certificate is not verified.",
21   faultCause = unset,
22   faultMessage = (vmodl.LocalizableMessage) [],
23   selfSigned = false,
24   thumbprint = 'E2:B0:AE:1A:EE:D1:DA:EE:16:99:FA:9D:D9:90:F3:3A:D1:28:D6:17'
25}

Hmm, what happened? Our AddHost task has failed with a SSLVerifyFault.

The documentation shows us that for the vim.host.ConnectSpec that we pass through to our AddHost() method has an optional attribute called sslThumprint. This attribute is only ‘optional’ if you have certificates that have been signed by a valid CA.
In this case we’re using self-signed certificates so we need to supply the sslThumbprint.

How do we find the sslThumbprint?

A quick google leads us to this article which gives us two options.

  1. Execute an openssl command on the ESXi host
  2. Remotely connect to port 443 on the ESXi host using openssl and extract the thumbprint.

Option 2 is a relatively simple option as it doesn’t require credentials for the host

From a bash perspective the following command can extract the sslThumbprint.

echo -n | openssl s_client -connect 192.168.10.101:443 2>/dev/null | openssl x509 -noout -fingerprint -sha1

To perform this we use python subprocesses and pipes to reflect the above command in a python construct.

Here is the complete program :

#!/usr/bin/python
"""
Using the pyVmomi python bindings for vSphere connect to a VCSA and configure Datacenters and Clusters
as per th spefified configuration file.
"""

import atexit
import yaml
import inspect
import time
import subprocess

class Vcenter(object):
    def __init__(self, vcenter_params):
        self.pyVmomi =  __import__("pyVmomi")
        self.server = vcenter_params['ip']
        self.username = vcenter_params['user']
        self.password = vcenter_params['pw']
        self.connect_to_vcenter()


    def create_datacenter(self, dcname=None, folder=None):
        datacenter = self.get_obj([self.pyVmomi.vim.Datacenter], dcname)
        if datacenter is not None:
            print("datacenter %s already exists" % dcname)
            return datacenter
        else:
            if len(dcname) > 79:
                raise ValueError("The name of the datacenter must be under 80 characters.")
            if folder is None:
                folder = self.service_instance.content.rootFolder
            if folder is not None and isinstance(folder, self.pyVmomi.vim.Folder):

                print("Creating Datacenter %s " % dcname )

                dc_moref = folder.CreateDatacenter(name=dcname)
                return dc_moref

    def create_cluster(self, cluster_name, datacenter):

        cluster = self.get_obj([self.pyVmomi.vim.ClusterComputeResource], cluster_name)
        if cluster is not None:
            print("cluster already exists")
            return cluster

        else:
            if cluster_name is None:
                raise ValueError("Missing value for name.")
            if datacenter is None:
                raise ValueError("Missing value for datacenter.")

            print("Creating Cluster %s " % cluster_name )

            cluster_spec = self.pyVmomi.vim.cluster.ConfigSpecEx()

            host_folder = datacenter.hostFolder
            cluster = host_folder.CreateClusterEx(name=cluster_name, spec=cluster_spec)
            return cluster

    def add_host(self, cluster_name, hostname, sslthumbprint,  username, password):
        host = self.get_obj([self.pyVmomi.vim.HostSystem], hostname)
        if host is not None:
            print("host already exists")
            return host
        else:
            if hostname is None:
                raise ValueError("Missing value for name.")
            cluster = self.get_obj([self.pyVmomi.vim.ClusterComputeResource], cluster_name)
            if cluster is None:
                error = 'Error - Cluster %s not found. Unable to add host %s' % (cluster_name, hostname)
                raise ValueError(error)

            try:
                hostspec = self.pyVmomi.vim.host.ConnectSpec(hostName=hostname,userName=username, sslThumbprint=sslthumbprint, password=password, force=True)
                task=cluster.AddHost(spec=hostspec,asConnected=True)
            except self.pyVmomi.vmodl.MethodFault as error:
                print "Caught vmodl fault : " + error.msg
                return -1
            self.wait_for_task(task)
            host = self.get_obj([self.pyVmomi.vim.HostSystem], hostname)
            return host

    def get_obj(self, vimtype, name):
        """
        Get the vsphere object associated with a given text name
        """
        obj = None
        container = self.content.viewManager.CreateContainerView(self.content.rootFolder, vimtype, True)
        for c in container.view:
                if c.name == name:
                    obj = c
                    break
        return obj

    def wait_for_task(self, task):

        while task.info.state == (self.pyVmomi.vim.TaskInfo.State.running or self.pyVmomi.vim.TaskInfo.State.queued):
            time.sleep(2)

        if task.info.state == self.pyVmomi.vim.TaskInfo.State.success:
            if task.info.result is not None:
                out = 'Task completed successfully, result: %s' % (task.info.result,)
                print out
        elif task.info.state == self.pyVmomi.vim.TaskInfo.State.error:
            out = 'Error - Task did not complete successfully: %s' % (task.info.error,)
            raise ValueError(out)
        return task.info.result


    def connect_to_vcenter(self):
        from pyVim import connect

        print("Connecting to %s using username %s" % (self.server, self.username))
        self.service_instance = connect.SmartConnect(host=self.server,
                                                user=self.username,
                                                pwd=self.password,
                                                port=443)
        self.content = self.service_instance.RetrieveContent()
        about = self.service_instance.content.about
        print("Connected to %s, %s" % (self.server, about.fullName))
        atexit.register(connect.Disconnect, self.service_instance)

    def getsslThumbprint(self,ip):

        p1 = subprocess.Popen(('echo', '-n'), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        p2 = subprocess.Popen(('openssl', 's_client', '-connect', '{0}:443'.format(ip)), stdin=p1.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        p3 = subprocess.Popen(('openssl', 'x509', '-noout', '-fingerprint', '-sha1'), stdin=p2.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        out = p3.stdout.read()
        ssl_thumbprint = out.split('=')[-1].strip()
        return ssl_thumbprint


def main():

    config = yaml.load(open("labconfig.yaml"))

    vc=Vcenter(config["lab"]["vsphere"]["vcenter"]);
    dc=config["lab"]["vsphere"]["topology"]
    hostlist = config['lab']['vsphere']['host']

    # for each datacenter, get the name and optionally if there is a key for clusters then create matching clusters
    # in this datacenter, otherwise just create the datacenter.

    for i in dc:
       datacenter = vc.create_datacenter(dcname=i['dc']['name'])
       if i['dc'].has_key('clusters') :
          for j in i['dc']['clusters']:
             cluster=vc.create_cluster(j['name'],datacenter)
             if j.has_key('members') :
                for k in j['members']:
                   iptoadd=None
                   user=None
                   pw=None
                   for l in hostlist:             # lookup the member details in the hosts section in the YAML to find the
                      if l['ip']==k['ip']:        #
                         iptoadd=l['ip']          #
                         user=l['user']           # user and pasword details
                         pw=l['pw']
                   if iptoadd == None:
                       print("Couldnt find credentials for ip %s" % k['ip'])
                   else:
                       sslthumbprint=vc.getsslThumbprint(iptoadd)
                       vc.add_host(j['name'],iptoadd,sslthumbprint,user,pw)


    return 0

# Start program
if __name__ == "__main__":
    main()

Here is what the VCSA looked like before the job executed.

VCSA Before

 ./buildLabConfig.py
Connecting to 192.168.10.11 using username administrator@vsphere.local
/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
Connected to 192.168.10.11, VMware vCenter Server 5.5.0 build-2183111
Creating Datacenter LABDC
Creating Cluster LABC
Task completed successfully, result: 'vim.HostSystem:host-142'
Task completed successfully, result: 'vim.HostSystem:host-148'

and this is what it looks like after execution.

VCSA After

Other parts in the series can be found below :

  1. Part 1 :- Connecting to the VCSA
  2. Part 2 :- Creating Datacenters and Clusters
  3. Part 3 :- Adding Hosts to Clusters
  4. Part 4 :- Adding a NFS Share to a Cluster of Hosts