ESX41i Scripted deployment

ESX41i deployment

Useful Pages
http://www.van-lieshout.com/2009/10/uda2-0-test-drive/
http://www.rtfm-ed.co.uk/docs/vmwdocs/uda20-beta.pdf
http://www.virtuallyghetto.com/2010/09/automating-esxi-41-kickstart-tips.html
http://www.kendrickcoleman.com/index.php?/Tech-Blog/esxi-41-kickstart-install-wip.html

PXE deployment of ESX41i to hosts,
Although there are only 24 ESX hosts in the firm and deploying them manually would not be a
significant undertaking as the install per machine would be approx 30 minutes , the benefits ,
in our case to doing the deployment via a scripted PXE install are :

Identical repeatable builds - Apply known good working configurations

easily cope with hardware failures - failed array controller , CPU's etc , swap out the blade or server and kick off install

Perform upgrades of the OS by rolling out new install rather than patching -point PXE server to new OS and deploy our ESX servers have been gradually diverging in terms of configuration , this would be an ideal way to homogenize the builds


I downloaded and configured a copy of the Ultimate deployment appliance from
http://www.ultimatedeployment.org/
Deployed this to a VM , within the low resource usage pool , started it up and configured
the appliance , give it an IP address , hostname etc etc disabling DHCP as i did not want
the appliance handing out IPs on the server VLAN.

Once the machine was deployed the rest of the configuration was handled over the web based interface,
from vSphere i presented the ESX41i iso over the CD drive and from web interface of the UDA i mounted the CD drive.
Then click the OS tab and click new: enter the following
Flavor Name : ESX41i
operating system : VMware ESX...... click next
browse to the image location and click finish.
This imports the deployment image source files to the UDA

Now the UDA has the OS source files and is ready to hand out the base  image, indeed we could
use the UDA as a store for unconfigured images in this state.
As we want to automate the deployment there are a number of things we need to do
we also need to secure and allow the windows DHCP servers to hand out the PXE host IP and the image name.

 i will be allowing all servers on the server VLAN to potentially start up and unconditionally format and deploy ESX i have set the Appliance to request a password , this is to prevent accidental deployment of ESX41 to windows boxes.

To deploy the configured image successfully i need to customise it for our infrastructure , this requires
a kickstart script which contains settings for amongst others the source for the deployment media, the root password
disk partitioning , networking configuration , port groups , vmotion portgroup etc etc

( Some of the following code is plagiarised , specifically the joining AD part)

Within the templates tab , select advanced and paste the kickstart script in here

# Location script
# Standard install arguments
vmaccepteula
rootpw password
install url http:[UDA_IPADDR]/esx41i/ESX41i/
autopart --firstdisk --overwritevmfs
network --bootproto=static --device=vmnic0 --ip=[IPADDR] --gateway=10.128.134.229 --nameserver=10.128.100.100 --netmask=255.255.255.0 --hostname=[HOSTNAME].domainname.com --addvmportgroup=0

# We will configure some basic things during the first boot from the commandline before we add the host to vCenter
%firstboot --unsupported --interpreter=busybox

# Add an extra nic to vSwitch0 and a VLAN ID
esxcfg-vswitch -L vmnic1 vSwitch0
esxcfg-vswitch -v 26 -p 'Service Console' vSwitch0

# Add Server LAN and Deployment LAN portgroups to vSwitch 0
esxcfg-vswitch -A ''Server LAN' vSwitch0
esxcfg-vswitch -v 0 -p ''Server LAN' vSwitch0

esxcfg-vswitch -A ''win7 Deployment SP' vSwitch0
esxcfg-vswitch -v 29 -p 'win7 Deployment SP' vSwitch0


# Add new vSwitch for VM traffic (vmnic2 and vmnic3)
esxcfg-vswitch -a vSwitch1

# Add NICs to the new vSwitch1
esxcfg-vswitch -L vmnic2 vSwitch1
esxcfg-vswitch -L vmnic3 vSwitch1

# Add iSCSI Service Console and vmotion portgroups to vSwitch 1
esxcfg-vswitch -A 'iSCSI Service Console' vSwitch1
esxcfg-vswitch -v 27 -p 'iSCSI Service Console' vSwitch1
# Assign an ip-adress to the vMotion VMkernel and a VLAN ID to the Portgroup
esxcfg-vmknic -a -i [iSCSIsVCiP] -n 255.255.255.0 'iSCSI Service Console'

# Add 'RDM Switch' port group to vSwitch 1
esxcfg-vswitch -A 'RDM Switch' vSwitch1
esxcfg-vswitch -v 27 -p 'RDM Switch' vSwitch1

esxcfg-vswitch -A 'iSCSI-Vmotion' vSwitch1
esxcfg-vswitch -v 27 -p 'iSCSI-Vmotion' vSwitch1
# Assign an ip-adress to the vMotion VMkernel and a VLAN ID to the Portgroup
esxcfg-vmknic -a -i [vMotioniSCSIiP] -n 255.255.255.0 'iSCSI-Vmotion'


# Enable vMotion on the newly created VMkernel vmk0
vim-cmd hostsvc/vmotion/vnic_set vmk0

# Try to Add NFS datastores

esxcfg-nas -a -o 192.168.100.100-s /vol/Storageswap Storageswap
esxcfg-nas -a –o 192.168.100.100 -s /vol/Storagesw Storagesw
esxcfg-nas -a -o 192.168.100.100 -s /vol/Storagepage Storagepage
esxcfg-nas -a -o 192.168.100.100 -s /vol/Storageprod Storageprod
esxcfg-nas -a -o 192.168.100.100 -s /vol/Storagesrmph Storagesrmph
esxcfg-nas -a -o 192.168.100.100 -s /vol/Storagevol0 Storagevol0

esxcfg-advcfg -s 30 /Net/TcpipHeapSize
esxcfg-advcfg -s 120 /Net/TcpipHeapMax
esxcfg-advcfg -s 10 /NFS/HeartbeatMaxFailures
esxcfg-advcfg -s 12 /NFS/HeartbeatFrequency
esxcfg-advcfg -s 5 /NFS/HeartbeatTimeout
esxcfg-advcfg -s 64 /NFS/MaxVolumes

vim-cmd hostsvc/net/refresh

There were a number of issues with the above script , well the script itself was fine but it highlighted a few issues with the coexistance of ESX and ESXi in our organisation.
once the host was deployed everything worked as expected , except HA . This appears to have been due to a misconfiguration between the Service console and the VMKernel port
ESXi no longer has a "service Console" it does however have a VMkernel port.
with our ESX setup the Service console was transmitting the HA heartbeats , obviously it was looking for an identical port on all other hosts to transmit to.
one workaround was to modify the HA settings and add a das.allownetworks0 "NetworkName" and das.allownetwork1 "NetworkName"// , the effect of this would have been
to permit other networks to send/recieve HA heartbeats.
The above fix worked ... but once i attempted to reset the HA config by renabling HA on the cluster it failed for all hosts.
The fix in the end turned out to be a lot simpler and tidier , when we initially built the hosts we followed VM best practice and
segregated the traffic at the portgroup level by putting the Service console and the VMkernel(NFS/iSCSI) out on different portgroups but on closer examiniation
this was segregation in name only as both portgroups were going out the same uplinks and both were on the same VLAN.
I cleaned up the config by putting all traffic out the same portgroup (still following best practise for ESXi), this resolved the HA issue as all HA heartbeat traffic was now going out over the
portgroup with he same name.

The script below is the updated script which also joins the ESX host to the domain and just adds one portgroup to the vswitch

1. Stokes place script
2. Standard install arguments
3. Add DNS servers and NTP config

#
vmaccepteula
rootpw Pa55worD
install url http://[UDA_IPADDR]/esx41i/HPESX41i/
autopart --firstdisk --overwritevmfs
network --bootproto=static --device=vmnic0 --ip=[IPADDR] --gateway=10.128.134.229 --nameserver=10.128.100.100 --netmask=255.255.255.0 --hostname=[HOSTNAME]Domainname.com --addvmportgroup=0

# We will configure some basic things during the first boot from the commandline before we add the host to vCenter

%firstboot --unsupported --interpreter=busybox

# Add an extra nic to vSwitch0 and a VLAN ID

esxcfg-vswitch -L vmnic1 vSwitch0
esxcfg-vswitch -v 26 -p 'Management Network' vSwitch0

# Add Server LAN and Deployment LAN portgroups to vSwitch 0

esxcfg-vswitch -A 'Server LAN' vSwitch0
esxcfg-vswitch -v 0 -p ' Server LAN' vSwitch0

esxcfg-vswitch -A 'GDv5 Deployment SP' vSwitch0
esxcfg-vswitch -v 29 -p 'GDv5 Deployment SP' vSwitch0

# Add new vSwitch for NFS/iSCSI traffic (vmnic2 and vmnic3)

esxcfg-vswitch -a vSwitch1

# Add NICs to the new vSwitch1

esxcfg-vswitch -L vmnic2 vSwitch1
esxcfg-vswitch -L vmnic3 vSwitch1

# Add iSCSI Service Console and vmotion portgroups to vSwitch 1

esxcfg-vswitch -A 'iSCSI Service Console' vSwitch1
esxcfg-vswitch -v 27 -p 'iSCSI Service Console' vSwitch1

# Assign an ip-adress to the vMotion VMkernel and a VLAN ID to the Portgroup

esxcfg-vmknic -a -i [vMotioniSCSIiP] -n 255.255.255.0 'iSCSI Service Console'

# Add 'RDM Switch' port group to vSwitch 1

esxcfg-vswitch -A 'RDM Switch' vSwitch1
esxcfg-vswitch -v 27 -p 'RDM Switch' vSwitch1

# Enable vMotion on the newly created VMkernel vmk1

vim-cmd hostsvc/vmotion/vnic_set vmk1

# Add DNS servers

vim-cmd hostsvc/net/dns_set --ip-addresses=10.128.100.100,10.128.100.3

# Try to configure NTP

Echo restrict default kod nomodify notrap noquerynopeer > /etc/ntp.conf
echo restrict 127.0.0.1 >> /etc/ntp.conf
echo server 10.128.100.100 >> /etc/ntp.conf
echo driftfile /var/lib/ntp/drift >> /etc/ntp.conf
/etc/init.d/ntpd stop
/etc/init.d/ntpd start

 # Try to Add NFS datastores

esxcfg-nas -a -o 192.168.100.100 -s /vol/Storageswap Storageswap
esxcfg-nas -a -o 192.168.100.100 -s /vol/Storagesw Storagesw
esxcfg-nas -a -o 192.168.100.100 -s /vol/Storagepage Storagepage
esxcfg-nas -a -o 192.168.100.100 -s /vol/Storageprod Storageprod
esxcfg-nas -a -o 192.168.100.100 -s /vol/Storagesrmph Storagesrmph
esxcfg-nas -a -o 192.168.100.100 -s /vol/Storagevol0 Storagevol0

esxcfg-advcfg -s 30 /Net/TcpipHeapSize
esxcfg-advcfg -s 120 /Net/TcpipHeapMax
esxcfg-advcfg -s 10 /NFS/HeartbeatMaxFailures
esxcfg-advcfg -s 12 /NFS/HeartbeatFrequency
esxcfg-advcfg -s 5 /NFS/HeartbeatTimeout
esxcfg-advcfg -s 64 /NFS/MaxVolumes

# Try to join the Domain

cat > /tmp/joinActiveDirectory.py << JOIN_AD


import sys,re,os,urllib,urllib2,base64

# mob url

url = "https://localhost/mob/?moid=ha-ad-auth&method=joinDomain"

# mob login credentials -- use password = "" for build scripting

username = "root"
password = ""

1. which domain to join, and associated OU
2. e.g.
3. "primp-industries.com"
4. "primp-industries.com/VMware Server OU"

domainname = "Domain.com/ou=esxhosts,ou=servers,ou=OU,dc=domain,dc=com"

# active directory credentials using encoded base64 password

ad_username = "username"
encodedpassword = "XXXXXXXXXXXXXXXXXX"
ad_password = base64.b64decode(encodedpassword)

# Create global variables

global passman,authhandler,opener,req,page,page_content,nonce,headers,cookie,params,e_params

# Code to build opener with HTTP Basic Authentication

passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None,url,username,password)
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)

# Code to capture required page data and cookie required for post back to meet CSRF requirements ###

req = urllib2.Request(url)
page = urllib2.urlopen(req)
page_content= page.read()

# regex to get the vmware-session-nonce value from the hidden form entry

reg = re.compile('name="vmware-session-nonce" type="hidden" value="?([^\s^"]+)"')
nonce = reg.search(page_content).group(1)

# get the page headers to capture the cookie

headers = page.info()
cookie = headers.get("Set-Cookie")

# Code to join the domain

params = {'vmware-session-nonce':nonce,'domainName':domainname,'userName':ad_username,'password':ad_password}
e_params = urllib.urlencode(params)
req = urllib2.Request(url, e_params, headers={"Cookie":cookie})
page = urllib2.urlopen(req).read()


JOIN_AD


#execute python script to Join AD
python /tmp/joinActiveDirectory.py

vim-cmd hostsvc/net/refresh

Halt






There are a number of variables in [Brackets] , the purpose of these is to pipe values defined in the subtemplates tab into the kickstart script as it is running.


#boot into nic
2. request DHCP and PXE
3. Receive DHCP offer , PXE address and boot file name
4. boot into UDA , user is prompted for OS deployment type and prompted for password
5. User is prompted for subtemplate
6. OS Deployment starts, ESX41 is installed to host
7. kickstart script runs
8 machine is available