to avoid confusion and for brevity i will use the following terms in this documnet
pESX - Physical ESX
vESX - Virtual ESX
A very basic network / logical layout of the pESX/vESX infrastructure
DRAFT
I decided to create an vESX cluster within an pESX server for the purpose of going over material for the VCAP, I needed to be able to break things without worrying about impacting the production environment along with the fact that change control and making changes for the purpose of testing configs for vcap study material are mutually exclusive.
I had read a few articles previously but never bothered to absorb any of the information , i had a few hours free and an pESX server sitting there doing nothing so i decided to see what would happen if i tried to install ESX on ESX and would i be able to get a fully working esx cluster without referring to somebody elses work.
One very important consideration was that the vESX traffic could not touch the production network
I will briefly describe the steps i took to build and configure the environment and the challenges i encountered
The architecture:
Physcial
HP BL 460 G1
4 X 3Ghz CPU's
32 GB RAM
4 X 146Gb HDD's (2 X raid 1 volumes 131GB and 143GB usable storage)
3 X Standard switches mapped to no physical uplinks
Virtual
2 vESX servers(2 pseudo Physical CPUs , 2GB RAM)
2 vESXi servers(2 pseudo Physical CPUs , 2GB RAM)
1 Open Filer which had 140GB of storage configured over 2 LUNs
1 W2K8r2 server acting as DC and VC
1 router (zeroshell)
IP Addressing
ESX01
SC - 192.168.100.1
VmotionFTvmKernel - 192.168.100.5
vswif1 - 192.168.150.21
vmk0 - 192.168.150.1
ESX02
SC - 192.168.100.2
VmotionFTvmKernel - 192.168.100.5
vswif1 - 192.168.150.21
vmk0 - 192.168.150.1
ESXi03
VMK0 - 192.168.100.3
VMK1 - 192.168.150.3
ESXi04
VMK0 - 192.168.100.4
VMK1 - 192.168.150.4
TLOpenFiler
Management IP - 192.168.100.11
iSCSI teamed IF - 192.168.150.13
TLFWrouter
192.168.100.254
192.168.150.254
192.168.200.254
TLDCVS01
192.168.200.100
Networking
In keeping with best practice i made the decision to split out the networking by traffic type,
on the pESX host i created 3 standard switches and did not bind them to any physical adapter
They were labeled as follows
"TL ESX vmK - isolated" 192.168.100.0/24 -> VMkernel and management traffic
"TL ESX iSCSI - isolated" 192.168.150.0/24 -> iSCSI Traffic
"TL ESX VMs - isolated" 192.168.200.0/24 -> Virtual Machine Traffic
Promiscous mode must be enabled on the virtual switches on the pESX host , the reason for enabling this security risk is the nested ESX host would otherwise only see the MAC addresses of virtual machines placed on the virtual network of that vESX host , we enable it so that all vESX servers see all MAC addresses.
Routing
As simple as putting a zeroshell router on the pESX host and giving it a NIC + an IP on each subnet.
Vlans
Not yet
Storage layout
Storage available on the pESX host was 4 X 146 GB HDD's split into 2 volumes , ESXi was installed on one of the volumes so there was 132 GB available on 1 volume (Datastore01) and 140 GB (esx07localvol1) available on the second.
in order to seperate storage by function (i.e vVMs on seperate phsycial spindles to the vESX and management/infrastucture systems) the vESX servers , DC/VC , openfiler and router were all installed on Datastore1.
In order to present storage to the openfiler i opened VC and pointed to the pESX server , and added a number of disks to the "TLOpenfiler" virtual machine (the only reason for adding a number of smaller disks rather than one big disk was simply to be able to add a number of disks within openfiler to create a LUN composed of a number of smaller disks as i felt this was more representative of a near production environment )
Openfiler configuration
The openfiler was installed as a VM directly on the pESX host , storage was allocated to it as above , and the VM was powered on and configured with boot , / (root) and swap all on the same disk (we dont care about performance in this setup) defaults were followed for the remainder.
The openfiler was initially configured with 1 interface which was on the "TL ESX vmK - isolated"
I added two more interfaces on the "TL ESX iSCSI - isolated" and created a bonded interface , assigning a 192.168.150.13 address.
create storage ......... as per http://greg.porter.name/wiki/HowTo:Openfiler
createLUN
createiSCSItarget
vESX server build
As the ESX Service console is simply an instance of Red hat linux i created the new virtual machine in the pVSphere and set the OS type as Redhat enterprise 64bit , i set the NIC type to be E1000 and the disk controller to be SAS parallel.
After the VM had been created i mounted the ESX media and started the VM following the defaults to get a good install.
ESXi does not have a service console instead it uses busybox to issue commands to the vmkernel , in this case i took a chance and created the vESXi machine as a generic 64bit version of linux , again NIC type was set to E1000 and the disk controller was set as SAS parallel.
ESXi image was mounted and installation was kicked off.
vESX Networking
on the pESX server for each vESX server , Add vNIC's mapped to the unbound vNICs created on the the pESX server this will give the vESX server physical adapters. in each case i added 2
NICs for each traffic type to more accurately reflect real world configurations
Attaching to iSCSI storage
On Each ESX server using the vSphere client to connect to each one the following steps must be
performed in order to access iSCSI LUNS (Which have already been created)
on vESX add service console port on the iSCSI vSwitch, assign IP, SN mask etc
Open iSCSI (3260) service console FW port under security configuration.
on the vspehere client attached to the vESX server go to configure , Storage adapters , iSCSI ,
Configured (enabled) and finally click OK
Now ESX is ready to attach to the iSCSI LUN created on the Openfiler
on the iSCSI software intitator , select the Static Discovery tab and enter the IP and port of the
OpenFilers IP on the Storage subnet and enter the target name copied from the Openfiler config page
MPIOing the storage
To enable Redundant MPIOing for the storage i added a second VMkernel port on the vswitch
and named it "TL ESX iSCSI Pathb - Isolated" and gave it an IP on the 150 Subnet.
Now to enable the redundancy in this environment i promoted NIC1 to the active and demoted
NIC2 to be the standby adapter on the "TL ESX iSCSI - Isolated" VMK Port ,
and the reverse of this NIC Order on the "TL ESX iSCSI Pathb - Isolated"
then on the ESX host i ran the following command:
esxcli swiscsi nic add -n vmk0 -d vmhna33
esxcli swiscsi nic add -n vmk1 -d vmhba33
RDM
I also created an RDM to the FileServer01 , to do this i added a virtual machine portgroup to the iSCSI vswitch, naming it "TL ESX iSCSI RDM - Isolated" , added a vNIC bound to that network to the VM , installed the iSCSI initiator and accessed the TLiSCSI1 LUN....except...i couldnt i found that W2K8 does not not seem to be compatible with the version of OpenFiler that i installed(2.3), i chose to not use CHAP authentication on the Openfiler ( Real world i would not use CHAP either , instead i would rely on the non routed segregated iSCSI network and switch port security, CHAP is cleartext and very easy to intercept)
Issues
Power on VM fails with an error about running ESX as a virtual machine
Cannot virtualise vcentre on a vESX (64bit problem)
vcenter install fails saying you cant install on this version of 64bit windows
(This is actually an issue with ADAM being installed on a DC and trying to listen on 389)
Cannot join esxi hosts to a HA clister that already contains esx hosts
(this is to do with the vmkernel and service consle ports having different names and types , create das.allownetworks entry on cluster)
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006541
Cannot Start VMA within the vESX sinfrastructure
(64 bit OS cannot start in vESX)
No comments:
Post a Comment