Increasing Max vCPUS per core in VMware ESX

When creating a large number of Virtual Machines on a not-so-powerful hardware (in my case it was 6 core CPU), you might see the following error when trying to power on the virtual machines.

An error was received from the ESX host while powering on VM TestRHEL6_B90.
Failed to start the virtual machine.
Failed to power on VM.
Could not power on VM : msg.vmk.status.VMK_VCPU_LIMIT_EXCEEDED.
Max VCPUs limit reached: 192 (32768 worlds)

• Check what's the default maxVCPUsPerCore

esxcli system settings kernel list -o maxVCPUsPerCore
Name             Type Description Configured  Runtime  Default
---------------  ------ -------------------------------------------------------------------------- ----------  -------  -------
maxVCPUsPerCore  uint32  Max number of VCPUs should run on a single core. 0 == determine at runtime  85          0        0

Looks like by default there is max no.of vCPUS per core, decided internally by VMWare.

• Increase the vCPUS per core to a value that suits you. In my case around 85VMs per core.

esxcli system settings kernel set -s maxVCPUsPerCore -v 85 

• Reboot the ESX host (sadly, yes). So ensure you set this param before you create your bulk VMs. Then check the value, to ensure it is set right.

esxcli system settings kernel list -o maxVCPUsPerCore
Name             Type    Description                                                                 Configured  Runtime  Default
---------------  ------  --------------------------------------------------------------------------  ----------  -------  -------
maxVCPUsPerCore  uint32  Max number of VCPUs should run on a single core. 0 == determine at runtime  85          85       0

This still didn't guarantee that all VMs you create will power ON. I still hit a ceiling and didn't manage to power ON many VMs. No, this was below 512 VMs, which is the max VMs you can run on any ESX host.

How to check I/O speeds

While working on a VMware ESX image, I came across this situation that it was working really well when hosted on Datastore1. However, if I moved it to Datastore2, the response of the image became very poor.
Since the VMware ESX machine was same, it was abundantly clear there was something wrong with Datastore2. Unfortunately VMware ESX doesn't seem to complaint if the I/O speeds degrade - atleast I am not aware of this. So the only way to find out was by testing the I/O speeds seen by the VMware Image.
The tool I used is a pretty familiar one in Linux world - dd. Here's what you do:

Image is on Datastore1
Step 1: Login to your linux machine
Step 2: Run this command on shell `dd if=/dev/zero of=/dev/null& pid=$!`
Step 3: Run this command after few seconds `kill -USR1 $pid; sleep 1; kill $pid`

You'll see an output like this below. Pay attention to the last line.

[root@mylinux ~]# dd if=/dev/zero of=/dev/null& pid=$!
[1] 10055
[root@mylinux ~]# kill -USR1 $pid; sleep 1; kill $pid
23469330+0 records in
23469329+0 records out
12016296448 bytes (12 GB) copied, 11.711 s, 1.0 GB/s

Repeat the above steps on Datastore2.

[root@mylinux ~]# dd if=/dev/zero of=/dev/null& pid=$!
[1] 4430
[root@mylinux ~]# kill -USR1 $pid; sleep 1; kill $pid
11572320+0 records in
11572319+0 records out
5925027328 bytes (5.9 GB) copied, 10.962 seconds, 541 MB/s

As you can see the I/O speed on Datastore2 is 541 MB/s against 1GB/s on Datastore1. Hence I was seeing degraded performance on Datastore2.

VMware ESX Clones and Windows SID.

I had made a single installation of Windows Server 2008 R2 and saved this as a master clone - using this master clone I made no.of other clones for my use. Turns out if you don't chose the option to customize the new virtual machine (which is the default option :-( ) , the new virtual machine will bear the same SID (security identifier) as the main cloned image.

This is still OK (perhaps not) if you plan to use these cloned VM's in different environments. However, if you plan to create a Active Directory Domain and add these VM's on the same domain, authentication gets messed up seriously. Worst part is that when you add machines to domain, it won't complaint, only when you chose to use domain accounts to auth (like logging into Member Server using Domain account) it will spit out error on event viewer logs.
It took lot of google time to realize that domain account on member server was failing because of duplicate SID which was because I was using clones of the same image.

Lessons:
1) If you ever take clones of a VM, ensure you chose to customize the new virtual machine, by choosing the option "Customize using the Customization Wizard".
2) If you already have clones now, you can accomplish #1 above using VMware Convertor tool, by using the option "Configure machine". This will force SID change on the VM amongst other things.

Running VMware ESX 4 inside VMware Workstation 6.x.

There are lot of posts which describe how to install VMware ESX inside a VMware workstation, for eg: http://vmblog.com/archive/2007/11/01/more-on-installing-vmware-esx-server-inside-a-workstation-6-vm.aspx

This is easy and can be done very quickly. The next step of course is to create a VM. In my test bed I tried to install a RHEL4 OS inside VMware ESX using vCenter. I could create the VM very easily using the Create New Virtual Machine wizard however, the VM wouldn't power ON!. When you try to power the VM ON, the error that it would throw is

"you may not power on a virtual machine inside a virtual machine"

The reason why this happens is that when you installed VMware ESX inside a virtualized environment, you chose to install it as a "Red Hat Enterprise Linux 5" OS. As a result the VMware virtual machine configuration file stores parameter guestOS as follows:

guestOS = "RHEL5"


What you have to do is, edit the VMware virtual machine configuration file (.vmx file inside the folder where you've created the ESX VM) using notepad/wordpad and change it to:

guestOS = "vmkernel"

Now, power ON the OS and voila, it boots. Connect it to your installation CD/ISO and install the system!