Migrate VMware vCenter from ESXi to Microsoft Hyper-V 2012 R2

Because vCenter was running on the ESXi that was supposed to be updated by the vCenter itself, it had to be moved elsewhere first.

To migrate to Hyper-V use the Microsoft tool
Microsoft Virtual Machine Converter 2.0 at
http://www.microsoft.com/en-us/download/details.aspx?id=42497

Check the manual, there are some caveats.

Installed on the Hyper-V server and shared the 'Disks' folder on the Hyper-V server to administrators. This share is required by the converter tool.

Shut down the vCenter machine and point the converter to the ESXi itself.

Use the shared drive above as the destination and a temp folder as the conversion working folder.

After conversion completes:

• uninstall the VMware tools and install the Microsoft Integration Services
• change the settings on the NIC as there's a completely new one installed. Uncheck IPV6
• update any DHCP lease reservations with the new MAC address if needed
• to keep the old VM from interfering, disconnect the NIC for the VM by unchecking the 'connected' box in ESX

The migrated vCenter machine had some issues and was unable to start the vpxd service

There was an error something to the effect of "unable to create sso facade" in the log file at

  %ProgramData%\VMware\VMware VirtualCenter\Logs\vpxd.log

Two possible fixes 

vmware KB 2056296

vmware KB 2036170

• Check your hosts file and make sure when you ping the FQDN you get IPV4 responses only.
• Run the fix on the second article 

start -> run -> cmd

set JAVA_HOME=C:\Program Files\VMware\Infrastructure\jre
cd "C:\Program Files\VMware\Infrastructure\SSOServer\Utils
rsautil manage-secrets -a recover -m <masterPassword>

Try to manually start service or reboot

VMware vCenter: A general system error occurred: Authorize Exception

Intro 

Trying to log in to ESXi and vCenter 5.1, getting error 'Authorize Exception'. These are the troubleshooting steps taken to resolve.
There was a specific cause tied to a scenario and there are solution steps taken to remedy.

Troubleshooting

Luckily there is a KB article that shows up when you search for the error text on a search engine. We'll start with that as a guide

vCenter Server login fails with the error: A general system error occurred: Authorize Exception (1015639)

VMware advises to check these 3 things

1. SSO identity source not configured correctly
2. vCenter Server disconnected from Active Directory
3. DNS resolution

In order of difficulty, start from the bottom.

3. DNS resolution

To verify that the DNS is working, on the vCenter window server, do one or all of these

• check the ipconfig/all from the command line and verify the DNS servers listed are correct
• open cmd, type nslookup, and try to resolve the hostname and FQDN of your vCenter server

2. vCenter Server AD connectivity

To verify, RDP to vCenter and open the user groups in Windows Server, then review the Administrators group. 

If you are seeing the Security IDs instead of DOMAIN\USER, the AD connectivity is lost and you may have to rejoin the AD. 

The KB article above has some more details before you just re-add blindly.

1. SSO identity source

Here's the guide on how to configure the SSO

Configuring a vCenter Single Sign-On 5.1 Identity Source using LDAP with SSL (LDAPS) (2041378)

To verify this configuration we are going to go backwards

• Navigate to vCenter web Client
• https://vcenter.corp.potato.com:9443/vsphere-client/ 
• login using the following
• username: admin@System-Domain
• password:  set up during vCenter installation. If you used Simple install, the password may be blank (?)

• Go to Administration -> Sign-On and Discovery, Configuration
• under Identity Sources, click on the ldaps entry and then the little pencil icon to edit it.
• Verify your settings and check the Primary server URL first
• Note the button 'Choose Certificate' does not show up until you type something in any box
• You can test the connection but the fail message is non-descript here.

Check Certificate 

There are different ways about this, but you want to verify the machine certificate of the 'Primary server URL'. This is needed for secure LDAP to work.

• RDP to the windows machine (dynamics.corp.potato.com). Start -> run -> mmc -> add remove snap-in -> Certificates -> Add -> Computer account -> Local -> finish
• Expand Certificates -> Personal -> Certificates
• find the matching cert and verify
• Valid from: check this date is correct!
• Issued to: dynamics.corp.potato.com
• Issued by: your AD cert authority 
• Intent: "Proves your identity to a remote computer" AND "Ensures the identity of a remote computer"
• Certificate Template Name : "DomainController"
• Enhanced Key Usage: "Client Authentication (1.3.6.1.5.5.7.3.2)" AND "Server Authentication (1.3.6.1.5.5.7.3.1)"

Check LDAPS is working

• connect to any domain controller that works and go start -> run -> ldp
• Connection -> connect -> enter the FQDN of the 'primary server' (dynamics.corp.potato.com)
• use port 636 and check 'SSL'
• if it's failing you will get an error that you can look up, hopefully. In this case it was 

0x0 = ldap_unbind(ld);
ld = ldap_sslinit("dynamics", 389, 1);
Error 0 = ldap_set_option(hLdap, LDAP_OPT_PROTOCOL_VERSION, 3);
Error 81 = ldap_connect(hLdap, NULL);
Server error: <empty>
Error <0x51>: Fail to connect to dynamics.
ld = ldap_sslinit("dynamics", 6389, 1);
Error 81 = ldap_set_option(hLdap, LDAP_OPT_PROTOCOL_VERSION, 3);
Error 81 = ldap_connect(hLdap, NULL);
Server error: <empty>
Error <0x51>: Fail to connect to dynamics.

• If it's successful you will get something like this

ld = ldap_sslinit("dynamics", 636, 1);
Error 0 = ldap_set_option(hLdap, LDAP_OPT_PROTOCOL_VERSION, 3);
Error 0 = ldap_connect(hLdap, NULL);
Error 0 = ldap_get_option(hLdap,LDAP_OPT_SSL,(void*)&lv);
Host supports SSL, SSL cipher strength = 128 bits
Established connection to dynamics.
Retrieving base DSA information...
Getting 1 entries:
Dn: (RootDSE)
configurationNamingContext: CN=Configuration,DC=corp,DC=potato,DC=com;
          [many lines]
....... 

Cause

In this case, the following scenario happened:

• A domain controller was removed from the forest. 
• This DC by chance was also the Cert Authority. 
• The replacement DC was not made a CA
• There was no CA on the domain, and all the existing machine issued certs were valid for a year
• After a year has passed since the [automatic] issuing of the certs, the LDAPS ceased to function since the certs were past their expiry date.
• The clue was that during the cert check of 'dynamics' the machine cert was showing as expired and LDP tool was failing to connect as well

Solution

Issue new cert to the 'Primary server' (dynamics)

• A Cert Authority was installed in the domain forest. 'New CA' was added since old private keys were lost
• CA automatically propagated on the domain and issued new machine certs to all the existing DCs and signed with the new CA as 'issued by'. 
• Reboot of the 'dynamics' server was performed here.
• at this point verified LDAPS with LDP tool connecting to 'dynamics' successfully 

Export the cert from 'Primary server' (dynamics)

• Connect to the 'Primary server' (dynamics) 
• Start -> run -> mmc -> add remove snap-in -> Certificates -> Add -> Computer account -> Local -> finish
• Expand Certificates -> Personal -> Certificates
• Find the matching cert by the FQDN -> properties
• Details tab -> Copy to File -> Next -> No, do not export the private key -> Base-64 encoded X.509 (.CER)
• Save this file to a common location

Import the new cert to vCenter server 

This next bit may not be the formal way to fix, but it worked since all the config in place was valid and only the cert was needed

• Navigate to vCenter web Client as instructed above
• https://vcenter.corp.potato.com:9443/vsphere-client/ 
• .... like above and refer to the picture....
• Add any text in either 'Primary' or 'Secondary server URL' and undo it. This will cause the button 'Choose Certificate' to appear, but keep all the existing values as they were. 
• Click the 'Choose Certificate' and navigate to the .CER file you exported above
• The successful message is 'Certificate added' / 'The selected certificate has been successfully added to the vCenter Single Sign On keystore.'
• You can test the connection now as well, 
• since we're logged in as admin@System-Domain change to 'Password' 
• use any domain user/pw that is authorized
• 'The connection has been established successfully.'
• Click OK to save

Logging in to vCenter using the vSphere client worked at this point without any more reboots.

The trust relationship between this workstation and the primary domain failed

Had this error happen with a Microsoft Windows 2008 R2 server running in a VM.
When logging in to Windows, if using any domain accounts, getting the error

“The trust relationship between this workstation and the primary domain failed"


The obvious fix is to rejoin the domain but a much simpler and probably less invasive method is to run the netdom command

Login with a local machine admin account and run:

netdom.exe resetpwd /s:<server> /ud:<user> /pd:*


<server> = a domain controller in the joined domain
<user> = DOMAIN\User format with rights to change the computer password. I used domain admin.

More info at
http://implbits.com/About/Blog/tabid/78/post/don-t-rejoin-to-fix-the-trust-relationship-between-this-workstation-and-the-primary-domain-failed/Default.aspx

ESX 4.0 upgrade to ESXi 5.1 with 3ware 9650SE

Intro

This is a straight up upgrade of an ESX 4.0 installation to ESXi 5.1. 

The hardware is Intel Xeon CPUs however the biggest obstacle was the RAID controller LSI/AMCC 3ware 9650SE which was not detected when installing with the VMware default ISO.

The RAID1 was also where the ESX 4 was installed, so it was not even possible to perform the upgrade without the extra steps.

Overview

1. Download the VMWare driver for the 3ware 9650SE
2. Integrate the driver into a custom boot ISO image
3. Perform the upgrade using a CD on site
4. Few observations and notes

--------------------------------------------------------------------------

1. Download the VMWare driver for the 3ware 9650SE

• Search the web for 'VMware Compatibility Guide'. At the time of writing, the URL is http://partnerweb.vmware.com/
• Change 
• 'What you are looking for' to 'IO Devices'
• select ESXi 5.1 
• and keyword '9650'
• In the results click on the '3ware 9650SE' link. Brand name is under 'AMCC'
• The footnotes refer to 
• '3w-9xxx version 2.27.08.036' and a link
• http://www.vmware.com/download/vsphere/drivers_tools.html
• Follow the link, and either 
• search for '3w-9xxx version 2.27.08.036' in 'All Downloads'
• or follow the links 
• VMware vSphere -> 
• Drivers & Tools tab -> 
• Driver CDs -> 
• look for '3ware' or '2.27.08.036'
• Final link is at VMware ESXi 5.0 Driver CD for LSI 3ware 96xx RAID Controllers
• File LSI_2.27.08.036vm50-637081.zip (110,254 bytes)

2. Integrate the driver into a custom boot ISO image

• Search the web for 'ESXi-Customizer'. At the time of writing, the URL is 
• http://www.v-front.de/p/esxi-customizer.html ver. 2.7
• Install and run the ESXi-Customizer.cmd
• Unzip the driver file, and feed the ISO image and the driver into the ESXi-Customizer
• Files are: 
• VMware-VMvisor-Installer-5.1.0-799733.x86_64.iso
• scsi-3w-9xxx-2.27.08.036-1OEM.500.0.0.472560.x86_64.vib
• Once the ISO image is built, burn it to a CD or DVD, rewritable also works.

3. Perform the upgrade using a CD on site

• This is a straight forward procedure which should not take more than 30min
• Schedule a maintenance window, and reboot with the CD in the host
• The RAID will now be detected as a storage device and you can perform the 'force upgrade'
• After the reboot, make sure you have your new licence ready because it will be in a 60 day trial mode

4. Few observations and notes

• The 4.0 was actually updated (not upgraded) using the Update Manager first. There was a few hundred patches and updates, however I don't believe this was necessary prior to the upgrade to 5.1
• The Update Manager could most likely use the custom ISO as a new upgrade baseline, however this was not tested.
• First attempt was the 5.1 default ISO which was used as an upgrade baseline in the Update Manager, however this failed and caused the host to go in an infinite reboot loop. There is a boot menu that was added that allows either boot from 4.0 or to perform the 5.1 upgrade. Not sure how to edit this menu and remove the 5.1 upgrade as default option which kept failing

Update VMWare Tools on Cisco CallManager 7.1 to ESXi 5.1

Intro

We have CUCM 7 running on ESXi 4.0 that was upgraded to 5.1. VMWare tools need to be installed or updated on the CallManager

Overview of the steps

1. Log in to the CUCM shell via "Remote Support" account
2. Initiate the VMWare tools install
3. Bonus: Troubleshooting an error 'No module mptscsi found'

1. CUCM Remote Support login

• Download the CentOS install disk. Tested working is 
• CentOS-5.5-i386-bin-1of7.iso (653,910,016 bytes)

• Log in to the CUCM console, and add a 'remote support' user
•  admin:utils remote_account enable
•  admin:utils remote_account create [username] [num of days active]

• Mount the CentOS ISO to the VM and reboot the CM
•  admin:utils system restart

• Boot to the CentOS CD, press 'esc' key during boot to get to the BIOS boot manager
• Select the 'recovery disk' option
• Mount the root partition '/' as there may be '/partB' as well
• Once you get to shell prompt, check these files
• #lsattr /etc/passwd /etc/group /etc/shadow /etc/gshadow

• In my case they were all --------, but if there's an immutable flag 'i' you will have to take it out
• #chattr -i /etc/passwd /etc/shadow /etc/group /etc/gshadow

• Change the password for the [username] used above in step 2
• #passwd [username]

• Reboot and unmount the CD from the VM
• Log in to the console using the [username] and password you reset in step above

2. VMWare Tools install

• In the vSphere client, initiate the vmware tools installation by going to Guest-> 'Install/Upgrade VM Tools'. Don't select the automatic install.
• Go to the console where you logged in with [username] and check that you have a folder /media/cdrom
• In my case, there was no /media/cdrom. Because of an upgrade it was located in /partB/media/cdrom however the /etc/fstab was still pointing to /media/cdrom
• I had to create folder /media/cdrom before mounting the CD, then just issue
• # mount /media/cdrom
• Copy the tools tar.gz from the CD to your home folder then untar and run the install script
• # cp /media/cdrom/VMwareTools-9.0.1-913578.tar.gz ~
• # cd ~
• # tar zxvf VMwareTools-9.0.1-913578.tar.gz
• # cd vmware-tools-distrib/
• # ./vmware-install.pl

• Take all the defaults the script offers. If you don't have errors, this should complete fine, in which case just reboot the server and enjoy the job well done.

3. BONUS. Errors!

During the install, just at the end it fails with an error message 

'No module mptscsi found for kernel x.x.x.x'

There's a lovely KB article on VMWare site at 

Upgrading VMware Tools fails with the error: No module mptscsi found for kernel x.x.xx-xx, aborting

Even though the kernel versions don't match, following the instructions to comment out 'mptscsi' from /etc/modprobe.conf worked. In my case I had to comment out two entries, 

/etc/modprobe.conf :
...
#alias scsi_hostadapter1 mptscsi
...
#alias scsi_hostadapter5 mptscsih
...
~

Re-running the 'vmware-install.pl' script completed successfully this time, and I uncommented those entries before the final reboot.

Enable VNC in VMware ESXi 5.1

Steps

1. create custom firewall rule file
2. refresh rules
3. make sure the changes are persistent across reboots
4. NOTE: these changes may not be persistent across upgrades or updates.
5. Enable VNC on the target VM

Procedure

Custom Firewall rule file

Service ID

First, find the service ID to use for your custom firewall rule.

Example:

/etc/vmware/firewall # cat service.xml
...
<service id='0032'>
    <id>vprobeServer</id>
    <rule id='0000'>
      <direction>inbound</direction>
      <protocol>tcp</protocol>
      <porttype>dst</porttype>
      <port>57007</port>
    </rule>
    <enabled>false</enabled>
    <required>false</required>
  </service>

</ConfigRoot>

/etc/vmware/firewall #

We are going to use 0033

Custom File

Create a file on your computer called enablevnc.xml


<!-- Firewall configuration information -->
<ConfigRoot>

  <!-- VNC -->
  <service id="0033">
    <id>VNC</id>
    <rule id='0000'>
      <direction>outbound</direction>
      <protocol>tcp</protocol>
      <porttype>dst</porttype>
      <port>
        <begin>5800</begin>
        <end>5999</end>
      </port>
    </rule>
    <rule id='0001'>
      <direction>inbound</direction>
      <protocol>tcp</protocol>
      <porttype>dst</porttype>
      <port>
        <begin>5800</begin>
        <end>5999</end>
      </port>
    </rule>
    <enabled>true</enabled>
    <required>false</required>
  </service>

</ConfigRoot>

Copy the file to a datastore on your VM

in this example copied to 

/vmfs/volumes/500G2/enablevnc.xml

You can also edit in place in SSH and just copy/paste.


Copy the file from the datastore to the firewall folder

/etc/vmware/firewall # cp /vmfs/volumes/500G2/enablevnc.xml /etc/vmware/firewall/

Refresh firewall rules

/etc/vmware/firewall # esxcli network firewall refresh 

Confirm the changes

/etc/vmware/firewall # esxcli network firewall ruleset list

...

vSPC                  false

remoteSerialPort      false

vprobeServer          false

VNC                    true

Make changes peristent

Edit local.sh

ESXi 5.1 made changes so you have to edit /etc/rc.local.d/local.sh now.

/etc/vmware/firewall # vi /etc/rc.local.d/local.sh 

#!/bin/sh

# local configuration options

# Note: modify at your own risk!  If you do/use anything in this

# script that is not part of a stable API (relying on files to be in

# specific places, specific tools, specific output, etc) there is a

# possibility you will end up with a broken system after patching or

# upgrading.  Changes are not supported unless under direction of

# VMware support.

cp /vmfs/volumes/500G2/enablevnc.xml /etc/vmware/firewall/

      

exit 0

~

/etc/vmware/firewall # 

Run auto-backup.sh

/etc/vmware/firewall # /sbin/auto-backup.sh 

Files /etc/vmware/dvsdata.db and /tmp/auto-backup.61769//etc/vmware/dvsdata.db differ

Saving current state in /bootbank

Clock updated.

Time: 03:54:44   Date: 01/20/2013   UTC

/etc/vmware/firewall #

Enable VNC on target VM

Make sure your VM is shut down

edit the .vmx file manually over SSH or change the 'Configuration Parameters' in vSphere

These are the two config parameters to add. 

There is no password configured in this example.

remotedisplay.vnc.enabled = "true"

remotedisplay.vnc.port = "5900"

Connect

To connect to the desktop of the target VM, use a free VNC client, IP address of your ESXi server and port 5900

Transcend USB 3.0 controller drivers

So the USB controller model TS-PDU3 comes with an NEC chipset.
NEC is called RENESAS now.

You can see the lineup at
www.renesas.com/prod/usb/map/assp_host.html
EDIT: Updated link : http://www.renesas.com/products/soc/usb_assp/index.jsp

I think it's the µPD720200, but I couldn't find a way to download drivers for it from their site.
http://www2.renesas.com/usb/en/product/upd720200.html



Download the transcend driver and install.

filename : RENESAS-USB3-Host-Driver-20320-setup.exe

It will install as

date : 12/10/2010
version : 2.0.32.0

Then go to intel's website and download their updated drivers
http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=19880&lang=eng&wapkw=

filename : USB3.0_allOS_2.1.28.0_PV.exe
EDIT: updated file same link is USB3.0_allOS_2.1.28.1_PV.exe


Run and it will automatically update to

date : 10/25/2011
version : 2.1.28.0

Maybe there are newer drivers but this fixes the issues with recognizing some USB drives.

This was all tested on Windows 7 x64.