March 27, 2015

Quickly recover VMware orphaned virtual machines

After the ESXi OS crashed, it was quickly reinstalled to a fresh USB.

Once 'reconnected' in vCenter the existing datastore was recognized automatically however few things were missing:
- networking settings
- firewall rules
- auto-start for any VMs
- all VMs showed as 'orphaned'

Quick way to restore the VMs :
- enable SSH
- find your volume for the datastore in /vmfs/volumes
- run these line

# find /vmfs/volumes/53456cd6-ee79d800-ad57-002590e2fde0/ -name "*.vmx" | sed
's/\(.*\)/vim-cmd solo\/registervm "\1"/' > /restorevmx

# sh /restorevmx
# rm /restorevmx

What it does is search for all .vmx files in the datastore and runs the 'vim-cmd solo/registervm' command. 
Running it on existing VMs that are not 'orphaned' will just cause the tool to skip them

The best part is that the VMs don't have to be removed from inventory to be re-added

There doesn't seem to be any downsides as long as you don't remove them in vCenter.

References: VMware KB 1006160

February 10, 2015

automating winhelp2002 MVP HOSTS file to pfSense 2.2

Intro

looking to implement - "...HOSTS file to block ads, banners, 3rd party Cookies, 3rd party page counters, web bugs, and even most hijackers."


pfSense uses dnsmasq and we need to convert entries from HOSTS file to the dnsmasq format

0.0.0.0 static.a-ads.com -> address=/static.a-ads.com/127.0.0.1

We will have it scheduled to run daily at midnight to keep the list up to date

noteOn pfSense 2.2, The DNS Forwarder is not active by default. It has been replaced by Unbound as a DNS Resolver. It may still be used, and is still active on upgraded configurations. To use the DNS Forwarder (dnsmasq) on 2.2, first disable Unbound and then enable the DNS Forwarder.Dec 26, 2014
DNS Forwarder - PFSenseDocs
https://doc.pfsense.org/index.php/DNS_Forwarder

Therefore this applies only if you're using dnsmasq

Preparation

install package cron 0.1.8

create folder in ssh 
  # mkdir /usr/local/etc/dnsmasq.d/

go to pfSense: 
Services -> DNS Forwarded
  enable advanced button and enter: 
  conf-dir=/usr/local/etc/dnsmasq.d



Command

fetch -qR http://winhelp2002.mvps.org/hosts.txt /root/hosts.txt && perl -e 'while(<>){ chomp; lc; next if /^#/; if (/^0\.0\.0\.0\s([-a-z0-9.]*)/) { print "address=\/$1\/127.0.0.1\n"; } }' /root/hosts.txt > /usr/local/etc/dnsmasq.d/entries && pfSsh.php playback svc restart dnsmasq

broken down and explained

fetch -qR http://winhelp2002.mvps.org/hosts.txt /root/hosts.txt 
&& 

# -q, --quiet
Quiet mode.
# -R, --keep-output
The output files are precious, and should not be deleted
under any circumstances, even if the transfer failed or was
incomplete.

# saves to /root/hosts.txt
---------------------------------------------------------------------------

perl -e '
  while(<>) {    # for every line in input file (specified as argument below)

    chomp;         # remove newlines

    lc;                 # lowercase all characters

    next if /^#/;  # skip if it matches character '#' at the begining - comments

    if (/^0\.0\.0\.0\s([-a-z0-9.]*)/)     
        # if it matches this 0.0.0.0(space)(url - can only contain a-z0-9 and a dash)
        # example 0.0.0.0 static.a-ads.com

    { 
       print "address=\/$1\/127.0.0.1\n"; 
        # print what we matched $1 and the rest of the text so it looks like: 
        #   address=/static.a-ads.com/127.0.0.1

     } 
  }' 

/root/hosts.txt                                          # our input file
> /usr/local/etc/dnsmasq.d/entries          # save to this file
&& 
---------------------------------------------------------------------------

pfSsh.php playback svc restart dnsmasq   # restart dnsmasq service to reload file


Application


test the line by running it in the SSH - check the output of
/root/hosts.txt
/usr/local/etc/dnsmasq.d/entries


when everything runs and you have the 'entries' file populated correctly, schedule it to run daily 

Schedule


After you install the package, go to pfSense: 
Services -> Cron
Pay attention to add full paths





June 13, 2014

Migrate VMware vCenter from ESXi to Microsoft Hyper-V 2012 R2

Because vCenter was running on the ESXi that was supposed to be updated by the vCenter itself, it had to be moved elsewhere first.

To migrate to Hyper-V use the Microsoft tool
Microsoft Virtual Machine Converter 2.0 at
http://www.microsoft.com/en-us/download/details.aspx?id=42497

Check the manual, there are some caveats.
Installed on the Hyper-V server and shared the 'Disks' folder on the Hyper-V server to administrators. This share is required by the converter tool.

Shut down the vCenter machine and point the converter to the ESXi itself.
Use the shared drive above as the destination and a temp folder as the conversion working folder.


After conversion completes:
  • uninstall the VMware tools and install the Microsoft Integration Services
  • change the settings on the NIC as there's a completely new one installed. Uncheck IPV6
  • update any DHCP lease reservations with the new MAC address if needed
  • to keep the old VM from interfering, disconnect the NIC for the VM by unchecking the 'connected' box in ESX

The migrated vCenter machine had some issues and was unable to start the vpxd service
There was an error something to the effect of "unable to create sso facade" in the log file at
  %ProgramData%\VMware\VMware VirtualCenter\Logs\vpxd.log

Two possible fixes 

  • Check your hosts file and make sure when you ping the FQDN you get IPV4 responses only.
  • Run the fix on the second article 
start -> run -> cmd
set JAVA_HOME=C:\Program Files\VMware\Infrastructure\jre
cd "C:\Program Files\VMware\Infrastructure\SSOServer\Utils
rsautil manage-secrets -a recover -m <masterPassword>

Try to manually start service or reboot


January 19, 2014

VMware vCenter: A general system error occurred: Authorize Exception

Intro 

Trying to log in to ESXi and vCenter 5.1, getting error 'Authorize Exception'. These are the troubleshooting steps taken to resolve.
There was a specific cause tied to a scenario and there are solution steps taken to remedy.


Troubleshooting

Luckily there is a KB article that shows up when you search for the error text on a search engine. We'll start with that as a guide

vCenter Server login fails with the error: A general system error occurred: Authorize Exception (1015639)

VMware advises to check these 3 things

  1. SSO identity source not configured correctly
  2. vCenter Server disconnected from Active Directory
  3. DNS resolution


In order of difficulty, start from the bottom.



3. DNS resolution

To verify that the DNS is working, on the vCenter window server, do one or all of these

  • check the ipconfig/all from the command line and verify the DNS servers listed are correct
  • open cmd, type nslookup, and try to resolve the hostname and FQDN of your vCenter server



2. vCenter Server AD connectivity

To verify, RDP to vCenter and open the user groups in Windows Server, then review the Administrators group. 

If you are seeing the Security IDs instead of DOMAIN\USER, the AD connectivity is lost and you may have to rejoin the AD. 
The KB article above has some more details before you just re-add blindly.



1. SSO identity source

Here's the guide on how to configure the SSO


To verify this configuration we are going to go backwards
  • Navigate to vCenter web Client
  • https://vcenter.corp.potato.com:9443/vsphere-client/ 
  • login using the following
    • username: admin@System-Domain
    • password:  set up during vCenter installation. If you used Simple install, the password may be blank (?)
  • Go to Administration -> Sign-On and Discovery, Configuration
  • under Identity Sources, click on the ldaps entry and then the little pencil icon to edit it.
  • Verify your settings and check the Primary server URL first
  • Note the button 'Choose Certificate' does not show up until you type something in any box
  • You can test the connection but the fail message is non-descript here.



































Check Certificate 


There are different ways about this, but you want to verify the machine certificate of the 'Primary server URL'. This is needed for secure LDAP to work.
  • RDP to the windows machine (dynamics.corp.potato.com). Start -> run -> mmc -> add remove snap-in -> Certificates -> Add -> Computer account -> Local -> finish
  • Expand Certificates -> Personal -> Certificates
  • find the matching cert and verify
    • Valid from: check this date is correct!
    • Issued to: dynamics.corp.potato.com
    • Issued by: your AD cert authority 
    • Intent: "Proves your identity to a remote computer" AND "Ensures the identity of a remote computer"
    • Certificate Template Name : "DomainController"
    • Enhanced Key Usage: "Client Authentication (1.3.6.1.5.5.7.3.2)" AND "Server Authentication (1.3.6.1.5.5.7.3.1)"

Check LDAPS is working

  • connect to any domain controller that works and go start -> run -> ldp
  • Connection -> connect -> enter the FQDN of the 'primary server' (dynamics.corp.potato.com)
  • use port 636 and check 'SSL'
  • if it's failing you will get an error that you can look up, hopefully. In this case it was 
0x0 = ldap_unbind(ld);
ld = ldap_sslinit("dynamics", 389, 1);
Error 0 = ldap_set_option(hLdap, LDAP_OPT_PROTOCOL_VERSION, 3);
Error 81 = ldap_connect(hLdap, NULL);
Server error: <empty>
Error <0x51>: Fail to connect to dynamics.
ld = ldap_sslinit("dynamics", 6389, 1);
Error 81 = ldap_set_option(hLdap, LDAP_OPT_PROTOCOL_VERSION, 3);
Error 81 = ldap_connect(hLdap, NULL);
Server error: <empty>
Error <0x51>: Fail to connect to dynamics.

  • If it's successful you will get something like this
ld = ldap_sslinit("dynamics", 636, 1);
Error 0 = ldap_set_option(hLdap, LDAP_OPT_PROTOCOL_VERSION, 3);
Error 0 = ldap_connect(hLdap, NULL);
Error 0 = ldap_get_option(hLdap,LDAP_OPT_SSL,(void*)&lv);
Host supports SSL, SSL cipher strength = 128 bits
Established connection to dynamics.
Retrieving base DSA information...
Getting 1 entries:
Dn: (RootDSE)
configurationNamingContext: CN=Configuration,DC=corp,DC=potato,DC=com;
          [many lines]
....... 

Cause

In this case, the following scenario happened:

  • A domain controller was removed from the forest. 
  • This DC by chance was also the Cert Authority. 
  • The replacement DC was not made a CA
  • There was no CA on the domain, and all the existing machine issued certs were valid for a year
  • After a year has passed since the [automatic] issuing of the certs, the LDAPS ceased to function since the certs were past their expiry date.
  • The clue was that during the cert check of 'dynamics' the machine cert was showing as expired and LDP tool was failing to connect as well


Solution

Issue new cert to the 'Primary server' (dynamics)

  • A Cert Authority was installed in the domain forest. 'New CA' was added since old private keys were lost
  • CA automatically propagated on the domain and issued new machine certs to all the existing DCs and signed with the new CA as 'issued by'. 
  • Reboot of the 'dynamics' server was performed here.
  • at this point verified LDAPS with LDP tool connecting to 'dynamics' successfully 

Export the cert from 'Primary server' (dynamics)

  • Connect to the 'Primary server' (dynamics) 
  • Start -> run -> mmc -> add remove snap-in -> Certificates -> Add -> Computer account -> Local -> finish
  • Expand Certificates -> Personal -> Certificates
  • Find the matching cert by the FQDN -> properties
  • Details tab -> Copy to File -> Next -> No, do not export the private key -> Base-64 encoded X.509 (.CER)
  • Save this file to a common location

Import the new cert to vCenter server 

This next bit may not be the formal way to fix, but it worked since all the config in place was valid and only the cert was needed
  • Navigate to vCenter web Client as instructed above
  • https://vcenter.corp.potato.com:9443/vsphere-client/ 
  • .... like above and refer to the picture....
  • Add any text in either 'Primary' or 'Secondary server URL' and undo it. This will cause the button 'Choose Certificate' to appear, but keep all the existing values as they were. 
  • Click the 'Choose Certificate' and navigate to the .CER file you exported above
  • The successful message is 'Certificate added' / 'The selected certificate has been successfully added to the vCenter Single Sign On keystore.'
  • You can test the connection now as well, 
    • since we're logged in as admin@System-Domain change to 'Password' 
    • use any domain user/pw that is authorized
    • 'The connection has been established successfully.'
  • Click OK to save

Logging in to vCenter using the vSphere client worked at this point without any more reboots.


April 8, 2013

The trust relationship between this workstation and the primary domain failed

Had this error happen with a Microsoft Windows 2008 R2 server running in a VM.
When logging in to Windows, if using any domain accounts, getting the error

“The trust relationship between this workstation and the primary domain failed"


The obvious fix is to rejoin the domain but a much simpler and probably less invasive method is to run the netdom command

Login with a local machine admin account and run:

netdom.exe resetpwd /s:<server> /ud:<user> /pd:*


<server> = a domain controller in the joined domain
<user> = DOMAIN\User format with rights to change the computer password. I used domain admin.

More info at
http://implbits.com/About/Blog/tabid/78/post/don-t-rejoin-to-fix-the-trust-relationship-between-this-workstation-and-the-primary-domain-failed/Default.aspx

March 5, 2013

ESX 4.0 upgrade to ESXi 5.1 with 3ware 9650SE

Intro

This is a straight up upgrade of an ESX 4.0 installation to ESXi 5.1. 
The hardware is Intel Xeon CPUs however the biggest obstacle was the RAID controller LSI/AMCC 3ware 9650SE which was not detected when installing with the VMware default ISO.
The RAID1 was also where the ESX 4 was installed, so it was not even possible to perform the upgrade without the extra steps.

Overview

  1. Download the VMWare driver for the 3ware 9650SE
  2. Integrate the driver into a custom boot ISO image
  3. Perform the upgrade using a CD on site
  4. Few observations and notes

--------------------------------------------------------------------------


1. Download the VMWare driver for the 3ware 9650SE


  • Search the web for 'VMware Compatibility Guide'. At the time of writing, the URL is http://partnerweb.vmware.com/
  • Change 
    • 'What you are looking for' to 'IO Devices'
    • select ESXi 5.1 
    • and keyword '9650'
  • In the results click on the '3ware 9650SE' link. Brand name is under 'AMCC'
  • The footnotes refer to 
  • Follow the link, and either 
    • search for '3w-9xxx version 2.27.08.036' in 'All Downloads'
    • or follow the links 
      • VMware vSphere -> 
      • Drivers & Tools tab -> 
      • Driver CDs -> 
      • look for '3ware' or '2.27.08.036'
  • Final link is at VMware ESXi 5.0 Driver CD for LSI 3ware 96xx RAID Controllers
  • File LSI_2.27.08.036vm50-637081.zip (110,254 bytes)



2. Integrate the driver into a custom boot ISO image


  • Search the web for 'ESXi-Customizer'. At the time of writing, the URL is 
  • Install and run the ESXi-Customizer.cmd
  • Unzip the driver file, and feed the ISO image and the driver into the ESXi-Customizer
  • Files are: 
    • VMware-VMvisor-Installer-5.1.0-799733.x86_64.iso
    • scsi-3w-9xxx-2.27.08.036-1OEM.500.0.0.472560.x86_64.vib
  • Once the ISO image is built, burn it to a CD or DVD, rewritable also works.


3. Perform the upgrade using a CD on site


  • This is a straight forward procedure which should not take more than 30min
  • Schedule a maintenance window, and reboot with the CD in the host
  • The RAID will now be detected as a storage device and you can perform the 'force upgrade'
  • After the reboot, make sure you have your new licence ready because it will be in a 60 day trial mode


4. Few observations and notes


  • The 4.0 was actually updated (not upgraded) using the Update Manager first. There was a few hundred patches and updates, however I don't believe this was necessary prior to the upgrade to 5.1
  • The Update Manager could most likely use the custom ISO as a new upgrade baseline, however this was not tested.
  • First attempt was the 5.1 default ISO which was used as an upgrade baseline in the Update Manager, however this failed and caused the host to go in an infinite reboot loop. There is a boot menu that was added that allows either boot from 4.0 or to perform the 5.1 upgrade. Not sure how to edit this menu and remove the 5.1 upgrade as default option which kept failing

March 4, 2013

Update VMWare Tools on Cisco CallManager 7.1 to ESXi 5.1

Intro


We have CUCM 7 running on ESXi 4.0 that was upgraded to 5.1. VMWare tools need to be installed or updated on the CallManager


Overview of the steps

  1. Log in to the CUCM shell via "Remote Support" account
  2. Initiate the VMWare tools install
  3. Bonus: Troubleshooting an error 'No module mptscsi found'

1. CUCM Remote Support login

  • Download the CentOS install disk. Tested working is 
    • CentOS-5.5-i386-bin-1of7.iso (653,910,016 bytes)
  • Log in to the CUCM console, and add a 'remote support' user
    •  admin:utils remote_account enable
    •  admin:utils remote_account create [username] [num of days active]
  • Mount the CentOS ISO to the VM and reboot the CM
    •  admin:utils system restart
  • Boot to the CentOS CD, press 'esc' key during boot to get to the BIOS boot manager
  • Select the 'recovery disk' option
  • Mount the root partition '/' as there may be '/partB' as well
  • Once you get to shell prompt, check these files
    • #lsattr /etc/passwd /etc/group /etc/shadow /etc/gshadow
  • In my case they were all --------, but if there's an immutable flag 'i' you will have to take it out
    • #chattr -i /etc/passwd /etc/shadow /etc/group /etc/gshadow
  • Change the password for the [username] used above in step 2
    • #passwd [username]
  • Reboot and unmount the CD from the VM
  • Log in to the console using the [username] and password you reset in step above

2. VMWare Tools install

  • In the vSphere client, initiate the vmware tools installation by going to Guest-> 'Install/Upgrade VM Tools'. Don't select the automatic install.
  • Go to the console where you logged in with [username] and check that you have a folder /media/cdrom
  • In my case, there was no /media/cdrom. Because of an upgrade it was located in /partB/media/cdrom however the /etc/fstab was still pointing to /media/cdrom
  • I had to create folder /media/cdrom before mounting the CD, then just issue
    • # mount /media/cdrom
  • Copy the tools tar.gz from the CD to your home folder then untar and run the install script
    • # cp /media/cdrom/VMwareTools-9.0.1-913578.tar.gz ~
    • # cd ~
    • # tar zxvf VMwareTools-9.0.1-913578.tar.gz
    • # cd vmware-tools-distrib/
    • # ./vmware-install.pl
  • Take all the defaults the script offers. If you don't have errors, this should complete fine, in which case just reboot the server and enjoy the job well done.

3. BONUS. Errors!


During the install, just at the end it fails with an error message 
'No module mptscsi found for kernel x.x.x.x'

There's a lovely KB article on VMWare site at 

Even though the kernel versions don't match, following the instructions to comment out 'mptscsi' from /etc/modprobe.conf worked. In my case I had to comment out two entries, 

/etc/modprobe.conf :
...
#alias scsi_hostadapter1 mptscsi
...
#alias scsi_hostadapter5 mptscsih
...
~

Re-running the 'vmware-install.pl' script completed successfully this time, and I uncommented those entries before the final reboot.