Crucial Paradigm

For those of you have used vanilla Xen will know about the xm command and what it does.  While I’ve found in a few places people are saying the xm command has been replaced with xe in XenServer – which is sort of true.  The xe should used whenever you can, as it is ensuring all data in the DB is kept in tact, etc.

If you really want to get access to xm (which I do not recommend unless you really know what you are doing), you can use xl on a XenServer machine.

This should only be use as a visibility tool, and should not be used to make any modifications!

If you are receiving the following error in XenServer/XenCenter, there is a quick fix!

4/05/2011 4:05:40 PM Error: Migrating VM ’36′ from ‘blXXX’ to ‘blXXX’ – Internal error: INTERNAL_ERROR: [ Failure("The VDI 996b046b-1960-4315-bad7-48830254d4c3 is already attached in RO mode; it can't be attached in RW mode!") ]

Or

5/05/2011 7:09:08 AM Error: Migrating VM ‘n3 – NTP’ from ‘blXXX’ to ‘blXXX’ – Internal error: INTERNAL_ERROR: [ Xb.Invalid ]

This can be fixed by running:

xe-toolstart-restart

During our testing we have had this error pop up a few times while trying to start VMs after we have  unplugged a blade without correctly shutting it down (simulating a failure scenario).

NOTE: This is the error that is shown in XenCenter while trying to start the VM.

4/05/2011 3:33:15 PM Error: Starting VM ’36′ – Internal error: another frontend device is already connected to this domain (frontend (domid=0 | kind=vbd | devid=51712); backend (domid=0 | kind=vbd | devid=51712))

It usually fixes itself if you wait a few hours or days, and attempt to start the VM again later.  This however is not an ideal situation, as it means your VM is down until the error disappears.

I fixed this in one instance (have not tested again, as I’m waiting for the error to pop up again) by doing the following.  This is for a VM with the name “36″:

[root@blXXX lib]# xe vbd-list vm-name-label=36
uuid ( RO)             : f84cfbef-1054-f8e8-5d95-43587b63060a
vm-uuid ( RO): b7fa54cf-4cc7-ec1e-0310-27b629a83d30
vm-name-label ( RO): 36
vdi-uuid ( RO): <not in database>
empty ( RO): true
device ( RO): xvdd

uuid ( RO)             : b4ff5293-46f6-6944-2b51-f5a0ea0aa92c
vm-uuid ( RO): b7fa54cf-4cc7-ec1e-0310-27b629a83d30
vm-name-label ( RO): 36
vdi-uuid ( RO): 996b046b-1960-4315-bad7-48830254d4c3
empty ( RO): false
device ( RO): xvda

[root@blXXX lib]#

As you can see the first VBD entry for this VM has:

vdi-uuid ( RO): <not in database>

By removing the VDB entry it seems to fix the problem:

[root@blXXX lib]# xe vbd-destroy uuid=f84cfbef-1054-f8e8-5d95-43587b63060a

I’ve seen some places have recommended using xenstore-ls and xenstore-rm the above method seems to work better – as the xenstore-rm command did not seem to address the issue.

UPDATE:

Upon further testing we found that if there are multiple entries with “vdi-uuid ( RO): <not in database>” in the vdb list for the VM – remove all of these.

During some testing of our up and coming highly available solution, we noticed an issue when a XenServer node in a pool had been hard rebooted and brought back online – it would have all its NICs missing. It would not appear up in the pool, and would appear down.

It was possible to ping other servers in the pool from the node, however for some reason it seemed to have issues with its network config.  ifconfig would show all interfaces as up, but the XenServer console showed all as missing.

The quick fix for this was running the following command:

xe-toolstack-restart

Update:

The above fix seemed to only temporarily fix the issue, I used the fix outlined in my blog post about the state.db corruption.

XenServer does not have any obvious stats on the number of VMs in a pool, or on a particular server.  These details can however can be found via the commend line:

Listing number of VMs in a pool (running, halted, or suspended):

xe vm-list  | grep -c running

Listing the number of VMs on a particular host/server (running, halted, or suspended):

xe vm-list resident-on=[host uuid] | grep -c running

You can replace “running” with halted, suspended, or if you want to see VMs regardless of whether they are on or not “power-state”.

If you are one of the many people getting one of the following errors this article will help you!

If your alarms list keeps showing the error message:

“Warning: Alarms may not be available.”

Or you have trouble installing the latest updates to SAN/iQ 9 with it giving an error message such as the following:

“Management group ‘xxxx’ has the following issues: There was a problem getting the alarms list for management group ‘xxxx’.. Canceling all further installations.”

After a fairly long list of troubleshooting, HP provided us with a fix (10105-00) for this issue.  The fix specically states:

“Patch 10105-00 fixes a behavior where the CMC displays Warning:Alarms may not be available that prevents online upgrade/patch install from working as the CMC cannot get the alarms list. This also causes the Export Support Bundle to not work and may prevent CLI commands from working as well.”

I’m not aware of this patch being publically available from HP yet, so if you have these symptons I suggest contacting them and going through the steps to obtain the patch.

Once you receive the patch, you can use the following steps to install the update (we came across a few issues while trying to patch our systems, that didn’t seem to be documented by HP):

1. Shutdown the CMC
2. Open \Users\[user]\.storage_system\preferences.txt
3. At the top of the file, add the following:

CmcSystemPreference.supportMode=true
CmcUpgradePreference.useOldUpgrades=true
CmcUpgradePreference.userUpgrade=true

4. Start CMC, under Configuration Summary there will now be a “Support  Upgrades” tab.
5. Locate the 10105-00.20110406.patch from HP, you will probably need to contact HP support for this.  And they may go through a bunch of troubleshooting with you before they provide you with the download.
6. Hit browse, and select the file 10105-00.20110406.patch
7. Select the storage node from the list that you wish to upgrade, you will need to them upgrade one at a time. I installed the patch on all storage systems, and the Failover Manager (FOM).

If you have ever needed to enable Support Upgrades in the CMC, to manually install patches as you used to be able to with SAN/iQ 8.5 and below – then this article will benefit you.

The following steps will enable Support Upgrades in the CMC, and will allow you to manually patch the storage systems with individual patches:

1. Shutdown the CMC

2. Open \Users\[user]\.storage_system\preferences.txt

3. At the top of the file, add the following:

CmcSystemPreference.supportMode=true
CmcUpgradePreference.useOldUpgrades=true
CmcUpgradePreference.userUpgrade=true

4. Start CMC, under Configuration Summary there will now be a “Support Upgrades” tab.

It is important to add the lines to the top in step 3, otherwise the “Support Uprades” tab will not be available.

If you have had trouble getting access to the Distributed vSwitch Controller (DVSC) in XenServer, then you are not alone!

While doing some testing on XenServer’s vSwitch and following XenServers documentation we noticed it was not possible to using the “Switch to Graphical Mode” that was referenced in the documentation.  While the “Switch to Graphical Mode” was there in XenCenter, it was grayed out. Rebooting, or re-importing the DVSC appliance did not help. (more…)

If you have ever had problems with time on a cluster, and had trouble getting the licenses in sync in Citrix XenServer 5.6 FP1, you can run the following commend to disable HA:

xe host-emergency-ha-disable force=true

If you ever find your server won’t exist maintenance mode after a power failure, or even just normally you may have a corrupt state.db file.

The error message you may get while trying to make a change on the node is:

Cannot forward messages because the host cannot be contacted.  The host may be switched off or there may be network connectivity problems.

If this is in fact a state.db corruption in a pool, you can try the following steps to fix the issue:

1. cd /var/xapi
2. ls -al  -> check the results of this, usually the state.db will have an old time stamp – the time at which is became corrupted.
3. mv state.db state.db_bak
4. service xapi restart

If this does not fix the issue, I’ve heard of users opening state.db with an xml editor (its just an xml file) and fixing the errors in the xml file. Some other users have mentioned that the cause of the issue could be that the config in the node that is down in maintenance mode could have the incorrect config in /etc/xensource/pool.conf. The format should be “slave:[masterip]“.