Crucial Paradigm

One thing you notice online when working in virtualization is the lack of resource explaining the engines / drive trains / gear boxes what go into making system virtualization possible.

This actually goes for a lot of things, there will always be more howto’s on technical webserver stuff for linux than for windows… etc etc.

Today I am going to explain a bit about the relationship between processes and objects that come together to make your virtual machine work.

(more…)

For those of you have used vanilla Xen will know about the xm command and what it does.  While I’ve found in a few places people are saying the xm command has been replaced with xe in XenServer – which is sort of true.  The xe should used whenever you can, as it is ensuring all data in the DB is kept in tact, etc.

If you really want to get access to xm (which I do not recommend unless you really know what you are doing), you can use xl on a XenServer machine.

This should only be use as a visibility tool, and should not be used to make any modifications!

If you are receiving the following error in XenServer/XenCenter, there is a quick fix!

4/05/2011 4:05:40 PM Error: Migrating VM ’36′ from ‘blXXX’ to ‘blXXX’ – Internal error: INTERNAL_ERROR: [ Failure("The VDI 996b046b-1960-4315-bad7-48830254d4c3 is already attached in RO mode; it can't be attached in RW mode!") ]

Or

5/05/2011 7:09:08 AM Error: Migrating VM ‘n3 – NTP’ from ‘blXXX’ to ‘blXXX’ – Internal error: INTERNAL_ERROR: [ Xb.Invalid ]

This can be fixed by running:

xe-toolstart-restart

During our testing we have had this error pop up a few times while trying to start VMs after we have  unplugged a blade without correctly shutting it down (simulating a failure scenario).

NOTE: This is the error that is shown in XenCenter while trying to start the VM.

4/05/2011 3:33:15 PM Error: Starting VM ’36′ – Internal error: another frontend device is already connected to this domain (frontend (domid=0 | kind=vbd | devid=51712); backend (domid=0 | kind=vbd | devid=51712))

It usually fixes itself if you wait a few hours or days, and attempt to start the VM again later.  This however is not an ideal situation, as it means your VM is down until the error disappears.

I fixed this in one instance (have not tested again, as I’m waiting for the error to pop up again) by doing the following.  This is for a VM with the name “36″:

[root@blXXX lib]# xe vbd-list vm-name-label=36
uuid ( RO)             : f84cfbef-1054-f8e8-5d95-43587b63060a
vm-uuid ( RO): b7fa54cf-4cc7-ec1e-0310-27b629a83d30
vm-name-label ( RO): 36
vdi-uuid ( RO): <not in database>
empty ( RO): true
device ( RO): xvdd

uuid ( RO)             : b4ff5293-46f6-6944-2b51-f5a0ea0aa92c
vm-uuid ( RO): b7fa54cf-4cc7-ec1e-0310-27b629a83d30
vm-name-label ( RO): 36
vdi-uuid ( RO): 996b046b-1960-4315-bad7-48830254d4c3
empty ( RO): false
device ( RO): xvda

[root@blXXX lib]#

As you can see the first VBD entry for this VM has:

vdi-uuid ( RO): <not in database>

By removing the VDB entry it seems to fix the problem:

[root@blXXX lib]# xe vbd-destroy uuid=f84cfbef-1054-f8e8-5d95-43587b63060a

I’ve seen some places have recommended using xenstore-ls and xenstore-rm the above method seems to work better – as the xenstore-rm command did not seem to address the issue.

UPDATE:

Upon further testing we found that if there are multiple entries with “vdi-uuid ( RO): <not in database>” in the vdb list for the VM – remove all of these.

During some testing of our up and coming highly available solution, we noticed an issue when a XenServer node in a pool had been hard rebooted and brought back online – it would have all its NICs missing. It would not appear up in the pool, and would appear down.

It was possible to ping other servers in the pool from the node, however for some reason it seemed to have issues with its network config.  ifconfig would show all interfaces as up, but the XenServer console showed all as missing.

The quick fix for this was running the following command:

xe-toolstack-restart

Update:

The above fix seemed to only temporarily fix the issue, I used the fix outlined in my blog post about the state.db corruption.

XenServer does not have any obvious stats on the number of VMs in a pool, or on a particular server.  These details can however can be found via the commend line:

Listing number of VMs in a pool (running, halted, or suspended):

xe vm-list  | grep -c running

Listing the number of VMs on a particular host/server (running, halted, or suspended):

xe vm-list resident-on=[host uuid] | grep -c running

You can replace “running” with halted, suspended, or if you want to see VMs regardless of whether they are on or not “power-state”.

If you have had trouble getting access to the Distributed vSwitch Controller (DVSC) in XenServer, then you are not alone!

While doing some testing on XenServer’s vSwitch and following XenServers documentation we noticed it was not possible to using the “Switch to Graphical Mode” that was referenced in the documentation.  While the “Switch to Graphical Mode” was there in XenCenter, it was grayed out. Rebooting, or re-importing the DVSC appliance did not help. (more…)

If you have ever had problems with time on a cluster, and had trouble getting the licenses in sync in Citrix XenServer 5.6 FP1, you can run the following commend to disable HA:

xe host-emergency-ha-disable force=true

If you ever find your server won’t exist maintenance mode after a power failure, or even just normally you may have a corrupt state.db file.

The error message you may get while trying to make a change on the node is:

Cannot forward messages because the host cannot be contacted.  The host may be switched off or there may be network connectivity problems.

If this is in fact a state.db corruption in a pool, you can try the following steps to fix the issue:

1. cd /var/xapi
2. ls -al  -> check the results of this, usually the state.db will have an old time stamp – the time at which is became corrupted.
3. mv state.db state.db_bak
4. service xapi restart

If this does not fix the issue, I’ve heard of users opening state.db with an xml editor (its just an xml file) and fixing the errors in the xml file. Some other users have mentioned that the cause of the issue could be that the config in the node that is down in maintenance mode could have the incorrect config in /etc/xensource/pool.conf. The format should be “slave:[masterip]“.

You can use the following commands to change the Citrix XenServer pool’s master server:

First you need to disable high availibility:

xe pool-ha-disable

Then you need to find out which host you want to change the master to:

xe host-list

Then change the master:

xe pool-designate-new-master host-uuid=[uuid of new master host]

Turn HA back on:

xe pool-ha-enable