Crucial Paradigm

If you are receiving the following error in XenServer/XenCenter, there is a quick fix!

4/05/2011 4:05:40 PM Error: Migrating VM ’36′ from ‘blXXX’ to ‘blXXX’ – Internal error: INTERNAL_ERROR: [ Failure("The VDI 996b046b-1960-4315-bad7-48830254d4c3 is already attached in RO mode; it can't be attached in RW mode!") ]

Or

5/05/2011 7:09:08 AM Error: Migrating VM ‘n3 – NTP’ from ‘blXXX’ to ‘blXXX’ – Internal error: INTERNAL_ERROR: [ Xb.Invalid ]

This can be fixed by running:

xe-toolstart-restart

During our testing we have had this error pop up a few times while trying to start VMs after we have  unplugged a blade without correctly shutting it down (simulating a failure scenario).

NOTE: This is the error that is shown in XenCenter while trying to start the VM.

4/05/2011 3:33:15 PM Error: Starting VM ’36′ – Internal error: another frontend device is already connected to this domain (frontend (domid=0 | kind=vbd | devid=51712); backend (domid=0 | kind=vbd | devid=51712))

It usually fixes itself if you wait a few hours or days, and attempt to start the VM again later.  This however is not an ideal situation, as it means your VM is down until the error disappears.

I fixed this in one instance (have not tested again, as I’m waiting for the error to pop up again) by doing the following.  This is for a VM with the name “36″:

[root@blXXX lib]# xe vbd-list vm-name-label=36
uuid ( RO)             : f84cfbef-1054-f8e8-5d95-43587b63060a
vm-uuid ( RO): b7fa54cf-4cc7-ec1e-0310-27b629a83d30
vm-name-label ( RO): 36
vdi-uuid ( RO): <not in database>
empty ( RO): true
device ( RO): xvdd

uuid ( RO)             : b4ff5293-46f6-6944-2b51-f5a0ea0aa92c
vm-uuid ( RO): b7fa54cf-4cc7-ec1e-0310-27b629a83d30
vm-name-label ( RO): 36
vdi-uuid ( RO): 996b046b-1960-4315-bad7-48830254d4c3
empty ( RO): false
device ( RO): xvda

[root@blXXX lib]#

As you can see the first VBD entry for this VM has:

vdi-uuid ( RO): <not in database>

By removing the VDB entry it seems to fix the problem:

[root@blXXX lib]# xe vbd-destroy uuid=f84cfbef-1054-f8e8-5d95-43587b63060a

I’ve seen some places have recommended using xenstore-ls and xenstore-rm the above method seems to work better – as the xenstore-rm command did not seem to address the issue.

UPDATE:

Upon further testing we found that if there are multiple entries with “vdi-uuid ( RO): <not in database>” in the vdb list for the VM – remove all of these.

During some testing of our up and coming highly available solution, we noticed an issue when a XenServer node in a pool had been hard rebooted and brought back online – it would have all its NICs missing. It would not appear up in the pool, and would appear down.

It was possible to ping other servers in the pool from the node, however for some reason it seemed to have issues with its network config.  ifconfig would show all interfaces as up, but the XenServer console showed all as missing.

The quick fix for this was running the following command:

xe-toolstack-restart

Update:

The above fix seemed to only temporarily fix the issue, I used the fix outlined in my blog post about the state.db corruption.

XenServer does not have any obvious stats on the number of VMs in a pool, or on a particular server.  These details can however can be found via the commend line:

Listing number of VMs in a pool (running, halted, or suspended):

xe vm-list  | grep -c running

Listing the number of VMs on a particular host/server (running, halted, or suspended):

xe vm-list resident-on=[host uuid] | grep -c running

You can replace “running” with halted, suspended, or if you want to see VMs regardless of whether they are on or not “power-state”.

If you are one of the many people getting one of the following errors this article will help you!

If your alarms list keeps showing the error message:

“Warning: Alarms may not be available.”

Or you have trouble installing the latest updates to SAN/iQ 9 with it giving an error message such as the following:

“Management group ‘xxxx’ has the following issues: There was a problem getting the alarms list for management group ‘xxxx’.. Canceling all further installations.”

After a fairly long list of troubleshooting, HP provided us with a fix (10105-00) for this issue.  The fix specically states:

“Patch 10105-00 fixes a behavior where the CMC displays Warning:Alarms may not be available that prevents online upgrade/patch install from working as the CMC cannot get the alarms list. This also causes the Export Support Bundle to not work and may prevent CLI commands from working as well.”

I’m not aware of this patch being publically available from HP yet, so if you have these symptons I suggest contacting them and going through the steps to obtain the patch.

Once you receive the patch, you can use the following steps to install the update (we came across a few issues while trying to patch our systems, that didn’t seem to be documented by HP):

1. Shutdown the CMC
2. Open \Users\[user]\.storage_system\preferences.txt
3. At the top of the file, add the following:

CmcSystemPreference.supportMode=true
CmcUpgradePreference.useOldUpgrades=true
CmcUpgradePreference.userUpgrade=true

4. Start CMC, under Configuration Summary there will now be a “Support  Upgrades” tab.
5. Locate the 10105-00.20110406.patch from HP, you will probably need to contact HP support for this.  And they may go through a bunch of troubleshooting with you before they provide you with the download.
6. Hit browse, and select the file 10105-00.20110406.patch
7. Select the storage node from the list that you wish to upgrade, you will need to them upgrade one at a time. I installed the patch on all storage systems, and the Failover Manager (FOM).

If you have ever needed to enable Support Upgrades in the CMC, to manually install patches as you used to be able to with SAN/iQ 8.5 and below – then this article will benefit you.

The following steps will enable Support Upgrades in the CMC, and will allow you to manually patch the storage systems with individual patches:

1. Shutdown the CMC

2. Open \Users\[user]\.storage_system\preferences.txt

3. At the top of the file, add the following:

CmcSystemPreference.supportMode=true
CmcUpgradePreference.useOldUpgrades=true
CmcUpgradePreference.userUpgrade=true

4. Start CMC, under Configuration Summary there will now be a “Support Upgrades” tab.

It is important to add the lines to the top in step 3, otherwise the “Support Uprades” tab will not be available.

If you have had trouble getting access to the Distributed vSwitch Controller (DVSC) in XenServer, then you are not alone!

While doing some testing on XenServer’s vSwitch and following XenServers documentation we noticed it was not possible to using the “Switch to Graphical Mode” that was referenced in the documentation.  While the “Switch to Graphical Mode” was there in XenCenter, it was grayed out. Rebooting, or re-importing the DVSC appliance did not help. (more…)

The following article is aimed at providing some quick details on IPv6, which can be used as an IPv6 Subnet Cheat Sheet/Reference: (more…)

Today APNIC announced it had officially depleted its general reserves for IPv4 space, leaving only a single /8 remaining in its reserves.

http://www.apnic.net/community/ipv4-exhaustion/graphical-information

What does this mean?

This basically means that APNIC (Asia and Pacific region) will no longer be issuing out IPv4 space as it used to, it will now only issue a maximum of a /22 (1024 IPs) to companies.  Content providers and eye ball networks alike will now be forced to deploy IPv6 as their IPv4 reserves deplete.  Companies were only allowed to request a maximum of 1 year in advance for IP address allocations previous to this event, so most companies will only have a maximum 1 year supply of IPv4 space remaining.

Over the following months to year it is likely we will see companies deplete their remaining IPv4 space, and only be able offer IPv6  support to their customers (whether an ISP or a hosting/content provider).

While other RIRs (Regional Internet Registry) such as ARIN may have IPv4 space left, this will not last for very long. (more…)

Recently Crucial has been gearing up for IPv6 inclusion in our services, it’s been my task to deliver this in the best way possible.

Some of the features that IPv6 provide which weren’t included in IPv4 are:

1. More efficient address space allocation
2. End-to-end addressing; no NAT anymore!
3. Fragmentation only by the source host
4. Routers do not calculate header checksum (speedup!)
5. Multicasting instead of broadcasting
6. Built-in security mechanisms
7. Single control protocol (ICMPv6)
8. Modular headers structure

But most importantly AUTO CONFIGURATION!

IPv6 allows for auto-configuration using the EUI-64 specification and SLAAC discovery.

SLAAC is a stateless configuration, though it generates network traffic it doesn’t need a server or client configuration nor does it communicate with a centralized administrator.

(more…)