Jon Owings | CrashLoopBackoff

Trunks – Dell Power Connect and Cisco

I recently needed to install a stack of Dell 6224 Power Connect switches. The core of the network was actually a Cisco 3560 (no G). While there are already posts existing from Scott Lowe about using the “General” mode to keep VLAN 1 untagged and also have other VLAN’s tagged. Dell’s General mode traditionally works just like a default dot1q trunk in Cisco. However when VLAN 1 is in use I secretly grumble because I know the fact that Dell’s general mode is finicky when interoperating with some devices. Most of the time general mode works like a charm but not on this day.

Dell’s “trunk” mode worked fine. Any tagged VLAN would pass fine to the Cisco. Except that pesky native VLAN 1. We HAD to have VLAN 1 passed down to the ESX servers. So after kicking around wondering what I did wrong I decided to just work around the problem. I tagged vlan 1 on the Dell port and changed the native vlan on that specific trunk on the Cisco to another vlan (not being used on the Dell). BAM it worked.

Note: Dell was running their newest firmware on that day – 3.2.0.9 (they have since released 3.2.0.10)
Note 2: I am all about auto-negotiation at Gigabit but still like 100Mbps switch links to be hard coded.

Cisco 3560 (no G).

interface FastEthernet 0/24
speed 100
duplex Full
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,10,11
swtichport trunk native vlan 8
switchport mode trunk

Dell 6224

interface Ethernet 1/g24
no negotiation
speed 100
duplex full
switchport mode trunk
switchport trunk allowed vlan add 1,10,11

Dynamic Cluster Pooling

Dynamic Cluster Pooling is an idea that Kevin Miller ( @captainstorage) and I came up with one day while we were just rapping out some ideas on the whiteboard. It is an incomplete idea, but may have the beginnings of something useful. The idea is that clusters can be dynamically sized depending on expected workload. Today a VMware Cluster is sized based on capacity estimates from something like VMware Capacity Planner. The problem is this method requires you apply a workload profile across all time periods or situations. What if only a couple days of the month require the full capacity of a cluster. Could those resources be used elsewhere the rest of the month?

Example Situation
Imagine a scenario with a Virtual Infrastructure with multiple clusters. Cluster “Gold” has 8 hosts. Cluster “Bronze” has 8 hosts. Gold is going to require additionally resources on the last day of the month to process reports from a database (or something like that). In order to provide additional resources to Gold we will take an ESX host away from the Bronze cluster. This allows us to deploy additional Virtual Machines to crunch through the process or allow less contention for the existing machines.

You don’t have to be a powercli guru to figure out how to vMotion all the machines off of a ESX host and place it in maintenance mode. Once the host is in maintenance mode it can be moved to the new cluster, removed from maintenance mode and VM’s can be redistributed by DRS.

Sample Code more to prove the concept:
#Connect to the vCenter Connect-VIServer [vcenterserver] #indentify the host, you should pass the host or hosts you want to vacate into a variable Get-Cluster Cluster-Bronze | get-vmhost


#Find the least loaded host(skipping for now)
#Vmotion the machines to somewhere else in that cluster

Get-VMHost lab1.domain.local | Get-VM| Move-VM -Destination [some other host in the bronze cluster]
#Move the host

Set-VMHost lab1.domain.local -State Maintenance

Move-VMHost lab1.domain.local -Destination Cluster-Gold

Set-VMHost lab1.domain.local -State Connected

#Rebalance VM's Get-DrsRecommendation -Cluster Cluster-Gold | Apply-DrsRecommendation

I was able to manually make this happen in our lab. Maybe if this sparks any interest someone that is good with “the code” can make this awesome.

Storage Caching vs Tiering Part 2

Recently I had the privilege of being a Tech Field Day Delegate. Tech Field Day is organized by Gestalt IT. If you want more detail on Tech Field Day visit right here. In interest of full disclosure the vendors we visit sponsor the event. The delegates are under no obligation to review good or bad the sponsoring companies.

After jumping in with a post last week on tierless caching I wanted to jump in with my thoughts on a second Tech Field Day vendor. Avere presented a very interesting and technical presentation. I appreciated being engaged on an engineering level and not a marketing pitch.

Avere tiers everything. It is essentially a scale out NAS solution (they called it a FXT Appliance) that can front end any existing NFS. Described to me by someone else as file acceleration. The Avere NAS stores data internally on a cluster of NAS units. The “paranoia meter” lets you set how often the mass storage device is updated. If you need more availability or speed you add Avere devices. If you need more disk space you add to your mass storage. In their benchmarking tests they basically used some drives connected to a CentOS machine running NFS front-ended by Avere’s NAS units. They were able to get the required IOPS at a fraction of the cost of NetApp or EMC.

The Avere Systems blog provides some good questions on Tiering.

The really good part of the presentation is how they write between the tiers. Everything is optimized for that particular type of media, SSD, SAS or SATA.
When I asked about NetApp’s statements about tiering (funny they were on the same day). Ron Bianchini responded, “that when you sell hammers, everything is a nail.” I believe him.

So how do we move past all the marketing speak to get down to the truth when it comes to Caching and Tiering. I am leaning toward thinking of any location where data lives for any period of time as a tier. I think a cache is a tier. Really fast cache for reads and writes is for sure a tier. Different kinds of disks are tiers. So I would say everyone has tiers. The value comes in when the storage vendor innovates and automates the movement and management of that data.

My questions/comments about Avere.

1. Slick technology. I would like to see it work in the enterprise over time. People might be scared because it is not one of the “big names”.
2. Having came from Spinnaker. Is the plan to go long term with Avere, or build something to be purchased by a big guy?
3. I would like to see how the methods used by the Avere FXT appliance can be applied to block storage. Plenty of slow inexpensive iSCSI products that would benefit from a device like this on the front end.

Storage Caching vs Tiering Part 1

The first place hosting the delegates was NetApp. I basically have worked with several different storage vendors but I must admit I have never experienced NetApp in any way before. Except for Storage vMotioning Virtual Machines from an old NetApp (I don’t even know the model) to a new SAN.

Among the 4 hours of slide shows I learned a ton. One great topic is Storage Caching vs Tiering. Some of the delegates have already blogged about the sessions here and here.

So I am going to give my super quick summary of Caching as I understood it from the NetApp session. Followed by a post about Tiering as I learned from one of our subsequent sessions from Avere.

1. Caching is superior to Tiering because Tiering requires too much management.
2. Caching outperforms tiering.
3. Tiering drives cost up.

The NetApp method is to use really quick Flash Memory to speed up the performance of the SAN. Their software attempts to predict what data will be read and keep that data available in the cache. This “front-ends” a giant pool of SATA drives. The cache cards provide the performance the the SATA drives provide a single large pool to manage. With a simplified management model and using just one type of big disk the cost is driven down.

My Take Away in Tierless-Caching

This is a solution that has a place and would work well for many situations. This is not the only solution. All in all the presentation was very good. The comparisons against tiering were really setup against a “straw-man”. A multi-device tiered solution requiring manual management off all the different storage tiers is of course a really hard solution. It could cost more to obtain and could be more expensive to manage. I asked about fully virtual automated tiering solutions. Solutions that manage your “tiers” as one big pool. These solutions would seem to solve the problem of managing tiers of disks, keeping the cost down. The question was somewhat deflected because these solutions will move data on a schedule. “How can I know when to move my data up to the top tier?” was the question posed by NetApp. Of course this is not exactly how a fully-automated tiering SAN works, but is a valid concern.

My Questions for the Smartguys:

1. How can the NetApp caching software choices be better/worse than software that makes tiering decisions from companies that have done this for several years?
2. If tiering is so bad, why does Compellent’s stock continue to rise in anticipation of an acquisition from someone big?
3. Would I really want to pay NetApp sized money to send my backups to a NetApp pool of SATA disks? Would I be better off with a more affordable SATA solution for Backup to Disk even if I have to spend slightly more time managing the device?

Fast Don’t Lie – Tech Field Day

Apologies to the new Adidas Basketball youtube campaign. I am going to steal their title for this post.

Time has flown by and it is now time to get going to Gestalt IT’s Tech Field Day. Thursday and Friday will be full of some pretty exciting companies. I have some familiarity with three of them: Solarwinds, NetApp and Intel. I am excited to get some in depth information from them though.

Then Aprius, Avere Systems, Actifio, and Asigra are companies I have never really heard anything about so it will be interesting to see what they do and see how it fits in to my perspective as a Virtualization dude.

For now I have one question on my list (I will come up with others), Is it Fast? Watch the videos, because when we talk about the cloud, Fast dont’ lie.

I’m Fast

I’m Fast 2

Fast Don’t Lie

Equallogic, VAAI and the Fear of Queues

Previously I posted on how using bigger VMFS volumes helps Equallogic reduce their scalability issues when it comes to total iSCSI connections. There was a comment about does this mean we can have a new best practice for VMFS size. I quickly said, “Yeah, make em big or go home.” I didn’t really say that but something like it. Since the commenter responded with a long response from Equallogic saying VAAI only fixes SCSI locks all the other issues with bigger datastores still remain. ALL the other issues being “Queue Depth.”

Here is my order of potential IO problems on with VMware on Equallogic:

Being spindle bound. You have an awesome virtualized array that will send IO to every disk in the pool or group. Unlike some others you can take advantage of a lot of spindles. Even then, depending on the types of disks some IO workloads are going to use up all your potential IO.
Solution(s): More spindles is always a good solution if you have unlimited budget. Not always practical. Put some planning into your deployment. Don’t just buy 17TB of SATA. Get some faster disk and break your Group into pools and separate the workloads into something better suited to the IO needs.
Connection Limits. The next problem you will run into if you are not having IO problems is the total iSCSI connections. In an attempt to get all of the IO you can from your array you have multiple vmk ports using MPIO. This multiplies the connections very quickly. When you reach the limit, connections drop and bad things happen.
Solution: The new 5.02 firmware increases the total maximum connections. Additionally, bigger datastores means less connections. Do the math.
Queue Depth. There are queues everywhere, the SAN ports have queues. Each LUN has a queue. The HBA has a queue. I would need to defer to a this article by Frank Denneman (a much smarter guy than myself.) That balanced storage design is best course of action.
Solution(s): Refer to problem 1. Properly designed storage is going to give you the best solution for any potential (even though unlikely) queue problems. In your great storage design, make room for monitoring. Equallogic gives you SANHQ USE IT!!! See how your front end queues are doing on all your ports. Use ESXTOP or RESXTOP to see how the queues look on the ESX host. Most of us will find that queues are not a problem when problem one is properly taken care of. If you still have a queuing problem then go ahead and make a new datastore. I would also request Equallogic (and others) release a Path Selection Policy plugin that uses a Least Queue Depth algorithm (or something smarter). That would help a lot.

So I will repeat my earlier statement that VAAI allows you to make bigger datastores and house more VM’s per store. I will add a caveat, if you have a particular application that needs a high IO workload, give it a datastore.

Gestalt IT – Tech Field Day

I am honored to be included in the upcoming Gestalt IT Field Day. Looks like a great group from the community will be in attendanc. I am looking forward to the collection of presenters. With how busy I have been delivering solutions lately it will be really good to dedicate some time to learning what is new and exciting. I plan to take good notes and share my thoughts here on the blog. For more information on the Field Day check it out right here: http://bit.ly/ITTFD4

How VAAI Helps Equallogic

I previously posted about the limits on iSCSI connections when using Equallogic arrays and MPIO. If you have lots of Datastores and lots of ESX hosts with multiple paths the numbers of connections multiplies pretty quickly. Now with VAAI support in the Equallogic 5.02 firmware (hopefully no recalls this time), the number of Virtual Machines per Datastore is not important. Among other improvements, the entire VMFS volume will not lock. As I understand VAAI the only the blocks (or files maybe?) are locked when exclusive access is needed.

Lets look at the improvement when using fewer larger EQ volumes:
Old way (with 500GB Datastores for example):
8Hosts x 2(vmkernel connections) x 10(Datastores) = 160 connections (already too many for the smaller arrays, PS 4000).

VAAI (with 1.9 TB* Datastores)
8 Hosts x 2(vmkernel connections) x 3(Datastores) = 48 connections

The scalability for Equallogic is much better with VAAI when trying to stay under the connection limits.

*Limit for VMFS is 2TB minus 512B so 1.9TB works out nicely.

Update Manager Problem after 4.1 Upgrade

A quick note to hopefully publicize a problem I had which I see is discussed in the VMware Community Forums already.

After building a new vCenter Server and Upgrading the vSphere 4.0 databases for vCenter and Update Manager. I noticed I could not scan hosts that were upgraded to 4.1. To be fair, by upgrading I mean rebuilt with a fresh install but with the exact same name and IP addresses. Seems that the process I took to upgrade has some kind of weird effect in the Update Manager Database. The scans fail almost immediately. I searched around the internet and found a couple of posts on the VMware Forums about the subject. One person was able to fix the problem by removing Update Manager and when reinstalling selecting the option to install a new database. I figured I didn’t have anything important in my UM database so I gave it a try and it worked like a champ.

Right now there is not any new patches for vSphere 4.1 but I have some Extension packages that need to be installed (Xsigo HCA Drivers). I wanted to note that I like the ability to upload extensions directly into Update Manager. This is a much cleaner process than loading the patches via the vMA for tracking and change control purposes.

ESXi 4.1 pNics Hard Coded to 1000 Full

I have recently made the transition to using ESXi for all customer installs. One thing I noticed was after installing with a couple different types of media (ISO and PXE install) the servers come up with the NIC’s hard coded to 1000 Full. I have always made it a practice to keep Gigabit Ethernet at auto-configure. I was told by a wise Cisco engineer many years ago that GigE and Auto/Auto is the way to go. You can also check the Internet for articles and best practices around using auto-configure with gigabit ethernet. Even the VMware “Health Analyzer” recommends using auto. So it is perplexing to me that ESXi 4.1 would start to default to hard set. Is it just me? Has anyone else noticed this behavior?

The only reason I make an issue is I was ready to call VMware support a couple weeks ago because nothing in a DRS/HA cluster just built with 4.1 would work. One vMotion would be successful, the next would fail. Editing settings on the hosts would fail miserably when done from the vSphere Client connected to vCenter. After changing all the pNics to auto everything worked just fine (matching the switches).

Hit me up in the comments or on twitter if you have noticed this.