Uncategorized | CrashLoopBackoff

vExpert 2010 – a High Five from VMware

Last Friday night I received an email from John Troyer at VMware inviting me to the vExpert 2010 program. It is an honor to be included with such a great list of people. I never know how to respond to be complimented. That is why I like high fives. It is like saying good job, but the only expected response is to high five back. No awkward thank you, or speeches needed. Just don’t leave me hangin’ and everything is understood.

I am extremely excited to get the VMware vExpert 2010, but since this is the internets, no one is here to high five.

So I will just say, “Thank you.”

Yo Gabba Gabba High Five

Really needed a reason to put this song in a post…

Update Manager and Isolated ESX Service Console Networks

Sometimes you may be required to run your vCenter server that has two network interfaces. One in the network it can be reached for remote desktop access and the other where it has access to the ESX servers in order to manage the VMware hosts. This is sort of a hybrid model of an isolated management network. Where only one host can reach the management ports. One thing to think about in this model is Update Manager by default will not like it. Everything may look ok, but trying to scan a host will fail. Luckily though it is an easy fix.

In the update manager configuration tab change the ip in the picture to the IP accessible by the ESX servers. Then remember to restart the Update Manager services. Now go back and run the ESX scan/stage/remediation.

B.Y.O.P – The Alternative Vblock

In college I often would be invited to a get together that could often include the letters BYOB, Bring Your Own Beer. Sometimes a cookout would be BYOM, Bring Your Own Meat (or meat alternative for the vegetarians). So today I want to leverage this to push my new acronym B.Y.O.P. Bring Your Own Pod. Lately I have been seeing people talk about Vblocks. If I can venture a succinct definition a Vblock is a pre-configured set of Cisco, EMC and VMware products tested by super smart people, approved by these people to work together, then supported by these organizations as a single entity. Your reseller/solutions provider really should already be doing this very thing for you. You may choose to buy just the network piece, or the hypervisor but your partner should be able to verify a solution to work from end to end and provide unified support.

So You can’t call it BYOPCVCEP

Why not Vblock? This might get me blacklisted by the Elders of the vDiva council, but VCE doesn’t exist to make your life in the datacenter easier, they exist to sell you more VMware, Cisco and EMC. Vblock for sure simplifies your buying experience. I believe they are all great products and may very well do just what you need. Without competition though the only winner is VCE. Do not by forced into a box by the giant vendors. Find someone that can help determine your end goal, provide you vendor neutral analysis of the building blocks needed to achieve your end goal. Then provide the correct vendors and unified support to Build Your Own Pod.

So What is the Alternative Vblock

Originally I was going to draw up a sweet solution of 3par, Xsigo and Dell R610’s and say, “Hey everyone! This is some cool stuff. Try to quiet the overwhelmingly loud voice calling from VCE and give this Alternative Vblock a try.” As I thought more and more about it I think doing that is contrary to my main point. I would like more to provide the discussion points or some possible products among others that can be used to Build Your Own Pod. I am a firm believer in getting what is right for your datacenter needs. So here is a few links to help begin the discussion.

Xsigo and Pod – Jon Toor
3par and iBlocks – Marc Farley

You might be a vDiva if…

I am avoiding a post where I have to think really hard about a topic. That makes me procrastinate and come up with even crazier ideas. I am writing this one down now. Most of these apply to me so if you are offended by any of them you are probably a vDiva.

You might be a vDiva if…

… you roll your eyes when someone talks about installing a PHYSICAL server.

… you are on twitter to see how many people you can get to look at your blog, but you never stoop so low to interact with the common folk.

… you are surprised when the guy at the table at the VMUG doesn’t know who you are.

… you constantly check your Google Analytics account to see how many views you have. (guilty)

… you refer to yourself as @… (your twitter account)

… you hunt down @jtroyer if you latest post takes too long to get on the v12n board.

… your require a signed rider agreement with your speaking topic for VMworld, saying you need 800 green M&M’s, a copy of Lord of the Rings in your hotel room, and direct phone access to Steve Herrod’s iPhone.

I probably ticked a bunch of people off. I am just having fun. Have a great day! Go ahead and add your own in the comments.

VMworld 2010 voting – Check out this VDI Session

I try to not “self promote” too much. A co-worker and I submitted a topic in the Desktop Virtualization track and I am giving in and spreading the word:

Thinning Down to Scale Out

Abstract:

Desktop Virtualization provides the ability run hundreds even thousands of desktops. Each small performance enhancement can make a difference when multiplied across an entire enterprise. This presentation will demonstrate the steps necessary to thin down your guest desktop image in order to provide overall better user experience.

Speaker: Kevin Miller, VeriStor Systems

We are not popular so this doesn’t qualify for the popularity contest that many other sessions have, plus my name doesn’t even show on the Session. One thing I will say is this will be a Practical technical session. You should leave saying, “here is some stuff I can do to improve my VDI setup.”

So go vote and if we get to present the session I will give you a high five. Vote now because voting ends May 26.

Operational Readiness

One thing I am thinking about due to the VCDX application is operational readiness. What does it mean to pronounce this project or solution good-to-go? In my world it would be to test that each feature does exactly what it should be doing. Most commonly this will be failover testing, but could reach into any feature or be as big as DR plan that involves much more than the technical parts doing what they should. Some things I think need to be checked:

Resources

Are the CPU, Memory, Network and Storage doing what they should be? Some load generating programs like IOmeter can be fine to test network and storage performance. CPU busy programs can verify Resource Pools and DRS are behaving the way they should.

Failover

You have redundant links right? Start pulling cables. Do the links failover for Virtual Machines, Service Console, and iSCSI? How about the redundancy of the physical network, even more cable to pull! Also test that the storage controllers failover correctly. Also, I will make sure HA does what it is supposed to, instantly power off a host and make sure some test virtual machines start up somewhere else on the cluster.

Virtual Center Operations

Deploy new virtual machines, host and storage VMotion, deploy from a template, and clone a vm are all things we need to make sure are working. If this is a big enough deployment make sure the customer can use the deployment appliance if you are making use of one. Make sure the alarms send traps and emails too.

Storage Operations

Create new luns, test replication, test storage alarms and make sure the customer understands thin provisioning if it is in use. Make sure you are getting IO as designed from the Storage side. Making use of the SAN tools to be sure the storage is doing what it should.

Applications

You can verify that each application is working as intended within the virtual environment.

There must be something I am missing but the point is trying to test out everything so you can tell that this virtualization solution is ready to be used.

Firewalls are not Routers

I am no network super-genius but I do enough with networking to be able to get by. Two common mistakes I find many times are flat networks and firewalls as the default gateway. A flat network is when generally switches are connected to one another without any configuration. There is one broadcast domain which means every packet that the switch does not have an entry in the MAC address table, is sent out all the ports but the originating port. This repeats across all of the switches until the layer 2 destination is found. Now, this means your expensive Cisco switches are barely better than hubs. You don’t have collisions like you would on a hub and once the switch learns where the MAC address lives it keeps that information for a certain amount of time. Then again in this network setup the logs are most likely not monitored so if there where collisions and other errors it goes unnoticed.
That is not the title of this post though. Although related to a flat network using the firewall is a different issue. Using the firewall as the router works just fine when you have a flat network. You may never notice the problem in a small network, but as your network grew you noticed how problems can come up when there is just one big network. So someone smart said use vlans to segment the network, create smaller broadcast domains. Then when you try to fix or change the flat network with subnets and vlans can you find out the new vlans can not reach the rest of the original network.

The current flat network with switches and the firewall used as the default gateway or router.

The problem comes when you add subnets that are different than the interface ip of the firewall. Firewalls in general have issue with redirecting traffic bound for other networks back out of the same interface. So in the picture above traffic from vlan 1 that is using the firewall as the default gateway trying to reach the subnet on vlan 10. Since the host on vlan 1 does not know where that network lives it sends the traffic to the default gateway. Even if you added a static route to the firewall the traffic will often fail. That is because firewalls are not meant to route but rather send traffic between trusted and untrusted networks and vice-versa. So the question becomes how do you actually fix your flat network that has the firewall as the router. There is of course more complicated solutions to provide high availability using VRRP or HSRP.
First get a real layer 3 device. That is a router or a switch capable of routing between multiple vlans. The good news is many of your newer switches are capable of layer 3, it is included in many Dell and HP switches, it may still be an add-on with Cisco. I haven’t used a new switch in the last year that did not have layer 3.
Next important step is use the layer 3 device (switch or router) to route everything. Set a default route in the layer 3 device to send only outbound traffic to the firewall and bam everything works. Why is this so hard. Many times there is hundreds of servers and desktops already configured to use the firewall as their router. We will do a lot of work to avoid having to do a bunch of manual work.

Now you are using a router to route and the firewall to block bad things and maybe even do NAT. (note: If you are doing NAT be sure to add your new VLANs to your NAT rules so the new networks can reach the outside of your firewall.)

Using Network Load Balancing with View

If you have a smaller View deployment but still want to have redundant connection servers look no further than Microsoft NLB. Solve this problem without the need for an expensive hardware loadbalancer. Will it have all of the bells and whistles? No. If you have less than a 1000 users you probably would not see the benefit of the advanced features in a hardware load balancer. Make sure to read the whitepaper from VMware about NLB in Virtual Machines.

I am making the assumption you are like me and want everything to be as virtual as possible. So the View Connection Manager servers will be VM’s

Setup the primary and replica View Servers

I won’t go over installing View. Just be sure to setup the initial manager server. Then go ahead and setup the replica VM.

Configure NLB

Go the the Administrative tools and open the Network Load Balancing Manager. Right click the top node in the tree menu on the left and select New Cluster.
Set the IP and other information you will used for the Load Balanced cluster. This is a new IP not used by your View Manager servers.
In the VMware document referenced above VMware recommends setting the Cluster operation mode to Multicast.
Click Next then next again. When asked to configure port rules I leave it on the default and click next. You can chose limit this to certain ports.

Click Next again and enter localhost in the wizard to configure the local interfaces for NLB. Click next and make sure to note the priority. When setting up the replica server this number needs to be different. Finally click finish and wait for the configuration to finish. You should now be able to ping your new cluster IP address.

Setup the Replica Server in the Load Balancer

Righ Click the node in the tree menu for the NLB Cluster you just created and select Add new host to cluster. Enter the IP for the Replica Server and click connect. Select the interface that will be used for the Load Balancing and click next. Make sure the Priority is unique from the first server. If it gives you any grief after this point close and re-open the Network Load balancing Manager. The working cluster should look like this:

Test the Failover

Start a continual ping to the cluster IP. Now use the vSphere Client to disconnect the network from one of the servers. Watch the pings continue to come back.

Finally, create a DNS A record (something like desktop.yourdomain.com) and point it to the cluster IP. You now have some decent failover in case of a VM failure and even a host failure (suggestion would be to use seperate hosts for the VM’s).

Note – You may need to add static ARP entries into your switching depending on your network topology. Be sure to test this fully and consult your network manufacturer’s documention for help with static ARP.

Adaptive Queuing in ESX

While troubleshooting another issue a week or two ago I came across this VMware knowledge base article. Having spent most of the time with other brand arrays in the past, I thought this was a pretty cool solution verses just increasing the queue length of the HBA. I would recommend setting this on your 3par BEFORE you get QFULL problems. Additionally, Netapp has an implementation of this as well.

Be sure to read the note at the bottom especially:

If hosts running operating systems other than ESX are connected to array ports that are being accessed by ESX hosts, while the latter are configured to use the adaptive algorithm, make sure those operating systems use an adaptive queue depth algorithm as well or isolate them on different ports on the storage array.

I do need to dig deeper how this affects performance as the queue begins to fill, not sure if one method is better than another. Is this the new direction that many Storage Vendors will follow?

Until then, the best advice is to do what your storage vendor recommends, especially if they say it is critical.

Here is a quick run through for you.

In the vSphere Client

Select the ESX host and go to the configuration tab and click on the Advanced Settings under Software.

In the Advanced Settings

Select the option for Disk and scroll down to the QFullSampleSize and QFullThreshold.
Change the values to the 3par recommended values:
QFullSampleSize = 32
QFullThreshold = 4

My Fun with the VMware Enterprise Administration and Design Exams

Sorry I have been missing for a few weeks. I know many were quite worried why I hadn’t blogged for a couple weeks (not really).

Back in February I sat for the Enterprise Administration Exam at PEX in Las Vegas. It was scheduled the day after the Super Bowl, what a bunch of distractions. Thankfully I passed and I want to give my experience so as to not violate any rules or anything I agreed to. This was a technical test. A lot of settings and configurations and information like that. Still multiple choice so at least you know the right answer is on the screen (hopefully, I did have one I thought none of these are right). The lab section was actually as fun as test taking could be. I wish there was more lab practical type things when it comes to these kinds of tests. Overall there is more intricate settings and config questions then you will find on the VCP exam.

At the end of April I took the Design Exam. This was a much different experience. I had a extremely hard time finding a study list of things that would help. Know the Exam Blueprint is all I would say. Also, this I think is where VMware can start finding out who does Architecture work and who may be an Administrator. I could say you could read every PDF on VMware.com and still not know how to pass this test unless you work with the solutions multiple times. The design drawing was a challenge, I wasted too much time reading the requirements document and ran of time, but I feel I was able to get a good portion of what I needed up on the page. Technically the interface was kind of quirky.

I felt both exams were challenging and but were fair to the Exam Blueprints. Nothing on there made me scream, “they didn’t say they would test on THAT!” The design exam needs some technical improvement (matching questions were buggy).

Now begins the harder and more involved process. The Design submission and hopefully an invitation to a defense.