Jon Owings | CrashLoopBackoff

Using Network Load Balancing with View

If you have a smaller View deployment but still want to have redundant connection servers look no further than Microsoft NLB. Solve this problem without the need for an expensive hardware loadbalancer. Will it have all of the bells and whistles? No. If you have less than a 1000 users you probably would not see the benefit of the advanced features in a hardware load balancer. Make sure to read the whitepaper from VMware about NLB in Virtual Machines.

I am making the assumption you are like me and want everything to be as virtual as possible. So the View Connection Manager servers will be VM’s

Setup the primary and replica View Servers

I won’t go over installing View. Just be sure to setup the initial manager server. Then go ahead and setup the replica VM.

Configure NLB

Go the the Administrative tools and open the Network Load Balancing Manager. Right click the top node in the tree menu on the left and select New Cluster.
Set the IP and other information you will used for the Load Balanced cluster. This is a new IP not used by your View Manager servers.
In the VMware document referenced above VMware recommends setting the Cluster operation mode to Multicast.
Click Next then next again. When asked to configure port rules I leave it on the default and click next. You can chose limit this to certain ports.

Click Next again and enter localhost in the wizard to configure the local interfaces for NLB. Click next and make sure to note the priority. When setting up the replica server this number needs to be different. Finally click finish and wait for the configuration to finish. You should now be able to ping your new cluster IP address.

Setup the Replica Server in the Load Balancer

Righ Click the node in the tree menu for the NLB Cluster you just created and select Add new host to cluster. Enter the IP for the Replica Server and click connect. Select the interface that will be used for the Load Balancing and click next. Make sure the Priority is unique from the first server. If it gives you any grief after this point close and re-open the Network Load balancing Manager. The working cluster should look like this:

Test the Failover

Start a continual ping to the cluster IP. Now use the vSphere Client to disconnect the network from one of the servers. Watch the pings continue to come back.

Finally, create a DNS A record (something like desktop.yourdomain.com) and point it to the cluster IP. You now have some decent failover in case of a VM failure and even a host failure (suggestion would be to use seperate hosts for the VM’s).

Note – You may need to add static ARP entries into your switching depending on your network topology. Be sure to test this fully and consult your network manufacturer’s documention for help with static ARP.

Adaptive Queuing in ESX

While troubleshooting another issue a week or two ago I came across this VMware knowledge base article. Having spent most of the time with other brand arrays in the past, I thought this was a pretty cool solution verses just increasing the queue length of the HBA. I would recommend setting this on your 3par BEFORE you get QFULL problems. Additionally, Netapp has an implementation of this as well.

Be sure to read the note at the bottom especially:

If hosts running operating systems other than ESX are connected to array ports that are being accessed by ESX hosts, while the latter are configured to use the adaptive algorithm, make sure those operating systems use an adaptive queue depth algorithm as well or isolate them on different ports on the storage array.

I do need to dig deeper how this affects performance as the queue begins to fill, not sure if one method is better than another. Is this the new direction that many Storage Vendors will follow?

Until then, the best advice is to do what your storage vendor recommends, especially if they say it is critical.

Here is a quick run through for you.

In the vSphere Client

Select the ESX host and go to the configuration tab and click on the Advanced Settings under Software.

In the Advanced Settings

Select the option for Disk and scroll down to the QFullSampleSize and QFullThreshold.
Change the values to the 3par recommended values:
QFullSampleSize = 32
QFullThreshold = 4

My Fun with the VMware Enterprise Administration and Design Exams

Sorry I have been missing for a few weeks. I know many were quite worried why I hadn’t blogged for a couple weeks (not really).

Back in February I sat for the Enterprise Administration Exam at PEX in Las Vegas. It was scheduled the day after the Super Bowl, what a bunch of distractions. Thankfully I passed and I want to give my experience so as to not violate any rules or anything I agreed to. This was a technical test. A lot of settings and configurations and information like that. Still multiple choice so at least you know the right answer is on the screen (hopefully, I did have one I thought none of these are right). The lab section was actually as fun as test taking could be. I wish there was more lab practical type things when it comes to these kinds of tests. Overall there is more intricate settings and config questions then you will find on the VCP exam.

At the end of April I took the Design Exam. This was a much different experience. I had a extremely hard time finding a study list of things that would help. Know the Exam Blueprint is all I would say. Also, this I think is where VMware can start finding out who does Architecture work and who may be an Administrator. I could say you could read every PDF on VMware.com and still not know how to pass this test unless you work with the solutions multiple times. The design drawing was a challenge, I wasted too much time reading the requirements document and ran of time, but I feel I was able to get a good portion of what I needed up on the page. Technically the interface was kind of quirky.

I felt both exams were challenging and but were fair to the Exam Blueprints. Nothing on there made me scream, “they didn’t say they would test on THAT!” The design exam needs some technical improvement (matching questions were buggy).

Now begins the harder and more involved process. The Design submission and hopefully an invitation to a defense.

Ask Good Questions

This happened a long time ago. I arrived at a customer site to install View Desktop Manager (may have been version 2). This was before any cool VDI sizing tools like Liquidware Labs. I am installing ESX and VDM I casually ask, “What apps will you be running on this install?” The answer was, “Oh, web apps like youtube, flash and some shockwave stuff.” I thought “ah dang” in my best Mater voice. This was a case of two different organizations thinking someone else had gathered the proper information. Important details sometimes fall through the cracks. Since that day, I try to at least uncover most of this stuff before I show up on site.

Even though we have great assessment tools now, remember to ask some questions and get to know what is your customers end goal.

Things I learned that day. As related to VDI.

1. Know what your client is doing, “What apps are you going to use?”

2. Know where your client wants to do that thing from, “So, what kind of connection do you have to that remote office with 100+ users?”

This is not the full list of questions I would ask, just some I learned along the way.

VMware View – User Profile Options

All the technology and gadgets for managing desktops are worthless if your users complain about their experience with the desktop. Something I learned administering Citrix Presentation Server. Differing methods exist to keep the technical presentation of the desktop usable, for example the mouse being in sync and the right pixels show the right colors. What is also included in the user experience is a consistent environment where their personal data and settings are where they should be. Here are a few methods for managing those bits when using VMware View.

Mandatory Profiles
This profile is kept on the a central file share. The profile is copied to the machine on login, when the user logs out the changes are not kept. Great way to keep a consistent profile on kiosk type and data entry desktops. Where customization is not needed and most likely not wanted mandatory profiles are worth exploring. Main change is you set up the profile just like you want it then rename the NTUSER.dat to NTUSER.man. A lot exists on the internet about setting up man profiles.

Local Profiles
If you go through life never changing a thing in your Windows environment, you are using a Local Profile. Not to say you don’t change settings, save files or customize your background. You just have Windows running as the default. This is an option I will usually discourage because it is hard to backup data that is often kept in the local profile. VMware View will redirect user data to a User Data Disk (or whatever it is called today) on Persistent Desktop Pools. This is a good way to get the data on another VMDK. This introduces problems when looking at data recovery. There is solutions, but just something you will need to remember to look into.

Roaming Profiles
Roaming profiles is a great way to redirect current profiles to a central location. In theory this works great. In a View environment you can keep a local copy on a users desktop profile and the changes are copied back and forth. I have often seen this work just great. Then from time to time, the profile will become corrupt, many times it does not unload correctly when users disconnect, or log out. Then you may have to pick through folders trying to find their “My Documents”. This is why I would suggest using this with Group Policy and Folder redirection which I will cover next.

Redirecting Folders
You may end up using a folder redirection group policy. This will move folders like the Desktop and My Documents for a user to a file server. This slims down the roaming profile as those locations are redirected to another location outside of the profile. This data is not copied from the machine to the server over and over. More information here.

Other Options
Immidio Flex Profiles
I really liked this option it was a way to combine mandatory profiles and a Roaming profile. This program would run some scripts on logon and log off to save files and settings. A really great paper on how to use it can be found here. Just like any great program that takes a new way to solve an annoying old problem, this is now not free.

RTO Virtual Profiles
I have never implemented this solution before. I have used it as part of a few training labs. I liked the feel. Now that VMware has purchased this software from RTO, the website redirects to a transition page. So I am looking for a way to test it in the lab, hoping the next set of bits of View includes RTO. Check this FAQ out for more information.

Maybe once it is built into View this will no longer be a serious issue. Profiles will be one of those things we tell stories to young padawan VM admins about, “We used to have to fight profiles, they were big and slow, and sometimes they would disappear!” Until that day…

Random Half Thoughts While Driving

So I often have epiphany teasers while driving long distances or stuck in traffic. I call them teasers because they are never fully developed ideas and often disappear into thoughts about passing cars, or yelling at the person on their cell phone going 15 MPH taking up 2 lanes.

Here is some I was able to save today (VMware related):

1. What if I DID want an HA cluster to be split in two different locations, Why?
2. Why must we over-subscribe iSCSI vmkernel ports to make the best use of the 1gbe phyical nics. Is it a just the software iSCSI in vSphere? Is just something that happens with IP storage? I should test that sometime…
3. If I had 10 GB nics I wouldn’t use them on Service Console or Vmotion that would be a waste. No wait, VMotion ports could use it to speed up your VMotions.
4. Why do people use VLAN 1 for their production servers? Didnt’ their Momma teach em?
5. People shouldn’t fear using extents, they are not that bad. No, maybe they are. Nah, I bet they are fine, how often does just 1 lun go down. What are the chances of it being the first lun in your extent? Ok maybe it happens a bunch. I am too scared to try it today.

VMware View and Xsigo

*Disclaimer – I work for a Xsigo and VMware partner.

I was in the VMware View Design and Best practices class a couple weeks ago. Much of the class is built on the VMware View Reference Architecture. The picture below is from that PDF.

It really struck me how many IO connections (Network or Storage) it would take to run this POD. Minimum (in my opinion) would be 6 cables per host with ten 8 host clusters that is 480 cables! Let’s say that 160 of those are 4 gb Fiberchannel and the other 320 are 1 gb ethernet. The is 640 gb for storage and 320 for network.

Xsigo currently uses 20 gb infiniband and best practice would be to use 2 cards per server. The same 80 servers in the above cluster would have 3200 gb of bandwidth available. Add in the flexibility and ease of management you get using virtual IO. The cost savings in the number director class fiber switches and datacenter switches you no longer need and the ROI I would think the pays for the Xsigo Directors. I don’t deal with pricing so this is pure contemplation. So I will stick with the technical benefits. Being in the datacenter I like any solution that makes provisioning servers easier, takes less cabling, and gives me unbelievable bandwidth.

So just in the way VMware changed the way we think about the datacenter. Virtual IO will once again change how we deal with our deployments.

iSCSI Connections on EqualLogic PS Series

Equallogic PS Series Design Considerations

VMware vSphere introduces support for multipathing for iSCSI. Equallogic released a recommended configuration for using MPIO with iSCSI. I have a few observations after working with MPIO and iSCSI. The main lesson is know the capabilities of the storage before you go trying to see how man paths you can have with active IO.

EqualLogic defines a host connection as 1 iSCSI path to a volume. At VMware Partner Exchange 2010 I was told by a Dell guy, “Yeah, gotta read those release notes!”
EqualLogic limits the number of hosts in the to 128 per pool or 256 per group connections in the 4000 series (see table 1 for full breakdown) and to 512/2048 per pool/group connections in the 6000 series arrays.
The EqualLogic MPIO recommendation mentioned above can consume many connections with just a few vSphere hosts.

I was under the false impression that by “hosts” we were talking about physical connections to the array. Especially since the datasheet says “Hosts Accessing PS series Group”. It actually means iSCSI connections to a volume. Therefore if you have 1 host with 128 volumes singly connected via 1 iSCSI path each, you are already at your limit (on the PS4000).

An example of how fast vSphere iSCSI MPIO (Round Robin) can consume available connections can be seen this this scenario. Five vSphere hosts with 2 network cards each on the iSCSI network. If we follow the whitepaper above we will create 4 vmkernel ports per host. Each vmkernel creates an additional connection per volume. Therefore if we have 10 300 GB volumes for datastores we already have 200 iSCSI connections to our Equallogic array. Really no problem for the 6000 series but the 4000 will start to drop connections. I have not even added the connections created by the vStorage API/VCB capable backup server. So here is a formula*:

N – number of hosts

V – number of vmkernel ports

T – number of targeted volumes

B – number of connections from the backup server

C – number of connections

(N * V * T) + B = C

Equallogic PS Series Array	Connections (pool/group)
4000E	128/256
4000X	128/256
4000XV	128/256
6000E	512/2048
6000S	512/2048
6000X	512/2048
6000XV	512/2048
6010,6500,6510 Series	512/2048

Use multiple pools within the group in order to avoid dropped iSCSI connections and provide scalability. This reduces the number of spindles you are hitting with your IO. Using care to know the capacity of the array will help avoid big problems down the road.

*I have seen the connections actually be higher and I can only figure this is because the way EqualLogic does iSCSI redirection.

Five Things you should know about VMware Certification – thanks @rickvanover

Tonight I was sitting with my Mac Book Pro, wife and kids all in bed, perfect blogging time. Except no idea what I wanted to write. After a burst of twitter encouragement from @rickvanover here we go:

Five things you should know about VMware Certs:

1. If you work for a VMware partner, start with the VSP/VTSP Certifications. They require a lot less initial investment and give you the start into VMware products. Partners only.

2. You can take the “Install and Configure” or “Design Secure and Analyze” or the “Fast Track” (which is some kind of combo of the previous 2) then study and take the VCP. The VMware Certified Professional is the standard in Virtualization industry and has personally opened many doors for me professionally. VMware Partners NEED them, but it is quite an investment. The class is far from free and the test isn’t cheap. Something to know though, if you go to VMworld the tests are discounted. So everyone at VMworld take advantage! Save a few bucks.

3. As good as the VCP is, there is still a demand for VMware knowledge more advanced than the base cert. I lost my job last year, while job hunting I saw listings asking for VCDX certified people. At the time no VCDX’s were publically known. It just shows the demand for proof of advanced knowledge in VMware and a possible dilution of the VCP due to brain dumping.

4. Testing experience is fairly standard technical testing. Although I wish it was all lab practical rather than memorization. Multiple choice type questions are the majority of the questions. My best test taking tip is to actually know how to Install and Manage Virtual Infrastructure. You can run it all in VMware Workstation 7, so there is no excuse to just memorize answers. I say that but you will have to memorize max and minimums and other facts in order to pass the VCP.

5. Check what is on the blue print on the VMware Certification website, this website is the final word on what is on the test.

Bonus: Use the communities/blogs/twitter to find information and answers. Don’t just post “Hey what is on the exam?”. Ask questions about technical topics, then make sure you could articulate the concept to someone else. I find if I understand enough to be able to teach it to someone else then I really am starting to learn things.

Bonus #2: Be willing to be always learning. If you already know everything you probably don’t need the certification.

*No clue who the original artist is of that picture. It is awesome, so if you know who to credit let me know.

New VMware KB – zeroedthick or eagerzeroedthick

Due to the performance hit while zeroing mentioned in the Thin Provisioning Performance white paper this article in the VMware knowledge base could be of some good use.

I would suggest using eagerzeroedthick for any high IO tier 1 type of Virtual Machine. This can be done when creating the VMDK from the GUI by selecting the “Support Clustering Features such as Fault Tolerance” check box.

So go out and check your VMDK’s.