design | CrashLoopBackoff

Start with Applications

I have been revisiting my work towards some advanced datacenter certifications and decided to journal some of the thoughts I have during the process. After a 3 year break I decided it was time to start pushing toward some of these goals.

This may sound eerily similar to something I have said before. It is a constant fight in the infrastructure technology field to get so weighed down by speeds and feeds and features. You begin to lose sight as to why you actually put servers, switches, storage and software together in the first place. While looking at the requirements guide for the VCAP-DCD the very first thing that is mentioned is getting the business requirements. How do I actually do that? What does the business actually require?

Know what the applications actually do.
Ask! What does this Microsoft SQL database do? How does email relate to our business doing deals? Find out how money goes in and out of the business. How does your company pay bills? How do you charge for whatever it is you produce? How do the MBA types make decisions about who, what, when, where and why for your business? In IT we often get so involved in rolling out a new widget from vendor X, Y and Z we often don’t realize what is the purpose to the business. Understand this from a high level first.
Map technology to the impact on the business.
Who cares if I can do a million IOPS if all I do is check email all day? How do I consolidate servers with no plan on how they impact the bottom line? How do I provide cloud like capabilities if no one really needs them? So start to map the capabilities to the benefits to the business. If the decisions being made can be done with data that is 5 minutes old instead of a 24 hours how can that change the landscape of your business? Does this give an advantage over competitors?
Know something about the Apps.
If your answer is I don’t know how are business runs or anything about SQL or Oracle I just make empty VM’s for people to put the apps on. I make sure they turn on and I move them around when they need performance or more capacity. Guess what? Those functions can be done by VMware Orchestrator. If you don’t know why you put 4 vCPU’s on a SQL VM because the batch jobs don’t ever use more than that and why, you need to learn. If you need tools to decipher the differences then get them. At least get the trial versions so you can see what happens. Get close to the queries that run at night. Do you know if they are CPU, Memory or Storage bound? Find out. Get off of reddit and check it out. Do you know if you put in faster servers will the app improve in a way that makes things better for business? Are you really going to gamble your budget on marginal improvements?

Can you connect how all of these things relate and benefit the business?

Just some small things I have been thinking about. In my job it is a constant temptation to push how many IOPS you can do with this thing or that. When I need to say “what process needs the performance? If that process is faster AND you get additional benefits of data reduction, floor tile reduction, power usage reduction what will it mean to your business users?”

My Fun with the VMware Enterprise Administration and Design Exams

Sorry I have been missing for a few weeks. I know many were quite worried why I hadn’t blogged for a couple weeks (not really).

Back in February I sat for the Enterprise Administration Exam at PEX in Las Vegas. It was scheduled the day after the Super Bowl, what a bunch of distractions. Thankfully I passed and I want to give my experience so as to not violate any rules or anything I agreed to. This was a technical test. A lot of settings and configurations and information like that. Still multiple choice so at least you know the right answer is on the screen (hopefully, I did have one I thought none of these are right). The lab section was actually as fun as test taking could be. I wish there was more lab practical type things when it comes to these kinds of tests. Overall there is more intricate settings and config questions then you will find on the VCP exam.

At the end of April I took the Design Exam. This was a much different experience. I had a extremely hard time finding a study list of things that would help. Know the Exam Blueprint is all I would say. Also, this I think is where VMware can start finding out who does Architecture work and who may be an Administrator. I could say you could read every PDF on VMware.com and still not know how to pass this test unless you work with the solutions multiple times. The design drawing was a challenge, I wasted too much time reading the requirements document and ran of time, but I feel I was able to get a good portion of what I needed up on the page. Technically the interface was kind of quirky.

I felt both exams were challenging and but were fair to the Exam Blueprints. Nothing on there made me scream, “they didn’t say they would test on THAT!” The design exam needs some technical improvement (matching questions were buggy).

Now begins the harder and more involved process. The Design submission and hopefully an invitation to a defense.

Storage Design and VDI

Recently I have spent time re-thinking certain configuration scenarios and asking myself, “Why?” If there is something I do day to day during installs is this still true when it comes to vSphere? or will it still be true when it comes to future versions.
Lately I have questioned how I deploy LUNs/volumes/datastores. I usually deploy multiple moderate size datastores. In my opinion this was always the best way to fit in MOST situations. I also will create datastores based on need afterward. So will create some general use datastores then add a bigger or smaller store based on performance/storage needs. After all the research I have done and asking questions on twitter* I still think this is a good plan in most situations.
I went over a VMworld.com session TA3220 – VMware vStorage VMFS-3 Architectural Advances since ESX 3.0 and read this paper:
http://www.vmware.com/resources/techresources/1059
I also went over some blog posts at Yellow-Bricks.com and Virtualgeek.

An idea occurred to me when it comes to using extents in VMFS, SCSI Reservations/Locks, and VDI “Boot Storms”. First some things a picked up.
1. Extents are not “spill and fill” VMFS places VM files across all the LUNs. Not quite what I would call load balancing, since it does not take IO load into account when placing files. So in situations where all the VM’s have similar loads this won’t be a problem.
2. Only the first LUN in a VMFS span gets locked by “storage and VMFS Administrative tasks” (Scalable Storage Performance pg 9). Not sure if this implies all locks.

Booting 100’s of VM’s for VMware View will cause locking and even though vSphere is much better when it comes to how quickly this process takes. There is still an impact. So I am beginning to think of a disk layout to ease administration for VDI, and possibly lay the groundwork for improved performance. Here is my theory:

Create four LUNs with 200GB each. Use VMFS to extents to group them together. Resulting in an 800 GB datastore with 4 disk queues and only 1 LUN that locks during administrative tasks.

Give this datastore to VMware View and let it have at it. Since the IO load for each VM is mostly the same, and really at the highest during boot other tasks performed on the LUN after the initial boot storm will have even less impact. So we can let desktops get destroyed and rebuilt/cloned all day with only locking that first LUN. This part I still need to confirm in the LAB.

What I have seen in the lab is with same sized clones the data on disk was spread pretty evenly across the LUNs.

Any other ideas? Please leave a comment. Maybe I am way off base.

*(thanks to @lamw @jasonboche and @sakacc for discussing or answering my tweets)