datacenters | CrashLoopBackoff

Some Reality for us Infrastructure Peeps or Apps are cool too

Don’t’ you just love double titles?

For many years I have been an infrastructure guy. I really liked how the cables, and processors and Memory and blinking lights worked. Applications were often the necessary evil tolerated so that I can play with cool technology. During my own journey toward learning about the cloud it becomes increasingly important to consider the function of the application. Six years ago me would totally punch me in the face right now. Traitor. J

1 – Don’t get your App messed up in my resource buckets of awesomeness

So the reality check to the Infrastructure geek in me is this: The application teams really think of what you do as the network. That is why when anything is ever wrong it is always “the network’s” fault. What we love to do is getting abstracted more and more. I will still contend that is very important and very hard to do. Whether you are building reference architectures or deploying a converged infrastructure appliance almost no one but us cares. They just want the data to do their jobs. So while we have really great discussions about speeds and feeds, the guy in the picture below just wants the app. From the hypervisor down we need to design with the application in mind or we will risk becoming like that goth dude locked in the server room on IT Crowd.

2 Honey badger don’t care about FCoE

My next post will get into what I have been researching regarding what is out there and hopefully help us (infra. peeps) understand our App/Dev brothers better.

You are probably an Infrastructure person if:

You read this blog.
You work mainly with Virtualization
Storage Admin
Network Admin
You like to make fun of DBA’s

My First Big Data

Ok while I was on vacation away from all things virtual last week. Some reason I had some deep thoughts about Big Data. At least deep for me. So this is mostly incoherent rambling, but I want it written down in case it happens.

What I kept coming to is how will data be used in a way that is NOT trying to sell me some XYZ product?

Mostly I think about big data as pulling public (or private) data about me in order to gain some edge to get me to buy the next tablet or phone.

I don’t know much about the entire big data industry, but I thought there must be more, and isn’t just about finding what I think about a brand on the twiiter.

Researchers – create a bunch of data about X. Give us cool graphs on the internet about how X and Y intersect. I feel this is extremely important. The next cures will be created outside a lab and 50 test subjects. Obvious Jon, I bet someone has already said that.

My linked in network

Innovation – can the next big thing be the output of some insane amount of data? How do I create a solution to a problem everyone has but they don’t know about yet? How can it be beyond the Social media craze. Big data is bigger than social networks. The next idea needs crush Google, Apple, Twitter and Facebook. I know there are armies of geniuses at those companies but the next thing will be created by someone we don’t know. Maybe I can use big data to find that person before anyone else?

vSphere Metro Stretched Clusters – Some Info/Links

A lot of questions lately about vSphere Clusters across distance. I really need to learn for myself so I collected some good links.

Make sure you understand what “Only Non-uniform host access configuration is supported” means. Someone correct me if I have this wrong but your device that enables the distributed virtual storage needs to be sure that hosts in site A are writing to their preferred volumes in site A and vice versa in Site B. Probably way over simplifying it.

LINKS

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007545

http://virtualgeek.typepad.com/virtual_geek/2011/10/new-vmware-hcl-category-vsphere-metro-stretched-cluster.html

http://www.yellow-bricks.com/2011/10/07/vsphere-metro-storage-cluster-solutions-what-is-supported-and-what-not/

http://www.yellow-bricks.com/2011/10/05/vsphere-5-0-ha-and-metro-stretched-cluster-solutions/

Big thanks to Scott Lowe for clearing the details on this topic.

B.Y.O.P – The Alternative Vblock

In college I often would be invited to a get together that could often include the letters BYOB, Bring Your Own Beer. Sometimes a cookout would be BYOM, Bring Your Own Meat (or meat alternative for the vegetarians). So today I want to leverage this to push my new acronym B.Y.O.P. Bring Your Own Pod. Lately I have been seeing people talk about Vblocks. If I can venture a succinct definition a Vblock is a pre-configured set of Cisco, EMC and VMware products tested by super smart people, approved by these people to work together, then supported by these organizations as a single entity. Your reseller/solutions provider really should already be doing this very thing for you. You may choose to buy just the network piece, or the hypervisor but your partner should be able to verify a solution to work from end to end and provide unified support.

So You can’t call it BYOPCVCEP

Why not Vblock? This might get me blacklisted by the Elders of the vDiva council, but VCE doesn’t exist to make your life in the datacenter easier, they exist to sell you more VMware, Cisco and EMC. Vblock for sure simplifies your buying experience. I believe they are all great products and may very well do just what you need. Without competition though the only winner is VCE. Do not by forced into a box by the giant vendors. Find someone that can help determine your end goal, provide you vendor neutral analysis of the building blocks needed to achieve your end goal. Then provide the correct vendors and unified support to Build Your Own Pod.

So What is the Alternative Vblock

Originally I was going to draw up a sweet solution of 3par, Xsigo and Dell R610’s and say, “Hey everyone! This is some cool stuff. Try to quiet the overwhelmingly loud voice calling from VCE and give this Alternative Vblock a try.” As I thought more and more about it I think doing that is contrary to my main point. I would like more to provide the discussion points or some possible products among others that can be used to Build Your Own Pod. I am a firm believer in getting what is right for your datacenter needs. So here is a few links to help begin the discussion.

Xsigo and Pod – Jon Toor
3par and iBlocks – Marc Farley

Operational Readiness

One thing I am thinking about due to the VCDX application is operational readiness. What does it mean to pronounce this project or solution good-to-go? In my world it would be to test that each feature does exactly what it should be doing. Most commonly this will be failover testing, but could reach into any feature or be as big as DR plan that involves much more than the technical parts doing what they should. Some things I think need to be checked:

Resources

Are the CPU, Memory, Network and Storage doing what they should be? Some load generating programs like IOmeter can be fine to test network and storage performance. CPU busy programs can verify Resource Pools and DRS are behaving the way they should.

Failover

You have redundant links right? Start pulling cables. Do the links failover for Virtual Machines, Service Console, and iSCSI? How about the redundancy of the physical network, even more cable to pull! Also test that the storage controllers failover correctly. Also, I will make sure HA does what it is supposed to, instantly power off a host and make sure some test virtual machines start up somewhere else on the cluster.

Virtual Center Operations

Deploy new virtual machines, host and storage VMotion, deploy from a template, and clone a vm are all things we need to make sure are working. If this is a big enough deployment make sure the customer can use the deployment appliance if you are making use of one. Make sure the alarms send traps and emails too.

Storage Operations

Create new luns, test replication, test storage alarms and make sure the customer understands thin provisioning if it is in use. Make sure you are getting IO as designed from the Storage side. Making use of the SAN tools to be sure the storage is doing what it should.

Applications

You can verify that each application is working as intended within the virtual environment.

There must be something I am missing but the point is trying to test out everything so you can tell that this virtualization solution is ready to be used.

Tale of Two Datacenters

I wish I had my camera so I could share with you the difference between the last two server rooms I was in.

I will try to describe. The first was an organizational dream. Color coordinated patch cables, wire management actually used. You could actually see the port numbers on the Cisco 4507. Even more the configs were labeled. So when you were logged into the switch you knew what was going on. Just a brief picture but the rest of the room was the same, the blades, SAN and everthing else was how I wish everyone would be.

In contrast, the next day, I was trying to upgrade some ESX servers. The cables had no rhyme or reason. The switches had no labels and the trunks and access ports were done in the most backward way I could think of. It would be too hard to describe. I vMotioned all the vm’s to another host. Shutdown the host to upgrade to 32GB of Memory. After unplugging everything and trying to slide the Dell 2950 out of the rack I discover the fibre to the server above it is running through the “handle” on the back of the 2950. So the server will only slide about 1 foot. Great, so I hook everything back up and vMotion everything off of the host above. Move the troublesome fibre, and decide to upgrade that server. I start to slide it out for the memory upgrade. The server on top of it is not on rails and starts to slide with the 2950. So I now need to get downtime (that is a physical server) to take that server down so I can upgrade the memory of the ESX servers then upgrade them to ESX 3.5 update 2.

So I would guess one of the quickest ways to eat all of my billable hours is to spend hours fighting poor datacenter work and eventually not get anything done.