Indeed, we are designed to always stay running. In less than 6 hours we will begin moving a live production environment of nearly 120 servers from their current home approximately 60 miles away into new racks in our facility. Those servers are coming here in part because keeping up with the always-on demands of today’s infrastructures is increasing more difficult and expensive to do.  It has come to the point where if IT infrastructure is not your core business, it’s awfully difficult to justify the ever rising costs of running an always-on environment yourself.

We’ve spent months getting ready for this move, and now in these hours before “go time”, it is eerily quiet in that part of the data center. Other than the hum of CRAC units and core switch fans, not a creature is stirring.  The patch cables are sorted in boxes with care, in hopes that people with servers soon will be there (sorry, couldn’t resist).  But once completed, the large precisely choreographed effort required to move our new customer into a better home will result in always-on uptime at  a fraction of what it would cost to build from scratch or remodel.

Since all of the data center details for this move are already cared for, it’s now time to make sure the most important item is covered:  three shifts-worth of fine coffee.

//spk

Tweet This Delicious Digg This Post Stumble This Post

//spk

Tweet This Delicious Digg This Post Stumble This Post

The iPhone is turning out to be a bit more expensive for AT&T than perhaps they were expecting.   While the air time minutes aren’t all that interesting, the bandwidth numbers do tell a  story – unexpected hidden costs.  The bandwidth-hungry nature of iPhone web applications is seriously taxing the AT&T wireless network, and the company will have to add cell towers and back-haul lines to support the increasing load.   Who’s going to foot that bill?  Since iPhone plans don’t have bandwidth limits, it won’t be existing customers, and so it turns out that the locked-in revenue stream is really a double-edged sword.   AT&T can’t go back to existing customers for more money, but they need to fund the expansion to keep the service viable.   Sounds a little like blackmail – infrastructure blackmail to be exact.

The underlying message of the iPhone story has a familiar ring to it (no pun intended).  Do you have technology in your business that periodically puts you over a financial barrel?   Are you finding yourself forced to fund infrastructure improvements without additional revenue to offset the cost?

We all have critical business applications – those applications without which we cannot transact business.   Sooner or later the day comes when the system breaks and can’t be fixed without being upgraded.  Then we learn that the upgrade comes with a list of pre-requisite hardware and other ancillary software that has to be upgraded as well.   The total price tag is huge, but there is no choice.   We swallow hard and pay the cost, or else the vendor banishes us to the land of No Technical Support with a broken application.

Insofar as unexpected server hardware costs are concerned, virtualization can be a saving grace.   If you have a virtualized system with excess capacity, you can react quickly at no additional cost.    Our analysis of customer systems over the past decade reveals that on average, most servers are running at under 15% processor utilization.   If you still haven’t virtualized a portion of your environment, this is an easy strategy to deploy to avoid infrastructure blackmail.

Another strategy is to find a good hosting partner that can provide you with capacity on demand, either with physical or virtualized servers.    By moving to a hosted environment, you effectively transfer the burden of hidden infrastructure expense from yourself to the hosting company.   Large unexpected capital expenses can be transformed into low to moderate increases in operating expense.   AT&T is probably wishing they could do that very thing right about now…

//spk

Tweet This Delicious Digg This Post Stumble This Post

So what do we do all day in the world of IT?  In the smaller server closets of the world, very little needs to be done on a daily basis unless something breaks.   However, in shops of any significant size, here is just a starter list for daily computer TCB:

1. Did last night’s backups run OK?
2. Did last night’s batch processing run OK? 
3. Are the online applications healthy and responsive?
4. Did all of scheduled change activity go well?  
5. Are all of the servers operating at nominally (e.g. acceptable response time, no 100% hung CPUs, etc.)
6. Is free disk space at acceptable levels?
7. Are there any trouble tickets from the overnight hours?
8. Is the network performing normally?
9. Are there any suspicious entries in the security logs?
10. Are any equipment alarm lights lit?

Any wrong answers to the above questions become the day’s TCB workload. Intertwined with that comes the standing to-do list items:

1. Returning phone calls
2. Answering Email
3. Attending Meetings
4. Performing project work
5. Managing staff
6. Meeting with customers
7. Closing business
8. Handling the unexpected, which usually trumps everything else.

Are you tired yet? 

Regardless of the level of automation one can employ to keep the infrastructure running, there are certain activities that require human attention.   There simply is no substitute.   If infrastructure is not your core business, you may want to take a hard look at whether caring and feeding for your IT systems is the best use of your time and resources.

We don’t each generate our own power, but we certainly need it to run our businesses.   There simply is no reason for each of us to invest in the people and equipment to generate electricity – it’s impractical and unaffordable.   Likewise, you certainly need computing resources to run your business, but unless IT infrastructure is your business, the ever-increasing complexity and cost of IT is making in harder to find a compelling business reason to continue building and running it yourself.  

Not to mention that good IT staff is hard to find…

 It would be much better to find a good partner to handle your computing TCB.

//spk

Tweet This Delicious Digg This Post Stumble This Post

News  of Cisco’s intent to enter the server market with its Unified Computing System offering has set the industry pundit’s hair ablaze.   “How will IBM & HP respond?”, “How much market share will be lost to Cisco?”, “Do you want a plumber building your servers?” and on it goes.  The FUD truly has been flying.  You would think the Big Bad Wolf had just come back to Grandma’s house.

So, what does the announcement of UCS mean to us here in the non-rarified air of business computing?  Will it help us run our shops better?

Listen to Cisco CEO Chambers closely…

We look at this as bringing virtualization to life…unleashing the power of virtualization.   We go about it catching market transitions and trying to set timing, first in the data center, but make no mistake about it [UCS will make it] all the way in the home… [emphasis added]

What market transitions, pray tell, is he referring to?  Could it be anything other than the transition to utility based computing? It’s fairly clear he’s not talking about our server rooms and data centers.  No, it would seem Cisco has its sights on something much larger. Chamber’s message is unmistakeable.  If the coming world of utility-based computing were to be compared to The Matrix, Cisco would not be found content with simply supplying the network plumbing – they want to be the Matrix itself. Having already tucked away the network, we now see a move into processors. Can storage be far behind? Perhaps the Big Bad Wolf already has that in the oven.

It doesn’t seem on the surface that UCS is intended for the typical IT shop, but let’s assume otherwise for a moment.  Is there a compelling reason for us to consider (or fear) UCS?    What would make us willing to try a  brand new brand?

In many ways, owning server hardware is a lot like owning a vehicle. First, you make your purchase based on size, looks, performance, the features you need, reliability, serviceability, and of course the price. Sometimes you’re looking to save gas (power), but not always. Maybe you decide to lease it. If you end up with a lemon, you know that very early in the game, and you get the vehicle fixed or replaced under warranty. From that point on, if you put in decent gasoline (clean UPS power), do regular maintenance (clean the fan grids, do disk defrags), and operate it within its design limits (proper cooling), it will run well for a long time.   When it wears out, or after you simply get tired of it and want something new and sexy, you buy a new one, sell or trade the old one, or possibly keep it and run it until the wheels fall off.

In the final analysis, whether you buy Chevy, Ford, Chrysler, or a brand you’ve never tried before really doesn’t matter. You go through the same decision process and ultimately you buy what you like or what you feel comfortable with.  The care, maintenance, and disposal process is the same no matter what you buy. And statistically, the reliability is pretty much the same across the board, despite the religious fervor that surrounds each brand. They all run well on balance, and they all have an occasional breakdown. For every hardware horror story out there, there are scores of identical hardware instances that run their entire lifetimes without a glitch.

Of course, if you absolutely must be the first kid on the block with a new hardware vendor, your mileage may vary.

Early UCS adopters on the phone with Cisco Tech Support

For most of us, UCS is not going to help with the primary purpose of our infrastructure.  So what does make a difference in how well our business systems stay up and running?

If you put a good driver (software) behind the wheel of your vehicle, you can be confident it will stay on the road doing what you intend it to do.  If you put an unskilled, abusive or reckless driver behind the wheel, you can expect more mechanical breakdowns (minor outages), accidents (major outages), or worse (disaster declaration).

I resisted naming operating system names above, but ask yourself, when was the last time you had down time because an operating system or application went off into the weeds?   Do you schedule weekly or nightly reboots “just for good measure” because you can’t trust things to stay healthy?    It is an alarmingly common practice in our client base.

There’s a Red Hat 7.2 system that’s been hosting workload here for years that only comes down when we take it down to replace or upgrade the hardware.   We have a farm of VMWare ESX servers that behave just as well.   Yet we also have a number of Win32 servers running on the same hardware for which I can’t say the same. 

It’s not the hardware.

Lemon’s notwithstanding, the brand of hardware, be it IBM, HP, Dell, and now ostensibly Cisco, really is not the key factor in maintaining uptime.   In this day of clusters-everywhere and RAID-everything, it’s typically not the hardware that takes you down – it’s unreliable software, change  or human error.

As for UCS, it doesn’t look like the Big Bad Wolf is coming to our house anytime soon, but it is a good idea to keep a watchful eye on where he is going.  Cisco has cold hard cash and a big vision, but that vision seems cast for The Matrix, not our server rooms.

Buy what you’re comfortable with and put the right driver behind the wheel, or better yet, let us worry about that for you.

Tweet This Delicious Digg This Post Stumble This Post