Racks – Label both front and back doors. We use a scheme based on the row number and position within the row, such that the first rack in row 5 would be labeled “5A”, the second “5B’, etc.  Other folks use the time-honored  “Battleship”-style system, based on an XY grid that maps out the room, most often based on two-foot squares that make up a typical raised floor system.

An example would be “AJ06″, where the “X” coordinate is “AJ” and the “Y” is “06″. Neither method is necessarily superior to the other, and we happen to use both. We use the row/rack scheme for our racks and XY coordinates for infrastructure items like Air Handlers, floor PDUs, chilled water valves, etc. The reason we use row/rack rather than XY coordinates is that in a large room full of equipment, it is often hard to see the grid system on the walls and figure out where things are located. We believe it’s easier for a new sysadmin to find rack 5C (row 5, third rack) than to ask him to find rack BQ59 in a room chocked full of racks where he can’t see any latitude/longitude markers on the walls to get his bearings. Again, there is neither right nor wrong here; just a couple of different ways to approach it.

Servers and Network Gear – Label both the front and the back. The name is probably sufficient. Security-sensitive folks in your shop may balk at using IP addresses on server labels.

PDUs and power whips – Both floor and in-rack units. If you have an A+B redundant power distribution system, everything can be tagged with a number to identify the unit and a color to indicate to which  feed (“A” or “B”) it’s attached. Note in the pictures how this flows all the way through – even the whips are colored.

Patch Panels – We could talk about patch panel labeling schemes for days. The important thing to do is pick one system and stick with it. Here’s a look at ours. We label our panels using a a “source/destination” scheme, so in the photo “1A/1D (1-6)” means that these are the first 6 ports running from rack 1A (the rack this panel is in) to rack 1D. Very easy for new sysadmins to grasp. This does not follow the ballyhooed TIA standard for labeling patch panels, but we find it be very practical and easy for the people working in the room.

Cables – Labeling cables is a religious issue for another day also, but in our data center we typically only label the key cables in the network backbone and edges  so that trouble shooting is easier at 3 AM. When we do label a cable, we label each end with a wrap-around style label that identifies where the other end of the cable can be found. You can see an example of this in the photo above.  If you click on the photo to enlarge it, you can almost read the label.

Air Handler (a.k.a “CRAC”) units – Simple to do, and very helpful when the units send alarms to systems management tools.

Emergency Power Off (EPO) and Fire suppression controls – I actually think the EPO button should be labeled “Update Resume Before Pushing,” but that’s a topic for another day.

Mechanical Support Systems -  Here you need to not only identify the control accurately, but sometimes you need to be very specific about it’s operation:

After all the hard work of designing and implementing a label system is done, you’ll need to put ongoing enforcement into place for which a label shouldn’t be necessary:

Take-away: What’s most important is that you pick a labeling system that works for you and is easily maintainable, because it needs to be useful, and it’s a never ending process. If it’s confusing to use or a pain to keep up to date, even Bubba and his .50 cal isn’t going to help.

//spk

Tweet This Delicious Digg This Post Stumble This Post

If you Google “systems management” or “systems monitoring” you’ll quickly learn that these tools are now essentially commodity software.  There are dozens, if not several hundred reinvented wheels out there to choose from, whether they be free, ad-supported, or pay-for products.  In fact, the space is so full of purple cows at this point that one has trouble judging between the shades.  If you’re thinking of doing a start-up to produce Yet Another Systems Management Tool, differentiating yourself will be the primary challenge.  Writing the software will be only 10% of your effort.

So, with all the eligible prospects out there, how does one select a systems management tool?  All of the worthwhile products have the same set of basic features: discovery & polling, thresholding and alerting, and data collection and reporting. Many of them also sport attractive eye candy in the form of topographical status displays (the practical usefulness of which is often questionable and a subject for another day) and three dimensional displays of node relationships and performance data. Some can do performance base-lining from which they are able to tell when a system begins to deviate from it’s normal performance profile.  More ambitious products attempt to correlate the blizzard of event traffic blowing through the network and do root-cause analysis.  The best ones are highly configurable, extensible, and offer great API’s, because no matter what you product you select, you will invariably be faced with some amount of customization to make it really fit your environment.

We narrowed our search to products that reflected the ideas  guiding the design of our own in-house tool until we will ceased its development several years ago. We were headed to the next level of system managements: provide business views of our operations.  Most tools focus strictly on the status of discrete elements and their resources, e.g. a router is down, a server is out of disk space, etc.  While that’s certainly useful information to a degree, the real question is, what does it mean to my business?   If disk space is low on a test server, how much do I care?  Should a red light on that condition cause a severity 1 ticket to be opened?  Is the downed router affecting my ability to ship product or simply the archiving of my 2009 backups?

We looked for a tool that could take the various elements that make up our major business systems and allow them to be defined under a small set of  dashboard lights, one for each business system, that would help us understand the impact of the various events occurring across the enterprise.  So, for example, if the “Order Entry” light goes red, we know it’s an all-hands-on-deck situation because of the well-known equation: no orders=no revenue.  If on the other hand, the “R&D” light goes red where all of test systems are located, there is no need to call in the cavalry.

Because human resources are not easily cloned in a crisis, the relative business value of a red light is extremely important to know when multiple red lights are on at the same time. Which ones should get the immediate attention of our limited human resources, and in what order?   A business view answers those questions, especially if you happen to know the dollar cost of downtime for each of your business systems.

Being in the hosting business, we also needed a tool that could segregate our customers into their own groups and provide a similar business view capability. This feature, usually referred to as “multi-tenancy,” typically comes with the ability to give each customer a personalized view into the business systems hosted in our facility.

After several months of shaking the trees and examining their fruits, we settled on Traverse. It does the basics well, has multi-tenancy, and supports business views. The company appears to be stable. The team that came on-site to help with implementation was capable, and the price didn’t break the bank. While I’m not endorsing Traverse as the be-all-end-all nirvana gotta-have-it systems management tool, it does meet our requirements, and it’s definitely an option worthy of your evaluation if you happen to be in the market for a new system.

With all this goodness, it would seem like our new fiancé is a keeper, and yet the ghost of Systems Management Tools Past still lingers.  Hopefully this relationship will last. Hmm…maybe we should sign a prenup…

//spk

Tweet This Delicious Digg This Post Stumble This Post

At this time of year my focus changes from IT to LIONEL infrastructure.  It’s time to begin building this year’s Christmas layout!   Ah, the joy of no project plan or business case!   And looking at the calendar, Christmas day is quite visible on the horizon, so I need to get moving.

With Christmas being on a Friday this year, the early days of that week will likely not be the most productive in the IT shop as the staff winds down the year with inter-office daytime soireés , extended “lunch hours” for last-minute shopping, vacation time, and so on.  From a online availability standpoint, this is clearly not the time to be making any major changes to your critical systems.

Here are a few ideas for redeeming the time during the slow days before Christmas Eve – those days when few creatures are stirring in your shop:

1. Check your software patch levels.  Are you up to date on critical systems?  If not, make plans to get current after the holidays.
2. Check you hardware maintenance agreements.  Is all your gear covered? How about the hardware that’s going out of warranty Real Soon Now? Are all you agreements current?
3. Check your software licenses.  Are you using more than you own?  Funny how that creeps up without notice.  You may want to square up with your vendors at year-end fire sale prices rather than wait until January.
4. Have your sysadmin’s check the free disk space across your server farm.  Is it time to order more storage, or simply clean out the dead wood?  If a file hasn’t been referenced in the last 12 months, archive it or ask the file owner if you may simply delete it.
5. Check for unnecessary VM sprawl.   Do you have virtual servers that you can decommission?
6. Review your backup strategy.   Are all of your critical systems included properly?
7. Test your recovery capability.  Try to recovery a file, a database, and perhaps even an entire server from backup.
8. Declare email bankruptcy and ask your users to do the same.  Don’t start 2010 with 2+GB’s of personal email.  Refuse to be part of the highest form of pack rattery and digital waste known to man.
9. Review your Internet bandwidth usage.  Do you need more or can you do with less?  Do you need to have a chat with any abusive power surfers?
10. Review your private bandwidth usage and contracts.  Are you nearing the end of any contracts?  Is it time to start shopping for better rates?

This is not a list of really exciting stuff to be sure, but they are all important, low-risk things you can do in the inevitable pre-Christmas lull to get your shop off to a good start in 2010.

Christmas Movie Review Department

There are many renditions of Dicken’s “A Christmas Carol” to chose from.   Arguably two of the best star Alistar Sim and George C. Scott, respectively. I personally prefer the George version , but you can’t go wrong with either.  You need at least one of these in your collection.

Christmas Train Department

Also, if you don’t have a train under your Christmas tree and you have kids, grand kids, or you just know kids in your neighborhood, you really ought to head to a hobby shop and check out the Lionel starter sets.   This one will look especially fine under your tree:

//spk

Tweet This Delicious Digg This Post Stumble This Post

While Anderson’s assertion may seem to be true from a philosophical standpoint, it’s not yet true an economic truth, or we wouldn’t have stories in USA Today talking about how YouTube is costing Google money. In June, Michael Liedtke authored a piece shedding some harsh light on the reality of “Free”:

Technology consultants RampRate projects YouTube’s operating losses this year at $174.2 million — far below the $470.6 million estimated by Credit Suisse analysts Spencer Wang and Kenneth Sena in an April research report that became a hot topic on Wall Street and the Internet.The big mystery is how much it costs Google to store and distribute the 20 hours of video that are sent to YouTube every minute.

After conferring with industry experts, Wang and Sena concluded Google will spend nearly $380 million on Internet bandwith, computer hardware, software and data centers.

But RampRate — a specialist in managing technology expenses — believes Google will spend about $83 million to provide the same things to YouTube.

The lower expense estimate presumes Google has negotiated money-saving deals with broadband providers and other behind-the-scenes players that play an integral role in moving data through the Internet’s pipes. RampRate also believes Google’s own propriety technology has helped hold down YouTube’s costs, an idea that Pichette endorsed in his Maclean’s interview.

Although it has been cutting costs to cope with the U.S. recession, Google can still afford to subsidize YouTube with the money it makes through its search engine. Google earned $4.2 billion last year and started off this year with a first-quarter profit of $1.4 billion.

“When people run models, they generally use standard industry pricing for bandwidth, storage, but we build everything from scratch,” [Google's] Pichette said at the time. “So we know our cost position but nobody else does.”

[emphasis added]

Does it seem logical that a service based on raw materials “too cheap to meter” should have trouble making money?  It’s certainly not my intention to discredit Anderson’s thesis about Free, on the contrary, I tend to agree with his primary point that dramatically decreasing costs will change not only the technology we use, but the very way we do business.  YouTube, Facebook and Twitter, all of which are Free, have already revolutionized the way we can all go to market for basically zero cost.

My point is more of a practical one. Bandwidth and storage continue to cost less, but they cost something significant enough that we still negotiate price when we purchase. Even Google does, though they won’t tell us how well they do it. We therefore need to be not only mindful of the cost, but of continuing to properly manage the capacity. Abundance tends to breed waste, and sooner or later Parkinson’s Law comes into effect, at which point we find ourselves breaking out the checkbook to buy more even of  the commodities that are seemingly “too cheap to meter.”

It is still best practice to put tools in place to monitor, control, and efficiently utilize bandwidth. The Internet OC-12 I priced just a few weeks ago was still measured in dollars per megabit, not cents.

It is also still best practice to manage storage, because it still costs real money also. What I mean by “manage” in this sense is giving your sysadmins tools and the time to go through the bowels of your storage systems and look stale old data that can either be deleted, archived to removable media, or moved to cheaper online storage. You couldn’t possibly have heard it, but that was the sound of 90,000 syadmin’s heads simultaneously exploding. For many sysadmins this kind of grunt work is simply beyond the pale, but notice I didn’t say it had to be a manual effort. There are great tools out there to help, FolderSizes being a personal favorite on the cheap end. If you have really deep pockets for some serious automation, IBM’s Tivoli Productivity Center for Data is the luxury car bus option.

And if you’re the one that gets stuck doing the storage management, don’t forget to head over to YouTube for an occasional break…while it’s still Free!  You can find some great “how-to” tips on cleaning your hard disk with an acetylene torch.

//spk

Tweet This Delicious Digg This Post Stumble This Post

Yes that’s right, clowns and coffee don’t go together, particularly this clown. We seem to know that instinctively, yet the blogosphere reports that we are apparently ignoring sound instinct in the name of price. The Clown is cheaper than Starbucks, and so we are enticed away from something of known, predictable quality to something, shall we say, less predictable.

McTreachery

Despite being a loyal Starbucks customer, last week I too swung by the local golden arches for a McLatte. Here’s how it went down at the drive-up’s McSpeaker box (imagine Charlie Brown’s teacher’s voice as you read the McSpeaker’s part):

McSpeaker:  Hello, may I take your order?

Me:  Yes, I’d like a medium cafe latte with skim milk.

McSpeaker:  Would you like whole or non-fat milk?

Me: Non-fat please.

McSpeaker: Hot or Cold?

Me: Hot.

McSpeaker: That will be $2.39.  Please pull around.

So far so good, despite my deliberate avoidance of the word “McLatte” or my faux pas of asking for skim rather than non-fat McMilk. I pull around to window #1, pay and receive my change, and proceed to window #2.  At window #2 I wait an eternity for the window to open.  Eventually a young, ponderously pierced McDude opens the window and presents me with a see-through plastic cup holding a milky substance with ice cubes in it. A new dialogue ensues:

Me: I’m sorry, this should have been made as a hot latte.  (said very politely)

McDude: Uh…really?

Me: Yes.   (The petulant McDude now checks the overhead order display for confirmation.)

McDude: Uh…OK…one minute.

As I settle in for another eternal wait, my server surprisingly appears in less than 20 seconds with the proper looking drink: a brown paper McCup with a black plastic McLid. I drive away, yet something seems wrong. The cup should be warm even though there’s a McSleeve to prevent me from suing them over a burned hand.  One sip, and I realized I’ve just been had. The devious McDude has poured the cold latte into a different cup and simply ditched the ice cubes. Surely there must be something in the Geneva Conventions about messing with a person’s morning caffeine fix. Where is Jackie Chiles when you need him?

So, disgusted but realizing I should have known better, I pointed my truck toward Starbucks. I paid a little more, but got exactly what I asked for with exactly the quality I was expecting.

The McLesson

Clearly, coffee is not Ronnie Mac’s forté. This was not my first failed attempt at getting a fancy coffee McDrink though I’ve tried on multiple occasions. Each and every time they have either botched it badly or been visibly irritated to have to break their burger making ritual in order to do obeisance at the latte machine. They pretty much bat 1.000 on the burgers though, because that’s what they’re really good at.

The lesson?  Stick to what you do well and let the rest to somebody else, because you can’t do everything well.  Not even if you have the deep pockets of Ronnie Mac.  Industry type doesn’t matter either, as history shows us that Novell made this same mistake in 1994 when they bought WordPerfect. They strayed away from what they did best at the time (file and print sharing), got into applications, and everything went downhill from there. Their slow descent into mediocrity is well chronicled, but it all started when they took their eyes off of what they were really good at. Tragically, they are not alone.

R.I.P. Wordperfect.  We knew thee well.

McApplication

There are only so many things you or your business can do with excellence – probably less than four – and usually only one in which you can truly excel. If, like the vast majority of businesses today, yours is a consumer of IT infrastructure rather than a provider, it will quite naturally be difficult and more expensive for you to try to deliver IT as effectively as a professional provider can. There is simply too much to know. It may even be a major annoyance like the McLatte machine. Why? Because it’s not your sweet spot. Your strengths will suffer while you’re focusing on things best delegated to others.

Servers, storage, networking infrastructure continue to evolve into increasingly more complex creatures. Unless IT is the sweet spot of your business, it doesn’t make sense to try to keep chasing infrastructure on your own, for much the same reasons you don’t keep factory-trained mechanics on staff to fix company cars. It’s too much, too costly and certainly not worth it.  IT is no different.  You absolutely need to have technology in your business, but there is no reason to bear the burden of it yourself.

Like Starbucks, professional IT providers may cost a fistful of dollars or just a few dollars more, but the results are reliable, predictable, and therefore very much worth it because it frees you to focus on what you do best.  Just make sure your IT provider doesn’t start selling coffee.

Why the movie references to Fistful of Dollars and A Few Dollars More? Well it turns out that Clint Eastwood’s most famous line came after a bad cup of Joe. If gun violence offends you, please don’t go here, otherwise…take a three minute action coffee break with Dirty Harry.

//spk

Tweet This Delicious Digg This Post Stumble This Post

Why it is worth the cost and commotion of going for the maintenance visit, but not every time a new vaccine or pill is announced?  Because the cost/benefit equation is right for one and not the other.

A lot can be learned from Patches about the discipline of patching servers. We are occasionally asked “How often should I patch my servers?” and we get into discussions with a wide variety of customers with widely differing views on the subject. Often though, we find that it largely boils down to one’s view of the world – is your glass half empty or half full? Certainly, we need to keep systems patched to at least the minimum level supported by our software vendors, but given the cost and commotion (dare I say trauma) of the patching process, how far beyond that is necessary or prudent? If you have Internet facing assets, then clearly you want to keep those up to date with the latest security patches as soon as they’re available. But if you have private, stable, non-web assets well behind well-managed firewalls, a less rigorous approach is reasonable. There is no need or rational justification to blindly apply a patch willy-nilly simply because it’s available. Who has not been the victim of downtime because an ill-behaved patch did something that it was not supposed to do? And, lest we forget, rebooting a Windows server after patching is not always a trivial event – just ask the sysadmin of a Blackberry Enterprise server.

Remember the purpose of infrastructure is to keep running – the very namesake of this blog.  Our infrastructure does us no good when it’s down. Every patch brings with it the some level of risk to uptime. So, the obvious thing to do would be to test every patch before we apply it to a product system. Do you? Really? Every time? Or is easier to just apply the latest raft of fixes from say, Microsoft, and just hope for the best? For those of us who have to endure the regular water-boarding process of a SAS 70 Type II audit, hope is not a strategy. Not only do we have to test every patch before applying it to a live system, but we also have to prove that we did so, and that we have a defined process that meets the muster of the auditors.

This process of patching is costly in terms of time, money, and risk. So how often should we patch? Somewhere between hope and SAS 70 lies the right answer for most of us. Like maintaining your car, regular maintenance of a server is necessary to keep a system “on the road.”  This many mean spending time regularly (like an oil change) researching patches to see which of them you really need as opposed to those that make you feel warm and fuzzy, and then testing appropriately first. On the other hand, regular maintenance may not imply regular patching. If a system needs to be running the latest Windows server OS, or the application vendor forces your hand, then you will certainly be patching more often. If on the other hand, you have a functionally stable system that doesn’t change much, has been running well and isn’t the flagship of your ecommerce empire, then you will probably patch extremely infrequently if ever, and that’s OK. We’ve got a Red Hat 7.2 system here that sees heavy daily usage, has not been patched in years, and has not been hacked or had any problems over that same span of time. Sacrilege? Perhaps, but we believe it’s prudence. It could also be Pennsylvania Dutch stubbornness.

You do need a patching process, but it should reflect you particular situation and account for the nature of each of your servers. Like the Pirate Code, best practices in this arena are more like guidelines. You can spend a lot of money and create a lot of headaches with a one-size-fits-all approach.  A socialist patching approach sounds good on paper, but as you would expect with anything socialistic, it tends not to work out well in reality.

Weigh the risks and cost of downtime vs. the potential benefit of a patch.  Part of your process should include a justification phase where IT and business stakeholders have an opportunity to understand what is being patched, why it’s been deemed necessary, and what the possible ramifications are if things go awry. And, most importantly, the stakeholders should have both veto power and the power to determine the scheduling of patch activity.

Patching is a necessary evil, but it is manageable if you take the time to think through the process and come up with a practical plan that fits your business. Or, you could simply delegate the process to folks who know how to both open and close Pandora’s box.

//spk

Tweet This Delicious Digg This Post Stumble This Post

In-flight Wi-Fi is a new purple bovine that’s gaining momentum. The Wall Street Journal reported yesterday that more than 500 planes are already in our skies with wireless access, and Delta and American are leading the way. Interestingly, but not surprisingly, the airlines are having trouble figuring out how much to charge for the service. With price plans ranging from $5.95 for flights under 3 hours to $12.95 for long-haul flights, usage has not been what the airlines had been hoping for. Completely free service has been tried with great success, but as one airline found out, even a $1 plan caused usage to drop off considerably. In a telling comment that exposes a misunderstanding of trends in technology, Michael Planey, a consultant specializing in in-flight passenger technologies, had this to say about the free vs. pay model:

There’s a very substantial decline in passenger usage the minute you start charging for the service.  It really begins to invalidate the model on which this service is being built for the next 10 years.

I don’t know much about Mr. Planey, but I hope he’s wrong about the model for the sake of those building the systems.  If he’s right, we need to avoid investing  in companies building Wi-Fi for the airlines.  Why?

We all know Moore’s Law, but less well known is that Caltech professor Carver Mead was the first to focus on it’s economic corollary:

If the amount of computer power for a given cost doubles every two years, then the cost of a given unit of computing power must halve over the same period.

We can substitute the words “storage” and “bandwidth” for “computer power” without invalidating the claim. Storage is doubling in one year, and bandwidth every nine months.  That’s why you can ditch your TiVo in favor of Hulu. The upshot of all this is that the cost of computing power, storage and bandwidth is getting cheaper and cheaper by the day, to the point where it will almost be too cheap to meter.  (For an outstanding in-depth analysis of this topic, add Free, by Chris Anderson to your reading list.)

So any revenue model that is built on charging for these commodities that continue to fall in price is, as Al Gore might say, a risky scheme.  Moreover, by charging for Wi-Fi access, the airlines are missing better revenue opportunities by forcing people to stop and ask the basic question we all ask when there’s a price tag of any amount:  “Is it worth it?” Is it worth opening my wallet to pay for a few hours of internet access? The airlines have already proved that any price lowers usage dramatically. So why not  just give it away, and use it as an opportunity to connect with their customers to up-sell or cross-sell them on other services? Don’t force your customer to incur the mental transaction cost of deciding whether to purchase, particularly when you have a captive audience for a few hours. There are oh so many ways to make the Wi-Fi service free and yet profitable at the same time.  “Give ‘em the razor” as King Gillette did so successfully.

So what’s the application of all this in our IT shops?  Sort of the reverse, actually. While computing power, storage and bandwidth are getting cheaper, they are certainly neither free or cheap enough to ingore.  They occupy monetary space in our capital and operating budgets as well as three dimensional space in our data centers. From a financial standpoint, many of us are not in a post-scarcity environment where we can afford to let the user community run wild. If we make all of the resources available for free, they’re going to indiscriminately use every last cycle, byte, and bit. For example, Google used to offer free snacks at their on-site conferences, which resulted in half-eaten power bars and bags of chips strewn all over the room, illustrating that people tend not to care about things they don’t have to pay for. But if it costs something, be it monetary or accountability, the user will automatically stop and ask “is it worth it?”

Am I suggesting charge-back systems? That’s certainly one way to go, but impractical for most of us.  (Ironically, charge back systems will be de rigueur in cloud/utility computing). If cost and budget are issues, and you’re up against your resource limits, then simple accountability is a better strategy. Get some tools and actually measure your resource usage. Sounds obvious, but we see few customers do it. There are plenty of low or no-cost tools to measure storage and bandwidth usage, so it’s within everyone’s reach to do so. Then comes the less pleasant part: go to your more aggressive resource users and really challenge them to justify what they’re using. As Pavlov discovered, they will eventually respond appropriately, if for no other reason than to avoid the pain of future visits.

“Free” in the right context (marketing) can be a wonderful revenue generator, but “Free” in the wrong context (IT) tends to encourage wasteful habits, and we ought to be good stewards our of business resources regardless of their cost.

//spk

Tweet This Delicious Digg This Post Stumble This Post

(To protect the super secret location of our low profile facility, the arrow above is not actually pointed to our exact location   ).  A mere 35 feet to to my left is 15,000sf of raised floor with customer production infrastructure humming along quietly.  Am I worried about an outage?  Not in the least. Why? Is it because we have a lightning suppression system deployed or because we have 3Mw of generator backup?  Is it because we have paralleled UPS units and expensive battery strings?  It is because we have at least one spare of every key component waiting in standby mode?   None of the above. I’m not concerned because we TEST all this stuff, and we do so religiously.

Our generators are typically run weekly, and tested under full load regularly. Our switch gear is regularly maintained and tested by professionals.  Do you trust your switch gear enough to pull the plug on the whole building?  We do, and we test that too.

If you’re running infrastructure that you consider critical to your business, how comfortable are you that you’ll stay online if  lightning takes out the utility pole across the street?  Is your equipment protected  from the electrical surges that summer storms bring? Really protected?  If you’re using server-room class UPS units, such as the APC SmartUPS rack or free standing products, have you checked the health of those batteries lately?  How about the load on those units?  Has server sprawl quietly overloaded them? Will you stay up long enough to shut down gracefully or transfer to generator?  If you’re not fortunate enough to have a generator, how long can you afford to wait for power to come back on?

Lots of obvious questions to be sure.  But we find that in small to medium sized business,  they are often tragically ignored or neglected.  We know because these businesses in our area order their UPS units and parts from us, often after the worst has happened. When you discover that 92% of the businesses in the US are in fact 1-99 employees in size, neglected power infrastructure is a more widespread problem than you might imagine.

If you have a really big shop, it’s likely you have all of your power gear under maintenance and test it regularly just like we do, but if you’re a smaller shop, it would be good practice to begin regularly checking the health of your backup power gear and testing it. Today would not be too soon to start.

We care about all this because it’s par for the course if you’re running an enterprise-class data center as we do, but quite frankly it’s a pain in the butt if that’s not your business. If you’d rather focus on your core business and not worry about lightning storms, you could instead put your gear here. Outsource your power worries to folks who take them seriously and are well equipped to handle them.

//spk

Tweet This Delicious Digg This Post Stumble This Post

After spending several days this week listening to the major players in the industry outline their current offerings and future plans, it was quite obvious that what I was talking about back in April was not caused by lack of sleep or hallucinogenic substances:

A short history lesson tells us all we need to know about cloud computing.  In the 1800’s  power generation was the responsibility of  those who needed it. Be it steam, water, or electricity, if I had factory with electrical machinery and lights, I had to generate my own power, and if you needed power, so did you. And both of us had the hassles of building, operating, and maintaining a power generation infrastructure which, by the way, was not our core business.  Power was necessary to the operation, but it was not the product or service we delivered for profit.

Eventually Edison and Westinghouse figured out how to transmit electricity, and entrepreneurs realized if they could build a Really Big Generator and implement a delivery method, they could sell power to industrial users. The case from the entrepreneurs to business was clear: “Let us worry about the hassles of generating power so you can focus on your core business, and oh by the way, it’s going to cost a lot less than doing it yourself.”

Fast foward to the present…has the light just come on (pun intended)? Cloud computing is nothing more than the name-du-jour for the centralization of computing resources so that they can be delivered as a utility service.  Nothing more, nothing less.

Cloud computing is here now, perhaps in nascent form, but it’s here nonetheless.

The cloud is not new technology we’re going to go out and buy, per se.  It’s not in a box, and it doesn’t have a part number. It’s more of a paradigm shift back to way things used to be in the Golden Age of the Mainframe (which never died by the way…according to people who enjoy crunching the numbers, IBM’s System z sales last year were in the $3.5B range).  You can read the popular Wikipedia page on cloud computing, but I would suggest you think of the cloud more as a way that technology is delivered, much like electricity or water. You only consume the goods delivered rather than first having to generate or pump them yourself.

From a cloud provider’s standpoint, this means a virtualized and highly automated infrastructure, fast and easy provisioning, self-service, enormous scalability, a usage-based billing model, and a very granular portioning of resources. And that means programmers. Hosting companies that plan to operate cloud offerings are going to need a talented programming staff of substantial size along with a very sharp support staff to build and operate all of the plumbing to deliver computing resources to your doorstep.  Proof of this lies in the list of the brave few heavyweights that currently occupy the lion’s share of the cloudscape: Google, Amazon and Rackspace. Building these utility services is not for the faint-at-heart or low-on-cash, and the average hosting company is simply not going to be able to really play in the clouds until the Big Boys blaze the trail.

Currently, we can find cloud services offered to us in three flavors, or if you like, three different levels of abstraction:

Infrastructure as a service (IaaS). This is closest thing to hosting as we knew it before today – computing and network hardware offered as a utility service, but essentially without limits or long term commitments. Rackspace’s current cloud offerings are of this type. Here, the hardware layer is abstracted away, leaving you to worry only about the operating system and applications.

Platform as  a service (PaaS). PasS takes the hardware layer and adds the operating system and a development environment (software APIs), which you then use to develop your applications. The development environment ostensibly hides all the details of where your code executes and how your data is stored and protected.  So at this level, in addition to the hardware layer, the OS and the development environment are now effectively abstracted away.  Google’s AppEngine is a prime example of PaaS.  You may have already surmised a possible evil in PaaS.  If you develop an application on a particular vendor’s proprietary cloud platform (e.g. Google), they’ve got you locked in to their service, and there is no small amount of chatter going on about this in the industry. The open community is crying for an industry standard set of APIs, while the big players are fighting to establish dominance with their proprietary systems.  Obviously, it’s best to stay on the sidelines here until the dust settles.

Software as a service (SaaS). With SaaS, everything is abstracted – you are simply presented with a user interface for an application. Salesforce.com is the premier poster child for this model.

Cloud computing is becoming a deeper and more pertinent topic every day, and we’d do well to begin keeping a sharp eye on it. Cloud is not necessarily an either/or strategy. For small business, and to a large degree medium business, the server room will eventually disappear into the cloud, and that will be a blessing for many. For the enterprise space, however, cloud services  will be just another tool in the arsenal to solve business problems along with, dare I say it, the mainframe.

If you work in IT for a small or medium-size business, there is a career signpost up ahead in the clouds.  Make sure you don’t ignore it.

//spk

Tweet This Delicious Digg This Post Stumble This Post

Quite surprising, isn’t it? Before you cry blasphemy, read the disclaimer from that comes with the graph:

We realize that many readers of this e-mail are going to say our level of virtualization (images per server) is significantly higher than MBA’s.  That may be true for the servers that you have virtualized.  We obtain our average by dividing the number of images for all servers in a platform by the number of servers supporting the platform. Our best participant for each server platform averages approximately 2.5 images per servers as shown below:

Windows – 2.57
UNIX – 2.46
Linux – 2.52.

Indeed, our own shop is much higher than this, 6.63 to be exact, but it’s important to note that they’re throwing in the non-virtualized servers still on the floor, which one could easily be tempted to overlook. Clearly, there are a lot of shops out there that haven’t yet ventured into the deep end of the virtualization pool. It doesn’t look like the world is going to reach Gartnerian Nivrana on schedule, but there does seem to be compelling evidence to continue moving in that direction. Consider these numbers, also from MBA:

From 2007 to 2008, the percent decrease in the average cost per server image was:

Windows – 8.8%
UNIX – 8.1%
Linux – 10.2%

Our current best participant for each server platform is:

Windows – $9,031
UNIX – $13,965
Linux – $10,827.

This graph makes it quite clear that costs per image are dropping. How so? Unless you just moved your shop to India, your labor costs are the same as last year or more. Facilities costs (power, cooling, space) are going up, not down. Taking a look at the Windows category above, hardware price decreases alone could not have accounted for an 8.8% average decrease per image – it seems like way too much. Chairman Steve hasn’t offered any fire sales on Windows in recent memory, so it can’t be that either.  What gives?  The decreasing cost of hardware is certainly a factor, and we would expect to see the biggest savings in the pay-for Unix world because it’s not running on commodity hardware. Yet regardless of platform, the savings are apparently quite substantial. There has to be something more going on here. Could it be correlated to the rise in virtualization?

The folks at MBA tend to factor in everything when they calculate costs, including the kitchen sink. Software and hardware acquisition costs, facilities costs, the cost to virtualize – it’s all baked into the numbers. Could it be that as the level of virtualization increases, even just fractionally, the amount of annual savings across the enterprise increases significantly?  Looking at our own shop, we have roughly 48 Windows images trundling along. Let’s suppose we had increased our number of images per server just .08 as shown above from 1.27 to 1.35.  Our annual savings would seem to be $81,600.

Hmm, that seems like financial voodoo, you say. Perhaps so, but how do we explain the huge savings?  New hardware avoidance is certainly one way to generate numbers this big, and reduced software costs might also be in play depending on how your software is licensed, both of which are natural outcomes of virtualization.

Without seeing a more detailed breakdown of the numbers it’s hard to be absolutely dogmatic about these savings, but it’s an interesting thought to mull over.  Could there be onging opportunity for significant cost savings by continuing to crank up the virtualization factor a few clicks every year, or is there a practical upper limit?

//spk

p.s. See you in a few weeks.  It’s vacation time!

Tweet This Delicious Digg This Post Stumble This Post