Wednesday, April 30, 2008

Postcards from day #3, Uptime Institute's Green Enterprise Computing Symposium

Today is the last day at the Uptime Institute's 2008 Symposium, and while it had an abbreviated schedule, it might have been the most information-rich day of all.

Early morning:
We opened with 4 perspectives on public-sector data center work:

On Products:
Andrew Fanara (of the EPA Energy Star program) gave an overview of their progress toward an Energy Star spec for computer servers for Tier-I/Tier-II data center use. The first version of the spec has already gone out for review/comment; the 2nd draft is dues mid-June; the 2nd stakeholder meeting is set for July 7 in the Seattle area; the final version of the spec is targeted by the end of CY2008. Andrew expects the spec to focus on 5 areas:
1) Net AC/DC power efficiency
2) Standardized reporting/polling
3) Power and temperature reporting
4) Idle power levels
5) Power management and virtualization “hooks”

On Buildings & data centers:
Michael Zats (of the EPA buildings division) then went on to describe the Energy Star Labeled Buildings program, and plans to extend the ratings system to data centers. There are currently 1,700 partners leveraging the energy star for buildings program, representing 11 billion square feet of commercial/industrial space. Over 70,000 buildings are currently measured/tracked, and 4,000 buildings have the Energy Star lable (representing the top 20% of the population). Michael is not spinning up a similar program for data centers, and asked the audience for volunteers – EPA needs at least 125 data centers to populate their comparison data base (with 12 months of data) to begin the process… and if you’re interested, materials can be found on the Energy Star Site ; additional inquiries can be made at

On Data Center Technology:
Paul Scheihing of the US DOE reviewed their Save Energy Now program and their DC-Pro data center assessment too plans. Their goal is to drive a 25% reduction in energy intensity for data centers. They are/will be providing Tools, training, information, standards and assessment tools to the public. But what’s really great about this initiative (far ahead of any private-sector initiative, as far as I can tell) is that it’s got these multiple resources. The tool not only helps you rank the overall efficiency of your data center against others (confidentially, of course) but will provide tons of input/ideas as to where to look for savings and efficiencies. (I actually grabbed Mike Zats outside the hall, and he spoke highly of the DOE’s tool, almost to the point where it was too sophisticated). Paul also made mention of DOE’s sponsoring some technology demonstration projects, but I have yet to see how/when/where these will be presented.

On other public sector work
Ray Pfeifer of the Silicon Valley Leadership Group’s Energy Efficient Data Center Project then gave an overview of their initiatives, with a goal in CA of a 15% reduction in industrial sector consumption. (Cassatt, I might add, is an active member of this project).

Late Morning:

There was then an insightful panel covering external work being done by ASHRAE, SPEC/SPECpower, and the Green Grid – and then, somewhat of a treat: Tosh Shimizo, GM of IT and new business with Tokyo Electric Power Co. gave an overview of his company’s data center facilities and level of power grid reliability. They provide 64GW of power to Tokyo, and have only 3 mins of downtime/year… far better than what is found in the US, the UK, or France.

The second treat was Will Forest of McKinsey & Co. presenting a fantastic data center efficiency report to the group. It’s chock-full of great charts and analyses In short, it focused on improving Power Capacity planning, power efficiency, and asset management. Organizationally, he also advocated the “one throat to choke” approach to making the CIO accountable for the TCO of the entire data center (assets + facility). McKinsey’s recommendation was CADE (Combined Average Data center efficiency) essentially Facility efficiency x Asset efficiency. Deeper-down, facility efficiency is composed of facility energy efficiency * facility utilization, and asset efficiency is composed of IT energy efficiency * IT utilization. Simple…. But I suspect there will be disagreement on units…

Finally, for me, Christina Page, Yahoo!’s director of climate and energy strategy gave a lunchtime presentation on curbing carbon emissions, and what Yahoo! is doing to become carbon-neutral. The big takeaway: not all kilowatts are created equally – be careful *where* you use them (is the source coal? Nuclear? Wind? US? India?) and *when* you use them (peak? Off-peak?). These factors are as important as the net kW that you consume.

Thanks for reading. Now it’s back to the airport…

… and don’t forget the theme: measure, measure, measure.

Tuesday, April 29, 2008

Postcards from day #2, Uptime Institute's Green Enterprise Computing Symposium

Here I am on Day #2 of what is otherwise an intense immersion into data center management war stories, best practices, studies and standards.

The morning
I do have to note the slow start today bsy ome vendor-sponsored Keynotes. It opened with Mark Thiele, VMware's Director of Business Operations, giving IMHO a distinctly plain-vanilla overview of virtualization's benefits. Yawn. But things picked up with Henry Wong of Intel (and a frequenter of meetings such as this, the Green Grid, and others) giving a somewhat more interesting retrospective -- he had found a presentation given to this group in 2006 called "picking the golden nuggets", 5 suggestions he made to the group a few years ago:
  1. Vitualize/consolidate
  2. Use power-saving features
  3. Turn-off comatose servers
  4. Eliminate bloatware
  5. Improve site infrastructure energy efficiency
On one hand, it's clear that this is "timeless" information; kudos to Henry. But on the other hand, I ask where have we come in 2 years? Not so far. There's lots of inertia in the system... But he also gave a really interesting perspective on "energy-proportional computing" - whereby you have a completely linear relationship between power consumed, and computing output. A similar proposal was put forward by Google engineers some time ago. But maybe Intel can make it happen.

The third, and most interesting am keynote was from Dave Anderson, IBM's Green Architect. He advocated a completely holistic approach to managing power and energy efficiency in data centers, and predicted that Cloud Computing is ultimately where IT is evolving to. Resources will be dynamically allocated -- with data centers compared to orchestras, where "ensembles" of similar equipment being purposed and re-purposed for like tasks. This is the only way we'll reap breakthrough IT efficiency. Kinda sounds familiar...

Next was a panel on "The business case for IT energy efficiency" moderated by Jon Koomey (of Stanford U as well as Lawrence Berkeley Labs) and Debra Grove, and including Andrew Fanara of the EPA/Energy Star program, and representatives from Allstate Insurance, Yahoo! and HSBC. Talk was wide-ranging, and included the (highly likely) legislation that could help cap/trade carbon emissions, and closing the social division between IT ops and Facilities. And I believe that all agreed that "green" ultimately has to result in $$ savings, not just goodwill.

Following lunch, I gave a talk on "Is it bad to turn off servers - and other myths hurting energy-efficient operation of your data center". Long title, short message: "Conventional wisdom" in data centers is confining our thinking, and limiting breakthrough efficiencies. If we look to combine efficient equipment with efficient operation of that equipment, then we can move beyond incremental efficiency increases.

I then attended an insightful presentation on Energy Star computer power supplies by Chris Calwell of Ecos, and a SPECpower overview by Klaus Lange of SPEC. These guys are the ones doing the heavy-lifting for our benefit. SPECpower had been initiated back in 2006, and is an elegant approach to rating compute load vs. power consumption. It's so easy to run that you can even do it at home -- even though vendors are expected to provide customers with a series of data for their own products. Finally we have a generally-accepted way to benchmark server power consumption under load! Similarly, studies are being done on power supplies for Tier I servers by Ecos (on behalf of Energy Star) for a forthcoming Energy Star ratings. It became clear that many power supplies are likely over-sized -- and, at low loads, are (1) terribly inefficient, and (2) have a seriously-degraded power factor. But there are products out there with superior (90-95%) performance at nearly all power levels.

We closed today with a reception as well as a series of Uptime Institute Green IT awards for a number of innovative projects - which I believe will be posted to the Uptime Institute Awards site shortly.

Oh - and a somewhat retold refrain heard from a number of participants today: "what are we doing when we get back to the office? measure, measure, measure..."

Monday, April 28, 2008

Postcards from Day #1, Uptime Institute's Green Enterprise Computing Symposium

Well, here we are in Orlando at the Uptime Institute's 2008 symposium on Green Enterprise computing. By Ken Brill's (the Institute's executive director) keynote estimates, we have 442 participants from 13 countries, and roughly a 50:50 split between technology providers and data center management.

Morning Keynote:
Ken Brill and Pitt Turner both gave pretty compelling keynotes -- first, by challenging the participants to providing a 50% energy reduction for data centers in the next 36 months. The possibility of this was driven home by the fact that Amazon Web Services can already deliver a CPU-Instance/Hour for $0.10. Could the rest of us? But more on all of this later.

Ken pointed out that at the current rate, there will be 15 million servers running by 2010, with that number doubling every 5 years or so. And what was really riveting was that the largest 1/3 of membership's data centers is growing at 27% CAGR; and even the average data center growth for membership was 12% CAGR. Clearly there is a massive construction (and energy use) boom in data centers in the U.S. This spells crisis.

Next, he focused on the excalating power/OpEx costs of operating a $2.5k server -- with much data taken from Jon Koomey's excellent studies. In a Tier II data center, that server's OpEx costs for power topped $5k; in a Tier III data center, OpEx costs were $7.7k, and in a Tier IV environment, power OpEx costs were a whopping $8.4k... all with power costs far outstripping the capital costs.

And so, Ken's request to the group: the only way to obtain a breakthrough 50% improvement on data center power consumption was to look at IT Operations, as driven from top-line management... rather than to pursue it from incremental facilities-based improvements.

Morning sessions
Ironically, two of the first three sessions of the morning - one on DC power, the other on power & cooling - focused on infrastructure improvements. And this, in my opinion, set the somewhat sobering theme for the day: while the participants would *like* to get to breakthrough energy strategies, the vendors are mostly stuck in pursing only incremental improvements. (Example: why would a leading power/cooling vendor focus on talking about hot/cold aisle containment, when they could have included forward-thinking adaptive management technologies that place cooling where it's needed while encouraging compute workload management?)

Next I attended a reasonably interesting roundtable panel titled "What can IT executives to Right Now to Increase Energy Efficiency?" co-chaired by Will Forest and Matt Stansberry. It became clear pretty early that the issue-at-hand wasn't one directly of technology. Rather, it was social: IT is incentivized on Uptime and on service delivery... not on power/energy cost. So, the concept of chargebacks, cost-transparency, and the "shining a flashlight" on cost are some of the places to begin to change behavior.

Afternoon Sessions
There were a number of other interesting breakout sessions including mine, chaired by Jon Koomey titled "Enabling Power Features to save 30+ percent". Jon kept this pretty focused, and we had a lively session with representives from HP, Microsoft, IBM, Visa and me from Cassatt. In the end, I think the room agreed that power-management and server hibernation features could indeed be leveraged to save significant power, but had to be implemented wisely, with potential tradeoffs between efficiency and high-availability. But, in a number of cases, this approach was highly warranted.

Other fun diversions today:
  • Chatting-it up with the very insightful bloggers Dave O'Hara of the Green Data Center Blog and Debra Grove of Grove Consulting (also with an excellent overview of the day in her blog)
  • Taking a tour of Rackable Systems' ICE Cube Data Center container
  • Sharing a drink (or two or three) with Kevin Heslin of Mission Critical Magazine

Tuesday, April 22, 2008

Cassatt IT Energy Survey & Roundup

Today we at Cassatt announced our very comprehensive survey of IT energy Efficiency, and released a companion White Paper as well. Although I initially blogged about initial findings back in January, since then we've been doing analyses and going back to respondents and collecting 1st-hand 1:1 data and quotes that elaborate on the raw data. (I might add that excerpts of these interviews are included in the white paper).

The Survey was a really wide-ranging look at the IT industry - who's taking what initiatives, where are the industry pain-points, what sources of information are trusted, etc., and we found a few surprising data points as well. Such as
  • Where most enterprises go for "green" IT information may not be where you think
  • The degree of energy waste in different departments is more acute than you might believe
  • The willingness of some professionals to take new/different approaches to power management is more aggressive than we believed
More Surveys:
I've been spending time researching a number of other very excellent data center surveys being conducted, and you should look at them together to provide a rounded-out picture of the state-of-the-industry. Here goes:
One last note: a significant finding in our own survey was that most IT managers still turn to their equipment vendors for guidance on energy efficiency -- rather than to independent sources. So my final plea is that you look hard at the surveys, conclusions and recommendations that come from independent sources as much (or more) than the data that comes from the big vendors. Besides the surveys above, fantastic work is also coming out of The Green Grid, The Uptime Institute, as well as from the EPA.

Monday, April 21, 2008

Mixed-Virtualization environments?

I was directed to an article by Adam Trujillo at TechTarget titled Is the mixed virtualization vendor data center possible? (I believe the original genesis of his article was from SearchServerVirtualization's Jan Stafford). His thesis:

I recently floated the idea of implementing a mix of virtualization products in your data center as a way to better customize your virtual environments. The query was part of a larger discussion I wanted to get going about how VMware will compete when Hyper-V is generally released. I threw out the notion that data center managers might use, for example, Hyper-V for end-user file servers; VMware ESX, for apps that require dynamic load balancing, sophisticated disaster recovery and migration; and Xen for commodity Linux boxes.

The idea behind that supposition was to match your enterprise investments to appropriate workloads because, let’s face it, running everything on ESX is going to be expensive compared to other options. Big deal if you don’t get ESX-level features because you may get enterprise level features on silver-medal products.

A Supposition? No - more like a reality. If you look around, there is already VMware pervasive in the market, Parallels Virtuoso staking a claim in the MSP/hosting market, Xen making inroads in the Linux market, and other folks like Virtual Iron, Oracle and others about to pounce. Add-in the fact that IBM, HP and Sun each have their flavors of containers/partitioning. Oh... and don't forget that sometime soon you'll see virtual appliances (like rPath) that have embedded VMs from various vendors. And then, of course, Hyper-V.

Inside of 12-18 months, it will be a fact that data centers will have multiple virtualization technologies.

Strategically, organizations will need to prepare themselves for this - rather than create "virtualization silos." As you would expect the Invisible Hand to act, there are already a number of vendors looking closely at this space -- multi-virtualization management. Cassatt clearly comes to mind - but there will be others.

The next data center management "high ground" will certainly be those platforms that can leverage/control the multiple VM technologies out there, and dynamically control workloads, failovers, scaling, security, etc., including managing those machines that are not appropriate to virtualize at all. And my prediction is that the incumbent management platforms (i.e. OpenView & Tivoli) will not be in a position to do so. Look to the agile startups for that.

Sunday, April 20, 2008

More of "Open Source Music"

Wow. It's validating to read Thomas Dolby's Blog (TED's musical director) where he's remixed Radiohead's "Nude". Radiohead is going a step further than anyone by offering "Stems" to the song on iTunes, and setting up a remix contest.

Why validating? This isn't just a remix where you mash-up two songs - this is the artist offering-up original unmixed tracks... It's exactly the open-sourcing of music I predicted on my March 4 blog.

Boy, would I love to see other artists begin to do this -- It could be just what the recording industry needs to spice-up the market. Turn fans into an extension of the artists themselves!

Monday, April 14, 2008

1,000+ servers, and hitting the power ceiling

I've had a number of conversations with a really insightful Architect with a major healthcare provider. He runs over 1,000 servers (each, I might add, consuming 200+ Watts) to deliver applications to hospital desktops/kiosks. On top of what he's using, he's got a built-in 15% buffer for peaks & failures. Oh -- and did I mention a remote failover site where this configuration is essentially duplicated?

And here's the kicker: He knows that during nights/weekends, user demand drops off precipitously.

Conventional wisdom has it that *all* of these servers should be on and ready 100% of the time. Hmmm. That's 400,000W (think 400 blow driers) and probably another 400,000W or so to keep them cool.

Fortunately, our friend realizes that even during peak hours, not all of these servers need to be on... and off-peak, many fewer. What he *does* need is to ensure availability -- that at any time he has headroom for a momentary peak, and that he can bring on more at a rate that keeps up with demand.

This is not a solution that virtualization, or more efficient power supplies can impact; but by breaking with conventional wisdom and power controlling the servers based on usage/demand, we hope that his organization will save $$ and even regain precious power/cooling headroom.

Sunday, April 13, 2008

Repurposing Technology + Power of YouTube

I just came across this video by Johnny Lee. In it he demos his Wii Remote hacks... taking a standard $40 game part and adapting it to power things like a digital whiteboard, a multitouch display and a head-mounted 3-D viewer. (Other projects by Johnny at Carnegie Mellon U are here)

His brilliant tenet is this: Millions of $ have been spent developing foundational technology like the Wii Remote -- which can then be applied to dozens of *different* applications. Factor in the ability of YouTube to share ideas, and you've got a fantastic acceleration of technology and solutions.

Friday, April 11, 2008

Differing approaches to IT power reduction

It's been pretty clear from a number of surveys, market research & industry orgs, that there are a variety of approaches to curbing IT Energy Consumption. And as you'd expect, there's somewhat of a tradeoff between complexity vs. immediacy.

First, everyone agrees that there are quick-hit actions to be taken around cooling (covering blanking panels, clearing air ductways, etc. etc.). These are super-simple, super-quick, and have a modest payback. To be sure, there are more complext/long-term cooling initiatives (e.g. air-side economizers, higher-efficiency chillers & CRAC units, etc.) but I'll skip these for the moment.

Second, there is the ballyhooed consolidation initiative. Clearly this is a high-impact approach to $ savings, but it's also going to be a year + before the full savings is achieved. (FYI, it took Sun Microsystems 12 months to complete their consolidation, and Gartner Research found that half of survey respondents expected that a consolidation project would take between 7-18 months). Also, assume that once you make the investment, it will take a year to reduce your server count. And, because most hardware is on a 3-4 year depreciation schedule, it could be as long as ~ 4 years or more before those machine expenses come off the books, and the project has complete payback.
Third, and the quickest-hit approach, is simple server power management. Implementing intelligent server shutdown (based on schedule, load, etc.) takes only a few months at most, and doesn't involve any SW configuration changes. Payback can be < 1 year. Granted, you don't remove any machines, but the power savings are essentially immediate -- and if you still want to go ahead and do a consolidation project later, you can continue to power-manage the consolidated machines.
I expect that we'll be seeing more from DOE, EPA/EnergyStar, Green Grid & Uptime Institute on these tradeoffs -- between impact/complexity, and between efficient equipment and efficient operation of that equipment.

Wednesday, April 9, 2008

Cassatt again cited by Gartner as "Cool Vendor"

John Rath's "Data Center Links" picked-up on Gartner's highlighting of Cassatt in their "Cool vendors in Sourcing and IT services 2008" report this year. This is primarily due to partnering with Tata Consulting to provide a
"...solution, built on TCS' best practices and proven methodologies, is enabled by Cassatt's software, which combines goal-driven automation with virtualization control to help customers create a real-time IT infrastructure that can achieve desired application service levels. TCS and Cassatt are now on their way to creating a highly visible Next Generation Infrastructure lab that will host Utility Computing enabled by Cassatt as a key solution station.
Despite the fact that John once believed that Cassatt's products were "just plain silly", it would appear that he's come around...

And, BTW, he writes an awesome data center blog, e.g. recently wrapping up Data Center World, Green Data Center Topics, and a number of newsworthy blog feeds.