Green Grid Technical Forum: Day #1

Since Cassatt is a new General Member to the Green Grid, I was able to attend this 2-day meeting. Today's session was quite well-attended... I estimate ~ 300+ participants in the room here in San Francisco for the opening talks and panels. Today/tomorrow appear to be engineered for members (and the press) as a comprehensive report-out of the excellent machinations of the organization and its workgroups.

The morning opened with some end-users (data center operators) such as Enterprise Rent-a-Car and AllState Insurance talking about comprehensive steps they took (and are still taking) to radically shift the efficiency of their operations. (I might interject that one of the strategies Allstate takes is to power-down idle equipment...)

But what was really interesting in the morning were a number of panels covering how the organization is reaching-out to other orgs and to government; panels featured Andrew Fanara of the EPA's EnergyStar program, Paul Sheihing of the DOE's Save Energy Now program, and Anson Wu who's helping develop the EU's CoC (Code of Conduct, roughly analogous to EnergyStar). A number of other organizations, such as Lawrence Berkeley National Labs were also present. It's good to see this degree of cooperation, but what was still left unsaid was specifically how these groups were specifically cooperating. Perhaps one answer lay in the fact that the Green Grid was still spinning-up liaison work groups -- but it's certainly on their radar.

Also really neat was the fact that EnergyStar will be creating a rating system for Data Centers, the same way that they have a rating for buildings, and that DOE will be creating an efficiency diagnostic/assessment tool ("DC-Pro") for Data Centers the same way they have tools for, say, chemical plants.

Also of special note were a number of talks having to do with metrics. Every org has them, and even more are being developed. There's alphabet soup like PUE, DCiE, DCP, SI-POM and DH-UE. In fact, nearly every organization above is proposing metrics to help characterize data center power consumption and efficiency. Finally, the question was asked regarding how to sort through who's creating what metric. I think that's one of the pivotal points -- what will be the generally-accepted set of metrics to use when assessing the efficiency of a data center? Hmmm. Maybe we'll find out on Day #2...

Huge Power Inefficiencies in Dev/Test Labs

Last week I was able to meet with the Director of Global Lab Services for a major networking company. We had a wide-ranging conversation, but here is an insightful summary that continues to reinforce my power management thesis:

• Company divisions are moving to individual P&L's (with chargebacks) and are under pressure to ID and eliminate wasteful costs
• He's overseeing the use of thousands of pieces of hardware (servers, switches, routers), being used by roughly 500 engineers (many overseas in India). Many of these pieces consume over 1kW each.
  
• Lab hardware is constantly being reserved for test-beds, checked-out, and then returned to unreserved status. But it's clear that even when a piece of hardware is part of a test-bed, it may be un-used for as much as 50% of the time.
• In labs alone, the company is paying nearly $100k per month in electricity bills
• He knows that there are massive power inefficiencies, and is looking for a solution.
  

Now, this company is not even on the Fortune 500 list (although on the F1000 list).... which says to me that there are probably hundreds of other enterprises facing the same problem. So as the industry focuses on "green data centers," virtualization and cooling technologies, *why* don't we simply look at the use rates (some would day the duty cycles) of the equipment itself? Simply using Active Power Management ought to be a quick/easy solution right off the bottom line.

And what better place to begin than in Dev/Test?

Dynamic Data Center, Dynamic Metrics

I'm happy to see that the Green Data Center Blog points out that data center efficiency metrics like PUE (Power Usage Effectiveness) need to be necessarily dynamic if/when the data center is dynamic:

"I've been meaning to write about PUE, and have been stumped in that It is defined as a metric, and in the Green Grid document referenced it makes no reference that is dynamic. In reality PUE will be a dynamic # that changes as the load changes in a room. How ironic would it be that your best PUE # is when all the servers are running at near capacity, and shutting down servers to save power will [actually] increase your PUE?

A broader explanation of Assessing the New Data Center Metrics is on an earlier entry.

If you look at Gartner's description of "Real-Time Infrastructure" (RTI), IDC's "Dynamic IT" or Forrester's "Organic IT" concept, they all point to evolutionary steps where data centers will adapt/respond to demand by re-purposing equipment in real-time. As this happens, the metrics that will define efficiency will need to track this as well.

EDS Outlines Key Data Center Energy Saving Measures for 2008

I Just came across this very validating article in the datacentres.com newsletter, Data Centre News from the BroadGroup in the U.K.

The EDS Fellows – a team of recognized IT leaders in EDS, the US outsourcing and IT company - have predicted 8 measures that need to be taken to reduce the impact of the Data Center on the environment and also prolong the use of IT components. They include:

1. Users should move away from deploying one application per server. By using server virtualization server utilization can be boosted from 15 to 20 per cent to up to 90 per cent.
2. Users should turn off their IT hardware – servers - when they are not being used. Automated operating processes should be deployed to bring on servers when they are required. [Personal note: I'm happy to see that Server Power Management made it to #2 on this list!]
   
3. Users should deploy power saving techniques which reduce server operating speeds and increase power supply efficiencies.
4. Users should reduce the number of software applications that are deployed within the Data Center and standardize as much as possible, and cut out the under-used applications.
5. Users should deploy higher density multi-core CPU’s which will as a result lead to fewer blades required per rack.
6. Users should implement a rigorous maintenance and replacement policy to improve the Data Center layout and IT configuration within the Data Center
7. Users should focus on the entire IT infrastructure operation rather than reducing the cost of computing, and should seek to measure the overall power impact of their facility. [Personal note: this is consistent with the metrics & recommendations I've found from the Uptime Institute]
   
8. Finally users should consider the implications of networked and integrated IT services, in particular to control heat, cooling and lighting in line with the use of the facility.

Data Center Energy Efficiency Survey: Redux

I received many cards and letters regarding my previous blog about the survey Cassatt ran last month - and yet I forgot to reference the raw results, which are here.

Meanwhile, I've also come across a few more sources of data and industry surveys on the state of Data Centers and IT energy use and efficiency:

• SearchDataCenters' study on Data Center construction: In 2007, 22% of survey respondents said they were involved in a data center construction project. More than 60% of respondents will be involved with construction in 2008. And only 18% of respondents will not undertake a data center construction project in 2007 or 2008...
• This very interesting study by AMD on power consumption and cooling in the data center; what makes it so intriguing are the trends it began to forecast back in 2005. Plus a more recent survey they conducted on Power and Heat Management.
• A more current study sponsored by AMD (but performed by a guy I hold in high esteem, Jonathan Koomey of Lawrence Berkeley National Laboratory) tracks shifting patterns in data center energy use across five world regions. It's a tour-de-force of estimating total power consumption around the world based on quantities of servers in use.

Survey Results in on Data Center Energy Efficiency

Here at Cassatt, we recently completed a survey of IT and Facilities professionals regarding data center energy efficiency; we were trying to get a handle on how many are pursuing what types of efficiency initiatives, why they did so, and what data informed them in making decisions.

Some results were surprising, some not. But one thing is for sure: This is a rapidly-developing area with few hard facts available as enterprises develop their own strategies. Maybe this explains why there were so many related surveys in 2007: These included surveys from Emerson Network Power and the Data Center User's Group in Sept '07, Digital Realty Trust in Sept '07, and Symantec in Nov '07. I'm sure there are many others.

Follows are some interesting factoids from our own study:

• Massive waste in Dev/Test: Looking at Development & Test labs, 30% of companies had 25% or more of their servers in Dev/Test. But consider this: Nearly 30% of those companies had over 60% of their Dev/Test servers idle during "off-peak" hours. And, fully 50% of companies had 20% or more of their serves idle in Dev/Test during off-peak. There's a lot of room for savings through Active Power Management here :)
• Hitting the ceiling: Nearly 43% of companies have a data center within 25% of their maximum power capacity, and 25% of companies are within 25% of their maximum cooling capacity.
• Motivation is still, ultimately, $: Even though we're talking about the benevolence of being "green," the strongest motivators for initiatives were still "Clear economic benefit" (68%) and "Constrained by power or cooling" (57%). But it's consoling to know that "Environmental responsibility" ranked reasonably well (39%)
• Self-serving advice?: We also asked where users look for guidance. Who came out on top? Systems vendors (49%) and Power & cooling vendors (39%). Now maybe that makes sense, but maybe it indicates the wolf guarding the sheep. Why was it that "neutral" sources like EPA and EnergyStar (29% and 33% respectively) narrowly scored below Peers/colleagues (34%)? (where do those peers get their advice, anyhow?)
  
• A little confused?: Also, an observation made by our friends at the Green Data Center Blog was that 61% of respondents were aware that the EPA recommends turning off idle servers to save energy -- but over 30% of respondents also said they didn't know how many servers are idle at any given time!
• Poor self-image: And finally, only about 19% of respondents felt that their data centers were "better than average" -- either we had a lot of pessimistic respondents, or there's a lot of room in the market for improvement :)

2007 Darwin Awards are final

I've got to say that each year, this stuff is pathetic.... it's all about folks who come from "the shallow end of the gene pool". The tally of the 2007 Darwin Awards is out, and my personal favorites are:

• (28 July 2007, Czech Republic) A pack of thieves attempted to steal scrap metal from an abandoned factory in Kladno. Unfortunately for them, they selected the steel girders that supported the factory roof. When the roof supports were dismantled, the roof fell, fatally crushing two thieves and injuring three others.

My other favorite (a vintage one, from a few years ago):

• (3 February 1990, Washington) The following mind-boggling attempt at a crime spree appeared to be the robber's first, due to his lack of a previous record of violence, and his terminally stupid choices:
  1. His target was H&J Leather & Firearms. A gun shop.
  2. The shop was full of customers - firearms customers.
  3. To enter the shop, the robber had to step around a marked police patrol car parked at the front door.
  4. A uniformed officer was standing at the counter, having coffee before work.
  Upon seeing the officer, the would-be robber announced a holdup, and fired a few wild shots. The officer and a clerk promptly returned fire, covered by several customers who also drew their guns, thereby removing the confused criminal from the gene pool. No one else was hurt. (BTW, this true story fits perfectly with one of the Darwin Awards Mottos: "Never bring a knife to a gunfight")
  

As you would expect, there are formal rules for nominations to the competition, the summary of which are:
- Reproduction - Out of the gene pool: dead or sterile.

- Excellence - Astounding misapplication of judgment.
- Self-Selection - Cause one's own demise.
- Maturity - Capable of sound judgment.
- Veracity - The event must be true.