Where the Server Industry Went Amiss

I've been doing an analysis regarding how "complexity" has evolved in the datacenter. Fundamentally, just why is it so hard to configure & provision new (physical) servers? Why is clustering inherently so complex? Why do we have data networks, storage networks and management networks (all distinct, I might add). How come we have all of these layered management systems?

OS virtualization has massively simplified complexity at the software level by abstracting-away the machine-level CPU commands, and has even contributed to simplifying networking between virtual machines. But we're still left with physical complexity at the physical I/O, networking and control levels - the other physical piece-parts (KVM ports, NICs, HBAs, etc.).

Ultimately, all of this complexity gradually resulted from incremental server hardware evolution… the motherboards to be exact. Way back when the computer industry was just getting started, motherboards harbored a simple CPU and remedial I/O (e.g. an audio jack to a cassette tape for storage...). But as processors got more sophisticated and datacenter environments grew, CPUs were integrated with more complex I/O (e.g. Network Interface Cards) as well as with storage connectivity (e.g. Host Bus Adaptors). Plus, there was usually a local disk, of course.

This meant that the server retained static data, specifically things like I/O addressing and storage connectivity naming, not to mention data on the local disk -- resulting in the server having a static “state". Usually the local network had state too – ensuring that the IP and MAC address of the motherboard were attached to switches and LANs in a particular way. Add to this the fact that with critical applications, all of these components (plus naming/addressing) were frequently duplicated for redundancy.

This meant that if you had to replace (or clone) a physical server, say because of a failure, you had to re-configure all of these addresses, names, storage connections and networks – and sometimes in duplicate. This resulted in lots of things to administer to, and lots of room for error. And frankly, this is where fundamental “data center complexity” probably arose from.

In response to dealing with failures and complexity, vendors developed special-purpose clustering and failover software – necessarily closely-coupled to specific software and hardware – to provide the re-assignment of state to the new hardware and networking. This software often required hand-crafted integration and frequent testing to ensure that all of the addressing, I/O, and connectivity operations worked properly. And many of these special-purpose systems are what are in use today.

Similarly, there are equally complicated software packages for scale-out and grid computing, that perform similar operations – not for the purpose of failure correction, but for “cloning” hardware to scale-out systems for parallel computing, databases, etc. But these systems are equally complex and usually application-specific, again having to deal with replicating Stateful computing resources.

So the industry, in an effort to add “smarts” and sophistication to the server – to enable it to fail-over or to scale – has instead created complexity and inflexibility for itself. Had the industry instead defined I/O, networks and addressing logically, then the way we assign/allocate servers would have been forever simplified and streamlined.

Fortunately, some technologies are being applied to somewhat revert/simplify:

• I/O virtualization appliances which logically consolidate all I/O into one reconfigurable physical card (e.g. Xsigo)
  
• Infrastructure virtualization software which logically defines all I/O, networking and switching so that any CPU-I/O-Network config. can be defined to take the place of any other CPU-I/O-Network config. (e.g. Egenera, Cisco UCS and to some degree HP's VirtualConnect)
• CPU pooling hardware/software which replace traditional I/O to make multiple physical servers act as large multi-core CPUs (e.g. 3leaf)

Unfortunately, the industry's own momentum sustains the level of complexity - most players continue to develop software products to handle/abstract the increasing complication. Nor is in the interest of the board designers & silicon manufacturers to _reduce_ the number of chips & cards associated with servers. So, we may not see a significant architectural change in stateful processing units - until the industry gradually acquiesces that there is an alternative to all of this madness.