Lighting the Path to'Lights Out'

Here are some strategies to reduce the technological and managerial complexities of reinventing your data center.

By: Galen Grumman

The twin pressures of doing more for the business and keeping IT costs down are forcing many CIOs to rethink their approach to one of their most critical responsibilities and one of their largest costs: the enterprise data center. A joint Computerworld/CIO survey of 389 IT executives shows that 63% are seeking increased efficiency from their data centers, while 60% are looking to bring in new capabilities.

“CIOs are seeing that the economics of yesterday’s data center isn’t going to work in the future,” says Michelle Bailey, research director at IDC. “You can’t have more people, more I/O and more servers with every new application,” she says.

New technologies, such as virtualization and automation, hold significant promise in changing the economics of the data center, largely by reducing both the labor and equipment needed to be maintained and optimized. The next-generation data center, promoted by vendors as “lights out,” moves away from the maintenance-and-repair model to the orchestrate-and-prevent model. In the lights-out data center, systems largely manage themselves and a smaller core of IT staff focuses on improving the data center’s ability to drive innovation and enhance the business.

But adoption of these next-generation technologies introduces architectural and integration challenges. The technologies’ varying levels of maturity will require a staged implementation approach and demand that CIOs rethink their data center staffing model. “It’s a lot to bite off,” says Galen Schreck, a senior analyst at Forrester Research Inc.

Virtualization: Promises

One new technology in particular — virtualization — can help CIOs significantly improve operations and flexibility in the data center. By separating software from the hardware on which it runs, virtualization can make the data center far more flexible and agile in five ways:

• It lets you treat multiple hardware devices, like servers or storage systems, as one device. Conversely, it allows you to separate individual physical servers into multiple virtual ones, making it easier to scale storage and applications up or down upon demand.
• It permits you to use cheaper, generic hardware and aggregate it into virtually unified systems that are more cost-effective.
• Virtualization makes it easier to handle equipment failure because you can move a virtual environment to other available hardware quickly.
• By treating hardware resources as one pool, virtualization lets you use that equipment more efficiently, so you need less equipment.
• Outside the production environment, virtualization makes it easier to set up, change and tear down test environments.

Clothing retailer Coldwater Creek Inc. is using virtualization to reduce its number of physical servers, leading CIO Michael Carper to predict that he will be able to “extend the life of the data center by five years.”

At Corrections Corporation of America (CCA), which runs about 60 prisons, virtualization not only helps reduce hardware needs, it also speeds provisioning of new servers. “It takes just hours to bring in additional capacity, not days,” says CIO Brad Wood.

Although server virtualization technology is advancing rapidly, there has been less progress on the storage side, notes Schreck. That’s because traditional large-systems vendors, such as EMC Corp. and Hitachi Data Systems Corp., don’t want to see storage become commoditized, says John Webster, senior analyst at Data Mobility Group LLC. But other vendors, such as Symantec Corp. and Kazeon Systems Inc., now offer software for heterogeneous environments that can virtualize generic storage systems and interact with proprietary systems.

Virtualization: Challenges

Virtualization technology is fairly new. Consequently, not all applications can work in a virtualized environment, cautions Kris Domich, president of data center consulting at Data Dimensions Inc. “You need to start with a feasibility assessment,” he says. Today, virtualization also requires a restart to change the environment, so it won’t work to redistribute computing and storage resources automatically on the fly.

An irony of the virtualized data center is that your staffing costs could increase even as your staffing levels decrease, says Wood. It takes more skill and experience to work in a virtualized environment than in a traditional one. As server and storage capacity is optimized, your buffer is diminished to handle fail-overs and unexpected loads. Furthermore, you have to know what resources are in use by which applications and understand which resources are now shared, since any failures or workload rebalancing can affect multiple systems simultaneously. “You need a highly skilled, highly paid staff to manage this more complex environment,” Wood says. “You can’t have just the generalist anymore.” The Computerworld/CIO survey bears this out: 44% of respondents say they expect to need a more highly skilled staff to run a virtualized data center.

The adoption of virtualization also raises political issues for CIOs. In a virtualized data center, business units may no longer have servers with their names inscribed on them. “Virtualization gets a lot of resistance [from business departments] because it’s shared services. People don’t play well in the sandbox together,” IDC’s Bailey notes. “A lot of people like to own their infrastructure.”

Even within the IT department, virtualization can sometimes bruise egos, says Schreck. “Virtualization cuts across ownership boundaries,” he says, leading each group to compete for resources and campaign for its own services. A related approach to virtualization is grid computing, in which multiple servers are joined together not as a single virtual machine, but as a coordinated set of computational engines, like a yoked team of horses.

Merrill Lynch & Co. has recently deployed a grid for its derivatives business, reducing the number of servers needed to handle the same computation load. “If there’s excess capacity in the grid, you can provision it in minutes,” says John Cislo, director of client infrastructure solutions. It makes sense to consider a grid rather than virtualization when your computations need to use full server capacity (not share it with others) and handle lots of floating-point calculations, which are common in engineering and financial analyses, he says.

Deploying a grid raises the same political issues that virtualization raises, notes Cislo, because it, too, is a shared service. And grid computing comes with other issues. One is the need for a more homogeneous server environment, which can be hard to maintain as vendors change their hardware over time. Another is the need to deal with small, less established vendors, since the commercial versions of the technology are still new. Third, setting up a grid is expensive, so it requires that you convince management that the investment will pay off as more business units take advantage of the system, says Brad Kurtzman, director of equity-linked products, the first business group at Merrill Lynch to use grid computing.

Automation: Promises

Automation is another area where new technology can help make the data center more efficient. Imagine that each time an employee is hired, is promoted or leaves, all application and network access systems are updated to reflect changes in permissions or roles. Or that as storage is added to the data center, application and file servers, disaster recovery systems and storage management policies adjust to use the new storage capacity. Or that as application servers are added, they could simply be attached to the network, and the operating system, applications and network connections would be installed automatically. Today, each system must be updated by an IT staffer, even if you have a centralized management console application to simplify the effort.

However, automation technology is even less mature than virtualization, reports Data Dimension’s Dominick. Although many automation tools are available, they tend to focus on specific subsystems, requiring IT staffers to manage their interactions and often develop their own tools to integrate them, says Forrester’s Schreck. “You need to get the integration before you can provision,” he says. The result: “Maybe about 15% of North American firms have started to make this transition,” Schreck notes, referring to an unpublished Forrester study. According to the Computerworld/CIO survey, 40% or more of respondents identified immaturity in each of four automation-related technologies as barriers to achieving a lights-out data center: policy-based, self-configuring systems for automated system adjustments; management and monitoring tools for security and compliance needs; standardized interfaces so management tools can work together; and support desk and system management tools that can work together automatically.

As automation products improve, this should change. “Today, they’re mostly point tools, but I see holistic umbrella tools emerging,” IDC’s Bailey says. “We see the ramp-up in the next 18 to 24 months.” In the meantime, enterprises should standardize their IT processes using techniques such as IT Infrastructure Library and Six Sigma to ensure that the data center has the right proc-esses to get ready for automation, Schreck says.

Automation: Challenges

As with virtualization, automation requires a more skilled, expensive IT staff. Troubleshooting becomes exponentially more difficult when the server that has failed is actually spread across several physical servers. And optimizing or just fixing problems in automated systems that interact with other automatic systems becomes quite difficult because so many decisions and assumptions are now hidden inside the software.

“The skill sets change dramatically,” says Coldwater’s Carper. He’s had to change his IT staff mix by hiring more-skilled, process-oriented staffers to focus on core systems management and automation so that errors and inefficiencies don’t ripple throughout the data center. He also has automated provisioning of new hardware and software wherever possible, so the IT staff doesn’t spend nearly as much time loading software onto machines. For example, if a laptop needed to be replaced, Coldwater’s IT staff would just drop off a new system to the user. All the user data, as well as applications and network access permissions, would be automatically restored over the network in an hour or two. That’s possible because Carper invested in automated backup and provisioning systems. After revamping Coldwater’s data center over the past two years, “we have fewer, more highly talented people,” he says. But the next-generation data center’s demand for a more highly skilled IT workforce presents a staffing risk for CIOs moving in that direction.

Consolidation

The third leg of a next-generation data center technology strategy is consolidation. Multiple data centers can eliminate redundancies in equipment, staff and telecommunications technology. Consolidation within a single data center — reducing the range of equipment used and using higher-capacity equipment — achieves similar results by simplifying technology deployments and process changes.

Incremental technology advances, such as the use of blade servers, denser storage and multiprocessor CPUs, have made it easier to consolidate data centers by increasing capacity in the same or less physical space. “We get economies of scale, with no additional management needed,” says Mike Kistner, CIO of Pegasus Solutions Inc., a payments and reservations service provider for hotels.

e maturation of remote monitoring and management tools lets central IT staff manage satellite offices confidently, notes Data Dimension’s Dominick. “But be careful not to get too much physical distance between the data center and the remote staff,” he warns. For example, if a remote office has its own disaster recovery application running on a local server, you’ll need to dispatch someone to fix it if it fails. Remote tools can handle standard processes and technologies, but specialty equipment and software will usually require someone at the scene.

“It’s expensive to consolidate, so you need executive buy-in,” CCA’s Wood cautions.

Lights Out: The Vision

Although vendors like to use the phrase “lights out” to describe the vision of future data centers, a fully automated, self-adjusting data center with no people (à la the HAL 9000 of 2001: A Space Odyssey) remains more fiction than science.

What will happen is that data centers will become increasingly automated and self-managing as virtualization, automation and monitoring technologies mature, letting computers respond to typical situations based on human-created scenarios. But people will still be needed to handle the unexpected and develop the improved processes and business applications that will keep the enterprise moving forward.

“There’s no standard framework” for a next-generation data center, says Dominick.

So the CIO still needs to figure out how to implement still-evolving technologies, rework and streamline the IT processes, and get the right skill mix in the organization to get closer to that lights-out vision.

Gruman is principal of The Zango Group in San Francisco and a regular contributor to CIO magazine. Contact him at ggruman@zangogroup.com.