Moving the Data Center, a LinuxWorld Keynote from Kevin Clark

Last week your author was in San Francisco attending LinuxWorld 2008. One keynote was from Kevin Clark, Director of IT Operations at Lucasfilm. Lucasfilm is the production company that brought us Star Wars, Indiana Jones and many other movies and related merchandise. As the Director of IT Operations, Kevin is responsible for the IT needs of four separate divisions in five locations. In 2005 the main data center was moved to a new facility; Kevin talked about the challenges and lessons learned in the process of moving a high availability data center, while making three movies and maintaining high security.

The four divisions of Lucasfilm all have different needs; to meet those needs, the data center has machines running Linux, Unix, Windows and few Macs. Industrial Light and Magic (ILM) is the biggest user of Linux. This is the division that does the special effects for Lucasfilm and many other movies such as Disney's "Pirates of the Caribbean" series. Lucas Arts, Lucas Licensing and Lucas Animation are the other three. These three divisions handle the production of movie-based video games, action figures, official web sites, animated films and other related endeavors.

When Hollywood producers want special effects, they want something that hasn't been seen before, something amazing. With each new movie the producer strives to out-do other movies. ILM must be on the bleeding edge of special effects technology, while maintaining high availability and high security. ILM Linux clusters run around the clock, producing "some of the best special effects the industry has to offer." Downtime is not an option, even for a major move.

Kevin's talk was about moving the data center, and not particularly about Linux. He did have some nice, short films showing off some of ILM's work. Did you know that Pirates of the Caribbean was not filmed on a ship at sea? It's just rendered that way.

For the new data center, Kevin knew he wanted to consolidate systems such as email, databases, storage and backup/recovery. He knew he needed flexible power and cooling requirements and a flexible distribution design with lots of storage for the rendering clusters and the backups and also web hosting for movie sites and other related businesses. The center has high bandwidth requirements, both internally and externally. Also, there are always many people trying to get the scoop on the latest movies and games, so high security is paramount. He chose technologies from AMD, Foundry, NetApp, HP and Juniper to accomplish his goals.

The new data center has over 700 miles of fiber and over 2000 miles of copper with a global WAN for sites at the Telco depot, Letterman Digital Arts Center, Skywalker Ranch, Big Rock Ranch and Singapore Animation. There are 400 terabytes of storage. The AMD blades have 32 gigabytes of memory and they stack them 66 blades per rack. There are lots of racks and floor to ceiling airflow cools them. When filming, all shots are archived, so there is high volume at all times and complete disaster recovery is required.

Kevin had a few lessons that he learned from the data center move: DC power has limitations, equipment interoperability is key and should be built to scale following a network design. The center has needs outside of IT to consider. All the pieces must be fully redundant. You always think that it is fully redundant until it fails. Power and cooling requirements must be balanced. Run the computers hotter to save power, but not so hot that they fail. The data center is a continually moving target with constant pressure to be more energy efficient. More virtualization could help. Getting light to move faster would help.

We were left to wonder how one might overcome the limitations of DC power, or how to get light to move faster. Those points did get a laugh from the audience though. All in all, one might wish for something more Linux related at LinuxWorld, but it was an entertaining presentation.

---

(Log in to post comments)

"Did you know that Pirates of the Caribbean was not filmed on a ship at sea?"

I thought it was pretty obvious myself ;)

Are there ANY movies that filmed large parts of their script on an actual ship at sea? (Surely
there mast be some)

>Are there ANY movies that filmed large parts of their script on an 
actual ship at sea? (Surely there mast be some)


According to wikipedia; much of Master and Commander (2003) was shot 
aboard the Rose (now named the Surprise and residing at the San Diego 
Maritime Museum) with the storm scenes 'digitally-composited footage of 
waves actually shot on board a modern replica of Cook's Endeavour 
rounding Cape Horn'.

Many (most?) of 'mutiny on the bounty' movies have used real ships; of 
course the last one, 'The Bounty' was made 24 years ago...

Good candidates would be "Dead Calm" (1989) and "Waterworld" (1995).