On ICE

Joe Casad: It looks like SGI has some exciting stuff coming up. What's coming with your next release of the Altix ICE HPC platform?

Bill Mannel: We're actually going to be announcing the latest generation of ICE – that's actually the fifth generation of the product. We basically went all the way back to first principles and looked at all the base installations we've done, all the customer feedback and field feedback we had, the wins and losses, and we essentially built it from scratch, although we did stay with some of the founding principles of the product, such as highest-performance weight architecture for HPC applications, flexibility for addressing a lot of different types of applications, and also keeping the customer satisfaction to the maximum. A recent survey found that this platform actually warrants our top customer satisfaction of any of our platforms, so we didn't want to do anything that would jeopardize that.

Based on the original ICE, we had both air cooling and water-cooled doors capability, and we're actually adding additional cooling capability as well. So, besides those two – air cooling and water-cooled doors – we also have the capability to put a cold sink directly on the processors themselves to take away the heat; we added warm water cooling as well, and the ability to use a contained hot aisle arrangement, as well as the standard front-to-back air flow arrangement.

We've got a couple different racking schemes. One is standard 19-inch, which is actually an improvement, because the current ICE is a 24-inch rack, which in some cases is somewhat problematic. In some installations, we actually had to put the enclosure on its side and plug it in and reduce some density, but with the new platform, we can get increased density in a standard 19-inch rack without having to do that. Also, we have this large, we call it a cell, which basically allows us to collapse a couple of compute aisles – a couple of hot aisles – to a constrained space, which allows us to deploy in a data center as well as a container.

We actually have a container offering as well. We are starting to see more and more that customers who are just plain running out of space, or their power is stranded somewhere on their site and it will cost a lot of money to get it moved from point A to point B. So, putting containers down is an easier solution than trying to move power around or get power in.

JC: So, it sounds like you're making improvements on the cooling technology, and also the use of space. Who is going to use this? Who are your biggest customers?

BM: The ICE platform does span a wide variety of customers. We have customers like NASA with the Pleiades, and I don't want to misquote the number of racks, but I think it's around 185 racks today. It's a LINPACK petaflops computer, so we have customers at that level, and they have a mixture of customers in both the capability and capacity type of regime.

So that's one use scenario: a large national resource. We have ICE systems at Wright Patterson, at ERDC (the Engineering and Research Development Center), at NASA, at CINES, which is the non-classified compute center for France, and at HLRN in Germany, so that's one typical use case – large scale opportunities – typically in a national lab type of environment. A second use case, and this really boils down to the capability around performance in InfiniBand, is in CFD (Computational Fluid Dynamics). We have a very large ICE footprint in manufacturers doing CFD. Toyota has Altix ICE platforms, and quite a lot of other automotive companies are using Altix ICE for CFD.

A couple months ago, we purchased the OpenFoam company, which has been supporting the OpenFoam product – an open source CFD program that is actually growing in popularity. I understand from some of the analysts that it's the number two mentioned CFD code, after Fluid, of course, which has the bulk of market share. But, our intent is actually to bring ICE and OpenFoam together and essentially design a really high-performing CFD platform out of those two products – hardware and software together in an optimized platform that will still be supporting the open distribution for OpenFoam. For those customers who need the extremes, for example in Formula 1 racing, we'll be able to optimize CFD performance.

BM: Yes, in fact that's one of the most important parts of it. We have management software that's on top of the hardware. We manage an ICE system in a hierarchy, so the customer has the ability to basically power up and power down racks. They can manage the system at the blade level, at the enclosure level, at the rack level, or at the entire system level. That gives you a lot of flexibility for doing updates and upgrades and troubleshooting. If you're running a big data center, you want to be able to provide a certain level of service back to your customer base even while you're doing preventive maintenance and upgrading the system.

Another part of the configuration is the topologies. A lot of folks in this industry offer one topology, typically fat-tree. We offer four, so we offer fat-tree, hypercube, enhanced hypercube, and all-to-all topologies. Typically, most of our customers use hypercube or enhanced hypercube – one of the derivatives of hypercube – because it gives you very similar performance to fat-tree without having nearly the amount of switch infrastructure, so you end up saving a bit on cost. The other thing hypercube does for you is that it allows you to configure a system in a certain way so that it's actually possible to wheel new racks up to a racked system and basically just plug them in without having to re-cable the existing system. That has tremendous benefits for some of our customers.

NASA did a release where they said they saved lots of compute hours because they had their Pleiades system, which around that time I think was 140-some racks. They had it running full of customer code, and they added another rank of racks into the system itself. A combination of our topologies and our hierarchical management let them manage that system as separate parts without having to shut the entire system down to do an upgrade or to add what we call an expansion, where we actually add racks into the system itself.

JC: Are these nodes running some form of Linux?

BM: Yeah. We support SLES, SUSE Linux, and Red Hat. Coming soon, in fact this quarter, we'll also support CentOS on the compute nodes, so if a customer wants to use CentOS as their distribution on the compute nodes, we can do that as well. There's a small number of service and support nodes that have to be either Red Hat or SLES, but the bulk of the cluster is in the compute nodes, and they can be CentOS.

In general, ICE systems are deployed without disks on the nodes themselves. On the blades, they typically go diskless. The earlier versions of ICE had to go diskless; we didn't have the capability of adding disks. But, with the latest version of the product, we can add either small SATA drives or SSDs, but most customers prefer not to. Every rack has what we call a rack leader node, or rack leader controller, which is actually the official term for it: RLC. And the nodes themselves boot off the rack leader controller, so they get their version of Linux across an internal network that is not used for compute, because the compute goes to the InfiniBand side of the house. And, they can boot themselves. So, the beauty of this is that, on a rack by rack basis, we can have a variety of different Linux distributions.

Let's say a customer wanted to bring in a new version of Linux and run it in a test environment on the rack. They can boot all the nodes in the rack off that particular version of Linux while the rest of the system is running an earlier version of Linux. Again, it gives you a lot of capability in controlling what you have and what you do on the system itself.

JC: I noticed a press release recently about some work that SGI has been doing with the Apache Hadoop (distributed computing) project.

BM: That's correct.

JC: Is this integrated with your system, too?

BM: Well, you know, we have one customer node that I can't talk about that is actually using ICE for Hadoop. Typically, most of our customers who are doing Hadoop are using a rackmount line and that's mostly because Hadoop by nature needs a fair amount of local storage. It wants to break up data on a node-by-node basis so it has both a fail-safe capability as well as distributed processing capability. So, typically, you want – it varies obviously – but you want 4 to 8 drives per node in a Hadoop cluster, and that's typically not something you would do on an Altix ICE type of system.

JC: Do you have your own internal distributed computing software platform, or do you use an open source tool?

BM: On Hadoop itself, there's a couple things we do. One, we have a number of customers that use the open source distribution themselves and spin it up. So, they get it out of the open source and basically go from there. We just recently signed an agreement with Cloudera, and we're going to be distributing their enterprise version of Hadoop on our platform by the end of November. So, if a customer is comfortable, and they're not all comfortable – most of the time it's very large Internet companies and a few customers in the defense space who've been working with Hadoop for a while – with pulling down the open source and dealing with bugs, then they can do that, but more and more of our customers want to have a distribution supported – very much like Red Hat. So Cloudera is like the Red Hat of Hadoop from that standpoint.

JC: On your ICE platforms, you said you don't tend to use Hadoop with ICE?

BM: We don't tend to use Hadoop on ICE, yeah. There's one or two sort of unusual customers who do that.

JC: Do you have some equivalent distributed computing software that's your own system?

BM: No. Altair is a great partner of ours, so a lot of our ICE customers use Altair PBS Pro as their workload manager. They tend to use PBS Pro, and also this quarter we will be supporting SLURM, which is an open source load balancer, so we have a number of customers that use that. Then, there's somewhat fewer customers that use Platform LSF as their distribution.

JC: So, what's the coolest thing about the new fifth generation release coming up? What are you proudest of? What do you think is going to have the most impact on the market?

BM: I think the thing that will have the most impact on the market is the fact that ICE itself is the highest performing InfiniBand-based system out there and, if you do some of the numbers, it's one of the highest performing systems in general.

JC: Great. Thanks very much for your time.