Rapidly decreasing cost of gigabit ethernet NICs and switches combined with steadily improving SMP capabilities in Linux 2.2 (stable) and especially 2.3 (beta) now make it feasible to network Celeron® dual-processor computers into parallel clusters at very low cost. Since it is already producing the key elements of such systems, the Dude! Computer Corporation has undertaken development and testing of a new system called the POW!er Cluster Dude! Beowulf cluster.
System Description
The cluster consists of nine POW!er 2000 Dude! boxes (8 nodes and one master) for a total of 18 Celeron® PPGA 366 processors operating at 550 MHz on 100 MHz FSB in dual-PPGA Abit BP6 motherboards.
The eight nodes each have additionally
- 128 MB of ECC PC100 SDRAM,
- one 9.1 GB ATA66 Quantum Fireball KA+ 7200 RPM drive, and
- one Netgear FA 310TX fast ethernet PCI adaptor card.
The master has additionally
- one 18.1 GB ATA66 Quantum Fireball KA+ 7200 RPM drive,
- 384 MB ECC PC100 SDRAM,
- one Netgear FA 310TX fast ethernet NIC,
- one Netgear GA620 gigabit ethernet adaptor,
- one 40X Plextor CDROM drive,
- one Viper V550 AGP video card,
- one Creative Labs Sound Blaster 64 sound card,
- floppy, keyboard, mouse, speakers and monitor.
Communication among the boxes is over a Netgear FS509 switch with 8 full-duplex fast-ethernet ports connected to the nodes plus one gigabit optical link to the master computer. The fast ethernet NIC on the master is used to communicate with the external network. The nodes and ethernet switch are housed in a standard 19" RS-310C rack with shelves. Four 650 VA UPS supplies are used for conditioning and emergency shutdown protection.
The operating system is a full-install of
- Red Hat 2.2.5-15 Linux including
- php.3,
- mysql,
- PVM [Parallel Virtual Machine] 3.4.1 and
- two versions of MPI [Message Passing Interface],
- LAM-6.3-b1 and
- MPICH 1.1.2,
For testing, various other software is also loaded including
- the benchmark programs
- hint,
- stream_d,
- blacs/linpack, and
- pvmpov/mpipov parallel versions of the Linux version of the PovRay raytracing program.
All programs have been successfully compiled and run except mpipov 3, which was written for an earlier version of MPICH. A rewrite of the mpipov program may be attempted since independent benchmark results suggest that MPI may outperform PVM for this application. Execution of the linpack benchmark is pending receipt of information on the exact criteria used for top 500 supercomputer evaluation.
The system has operated continuously (except for temporary shutdowns for power outages) for one month with complete stability. The master is completely functional as a workstation/server connected to the external network, and the nodes are accessible from the external network over the master, which acts as a bi-directional server running apache/php3 on two different IP addresses. The fast ethernet cards use the DEC tulip driver and the gigabit card uses the acenic driver. The entire network system was easily configured without problems and has operated flawlessly.
Communication with the nodes is via rsh and ssh, and no difficulty has been experienced accessing and controlling them from the master. Utilities have been written to check the operating speed of all nodes at boot up, and to permit simultaneous reboots and shutdowns with a single command. Of the three message passing wrappers, the LAM version of MPI is the most convenient to install and use, while PVM has the disadvantage of leaving running deamons following abnormal program termination which requires accessing the nodes to kill them manually. In LAM, this can be done for the entire system with the wipe and (for debugging) lamclean commands. Availability of all three wrappers is convenient for running application software which may be written for any one of the three. A wide range of such programs is available via the Internet at www-unix.mcs.anl.gov/mpi and elsewhere but applications have not yet been run except for pvmpov.
Summary of Performance Testing Results
This placed it ahead of 111 parallel systems in the database, and behind only
- the IBM MK-81 (2 seconds with 96 Pentium® III-500),
- Cray T3E-900-AC64 (3 seconds with 48 Alpha EV5-450),
- IBM Netfinity (3 seconds with 36 Xeon 400/450) and
- Gravitor I (6 seconds with 62 Pentium® II-400),
The single-processor result of 53 seconds placed us in 14th position out of 1017 results in the database.
The faster processors were
- the Emotion Engine (the amazing new PlayStation II chipset from Sony/Toshiba running Linux at only 5 seconds!!!),
- 4 Alpha (26-28 seconds),
- 4 Athlon (37-51 seconds),
- 1 HP PA8200 (42 seconds),
- 2 Celeron® 464/550 (45-51 seconds) and
- 1 Pentium® II-500 (49 seconds).
Results of the hint and stream benchmarks confirm the high performance of this Celeron®-based cluster. With all 18 processors running hint, the net QUIP result is 100.6 MQUIPS. [With optimization performed up to 9/23/99, the hint benchmark result has now reached 125.0 MQUIPS. Details are available in the appendix.]
This places it only slightly behind
- the IBM SP-2 (109.7 MQUIPS with 32 processors) and
- the Cray T3D (113.4 MQUIPS using 64 DEC EV4-150 processors),
- the Intel Paragon (46.2 MQUIPS with 64 i860 processors).
Such performance would not be possible with any weak links in the data processing and communication chain, and is a product of the network performance as well as that of the CPUs and memory subsystem. Running the systest network bandwidth program under MPICH yields 9 MByte/sec transfer rates between a single master processor and a single node processor (dual duplex messaging), with only a modest decline to 8 MByte/sec with all 18 processors in simultaneous bidirectional communication. These rates are for message lengths of about 16k up to over 512k, while shorter message rates are limited by network latency to lower but still very respectable speeds.
While disk subsystem performance has little effect on the benchmark data obtained to date, it can be critically important for certain applications. The srtest under MPICH shows a linear increase from zero processors at the origin up to a maximum write bandwidth for 18 processors of 1850 MByte/second following synchronization, and nearly the same rate preceding synchronization. Write bandwidth is 750 MBytes/second with 9 processors prior to synchronization, and remains at approximately this level with 10-18 processors in use. Write bandwidth following synchronization shows a similar linear increase from zero processors at the origin up to approximately 100 MBytes/second with 9 processors. and remains in the 55-85 MByte/second range for 9-18 processors. These results are for the ATA33 IDE hardware interface and driver; installation of the new Linux driver for the ATA66 interface has not yet been made, although it is planned.
Detailed data with graphs will be posted in an addendum to this summary report when time permits.
Conclusions
In addition to unmatched price:performance, this cluster computer has the further advantage of operational flexibility. With addition of inexpensive hardware to each node (video, monitor, keyboard, mouse, floppy, CDROM and possibly a second hard drive), each node becomes a powerful stand-alone workstation with high network connectivity. This makes it possible to run any operating system on the second drive during daytime working hours, and to switch to the Linux supercomputer mode for automatic batch processing at night and on weekends. Such a system is termed "Cluster of Workstations" or COW. We believe this would be of particular interest for applications such as graphics rendering, for which this system has been demonstrated to perform very well.
The system is also extremely flexible with respect to sizing. Eight additional nodes can be added through use of a second switch and gigabit ethernet card in the master computer with no other changes except for additions to a few configuration files. Or the system could be downsized to only 4 or 6 processors by connecting 1 or 2 nodes to the master using standard 10/100 NICs with crossover cables, saving the cost of the switch and gigabit NIC in the master.
