Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-) Softpanorama Search

Solaris Competitiveness on High End Servers

Large servers involve cutting edge technology not only in CPUs and memory area ( IBM mainframes has memory protection in 1962 and VM capabilities in 1968) but also speed of I/O and its organization (actually from the beginning mainframes peripherals were connected to central Unit with a special network and each peripheral has its own CPU capable for direct memory transfer into predetermined area of main processing unit memory).  This is the area where open source development is probably least effective and the pool of users is relatively small. Still it can help as IBM experience with Share and its brilliant move of converting VM/CMS into VM/Linux clearly demonstrated. Also here like with buyers of luxury cars and the price of gas, the price of electricity consumption generally does not matter: An organization which can buy those servers definitely can afford the current price of electricity.  The main consumers of mainframe style computers are large financial institutions and government, especially government funded research centers and universities.

Usually high end servers have 64G of RAM of more. Upper end servers of this class often have 4G of memory (for example Power 595). Dell don't even exist in that universe. HP used to, but the share is shrinking rapidly (situation may change with better Itanium CPUs in the pipeline). IBM (Power5 and, of course, their mainframe line) and Sun are top hardware vendors in this area of deployment. 

Here (in comparison with Linux) Solaris is the king of the hill (along with AIX which is the king of separate Power hill ;-).

Until version 2.6 Linux just cannot compete on its own outside several highly specialized areas (only SGI has limited success in this area with their highly modified fully 64-bit Linux). The situation is improved with version 2.6 that provides more or less full 64-bit support, but it might be too little and too late to change the dynamics of this flat (or even shrinking) and very conservative market, where each previous investment strongly suggests the next.  You just need to think about IBM mainframes which are still compatible with programs written in early 60th of the last century.  Linux track record in compatibility is simply horrible and as such Linux clearly is out of taste for mainframe buyers.

That's why few large companies consider linux for their mainframe class servers.  Both IBM and HP promote their proprietary OSes for their big iron and here Solaris has the advantage of being open sourced system, and enjoying (very limited in this area, but still non-zero) advantages of higher level of feedback typical for the open source mode of development.  Moreover both IBM and HP are sitting between two chairs (IBM actually manages to sit between four or even five chairs if we count the number of OSes it supports on big iron, so this is the company which probably most strongly competes with itself in this particular area :-). HP commitment to big iron is questionable in view of discontinuation of its own CPU line, slow adoption of Itanium and related attempts to push linux for low end and midrange servers.  Only Sun can boast to be a vendor that uses a single OS for the whole spectrum of its hardware offerings and that's translate into huge competitive advantage for Sun.

Currently IBM is on the roll with its Power 6 servers (IBM Power Systems Comparisons Performance):

 Highend Systems Performance  ¹

	SPECint_rate2006	SPECfp_rate2006	SPECjbb2005	2-tier SAP SD
64-core IBM Power 595	2,080	2,110	3,272,999	35,400
128-core HP Integrity Superdome	1,650	1,480	2,054,864	30,000
64-core HP Integrity Superdome	824	745	No Benchmark	9,265
128-core Sun SPARC Enterprise M9000	1,290	1,230	No Benchmark	No Benchmark
64-core Sun SPARC Enterprise M9000	650	636	No Benchmark	No Benchmark
144-core Sun Fire E25K	1,230	No Benchmark	2,105,264	10,175

 

The 64-core Power 595 server outperformed the 64-core HP Integrity Superdome server by at least 2 ½ to 1 and the 64-core Sun SPARC Enterprise M9000 server by over 3 to 1 on processor intensive benchmarks. Even when the Superdome and M9000 servers were tested with twice the number of cores, the Power 595 server was faster by 18% up to as much as 71%. 71% faster than a system with double the cores is a very impressive result, but that is what you can expect form the IBM Power 595 server, very impressive results.

I would like to stress that linux shortcomings are most visible in this particular segment of the market.  This is simply because this segment of the market is defined by forces and requirements that are completely outside of the ream of regular open source developers, and first of all I/O bandwidth.

One of the key issues with this market is the level of compatibility as the investments both in hardware and software are simply huge. And software on big iron boxes usually powers really critical for the firm applications, applications that are difficult to move due to inherent risks on any migration of critical applications or due to the long history of their development for a particular platform. Sun has an excellent record in maintaining compatibility and that's why Sun successfully competes with IBM for the share of shrinking IBM's mainframe market. It was the necessity to counter Sun's advances into this space that were one of the possible motivation of IBM's brilliant move with converting VM/CMS into VM/Linux (see IBM "100 featherless penguins in the steel VM cage" road show ).  In a sense linux was used by IBM as a Trojan horse to stall Sun advances into their mainframe turf. And one of the primary attraction of the UltraSparc is its ability to scale from small 1U servers all the way up to large supercomputers.  That means that an old $100 Ultra-5 desktop can be used for developing application that will run on huge 64 CPU mainframe on exactly the same operating system. Neither IBM not HP-UX with their push of linux for low end can boast a similar advantage.

Way of doing business here is completely different than in low-end servers and even mid-range servers. You will have an impression that your vendor rep is working for you. Just ask question and they're will be prompt professional response. It really feels that Sun reps into his area are working for you.

Solaris 8,9 and 10 are generally biased toward multi-CPU servers and are more efficient on servers that contain large number of  CPUs (Solaris scaled to 64 CPUs in 1996, in version 2.5.1 which is amazing if you remember the level of hardware in 1996; for example a typical PC at this time used to have 486 or Pentium 90 or 133 MHz CPU, 64M of memory and 1G or less harddrive). The side effect is that as we noted above is that Solaris is rather slow on uniprocessor machines. From what I have heard, with the added code from UnixWare which SUN bought, Solaris 10 should perform better on uniprocessor and dual processor configurations, but still on those configurations Linux might continue to have an edge, as this is Linux major hardware base.  But as the number of CPUs exceeds 4 Solaris advantage grows and on 8 CPUs or more Solaris might well be the preferable OS.  IBM does has impressive hardware (Power5-based servers) that can compete with Sun's UltraSparc line and impressive (although rather strange with many proprietary extensions and non standard features) flavor of Unix to compete with Solaris, but as I notes above it is sitting between three to five chairs. HP also is sitting between two chairs and in addition has a set back due to PR problems with Itanium.

And while Linux can run on SGI computers with more CPUs then Sun largest machines (SGI Altix)  those servers are very rare and use NUMA architecture (I actually do not understand the difference very well and it might be that SMP servers with large number of CPUs also has non-uniform memory access).  they does not represent major competitors in the mainframe segment which currently is owned by IBM, HP and Sun.  Actually numerical game with the number of the CPUs has its limits as the number of applications that can utilize huge number of CPUs is very limited. So in a sense we should not pay to much attention whether a particular OS can support more then a hundred CPUs: return of investment probably became marginal after 64 CPUs. 

Anyway none of those companies that promote linux is very excited about moving Linux in this segment. IBM limits Unix deployment to VM/CMS where linux actually is playing the role of DOS. Formally Linux can run on large HP and IBM Power5 boxes. But despite all real and PR-level love of Linux both IBM and HP will do their best to persuade you to buy their huge SMP boxes with AIX and HP-UX respectively. And not without a reason. Full support of complex mainframe hardware is available only with those OSes. So the possibility of running Linux on those boxes is just nice little PR trick. It will rarely happen. 

Moreover the fact that both IBM and HP have internally competing lines of product undermines their competitive position with Sun. Large Power5 boxes with AIX are actually cannibalizing IBM's mainframe market in a way very similar to Sun and I know several large corporation that replaced their aging mainframes with AIX boxes for running SAP/R3 despite all those exiting stories about VM/Linux. And I would like to stress it again that among major mainframe market players only Sun comes clean with "monotheistic" ("there no other OS then Solaris and Sun is the vendor")  religion and like in case of major world religions that translates into important marketing and technical advantages :-).  While IBM move from VM/CMS to VM/Linux was simply brilliant, their positioning of AIX is very confusing: they constantly talk about linux and try to sell you AIX.

And while UltraSparc-based large SMP configurations do not own many SMP sensitive benchmarks records, they have very respectable performance. For example on SPECjbb2005 ( Java performance benchmarks), Fujitsu PRIMEPOWER at 128 cores gets  respectable 1,251,024 running Solaris which is comparable with Altix 3700 BX2 (128P 1600 MHz Itanium 2) 1,828,349 which currently holds the record. Sun's own offerings (Sun Fire E25K) are getting over one million on this benchmark (1164995 to be exact).

The same picture is true for TPC-H transaction benchmarks where in 3,000 GB category Sun E25K has the fifth best result and in 10,000 GB category Sun is the leader.  Note that there is no IBM mainframes among leaders in this benchmark. Only Power5-based servers (clustered IBM eServer p5 575 in 10,000 GB category) represent "state of art" IBM's competitors to Sun.  HP has a very good performance numbers but costs per transaction are three times higher.

Here are selected  transactional benchmarks  relevant to our discussion (all of them are as of  June, 2006).

In April 2007 Sun confirmed its TPC-H Data Warehousing World Record. The Sun Fire E25K server has achieved new world-record results on the TPC-H data warehousing benchmark at the 3 Terabyte scale factor for both overall performance and single-system price/performance. The TPC-H benchmark was run on a 72-way 1.8 GHz UltraSPARC IV+ processor- based Sun Fire E25K with Sun StorageTek 6140 arrays, running Solaris 10 and Oracle Database 10g with Automatic Storage Management. The benchmark measures high-load multiple query throughput as well as single query performance. The Sun Fire E25K server outperformed the best competitor non-cluster system, the IBM p5-595 by 14% and had the best price/performance of the top six performing systems.