Network Management

Sun SPARC Enterprise T5120 - ALOM - Introduction

Abstract

UNIX Systems Manufacturers originated their markets as workstations, during a time when they used 32 bit systems and the rest of the PC market was concentrating on 8 and 16 bit systems, and some CPU vendors like Intel use segmentation to keep their 16 bit software alive while struggling to move to 32 bit architectures. Some of the original servers were stacked workstations on a rack in a cabinet. The former high-powered video cards were merely ignored, as remote management needed command line interfaces. Engineering quickly determined that console access needed to be built into a new class of systems: rack mounted servers. These early servers offered traditional ALOM compatibility shell as well as newer ILOM shell. The ALOM shell is quite functional.

Sun Enterprise T5120 LOM

The Sun Enterprise T5120 is a server with a second generation OpenSPARC processor. It comes with a Lights Out Management (LOM) capability referred to as Integrated Lights Out Management (ILOM.) The Advanced Lights Out Management (ALOM) shell may be it's default. Most remote systems management work can be done from the LOM. Oracle has an ILOM 3.0 manual. There are also manuals formerly published by Sun Microsystems for OpenBoot 3.x and OpenBoot 4.x manuals.

ALOM: Logging In

A 9600 Baud Serial Cable can be added to the Console port, to provide immediate access.

ALOM: Show System Controller & Network

The LOM can be enabled to perform DHCP, so access can be made over an ethernet cable.

{0} ok show-disks
a) /pci@0/pci@0/pci@8/pci@0/pci@9/SUNW,emlxs@0,1/fp@0,0/disk
b) /pci@0/pci@0/pci@8/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/disk
c) /pci@0/pci@0/pci@2/scsi@0/disk
d) /pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2/storage@3/disk
e) /pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2/storage@2/disk
f) /pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2/storage@1/disk
g) /iscsi-hba/disk
q) NO SELECTION
Enter Selection, q to quit: qOpenSPARC Systems are very viable platforms, offering tremendous Lights Out Management capabilities. The most recent SPARC Systems have been the fastest platforms in the world for nearly 4 years, so there is a tremendous growth potential for migrating from these older Open Source SPARC platforms. Being based upon Open Hardware, Open Firmware, and Open Hardware - new hardware vendors can also create their own newer generation platforms, to meet their own requirements if the Light Out Management capabilities are not a requirement.

  Solaris: Massive Internet Scalability
Abstract:
Computing systems started with single processors. As computer requirements increased, multiple processors were lashed together, using technology called SMP (Symmetric Multi-Processing) to add more computing power into a single system, breaking up tasks into processes and threads, but the transition to multi-threaded computing was a long process. The lack of scalability for some problems produced MPP (Massively Parallel Processing) platforms, lashing systems together using special software to load-balance jobs to be processed. MPP platforms were very difficult to program general purpose applications, so massively Multi-Core and Multi-Threaded processors started to appear. Oracle recently released the SPARC T5 processor and systems - producing an SMP platform scalable with massive sockets, cores, and threads into a single chassis - leveraging existing multi-threaded computing software, reducing the need for MPP in real-world applications, while placing tremendous pressure upon the Operating System layer.

SPARC Growth Rate:
The SPARC processors started a growth rate, with a movement to massively threaded software.

SPARC	Cores	GHz	Threads	Sockets	Total-Cores	Total-Threads
T1	8	1.4	32	1	8	32
T2	8	1.6	64	1	8	64
T2+	8	1.6	64	4	32	256
T3	16	1.6	128	4	64	512
T4	8	3	64	4	32	256
T5	16	3.6	128	8	128	1024
M5	6	3.6	48	32	192	1536

The movement to massively threaded processors meant that applications needed to be re-written to take advantage of the new higher throughput. Certain applications were already well suited for this workload (i.e. web servers) - but many were not.

Application Challenges:
The movement to massively threaded software, to take advantage of the higher overall throughput offered by the new processor technology, was difficult for application programmers. Technologies such as DTrace were added to advanced operating systems such as Solaris to assist developers and systems administrators in pin-pointing their code hot-spots for later re-write.

When the SPARC T4 was released, there was a feature called "Critical Thread API" in the S3 core, to assist application programmers who could not resolve some single thread bottlenecks. The S3 core could automatically switch into a single-threaded mode (with the sacrifice of throughput) to address hot-spots. The T4 (and T5) faster S3 core was also clocked at a higher rate, providing an overall boost to single threaded workflows over previous processors - even at the same number of cores and threads. The ability to perform out-of-order instruction handling in the S3 also increased speed in the execution of single-threaded applications.

The SPARC T4 and T5 processors finally offered application developers a no-compromise processor. For heavy single-threaded workloads, the SPARC M5 processor was released from Oracle, driving inreasing scales of higher single-threaded workloads, without having to rely upon systems produced by long-time SPARC partner & competitor - Fujitsu.


Operating System Challenges:

A single system scaling to 192 cores and 1536 threads offers incredible challenges to Operating System designers. Steve Sistare from Oracle discusses some of these challenges in a Part 1 article and solutions in a Part 2 article. Some of the challenges overcome by Solaris included:

CPU scaling issues include: •increased lock contention at higher thread counts
•O(NCPU) and worse algorithms

Memory scaling issues include:
•working sets that exceed VA translation caches
•unmapping translations in all CPUs that access a memory page
•O(memory) algorithms
•memory hotspots

Device scaling issues include:
•O(Ndevice) and worse algorithms
•system bandwidth limitations
•lock contention in interrupt threads and service threads

Clearly, the engineering team at Oracle were up for the tasks created for them by the Oracle SPARC engineering team. Innovation from Sun Microsystems continues under Oracle. It will take years for other Operating System vendors to "catch up".
Network Management Applications:

In the realm of Network Management, many polling applications used threads to scale, where network communication to edge devices was latency bottlenecked - making the SPARC "T" processors an excellent choice in the carrier based environment.
The data returned by the massively mult-threaded pollers needed to be placed in a database, in a consistent fashion. This offered a problem during the device "discovery" process. This is normally a single-threaded process, which experienced massive slow-downs under the "T" processors - until the T4 was released. With processors like the SPARC T4 and SPARC T5 - Network Management applications gain the proverbial "best of both worlds" with massive hardware thread scalability for pollers and excellent single-threaded throughput during discovery bottlenecks with the "Critical Thread API."

The latest SPARC platforms are optimal platforms for massive Network Management applications. There is no other platform on the planet which compares to SPARC for managing "The Internet".

[UltraSPARC T3 Micrograph]

CoolThreads UltraSPARC and SPARC Processors

Abstract:
Processor development takes an immense quantity of time, to architect a high-performance solution, and an uncanny vision of the future, to project market demand and acceptance. In 2005, Sun embarked on a bold path moving toward many cores and many threads per core. Since the purchase of Sun by Oracle, the internal SPARC road map from Sun had clarified.


[UltraSPARC T1 Micrograph]
Generation 1: UltraSPARC T1
A new family of SPARC processors was announced by Sun on 2005 November 14.

• Single die
• Single socket
  
• 64 bits
• 4, 6, 8 integer cores
• 4, 6, 8 crypto cores
  
• 4 threads/core
• 1 shared floating point core
• 1.0 GHz - 1.4 GHz clock speed
• 279 million transisters
• 378 mm2
• 90 nm CMOS (TI)
• 1 JBUS port
• 3 Megabyte Level 2 Cache
• 1 Integer ALU per Core
• ??? Memory Controllers
  
• 6 Stage Integer Pipeline per Core
• No embedded Ethernet into CPU
• Crypto Algorithms: ???
  

Platform designed as a front-end server for web server applications. With a massive number of cores, it was designed to provide web-tier performance similar to existing quad-socket systems leveraging a single socket.

To understand the ground-breaking advancement in this technology, most processors were single core, with an occasional dual core processor (with cores glued together through a more expensive process referred to as a multi-chip module, driving higher software licensing costs for those platforms.)


Generation 2: UltraSPARC T2
The next generation of the CoolThreads processor was announced by Sun on 2007 August.

• Single die
• Single Socket
  
• 64 bits
• 4, 6, 8 integer cores
• 4, 6, 8 crypto cores
  
• 4, 6, 8 floating point units
• 8 threads/core
• 1.2 GHz - 1.6 GHz clock speed
• 503 million transisters
• 342 mm2
• 65 nm CMOS (TI)
• 1 PCI Express port (1.0 x8)
• 4 Mageabyte Level 2 Cache
• 2 Integer ALU per Core
• 4x Dual Channel FBDIMM DDR2 Controllers
• 8 Stage Integer Pipeline per Core
• 2x 10 GigabitEthernet on-CPU ports
• Crypto Algorithms: DES, Triple DES, AES, RC4, SHA1, SHA256, MD5, RSA-2048, ECC, CRC32

This processor was designed for higher compute intensive requirements and incredibly efficient network capacity. Platform made an excellent front-end server for applications as well as Middleware, with the ability to do 10 Gigabit wire-speed encryption with virtually no CPU overhead.

Competitors started to build Single-Die dual-core CPU's with Quad-Core processors by gluing dual-core processors into a Multi-Chip Module.


[UltraSPARC T2 Micrograph]
Generation 3: UltraSPARC T2+
Sun quickly released the first CoolThreads SMP capable UltraSPARC T2+ in 2008 April.

• Single die
• 1-4 Sockets
  
• 64 bits
• 4, 6, 8 integer cores
• 4, 6, 8 crypto cores
  
• 4, 6, 8 floating point units
• 8 threads/core
• 1.2 GHz - 1.6 GHz clock speed
• 503 million transisters
• 342 mm2
• 65 nm CMOS (TI)
• 1 PCI Express port (1.0 x8)
• 4 Megabyte Level 2 Cache
• 2 Integer ALU per Core
• 2x? Dual Channel FBDIMM DDR2 Controllers
• 8? Stage Integer Pipeline per Core
• No embedded Ethernet into CPU
• Crypto Algorithms: DES, Triple DES, AES, RC4, SHA1, SHA256, MD5, RSA-2048, ECC, CRC32
  

This processor allowed the T processor series to move from the Tier 0 web engines and Middleware to Application tier. Architects started to understand the benefits of this platform entering the Database tier. This was the first Coolthreads processor to scale past 1 and up to 4 sockets.

By this time, competition really started to understand that Sun had properly predicted the future of computing. The drive toward single-die Quad-Core chips have started with Hex-Core Multi-Chip Modules being predicted.


Generation 4: SPARC T3
The market became nervous with Oracle purchasing Sun. The first Oracle branded CoolThreads SMP capable UltraSPARC T3 was launched in in 2010 September.

• Single die
• 1-4 Sockets
  
• 64 bits
• 16 integer cores
• 16 crypto cores
  
• 16 floating point units
• 8 threads/core
• 1.67 GHz clock speed
• ??? million transisters
  
• 377 mm2
• 40 nm
  
• 2x PCI Express port (2.0 x8)
• 6 Megabyte Level 2 Cache
• 2 Integer ALU per Core
• 4x DDR3 SDRAM Controllers
• 8? Stage Integer Pipeline per Core
• 2x 10 GigabitEthernet on-CPU ports
• Crypto Algorithms: DES, 3DES, AES, RC4, SHA1, SHA256/384/512, Kasumi, Galois Field, MD5, RSA to 2048 key, ECC, CRC32

This processor was more than what the market was anticipating from Oracle. This processor took all the features of the T2 and T2+ combined them into the new T3 with an increase in overall features. No longer did the market need to choose between multiple sockets or embedded 10 GigE interfaces - this chip has it all plus double the cores.

The market, immediately before this release, the competition was releasing single die hex-core and octal-core CPU's using multi-chip modules, by gluing them together. The T3 was a substantial upgrade over the competition by offering double the cores on a single die.


Generation 5: SPARC T4
Oracle indicated in December 2010 that they had thousands of these processors in the lab and predicted this processor will be released end of 2011.

After the announcement, a separate press release indicated processors will have a renovated core, for higher single threaded performance, but the socket will offer half the cores.

Most vendors are projected to have 8 core processors available (through Multi-Chip modules) by the time the T3 is released, but only the T4 should be on a single piece of silicon during this period.


[2010-12 SPARC Solaris Roadmap]
Generation 6: SPARC T5

Some details on the T5 were announced with the T4. Processors will use the renovated T4 core, with a 28nm process. This will return to 16 cores per socket again. This processor may be the first Coolthreads T processor able to scale from 1-8 processors. It is projected to appear in early 2013.

Some vendors are projecting to have 12 core processors on the market using Multi-Chip Module technology, but when the T5 is released, this should still be the market leader in 16 cores per socket.

Network Management Connection

Consolidating most network management stations in a globalized environment works very well with the Coolthreads T-Series processors. Consolidating multiple slower SPARC platforms onto single and double socket T series have worked well over the past half decade.

While most network management polling engines will scale linearly with these highly-threaded processors, there are some operations which are bound to single threads. These type of processes include event correlation, startup time, and syncronization after a discovery in a large managed topology.

The market will welcome the enhanced T4 processor core and the T5 processor, when it is released.