Network Management

Abstract:

Under SPARC Logical Domains, the Hypervisor is actually running in the firmware of the chassis, where the Control Domain sends commands to partition the hardware underneath the OS's. The hypervisor and all settings are completely in memory... which means if there is a power outage, all virtualization configuration can be lost. The ILOM has onboard storage, to hold the LDoms configuration, when saved, and the hypervisor in the firmware is smart enough to request the configuration from the ILOM on boot, and then simultaneously boot all Logical Domains (including the Control Domain.)

List LDom Configurations

To list all Logical Domain Configurations, which were stored to the ILOM:

sun1824-cd/root# ldm list-spconfig
factory-default
@post-migration [current]
default-config
20190301
20191002
20211014
20220908

Note: in the above example, the "@post-migration" means the configuration was saved the last time someone executed a live migration onto or off of this platform, with the "-s" flag for "save config".

Save Logical Domain Configuration

To save a copy of the LDom configuration:

sun1824-cd/root# ldm add-spconfig `date +%Y%m%d`
sun1824-cd/root#

List Saved Logical Domain Configurations

The newly saved logical domain configuration should show as the Year, Month, Day combination

sun1824-cd/root# ldm list-spconfig
factory-default
@post-migration
default-config
20190301
20191002
20211014
20220908
20230218 [current]
sun1824-cd/root#

Abstract:
The movement from physical to virtual servers has been happening for decades. First, the use of Physical Domains under SPARC "big-iron" became possible when Sun purchased Cray SPARC assets in 1996 while SGI purchased the remaining. The Sun Enterprise 10000 was introduced in 1997 with Physical Domains. With the release of Solaris 10 almost a decade ago, physical systems could be moved to logical Zones under a single kernel in 2004. With the line of T processors, the ability to load multiple OS's on the same platform at the firmware layer became possible in 2006. This article discusses LDom's.

P2V:
Physical to Virtual Migration or P2V is possible to consolidate physical Solaris platforms onto various virtualized Solaris platform destinations - such as Zones, Branded Zones, or LDom's. The P2V process uses an archive called a FLAR.

P2V and Branded Zone:
An Oracle Enterprise Manager blog was published recently, explaining how to move a physical server to a Branded Zone. The May 15th blog was titled: "How to go Physical to Virtual with Oracle Solaris Zones using Enterprise Manager Ops Center."

Logical Domains:
The documentation describing the deployment of the Logical Domains for Oracle Enterprise Manager is available under Oracle's web site. Each Logical Domain, sits on top of the firmware of T-Class processors, and can host Solaris 11, Solaris 10, and Solaris 10 can host older Solaris 8 & 9 Operating System under Branded Zones.

Network Management Implications:
Network Management platforms can very easily be consolidated onto newer platforms, with very little effort, using free drag-and-drop tools such as Oracle Enterprise Manager. If a network management center is still running under multiple older Physical platforms, one should consider Zones or LDom's, which offer virtually no overhead (in comparison to systems such as VMWare or HyperV which require a foreign software layer between their domains and the hardware, introducing problematic latency under heavy loads.)

VMWare 5.1 Resolves Some Issues

Abstract:
With the advent of simple and cost effective virtualization under Solaris 10, Zones, LDoms, and Virtual Box - pressure has been placed upon dominate virtualization vendors to create less expensive alternatives. VMWare, after being purchased by EMC, had decided to move in the opposite direction, making purchasing of VMWare very difficult, with odd pricing constraints in ESXi 5.0 in July 2011. The market has moved to 2012 and ESXi 5.1 has been released, fixing some of VMWare's problems.

Compatibility Issue Resolved:
If customers wanted to move an older VM to newer hardware, the VM's needed to be upgraded. In other words, there was compatibility issues which needed to be resolved. VM's created under ESX Server 3.5 and later will now run under ESXi 5.1 unchanged. This is good news for service providers.

No Longer Windows Bound:
Customers who had VMWare ESXi were required to use a lousy Microsoft Windows platform to manage the VMWare platform. When managing an ESXi server in a DMZ, this makes little sense for a service provider. This has now been resolved, with a web interface.

Memory Tax Issue Resolved:
The pricing constraints of ESXi 5.0 forced service providers to have to decide - is VMWare the correct hypervisor for the job... is Windows and/or Linux worth the aggravation of being nickel and dimed to death? When trying to determine hardware and hypervisor pricing for a new cluster where one does not know exactly how much memory will be required per instance because infrastructure is being purchased by a managed services provider before the first customer deal is sold, how does one know how much to buy?

Clearly, EMC's VMWare did not have a clue. The confusion that the pricing placed upon managed service providers negatively impacted purchasing of other EMC software products such as ITOI (aka Ionix, aka SMARTS) and RSA Archer, enVision, etc. If a managed service provider can not determine what to buy, they will not buy from that vendor. Solaris is clearly the better choice for Network Management, and other vendors are clearly the better choice for tools bound to VMWare & Windows.

The removing of the memory constraints for ESXi 5.1 was a good move, to simplify pricing. EMC Software is now in a better position to compete against other virtualized platforms.

Outstanding Core Issues:
For reasonable flexibility in the data center environment, when there is a spike in usage, there needs to be a way to easily migrate heavy usage live instances to lower utilized hypervisors. Dynamic migration with autobalancing is included with Oracle LDom's, but not quite there yet with VMWare.

When dealing with network virtualization, if one is trying to emulate a WAN environment, one could spin up dozens of zones under a Solaris 11 platform, and apply the WAN characteristics to the virtual network (latency, throughput, etc.) Technology like Solaris Crossbow is missing from VMWare.
Conclusions:
VMWare is a great benefit to the Windows and Linux world, but constraints by the vendor made purchasing difficult and implementation less desirable. Some of the issues have been resolved, but management is not yet what it needs to be for managed service providers.

EMC: Building The Cloud, Kicking Cisco Out?

Abstract:
EMC used to be a partner in the Data Center with a close relationship with vendors such as Sun Microsystems. With the movement of Sun to create ZFS and their own storage solution, the relationship was strained, with EMC responding by suggesting the discontinuance of software development on Solaris platforms. EMC purchased VMWare and entered into a partnership with Cisco - Cisco produced the server hardware in the Data Center while EMC provided VMWare software and with EMC storage. The status-quo is poised for change, again.


EMC World:
Cisco, being a first tier network provider of choice, started building their own blade platforms, entered into a relationship with EMC for their storage and OS virtualization (VMWare) technology. EMC announced just days ago during EMC World 2012 that they will start producing servers. EMC, a cloud virtualization provider, a cloud virtual switch provider, a cloud software management provider, a cloud storage provider, has now moved into the cloud server provider.

Cisco Response:
Apparently aware of the EMC development work before the announcement, Cisco released FlexPods with NetApp. The first release of FlexPods can be managed by EMC management software, because VMWare is still the hypervisor of choice. There is a move towards supporting HyperV, in a future release of FlexPods. There is also a movement towards providing complete management solution through Cisco Intelligent Automation for Cloud. Note, EMC's VMWare vCenter sits as a small brick in the solution acquired by Cisco, including NewScale and Tidal.


NetApp Position:
NetApp's Val Bercovici, CTO of Cloud, declares "the death of [EMC] VMAX." Cisco has been rumored to have been in a position to buy NetApp in 2009, 2010, but now with EMC marginalizing Cisco in 2012 - NetApp becomes more important, and NetApp's stock is dropping like a stone.
Cisco's Mishap:
Cisco, missing a Server Hardware, Server Hypervisor, Server Operating System, Tape Storage, Disk Storage, and management technologies, decided to enter into a partnership with EMC. Why this happened, when system administrators in data centers used to use identical console cables for Cisco and Sun equipment - this should have been their first clue.

Had Cisco been more forward-looking, they could have purchased Sun and acquired all their missing pieces: Intel, AMD, and SPARC Servers; Xen on x64 Solaris, LDom's on SPARC; Solaris Intel and SPARC; Storage Tek; ZFS Storage Appliances; Ops Center for multi-platform systems management.

Cisco now has virtually nothing but blade hardware, started acquiring management software [NewScale and Tidal]... will NetApp be next?


Recovery for Cisco:
An OpenSolaris base with hypervisor and ZFS is the core of what Cisco really needs to rise from the ashes of their missed purchase of Sun and unfortunate partnership with EMC.

From a storage perspective - ZFS is mature, providing a near superset of all features offered by competing storage subsystems (where is the embedded Lustre?) If someone could bring clustering to ZFS - there would be nothing missing - making ZFS a complete superset of everything on the market.

Xen was created around the need for OpenSolaris support, so Xen could easily be resurrected with a little investment by Cisco. Cloud provider Joyent created KVM on top of OpenSolaris and donated the work back to Illumos, so Cisco could easily fill their hypervisor need, to compate with EMC's VMWare.

SGI figured out they needed a first-class storage subsystem, and placed Nexenta (based upon Illumos) in their server lineup. What Cisco really needs is a company like Joyent (based upon Illumos) - to provide storage and a KVM hypervisor. Joyent would also provide Cisco with a cloud solution - a completely intregrated stack, from the ground on up... not as valuable as Sun, but probably a close second, at this point.

The Solaris Tab was recently updated with some white papers.

White papers were placed in date order, using shortened titles on the top, for easy access, while they were categorized with their full titles on the bottom according to topic.

Solaris Reference Material

2007-07 [PDF] Understanding and Deploying Logical Domains
2010-05 [PDF] Best Practices for Data Reliability with LDom's
2010-05 [PDF] Best Practices for Network Availability with LDom's
2010-05 [PDF] Increase Application Scalability and Improve Utilization with LDom's

Solaris LDoms / Oracle VM Server SPARC

• 2007-07 [PDF] Beginners Guide to Oracle VM Server for SPARC:Understanding and Deploying Logical Domains
  
• 2010-05 [PDF] Best Practices for Data Reliability with Oracle VM Server for SPARC
• 2010-05 [PDF] Best Practices for Network Availability with Oracle VM Server for SPARC
• 2010-05 [PDF] Increase Application Scalability and Improve System Utilization with Oracle VM Server for SPARC

Virtualizations: LPARs, LDoms, Xen, KVM, VMWare, and HyperV

IBM LPARs
IBM LPARs is a premium proprietary virtualization technology which sits on top of IBM POWER architecture. It leverages the Virtual I/O Server (VIOS) in order to manage operating system resource requests from other domains.

https://www.ibm.com/developerworks/wikis/display/virtualization/VIO

"This allows a single machine to run multiple operating system (OS) images at the same time but each is isolated from the others. POWER4 based machines started this in 2001 by allowing many Logical Partitions (LPAR) to run on the same machine using but each using different CPUs, different memory sections and different PCI adapter slots. Next came with POWER4, the ability to dynamically change the CPU, memory and PCI adapters slots with the OS running. With the introduction of POWER5 in 2005, further Virtualization items have been added."

http://publib.boulder.ibm.com/infocenter/powersys/v3r1m5/index.jsp?topic=/iphb1/iphb1_vios_virtualioserveroverview.htm

"The Virtual I/O Server is software that is located in a logical partition. This software facilitates the sharing of physical I/O resources between client logical partitions within the server. The Virtual I/O Server provides virtual SCSI target, virtual fibre channel, Shared Ethernet Adapter, and PowerVM™ Active Memory Sharing capability to client logical partitions within the system. As a result, client logical partitions can share SCSI devices, fibre channel adapters, Ethernet adapters, and expand the amount of memory available to logical partitions using paging space devices. The Virtual I/O Server software requires that the logical partition be dedicated solely for its use. The Virtual I/O Server is part of the PowerVM Editions hardware feature."

SPARC LDOM's or Oracle VM for SPARC
SPARC LDOM's (or now referred to as Oracle VM for SPARC) is analagous to IBM's LPARs. IBM's VIOS appears to be analagous to Control Domain under. The LDom Control Domain can be subdivided between Control, Service, and I/O Domains - to architect redundancy and additional performance in a SPARC platform. LDom's are a free Solaris SPARC bundled virtualization technology.

http://www.oracle.com/us/technologies/virtualization/oraclevm/oracle-vm-server-for-sparc-068923.html

"Oracle VM Server for SPARC (previously called Sun Logical Domains) provides highly efficient, enterprise-class virtualization capabilities for Oracle's SPARC T-Series servers. Oracle VM Server for SPARC allows you to create up to 128 virtual servers on one system to take advantage of the massive thread scale offered by SPARC T-Series servers and the Oracle Solaris operating system. And all this capability is available at no additional cost."

http://en.wikipedia.org/wiki/Logical_Domains

"The Control domain, as its name implies, controls the logical domain environment. It is used to configure machine resources and guest domains... The control domain also normally acts as a service domain. Service domains present virtual services, such as virtual disk drives and network switches, to other domains… Current processors can have two service domains in order to provide resiliency against failures. I/O domain has direct ownership of and direct access to physical I/O devices, such as a network card in a PCI controller… Control and service functions can be combined within domains."

There are basic technologies available through LDOM's to developers and architects such as cluster-in-a-box, redundant I/O domains, etc.
http://docs.oracle.com/cd/E19316-01/820-4676/ggtcs/index.html

"In this logical domains (LDoms) guest domain topology, a cluster and every node within that cluster are located on the same Solaris host. Each LDoms guest domain node acts the same as a Solaris host in a cluster. To preclude your having to include a quorum device, this configuration includes three nodes rather than only two."

Xen
There are some similarities to the way these former hypervisors and Xen is architected. Various implementations of Xen exist, such as Citrix Hypervisor, Oracle VM for x86, and OpenSolaris based Xen (now a project under Illumos.) Xen is an open-sourced hypervisor.

http://xen.org/files/Marketing/WhyXen.pdf

"A critical benefit of the Xen Hypervisor is its neutrality to the various operating systems. Due to its independence, Xen is capable of allowing any operating system (Linux, Solaris, BSD, etc) to be the Domain0 thereby ensuring the widest possible use case for customers. For example, many hardware manufacturers leverage NetBSD as their OS of choice for Domain0 and are able to deploy Xen in the manner of their choosing."

"This separation of hypervisor from the Domain0 operating system also ensures that Xen is not burdened with any operating system overhead that is unrelated to processing a series of guests on a given machine. In fact, more are beginning to break up the Domain0 from a single guest into a series of mini-OS guests each with a specific purpose and responsibility which drives better performance and security in a virtualization environment."

KVM
No, this is not a Keyboard switch. Late to the game was a Linux and OpenSolaris based virtualization technology, unfortunately called KVM, for Kernel Virtual Machine. First implemented under Linux.
http://wiki.linuxplumbersconf.org/_media/2010:02-lpc-kvmstoragestackperformance.pdf

Modern OS features such as DTrace and ZFS are now available to KVM after it was quickly ported to OpenSolaris source code base by Joyent for their Open Source SMARTOS cloud operating system and cloud offering
http://www.phoronix.com/scan.php?page=news_item&px=OTc5Ng

"Joyent has announced today they have open-sourced their SmartOS operating system, which is based on Illumos/Solaris. Additionally, this cloud software provider has ported the Linux KVM (Kernel-based Virtual Machine) to this platform.

Being derived from Illumos and in-turn from Solaris, SmartOS does ship with ZFS support, DTrace, and other former Sun Microsystems technologies."

Microsoft HyperV
Some vendors came very late to the hypervisor game. Microsoft HyperV have a similar architecture, available only under Intel & AMD processors, depend on hardware acceleration available under only certain CPU chips from both of those vendors.


VMWare ESXi
VMWare has a great deal of experience in hypervisors, growing out of a software-driven solution, before hardware handlers became popular (and leveraged) in the Intel/AMD world. They provide some of the best backwards-compatibility in the Intel/AMD world.

Solaris Tab - Secure Deployment of LDom's or VM Server for SPARC

An Oracle White Paper, Secure Deployment of Oracle VM Server for SPARC , was added to the Solaris Tab on Network Management.

Solaris Reference Material
2011-01 [PDF] Secure Deployment of LDom's or VM Server for SPARC

Solaris LDoms / Oracle VM Server for SPARC
Secure Deployment of LDoms or Oracle VM Server for SPARC

SPARC T4: Optimizing with Oracle VM Server for SPARC

Abstract:

Modern computing systems had found their footing through the history of computing. Some companies and architectures influenced the modern desktop computer more than others. One such company was Sun Microsystems, which had found it's way into Oracle. Oracle released their latest processor, the SPARC T4, with a dynamic new capability to offer the functionality to process two different workloads, via virtualization technology.

Processor History:

In 1985, Sun Microsystems produced their first Sun-3 workstation and servers based upon the 32 bit CISC Motorola 68000 processor. In 1987, Sun Microsystems produced their first Sun-4 workstations and servers upon the 32 bit RISC SPARC processor. In 1995, Sun Microsystems produced their first UltraSPARC system based upon 64 bit RISC UltraSPARC processor. In 2002, Sun Microsystems acquired Afara Web Systems, with a new high-throughput SPARC design. In 2005, Sun Microsystems released their first server (no desktops) based upon the UltraSPARC T1 processor, which was tuned for multi-threaded workloads. Oracle, who made their fortunes primarily from software upon SPARC, acquired Sun Microsystems and released their first server (no desktops) in 2010 based upon the SPARC T3. Oracle released the SPARC T4 in 2011, supporting both multi-threaded and single-threaded workload.


Workload History:

The workloads in the SPARC processors were traditionally single-threaded workloads from their early years. With the advent of RISC processors, the concept to reduce complexity allowed for the increase clock speed and thus the increase of single threaded performance. With the investment from AT&T and merger with SVR4, Solaris experienced multi-threaded workloads expansion. When SGI purchased Cray Research, Sun Microsystems purchased the Cray Superserver 6400 to create massive high-speed single threaded capability into massive multi-threaded workload throughput of 64 threads via racks of equipment.

With the release of UltraSPARC T1, Sun Microsystems managed to shrink 32 threads of slower integer and crypto capacity not only into a single socket, but onto single piece of silicon, performing outstanding aggregate capacity. With the subsequent release of the T2 processor, 64 threads were merged onto a chip. While the throughput was equivalent to racks of equipment in the T processors, the single threaded performance was a decade behind.

Workload Selection:

With the release of the Oracle SPARC T4 processor, a system can now be tuned to support single or multi-threaded workloads via Oracle VM Server for SPARC release 2.1, previously known as Logical Domains or LDom's.

The short tuning white paper from Oracle describes:

This paper describes how to use the Oracle VM Server for
SPARC 2.1 CPU threading controls to optimize CPU performance
on SPARC T4 platforms. CPU performance can be optimized for
CPU-bound workloads by tuning CPU cores to maximize the
number of instructions per cycle (IPC). Or, CPU performance
can be optimized for maximum throughput by tuning CPU cores
to use a maximum number of CPU threads. By default, the CPU
is tuned for maximum throughput

During the provisioning of a Logical Doman or VM under SPARC, the provisioner can choose the workload optimization required. This can be performed during ["add-domain"] or after ["set-domain"] provisioning.

ldm add-domain [mac-addr=num] [hostid=num]
[failure-policy=ignorepanicresetstop]
[extended-mapin-space=on]
[master=master-ldom1,...,master-ldom4]
[threading=max-throughputmax-ipc] ldom

ldm set-domain [mac-addr=num] [hostid=num]
[failure-policy=ignorepanicresetstop]
[extended-mapin-space=[onoff]]
[master=[master-ldom1,...,master-ldom4]]
[threading=max-throughputmax-ipc] ldom

The "threading" parameter defines the workload. The options from the white paper are defined as follows:

• max-throughput.
  Use this value to select the threading mode that maximizes throughput. This mode activates all threads that are assigned to the domain. This mode is used by default and is also selected if you do not specify any mode (threading=).
  


• max-ipc.
  Use this value to select the threading mode that maximizes the number of instructions per cycle (IPC). When you use this mode on the SPARC T4 platform, only one thread is active for each CPU core that is assigned to the domain. Selecting this mode requires that the domain is configured with the whole-core constraint.
  
  

Solaris Tab - SPARC T4 Workload Optimization

A new Oracle White Paper, Tuning the SPARC CPU to Optimize Workload Performance on SPARC T4, was added to the Solaris Tab on Network Management.

Solaris Reference Material
2011-09 [PDF] Tuning to Optimize Workload Performance on SPARC T4