It's a UNIX system!

Oracle have announced new systems based on the new T5 and M5 processors. The T5 has doubled the number of S3 cores from the T4 and increased the clock frequency to 3.6GHz.

The M5 processor is also based on the S3 core (rebranded M4) clocked at 3.6Ghz but is has 6 cores and 48MB L3$. The M systems supports up to 32 M5 processor so a fully configured systems will have 192 cores and 1536 strands (hardware threads). The M5-32 have 32TB of memory in a single system.

Old and existing processors for reference:
T3 16 cores @ 1.65GHz 6Mb L2$ 1-4 socket systems PCIe 2.0
T4 8 cores @ 3.0GHz 4MBL3$ 1-4 socket systems PCIe 2.0

New processors:
T5 16 cores @ 3.6GHz 8MB L31ドル-8 socket systems PCIe 3.0
M5 6 cores @ 3.6Ghz 48MB L3$ 32 socket system PCIe 3.0

Oracle claims 2.3x performance gain compared to the T4 with increased single-thread performance.

All new systems both T5 and M5 supports LDOM (Oracle VM for SPARC).

Did everyone read that the M5-32 Supports 32TB of memory? No wounder they had to rewrite the virtual memory subsystem in Solaris 11.1

SPARC Servers

Solaris 11.1 is now available for download from Oracle.

The usual SPARC/X86 [text|live|usb|AI] images are available as well as repository images. There are also pre-upgrade repository images available for those of you who are upgrading from 11/11 and have not upgraded to a recent SR or do not have a support contact.

Oracle Solaris 11 Download

Solaris 11.1 have been announced and will be released later this month. It is the first update of Solaris 11 since it release november last year. It contains a few interesting features, I've only list a few, over 200 projects have integrated into this release.

General

• New Virtual memory subsystem (VM2.0 or parts of it)
  Scales beyond 100TB, predicts and adapts to page demand, higher performance
• RSyslog
• USB3 support
• Install on UEFI/4K
• Interactive install on iSCSI
• FedFS

Zones

• Parallel zone update
• Must faster LOFI
• FS statistics per zone
• Physical to virtual Solaris 10 migration
• Better support for shared storage
• Remote Administration Daemon
  Secure zone administration with C, Java, Python API
• VNIC config in zone XML
• Faster install and attach

I asked the Solaris panel on Oracle Openworld about memory set for zones, it is not part of this release but might be implemented now that the initial part of VM2.0 is implemented.

Solaris 11.1 Whats's new

Oracle keeps quiet about upcoming Solaris release and their features. I've however figured out the name (and planned release month) for the possibly last Solaris 10 update. It seems like s10u11 will be named Solaris 10 10/12 indicating a October release.

Not much is known about this release more than that it will probably feature a ZFS tech refresh, fully integrate OCM and live upgrade enhancements. It could possibly also support the new T5 processors since they might be quite similar to the current T4 processors except for the doubling of cores.

I have also seen fragments of information indicating that Oracle was running Solaris 11 Update 1 on SPARC T5-8 machines as early as March.

Here is a good summary of a recent online forum, "Solaris 11: What's new since launch?"

Solaris 11 Update 1 (late this year)

• Updated Virtual memory subsystem
This is probably what has been known as vm 2.0 earlier
• Faster Solaris 11 updates with improved python performance
• Already running on the upcoming T5/M4 SPARC(R) chips
• VNIC configuration switch hosts with their zones

Summary: What's new with Solaris 11 since the launch?

I've finally managed to get some time to play with live migration on a pair of SPARC T4-2. This post is not really adding any new information but is a walk-trough and initial reflections. I am going to continue to write LDOM instead of Oracle VM for SPARC Domains or something like that, even Oracle people still say LDOM and everyone else knows what is.

An interesting note is that I've used Solaris 10 as I/O and Control domain for the T4 servers while the LDOM is installed with Solaris 11 11/11. The disks for the LDOM are on LUNs over FC and MPxIO is used for multipathing from the I/O domain:

t42-01# dskinfo list-long
disk size lun use p spd type lb
c0t5000CBA015B85D98d0 279G - rpool - - disk y
c0t5000CBA015B93B90d0 279G - - - - disk y
c0t50002870000254901593534030832420d0 33G 0x0 - 4 4Gb fc y
c0t50002870000254901593534030832420d0 33G 0x1 - 4 4Gb fc y

Examples of migrating and reconfiguring the LDOM while running:

t42-01# ldm list
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- UART 16 16G 0.1% 12d 6h 37m
ldms11-01 active -n---- 5000 16 8G 0.0% 24m

t42-02# ldm list
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- UART 16 16G 0.1% 12d 1h 26m

ldms11-01:~$ uptime 
 5:11pm up 19 min(s), 1 user, load average: 0.00, 0.00, 0.01
henrikj@ldms11-01:~$ prtconf -v |grep Mem
Memory size: 8192 Megabytes
henrikj@ldms11-01:~$ psrinfo | wc -l
 16

t42-02# ldm set-vcpu 96 ldms11-01
t42-02# ldm set-memory 200G ldms11-01

t42-02# ldm list
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- UART 16 16G 0.1% 12d 6h 50m
ldms11-01 active -n---- 5000 96 200G 0.1% 24m

ldms11-01:~$ prtconf -v |grep Mem
Memory size: 204800 Megabytes
ldms11-01:~$ psrinfo | wc -l
 96

When performing a live migration between the two hosts, running processes and open network connections are as expected intact, there is only a small delay in the network traffic visible. For my initial tests the delay was about 10 ms.

The live migration seems to work very well and the T4 seems to perform several times faster than the T2/T3 for general workloads. The only thing missing is that LDOM 2.1 is unable to dynamically reconfigure memory and CPU resources for a domain after migration. A reboot is then required, hopefully this will be fixed in the 3.0 release, which people at Oracle Open World said would be focused on removing current limitations (including migration between different types of sun4v processors).

Over a year ago Oracle closed the source for OpenSolaris, leaving contributors outside of Oracle left in the cold. That was a huge problem for the adoption of Solaris which had finally begun to rise again. Recently Solaris 11 was released, but without the source, this was likewise a huge problem, but now also for enterprise customers who are using and paying for Solaris. DTrace have lost part of it's value for Solaris 11 compared to the now dead OpenSolaris.

A perhaps even large problem is that when Oracle closed Solaris, many, many of the core developers left Oracle. Several of them now works outside of Oracle contributing to illumos, but these changes can not be ported back into Solaris 11 as long as Oracle keeps the source closed.

Solaris 11 has features not available in illlumos, but I chose to use illumos instead of Solaris 11 since I have access to the source and I am not locked to one OS-distribution. Also the licensing for Solaris 11 does not allow me to use it for small deployments without buying a whole support contract. If I "upgrade" a zpool to use new features available only in Solaris 11 I will be unable to import the pool using the free ZFS implementation that in illumos based distributions such as OpenIndiana or Nexenta or other operating systems such as FreeBSD.

I think this is a terrible move by Oracle, not only are the alienating new customers, they are also locking out great engineers who have implemented revolutionary features into Solaris. As Bryan Cantrill pointed out in his LISA '11 speech, Oracle has not made any official announcement about what they have done to OpenSolaris or what their future plans for the source are, this is very troubling and ignorant.

Solaris 11 is a great OS but it being treated terribly by Oracle, Oracle seems to think that the best way to make a profit out of Solaris is to keep it closed for everyone else, I don't agree.

I have had a few hours to try the final Solaris 11 release, overall I think it is far more stable and polished than the previous "Early Adaptors" release. Besides the fact that I am unable to use semi-old SPARC gear to test the release since only the latest generations of hardware are supported I have found few real problem so far.

The new packaging system finally fixes what I believe have been the biggest problem for Solaris the last five years or so, it is now repository based and it is simple to install software and dependencies are automatically solved. No more hassle of downloading and installing multiple software packages from SunFreeware to resolve dependencies. This also makes packaging faster and safer, brining the whole system to a known level and always with a safe recover option since it is used in conjunction with ZFS clones.

Zones are a fantastic tool for security/workload separation and virtualization so it's good to see that so many enhancements have been done in this area. The perhaps most noticeable is of corse that they now also use the new IPS system for packages and that makes a vanilla zones very lightweight without the hassle of a sparse zone. NFS service can now finally be provided from inside a zone. There is a tight integration with the new crossbow network virtualization making is possible to limit bandwidth to zones, use DHCP in a zone without having a separate NIC and build internal networks between zones inside a single Solaris 11 instance.

Imagine the power and flexibility of an T4-4 with 256 CPU threads and 1TB of memory running 50 zones with several high bandwidth/low latency networks inside the machine with no latency or overhead caused by virtualization.

Unfortunately the X86 version with the graphical desktop seems to be somewhat unstable compared to the Express release, I think it's related to the upgrade of the X server. I have been unable to use my laptop with to displays with the final release.

Solaris 11 is however focused on usage in servers and it seems stable for that, I have only found one disturbing problem so far, sharing ZFS filesystems does not seem to work ( zfs set -o sharenfs=on), but you can share each individual filesystem with the share command. Sadly if you are evaluating this you will probably have to wait until next year when there is a full release of Solaris since no updates are provided without a service contract. If you work for Oracle this is something you might want to fix for everyone, or tell me what I'm doing wrong.

Solaris 11 have many other new features such as per-user encryption of home directories with ZFS crypto, a new mirror/raidz hybrid bloc allocator for ZFS, numerous security enhancements among other thinks. I have only named a few of the changes I will probably keep posting Solaris 11 stuff as I find something interesting that is not directly highlighted it Oracle own what's new documents.

Update: As pointed out in the comments sharing of NFS together with ZFS works a bit differently now. If you share an existing dataset you have to set the share property. However if you set the sharenfs property when creating the dataset it works as in previous versions of Solaris 11 Express, OpenSolaris etc. Move information available in the documentation here.

Solaris 11 is available for download "SunOS Release 5.11 Version 11.0", based on build snv_175b.

There are of course many changes since Solaris 10, most of them have been available in the latest build of OpenSolaris but there are some new that are unique to the final release of Solaris 11.

Install images are available for download and works on all current SPARC machines which is the T and M-series. There are also images available for X86-based machines which also can be used in VirtualBox. Here is a quick reference for the brand new packaging system: IPS one liners.

I will post more detailed follow-up after I've had time to test it for more than a few hours.

Oracle Solaris 11 11/11 – What’s new
Download Oracle Solaris 11
Future features of Solaris 11

Solaris 11 will be release 2011年11月09日 (2011年11月11日 was not optimal for some reason).

Oracle will host a launch event in New York and you can register to attend to the live webcast.

Even if I have abandoned Solaris 11 for OpenIndiana for storage related installations, Solaris 11 have it's obvious place on bigger iron in the datacenter or for any mission critical workload that needs enterprise support. I would gladly have continued to use Solaris 11 for storage but the change made by Oracle to ditch the community and move to closed source and stricter licensing prevents that.

This will make fantastic features such as crossbow, IPS, Native CIFS and COMSTAR available for use in production environments. Many enhancements for zones have also been made, they can for example be NFS servers in Solaris 11.

Also if you want to make the most use of the new SPARC T4, Solaris 11 is the best choice since not every change usable to the T4 have been ported back to Solaris 10 8/11.

If you pay for support of Solaris 11, please demand that Oracle gives you access to the source, DTrace will loose it's value otherwise and I think Oracle needs to hear that.

Oracle Solaris 11 Launch webcast

A quick summary of he most interesting news from the first days of Oracle Open World that are related to Solaris/SPARC.

• Solaris 11 is slightly ahead of schedule, it was supposed to be released by the end of the year.

• OpCenter is now included in your support contract for no additional cost, including virtualization!

• A new Oracle VM for SPARC will be release (3.0) that supports live migration between processors with diffent clock frequency and even between T2 and T3.

• Future releases of Oracle VM for SPARC will try to remove limitations, for example enable live migration while using dedicated PCI hardware in guest domans.

• Some work on Solaris 12 have already been started

• Solaris 11 will have more focus on virtualization/cloud

• SPARC super cluster will support exadata instances (one per T4-4)

Now i'm of to a new session, a deep technical review of the SPARC T4.

• Out-oforder execution
• Dual instruction issue
• Data/instruction prefetch
• Deeper pipeline
• MMU Page size up to 2GB
• Level 3 cache

"All of these characteristics in the SPARC T4 have yielded improvements in single-thread performance by 5X while retaining networking and throughput performance equal to that of previous multicore processors from Sun/Oracle."

Oracle have now announced the new line of servers based on the T4 processor. They come in three configuration ranging from 1 CPU/8cores with up to 256GB memory to 4 processors and up to 1TB of memory. There is also a blade version which has one processor and up to 256GB memory. The T4-1 and T4-2 processors are running at 2.85GHz and processors in the T4-4 are running at 3.0GHz.

Oracle claims a 5 times increase in single thread performance compared to previous T-processors while keeping the same high thread count and cryptographic acceleration. All the new systems support Oracle VM for SPARC, previously known as Logical domains (LDOMs). LDOMs will finally be an option for general workloads, and the supported version 2.1 also support live migration.

Supported Solaris releases are Solaris 10 8/11, Solaris 11 or Solaris 10 9/10,10/09 with 10 8/11 Patch Bundle.

SPARC T-Series
Oracle's SPARC T4-1, SPARC T4-2, SPARC T4-4, and SPARC T4-1B Server Architecture
SPARC T4 announcement

A much updated pre-release of Solaris 11 is now available on OTN, it is supposed to contain all the final functionality of Solaris 11. Text installer for SPARC/X64 is available. live-CD for X64 as well as repository images.

This release should work all the same (64-bit) X86 systems as Solaris 11 Express, but on for SPARC you must have a T or M-series class machine, support for older UltraSPARC based system has been removed.

Thanks to Craig S. Bell for pointing this out in previous comments, I did not expect any new release before Oracle World.

Update: The build of this release is snv_171, the Solaris 11 Express release was based on snv_151a.

Solaris 11 Early Adopter Program

As other sources also have now also found out that the next release of Solaris will be named Solaris 10 8/11 which indicates a August release. As written previously we should expect Solaris 11 around Oracle World 2011, a more precise date have been disclosed in Joerg's blog: November 2011, a year after the Solaris 11 Express release. He also speculates in a 11/11/11 release driven by marketing. I will be very interesting to se what the last year of engineering under Oracles guidance have to deliver in the final Solaris 11 release.

Last but not least the OpenIndiana project is just about to release build 151. Update: there are even beta images available here.

Solaris 10 update 10 ZFS refresh

In the Solaris online forum last week there where refereces to another scheduled Solaris 10 update, update 11 (s10u11), schedule for release the second half of 2012.

There where no other information regarding the release but it's good to see that Solaris 10 will continue to be updated a while after Solaris 11 have been released.

The scheduled also included the expected Solaris 10 8/11 release and the Solaris 11 release which probably will be available at Oracle World 2011. There was also an update scheduled for Solaris 11 the first half of 2012.

Update: You can look at the roadmap here.

A rare opportunity is coming April 14, Oracle is sharing information on their Solaris strategy and upcoming Solaris 11 features.

The agenda taken from the registration site:

"9:00 a.m. - 9:45 a.m. PT
Oracle Solaris Strategy Overview
Bill Nesheim, VP Oracle Solaris Engineering

9:45 a.m. - 10:00 a.m. PT
An Industry Analyst's View of the Operating System Market
Gary Chen, IDC

10:00 a.m. - 10:45 a.m. PT
Manage Your Deployments With Image Packaging System and the Automated Installer
Bart Smaalders, Oracle Solaris Engineering
Dave Miner, Oracle Solaris Engineering
Glynn Foster, Oracle Solaris Product Management
Isaac Rozenfeld, Oracle Solaris Product Management

10:45 a.m. - 11:30 a.m. PT
Get More out of Your Oracle Solaris Environments With Virtualization
Dan Price, Oracle Solaris Engineering
Nicolas Droux, Oracle Solaris Engineering
Duncan Hardie, Oracle Solaris Product Management

11:30 a.m. - 12:15 p.m. PT
Learn How All New Features in Oracle Solaris 11 Raise The Bar For Operating Systems
Markus Flierl, Sr. Director Oracle Solaris Engineering
Liane Praza, Oracle Solaris Engineering
Joost Pronk, Oracle Solaris Product Management

Register and get more information FUSE team.

Projected Nevada integration build is snv_163."

OpenSolaris FUSE project

Solaris 11 is not the only Solaris release we can expect to see in 2011. The next update of Solaris 10, update 10 or Oracle Solaris 10 8/11 which seems to be what Oracle are aiming for in both name and release date is also coming in 2011. Earlier posts have included some of the expected content but the name and release window recently surfaced.

I found this in a publicly available bug report:

  Oracle Solaris 10 8/11 s10x_u10wos_02 X86
Copyright(c) 2010, Oracle and/or its affiliates. All rights reserved.
 Assembled 22 November 2010
 Early Access Build for Evaluation and Test Purposes Only.
 NOT SUPPORTED FOR PRODUCTION USE. Not patchable.

With the 8/11 target it seems that Oracle wants to release the update in time before Oracle World 2011, perhaps to have a new Solaris release to talk about in the sessions but it also seem likely that it will support the upcoming SPARC T4 (Yosemite Falls) processor that could be launched at the same time ( Oracles road maps have pointed to the second half of 2011 and Rick Hetherington stated in the recent interview that it was less than 12 month away ).

Another interesting thing is that in the Oracle roadmap, Solaris 11 was plotted at the same time as the SPARC T4, sometime in the second half of 2011. Could this mean both Solaris 11 and Solaris 10 8/11 will be released at the same time and at Oracle world 2011? Given all the announcement at last Oracle world it would make sense.

Take a look at the slides of John Fowler, Executive Vice President of Systems, page 8 for the roadmap.

This gives some more substance to Fowlers statement that they will continue to produce Solaris 10 updates for some time, the timeframe for the release was probably not very hard to guess but it is interesting that there now seems to be a preliminary name. It looks like next year will be an interesting year for both Solaris and SPARC.

Here is some information on some of the expected content of the update:
Solaris 10 update 10 update

A nice addition to easily use of ZFS encryption for home directories will be available in a future version of Solaris 11. This will allow you to have your password as key for your home dataset which is automatically used at login:

"For users with local ZFS storage we want to provide a very simple and as transparent as possible way of using encrypted ZFS datasets. The target for this is laptops and systems with local ZFS storage for the users home directory.

The goal is to provide as seemless as possible a way to have an encrypted home directory and additional encrypted datasets below the home directory. A new PAM module pam_zfs_key.so will be introduced. This module supports only pam_sm_setcred(3PAM) and pam_sm_chauthtok(3PAM), pam_sm_authenticate(3PAM) is provided but always returns PAM_IGNORE.

It assumes that the users login passphrase is also the passphrase used to protect thier ZFS encrypted home directory and will ensure that when users change their password the passphrase used for deriving the wrapping key for their encrypted ZFS home directory is changed as well."

Bugid: 6983112