It's a UNIX system!

Another year have passed with lots of changes for Solaris users, some of them good but others quite bad. Most of them are of course related to the Oracle acquisition of Sun which completed January 2010.

Besides ZFS data set encryption the upcoming Solaris Express release (Which seems to be tagged 2010.11) will contain several enhancements to ZFS besides dataset encryption that was mentioned earlier:

• RAID-Z/mirror hybrid allocator
"Normally mirror vdevs contain mirrored data, RAID-Z vdevs contain RAID-Z data, etc. However, for latency-sensitive metadata, we can use a mirrored layout across the children of a RAID-Z vdev. This ensures that such metadata can be read in a single I/O."


• Support for decompression, checksumming and raidz in zdb -R will make it much easier to debug ZFS errors on disk or just to learn more about the on-disk format of ZFS.


• Performance improvements for listing ZFS filesystems.

The downside of this is that these are features only present in Oracles internal gate and not other distribution besides Solaris 11/Express and the S7000/FishWork appliances can use them until and if Oracle releases the source after Solaris 11 is released.

I hope the Solaris 11 Express license will allow me to use Solaris 11 for my private storage needs, S7000 are nice and deployed at work but still a bit too expensive for my datacenter at home ;)

6977913 RAID-Z/mirror hybrid allocator
6757444 want zdb -R to support decompression, checksumming and raid-z

Support for encrypted ZFS datasets have been integrated into Solaris Next Development, build snv_148. This means that there is a good chance it will be available with the first release of Solaris 11 Express which is expected later this year.

This feature have been in the works for years with the original feature request filed in 2003. This will be a welcome addition for ZFS users with sensitive data and/or laptops.

After this only one major component is missing from ZFS, the block pointer rewrite feature. It will make it possible to encrypt/compress/deducplicate existing data in the pools as well as shrink pools and defragment them without the need to move the data out of the pool and in again.

Update:
As noted I failed to include a reference, here is bugid 4854202.

ZFS on disk encryption support
Darren in ZFS crypto video interview
ZFS crypto pushed to next year

The first release of Oracle Solaris 11 Express will be later this year with a full release of Solaris 11 in 2011. There is not much more information on what this release will contain but it is probably a later build of OpenSolaris rebranded for Oracle/Solaris 11.

A interesting note is that Solaris 11 will be powering the new Exadata X2-8 and in the new cloud-in-a-box Exalogic .

Oracle Outlines Next Major Release of Oracle Solaris

There is a another update planned for Solaris 10 after the yet to be released Solaris 10 9/10. It is still very unclear and probably not fully decided what this update will include, but i'we found indications on the following:
John Fowler mentioned the upcoming Solaris 10 updates in the systems webcast:

"We will continue to produce Solaris 10 updates for some time for customers who are used to and have deployed Solaris 10 and want to take advantage of new hardware platforms. Comming up with new product introductions this year and then extending in the previous years"

While looking at changes for the future packaging system in Solaris I stumbled on a putback that included pybonjour in the pkg(5) gate. Apparently Solaris of the future is going to be using zeroconf for auto discovery of packages repositories. Looking through other OpenSolaris project it seems like this and NWAM also will be using auto discovery. OpenSolaris have had support for mDNS discovery since build 72 but the pkg use-case is what got me interested.

This will allow clients on the same network to discover mirrors of the repository and use them to ease the load on the primary repository. The mirroring service is easily enabled:

# svcadm enable application/pkg/dynamic-mirror

From pkg.depotd(1M):

"The pkg depot is also able to act as a mirror server for local client images from pkg(5). This allows clients that share a subnet on a LAN to mirror their file caches. Clients may download files from one another, thereby reducing load on the package depot server. This functionality is available as an alternate depot service configured by smf(5). It uses mDNS and dns-sd for service discovery."

From pkg(1):

"mirror-discovery
(boolean) Mirror-discovery tells the client to discover
link-local content mirrors using mDNS and DNS-SD. If this is
set to True, the client will attempt to download package content
from mirrors it dynamically discovers. To run a mirror that
advertises its content via mDNS, see pkg.mdnsd(1m)."

# pkg set-property mirror-discovery True

Network Auto Magic (NWAM) will include service auto discovery of DNS and naming services:
http://hub.opensolaris.org/bin/view/Project+nwam/service-discovery

Aside from BP-rewrite, encryption is probably the most awaited ZFS feature, but there have been sparse information regarding the status of both projects for about a year.

The ZFS crypto project which are going to deliver on-disk encryption for ZFS have now at least showed a small life sign. This is after a long time of radio silence from Darren, Sun and Oracle regarding the ZFS crypto project since May last year when "PSARC/2009/443 ZFS Crypto Updates" was published. Now there are references to a new PSARC, but it's not public so we have only the name to to look at: PSARC/2010/329 ZFS Crypto additional keysource types.

The last crypto releated PSARC from laste year, which was public:
PSARC/2009/443 ZFS Crypto Updates

A supposedly internal memo from Oracle has been posted to the opensolaris-discuss list. The most important part of this is probably that it finally states that OpenSolaris as a distribution is dead, Oracle will no longer make OpenSolaris available in binary form as a release or in bi-weekly development builds. The source will for most parts continue to be open, but will not be pushed out in realtime as it does today, it will be posted after a Solaris 11 release has been made available. This way Oracle will always be first in line to productize their latest innovations and fixes. neither will Oracle continue to post most of the PSARC externally. There will however be options for partners to gain early access to source and binaries from Oracle. Solaris 11 will be made available in an Solaris 11 Express form as a preview to the next release of Solaris 11. It will be free to use for developers and has an optional support plan.

As am enterprise customer the message is great, to know that Oracle really believes and invests in Solaris and that Solaris 11 will be made available early. The availability of the source after a release is also very important. I also feel that Oracle genuinely are going to make Solaris even better and are willing to put in the resources to do so. Looking at all the new features available in OpenSolaris since Solaris 10, Solaris 11 will be a huge leap forward for Solaris.

As someone involved and personally interested in the OpenSolaris community this is not very good. I will be cut of from the day-to-day changes in the source and no free OpenSolaris distribution will be available to attract new customers outside of the large enterprises. I think there will be some other distribution that will try to take the place of OpenSolaris, but with the access to the source cut, except for snapshots from releases it will not be as interesting as the weekly builds have been. It would also stall months behind Solaris 11 as it has to wait for the release to be made available.

If Oracle just would make Solaris 11 available at a lower price with obviously less support commitment it might come something good out of this even for businesses with smaller budgets. I think this is really something Oracle must do, otherwise the adoption rate of Solaris will decline drastically. It would neither undercut the sales of the enterprise support contracts, they will still be sold, mission critical deployments are not run with patch or mail-only support options. I've been involved with several smaller startups which love Solaris and like to use it, but neither need or can afford the support contracts for Solaris today, it would be sad if there was no open option for them to continue to use Solaris for a, for them, reasonable price.

So in short, I think that this will make Solaris even better for the large customers, Oracle will put much effort in keeping Solaris the most capable and advanced operating system available. But smaller companies, startup and hobbyist will have a harder time to adopt Solaris 11. I will continue to use Solaris 11 express for my private servers if possible, but I can obviously not be as involved as before since I get cut off from frequent builds and changes to the source.

The full memo is available here.

John Fowel, executive vice president of system, now at Oracle have disclosed some a timeframe for the Solaris 11 release. It should be released in 2011 and contain many of the new features and enhancements that has been implemented in OpenSolaris. It should even contain a superset of what is currently in OpenSolaris, with new technologies created after the Oracle acquisition.

I've previously discussed that Oracle might be focusing on Solaris 11 instead of OpenSolaris, and that seems to have been correct. They does not mention any new release of OpenSolaris in this web event.

This also confirms the name "Solaris 11" for the next Solaris release which always officially have been Solaris next or Solaris.next since marketing wants to have the options open for the name of the next release. Many of you probably remember the change from Solaris 2.7 to Solaris 7.

I will write more when I and if more information is made available, the web event will be made available online later.

You can read the whole article from Serverwatch here.

Update: Eweek has some more information regarding this while awaiting the webcast. Oracle plans to double the performance of SPARC every year through 2015: Oracle Outlines SPARC, Solaris 11 Plans.

Read-only zpools have now been implemented in the source as proposed in PSARC/2010/306.

Previously individual filesystem have could be mounted as read-only, but now the whole pool can be imported this way:
# zool import -o readonly=on zpool01

When importing a pool with read-only, no writes will be performed to any of the devices in the pool, they are even opened read-only.

From the PSARC:
"When the 'readonly' property is 'on' (read-only) the following actions are performed:

1). pool is loaded but transaction processing is disabled
2). all filesystems and zvols are mounted in read-only mode
3). any intent-log replays are deferred (any pending synchronous
writes will be replayed once the pool is imported read-write)

The following restrictions apply when a pool is read-only:

1). Attempts to set additional zpool-level properties during import
are ignored.
2). All filesystem mounts are converted to include the 'ro'
mount option.
3). Additional mount options may be set on a temporary basis.

A pool which has been imported read-only can be restored back to read-write by exporting and reimporting it."

PSARC/2010/306 Read-only ZFS pools

A really nice feature have been implemented for ZFS, which allows you to list all file changes between two snapshots of a ZFS filesystem. The PSARC was filled in March but it has now been implemented. It will have a simple syntax just like diff(1):
# zfs diff zpool01/fs01@snap01 zpool01/fs02@snap02

With output in this format:

M /myfiles/
M /myfiles/link_to_me (+1)
R /myfiles/rename_me -> /myfiles/renamed
- /myfiles/delete_me
+ /myfiles/new_file

From the PSARC:
" There is a long-standing RFE for zfs to be able to describe
what has changed between the snapshots of a dataset.
To provide this capability, we propose a new 'zfs diff'
sub-command. When run with appropriate privilege the
sub-command describes what file system level changes have
occurred between the requested snapshots. A diff between the
current version of the file system and one of its snapshots is
also supported.

Five types of change are described:

o File/Directory modified
o File/Directory present in older snapshot but not newer
o File/Directory present in newer snapshot but not older
o File/Directory renamed
o File link count changed

PSARC/2010/105 zfs diff

Since build 134 of OpenSolaris which once was destined to become the OpenSolaris 2010.1H release quite a lot have changed. If there was a release based on that build today it would feel a bit old giving all the changes now available. I took a look on some of the larger or in my opinion more noticeable changes and made a list. It makes you want to get your hands on a Solaris Next developmen build soon, or at least update the home rolled builds to b146 once it is finished. The development efforts does not seem to have decreased, it's a shame we still don't have any public builds to test all these new features.

This list is just a overview of the enhancements, there have been over 1300 putbacks containing over 2200 bugfixes including many new features. This is only the OS/Net consolidation with the core part of OpenSolaris not including PKG, X, JDS or the many free software packages available for installation from the package repositories.

PSARC/2010/014 Open Source elxl replacement
PSARC/2010/042 increase max TCP_INIT_CWND
PSARC/2008/532 NWAM Phase 1
PSARC/2009/576 pam_krb5 PKINIT support
PSARC/2010/004 Logical Domains Information API and ldminfo
PSARC/2010/067 Interim modernization updates
PSARC/2010/045 x86gentopo enumeration of direct attached SATA
PSARC/2009/354 Always on / no reboot Solaris Audit
PSARC/2009/364 dlstat and flowstat
PSARC/2010/041 USB CDC ECM driver
PSARC/2010/062 increase number of realtime signals
PSARC/2009/689 Audio DDI Simplifications
PSARC/2009/534 SMB/CIFS Standalone DFS
PSARC/2010/009 Modified ZFS passthrough-x ACL inheritance
PSARC/2010/032 EC and SHA2 for KMF
PSARC/2010/013 SMF Early Manifest Import
PSARC/2009/104 Hot-Plug Support for ACPI-based Systems
PSARC/2009/306 Brussels II - ipadm and libipadm
PSARC/2009/421 Open Fabrics User Verbs (OFUV) - primary kernel components
PSARC/2009/019 SAS Management Protocol library
PSARC/2010/102 ikeadm dump algs
PSARC/2010/085 IPoIB Administration Enhancement
PSARC/2010/029 Improved ACL interoperability
PSARC/2010/101 in.iked preshared key file extensions
PSARC/2010/048 zfs-based ndmpd backup
PSARC/2010/119 "Console User" assignment, logindevperm and virtual console update
PSARC/2010/043 Reliable Datagram Service v3
PSARC/2009/377 In-kernel pfexec implementation.
PSARC/2010/108 zil synchronicity
PSARC/2010/127 ipadm hostmodel property
PSARC/2010/155 sshd(1M) PAM Service name options
PSARC/2010/174 TMPDIR support for Solaris tar
PSARC/2009/646 bd - generic block device driver
PSARC/2010/028 BSD List Interfaces
PSARC/2010/181 PRIV_SYS_RES_BIND privilege
PSARC/2010/106 DTrace TCP and UDP providers
PSARC/2010/151 new socket options for TCP timers
PSARC/2010/083 Kerberos Profile API
PSARC/2010/188 PKCS#11 URI parser for libcryptoutil
PSARC/2010/177 KMF Certificate Name mapping extensions
PSARC/2010/144 lofi(7D) in non global zones
PSARC/2009/590 Socket Filter Framework
PSARC/2007/477 Drivers registering devids
PSARC/2010/222 Relaxed Type Requirements for SMF Profiles
PSARC/2010/215 Add libstmf interfaces to set/get default stmf properties
PSARC/2009/505 IRM Framework Extension(s)
PSARC/2009/430 Default system CA (X.509) Certificates
PSARC/2010/195 PKCS11 KMS Provider
PSARC/2009/042 max-processes rctl
PSARC/2010/240 Allow ranges for the 'cpus' datalink property
PSARC/2010/166 layer-3 net properties for exclusive-IP zones
PSARC/2010/239 OFUV Userland Interface
PSARC/2010/235 POSIX 1003.1-2008 *at(2) syscalls
PSARC/2010/241 userattr
PSARC/2010/111 exclusive IP for s10c
PSARC/2010/269 KMF Certificate Validation Enhancements
PSARC/2010/164 interfaces for basic install network configuration
PSARC/2010/113 Boot Block Downgrade Avoidance
PSARC/2010/234 IBTF 2010.Q2 Enhancements
PSARC/2009/617 Software Events Notification Parameters CLI
PSARC/2009/618 snmp-notify: SNMP Notification Daemon for Software Events
PSARC/2009/619 smtp-notify: Email Notification Daemon for Software Events
PSARC/2010/278 FMA/SMF integration: instance state transitions
PSARC/2010/279 Modelling panics within FMA
6914369 enable s10brand on system booted under xvm
6889694 COMSTAR kstat to track IO statistics at the ITL
6655655 Single phase COMMIT processing for NFS3 and NFS4 server
6830088 Fast reboot support for SPARC platforms
6871356 Sata framework should support INQUIRY VPD page 0xB1
6782858 Solaris needs Broadcom 5718 network support
4404421 RFE: tar command -z and -j support
6202362 Solaris tar can't unpack files created with GNU tar
6922272 SATA framework does not handle>2TiB disks
6939349 RFE: Update ksh93 to ast-ksh.2010年03月09日
6923815 Implement FMA functionality in qlge driver
6889322 Add NLP2020 PHY support to nxge
6874797 Solaris needs to support I/O subtree reparenting
6945131 SMF should support multiple administrative profiles
6951174 Intel microcode 2010年02月09日 update
6923890 Add FMA to SiliconImage si3124 sata driver
6946100 Add machine check and performance counter support for new member of Nehalem family
6937229 Solaris needs to support BCM5717 and BCM5724
6953601 improve logical unit creation time for zvol backed logical units
6910752 mpt_sas driver performance can be improved
6494473 ZFS needs a way to slow down resilvering
6917066 zfs block picking can be improved
6932959 ZFS allows too many devices to be faulted
6716117 ZFS needs native system attribute infrastructure
6675946 'zpool status' should show the progress of resilvering for individual disk.
6668666 zpool command should put a bootblock on a disk added as a mirror of a root pool vdev
6733267 Allow a pool to be imported with a missing slog
6959846 DMU traverse prefetch size should be a global tunable
6866610 Add SATA TRIM support
6908227 pyzfs needs to move to python 2.6

If any of the links point to nowhere or in the case of PSARC:s empty or almost empty directories there are no public information available besides the source.

Last night i spend a few hours backporting recent changes from the ON gate to my own gate that is based on ONNV_134. I wanted to get a build that is as close to OpenSolaris b134 as possible while pulling fixes and a few enhancements for ZFS, COMSTAR, SATA and NFS. I went quite easy, the main problem was that i did want to get as close to the latest ZFS bits without including the changes that removed the iSCSI daemon (not because I will use it but it was part of 134). With a few modifications to the patches i now have a ONNV_134 build with a much more current ZFS. My plan is to test this for a while and then use it for my ZFS storage nodes at home, where of course ZFS is the most important part. There have been quite a few fixes for ZFS since February when build 134 closed.

This is obviously only something I will be playing with until Oracle decides to release a Solaris version based on a more current snapshot of the OpenSolaris source. Private builds and independent distributions can be both fun and useful but does not go trough the amount of testing as the "official" releases does, that kind of fragmentation decreases the testing for all distribution.

In the case of ZFS i like to use OpenSolaris and preferably a more current one. It is the growing ground of Fishworks, the software part of the S7000 series which I use for some production environments. The ZFS code in those should go trough a fair amount of testing both by Oracle and all of the thousands of enterprise deployments.

All this said, times goes by and with no binary release in sight a community effort to build and deliver binaries for the community for testing, preparation for the next release and recruit new users might be needed. But it would be sad if we ended up divided between an Oracle paid distribution and a community one with no involvement by Oracle other than providing almost all of the source and new features of the ON code, which they could decide to stop delivering or only deliver parts of it. But all this is in the hands of Oracle.

A common request from smaller shops or private users of Solaris is that they want email notifications when something is wrong with their system. This is most common with ZFS filers, they might not have any surveillance systems implemented but want to be sure to get a notification if something goes wrong with a zpool. A quick fix to this is of course a small script which can dump zpool or FMA errors over email but it is not a standardized or complete solution.

Now there have been a large putback in OpenSolaris that among other things delivers this functionality. FMA faults and SMF events can with these change be configured to deliver notifications over mail or SNMP. It also enables the fault management daemon in local zones so that they can deliver SMF state changes from within the zone. Using this framework you can also configure any of your own SMF services to deliver state change notification over any of these two protocols.

PSARC/2009/617 Software Events Notification Parameters CLI
PSARC/2009/618 snmp-notify: SNMP Notification Daemon for Software Events
PSARC/2009/619 smtp-notify: Email Notification Daemon for Software Events
PSARC/2010/225 fmd for non-global Solaris zones

The SATA TRIM command have been implemented in the OpenSolaris source, build 146. This can be used by filesystems to prevent performance degradation over time on SSD:s by giving the garbage collector feedback on which blocks are no longer in use. This is bugid 6866610. It it still not implemented in ZFS but this is the ground stone for making that possible and there is a feature request for that also 6957655.

Looking into the future the similar SCSI UNMAP command would be of great use for thin provisioning, both for thin devices located on a ZFS LUN and for ZFS residing on a thinly provisioned LUN for example in a SAN.

The later is described in this bug report 6913905.

Currently if you loose a non-redundant log device in your zpool it becomes impossible to import, there is no way to exclude the log device and import the rest of the pool. There are however light on the horizon, PSARC 2010/292 proposes a solution to this problem with a new switch for the zpool import command.

From the PSARC:
"This fast-track introduce a new command line flag to the
'zpool import' sub-command. This new option, '-m', allows
pools to import even when a log device is missing. The contents
of that log device are obviously discarded and the pool will
operate as if the log device were offlined."

PSARC 2010/292 zpool import despite missing log

Update
This has now been implemented in the source, the changeset is here.

There is a PSARC awaiting approval for providing on of the most long standing enhancements requests for Solaris Zones: NFS Shares from within a local zone. This functionality will not be limited to NFS and will initially also support CIFS. It seems to be planned for Solaris Next only, so no backport to Solaris 10.

From the PSARC:
"Patch binding is requested; however, there are no plans to backport any of
the proposed changes.

The proposed changes are needed to support an NFS server in a non-global
zone; however, the changes are not specific to NFS. They would also be used
when enabling any file sharing protocol (ie. CIFS) server for non-global zones."

PSARC/2010/2080

Good things come to those who wait? I've seen some indication that the latest release of Gnome (2.30) have been integrated into the snv_140 build. It's tagged as OS-2010.06, but the OpenSolaris 2010.06 name seems to have been in use since April so that might not mean anything. Firefox also have been upgraded to 3.6.x in build 135 and onwards. Time slider have got an update which includes replication to external filesystems (PSARC/210/129). These updates are available in Solaris Next Development builds post snv_134 so they have not been released even in the development repository for OpenSolaris.

As a response in a discussion weather Oracle will invest more in Solaris 10 maybe to also use as base for Fishworks or what they doing I wrote something similiar to this in the opensolaris-discuss list:

Solaris 10 Update 9 will indeed contain some new features including a refresh of ZFS, but it will not replace the OpenSolaris kernel in Fishworks, backporting both the CIFS server and COMSTAR is far to much work.

I think Solaris 10 9/10 will be released at OpenWorld together with at least a preview of "Solaris Next", "Solaris 11" or whatever name it will get. Solaris 10 is too old to be doing major development work in, it is time to release a major version soon, some things can't be done without a new major release. This would look good for Oracle and their commitment to Solaris (not OpenSolaris), that they have a new major Solaris release where Sun just keep developing it for year after year (I'm not saying that was bad, but Oracle may put it like that, look at what Larry said about Rock).

What happens with OpenSolaris as a distribution is unclear, my opinion is that they should have released the second respin on 134 without support at least, now they just get bad will from the community, at least until something is released or they decide to spread some light on the situation.

Solaris next not OpenSolaris
Solaris 10 9/10 and second zfs refresh

Larry regarding Rock:
"At the start, ellison shut down one of schwartz’s pet projects -- development of the “rock” microprocessor for sun’s high-end sparc server line, a semiconductor that had struggled in development for five years as engineers sought to overcome a string of technical problems. “this processor had two incredible virtues: it was incredibly slow and it consumed vast amounts of energy. it was so hot that they had to put about 12 inches of cooling fans on top of it to cool the processor,” said ellison. “it was just madness to continue that project.”

Many of us have patiently been awaiting the 2010.xx release of OpenSolaris frustrated by the lack of communication from Oracle. After some digging I realized that something quite interesting has been going on for a while. Oracle themselves are not building OpenSolaris anymore, not since build 135. They are building "Solaris Next Development", this is from /etc/release:

Solaris Next Development snv_140 X86
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
Assembled 29 April 2010

This kind of makes sense, since OpenSolaris always have been the development branch for the Solaris.next, but the distributions have been named OpenSolaris for a while. I also found some tags like "osol_2010.05u4" which seems to be related to build 140. So have there been a change in plan for OpenSolaris 2010.xx release or is this just something for the future, "Solaris 11". But since build 134[a|b] which once should have been the release build for the next release still has the old names in them there might sill be a "OpenSolaris" release.

This is also makes sense since Oracle is mostly talking about Solaris, not OpenSolaris, Solaris seems to be the way forward, time will tell (since Oracle doesn't) in which form. It looks like it will become more like a single distribution, Solaris, with some development releases, OpenSolaris had it's own life and even support side by side with Solaris while still being the bed for development.