OpsCenter 12c 12.4, Patches, and Solaris 11.4 SRU 48
Abstract
Sun Microsystems purchased a company, which performed data center management, across multiple platforms, and then decided to merge it into Solaris. The Sun Connect product was born, to help deliver patches more effectively. The N1 Compute Initiative was born, to treat all systems in the data center as a single entity. OpsCenter was made available for all customers, to do local provisioning, patch, health, and reporting. Oracle purchased Sun Microsystems and had their own management framework called Oracle Enterprise Manager, but it was always short on handling hardware through provisioning hardware & management of the ILOM hardware [without an OS]. Oracle refers to OpsCenter as Oracle Enterprise Manager OpsCenter.
Recent History
Oracle had been updating OpC pretty aggressively in 2022, this article talks about the path to accomplish this. The first set of updates was associated with Log4J vulnerability in December 2021, but introduction of Oracle Solaris 11.4 SRU 48 on agents actually caused a disconnection to the OpsCenter server, and some aspect of this procedure must be followed in order to restore connectivity to OpsCenter from managed Solaris 11.4 servers newer than 11.4.48.
OpsCenter Information Center
One of the most important sections to review is the OpsCenter Information Center, within Oracle's Support Network, for understanding what OpsCenter is and what updates are occurring.
Information Center: Overview of Enterprise Manager Ops Center (Doc ID 1507780.2)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=1507780.2
As of the writing of this article, there have been several CPU (Critical Patch Updates),
stemming from a Log4J vulnerability discovered in the industry in December 2021.
This is a good place to start, regarding the latest news on OpsCenter.
OpsCenter 12.4 Release
Oracle upgraded OpsCenter to the 12.4 release in 2019.
Release Announcement - Oracle Enterprise Manager Ops Center 12c Release 4 (12.4.0.0.0) ( Doc ID 2532906.1 ) April 2019
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2532906.1
The documentation is readily available to everyone, without My Oracle Support (MOS):
Oracle Enterprise Manager Ops Center 12c Release 4 (12.4)
https://docs.oracle.com/cd/ops-center-12.4/index.htm
Base Operating System
Ops Center 12c Release 12.4 is supported on Solaris 11.3 as well as Solaris 11.4, making it a well suited management tool which can be used on nearly any piece of Solaris SPARC Hardware.
Most installations will be on newer hardware, with security updates available for Solaris 11.4. Solaris 11.4, as of the time of this writing, is on SRU 50... this is the 50'th month after original release of 11.4!
To avoid installing a buggy Solaris 11.4.0 & applying a half-decade of patches to Solaris 11.4.50, Oracle released free for private use Solaris 11.4 CBE or Common Build Environment.
As of the time of this writing, this starts at Solaris 11.4 CBE starts at SRU 42, but OpsCenter will need to be upgraded to the support repository in order to get the required Perl XML parser.
OpsCenter 12c 12.4 Pre-Requisites
There are OpsCenter bugs which require workarounds,
for installation on later releases of Solaris.
There is a known
BUG (32548385) with OpsCenter, introduced by Solaris 11.4.30.
The python 'mediator' in Solaris 11.4 SRU 30 is set to
3.7 instead of 2.7. Ops Center requires 2.7.
Ops Center Will Not Start After Upgrading
to Solaris 11.4.3- SRU 30 - Svc:/application/scn/ajaxterm:default is Restarting
Too Quickly (Doc ID 2760685.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2760685.1
This mediator pre-requisite must be present for
OpsCenter to start up on Solaris 11.4.30 and later.
There is a known BUG (33622838) with OpsCenter, introduced by Solaris 11.4.39.
Older release of Perl 5.26 in Solaris 11.4 SRU 30 is removed. OpsCenter EC
requires Perl 5.2.2.
Ops Center 12.4 upgrades to Solaris 11.4
SRU 39 on an EC will fail (Doc ID 2826475.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2826475.1
This perl release pre-requisite must be present for
OpsCenter to install on Solaris 11.4.39 and later.
OpsCenter 12.4 Installation
OpsCenter should be installed or upgraded to its most recent base version.
A basic installation with a Single Enterprise Controller is readily available:
https://docs.oracle.com/cd/ops-center-12.4/doc.1240/e59965/GUID-0DE73AE5-1B0B-4403-890A-8F632AD30131.htm#OPCSO525
After upgrade or installation, patches should be applied.
[Byzantine Mosaic: Jesus Christ Pantocrator, Courtesy Ricard MN Photography]
OpsCenter 12.4 Critical Patch Updates
Normally, Critical Patch Updates are cumulative, but this ceased to be the case after April 2022.
A circuitous path to follow, to deal with bureaucracy, was un-affectionately referred to as Byzantine.
This is where our Byzantine journey begins!
OpsCenter 12.4 April 2022 Critical Patch Update
The April 2022 CPU resolved a variety of issues, including Log4J.
(The January 2022 release, with Log4J patches, is also bundled in the April 2022 CPU.)
Ops Center 12.4 companion document for the April 2022 CPU (Doc ID 2865470.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2865470.1
Non-intuitively, this refers the user to another document, that says EM-only, but includes OpsCenter:
Critical Patch Update (CPU) Program Apr 2022 Patch Availability Document (EM-only) (Doc ID 2844807.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2844807.1
A "Bundle Patch" was created, which must be downloaded, and applied according to the instructions:
OPSS BUNDLE PATCH 12.2.1.4.210418 Patch 32784652 or later
https://support.oracle.com/epmos/faces/ui/patch/PatchDetail.jspx?patchId=32784652
These are the major bugs which have been resolved:
Bug 33601961 - Ops Center 12.4: CVE-2021-40438 Apache HTTPD server
Bug 33490456 - CVE-2021-2351: UPDATE THE C CLIENT LIBRARY FOR NNE VULNERABILITY
Bug 33735042 - CVE-2021-44832: APACHE LOG4J UPDATE TO 2.3.2, 2.12.4, OR 2.17.1
This must be applied only on an installation of OpsCenter 12.4
OpsCenter 12.4 July 2022 Critical Patch Update
The July 2022 CPU resolved a variety of issues...
Ops Center 12.4 companion document for the July 2022 CPU (Doc ID 2885006.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2885006.1
Non-intuitively, this refers the user to another document, that says EM-only, but includes OpsCenter:
Critical Patch Update (CPU) Program Jul 2022 Patch Availability Document (EM-only) (Doc ID 2867874.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2867874.1&_afrWindowMode=0&_adf.ctrl-state=1b5ay5nont_123#babfaaai
A "Bundle Patch" was created, which must be downloaded, and applied according to the instructions:
Ops Center UCE patches for Jul CPU 2022 Patch 34332927 or later
https://support.oracle.com/epmos/faces/ui/patch/PatchDetail.jspx?patchId=34332927
These are the major bugs which have been resolved:
Bug 34259326 - Ops Center 12.4: CVE-2022-22720 in Apache 2.4.52
Bug 34259352 - Ops Center 12.4: CVE-2022-22721 in Apache 2.4.52
Bug 34269953 - Ops Center 12.4: Upgrade OpenSSL to 1.1.1o
This must be only applied after the April release.
OpsCenter 12.4 October 2022 Critical Patch Update
The October 2022 CPU resolved a variety of issue...
Ops Center 12.4 companion document for the Oct 2022 CPU (Doc ID 2904332.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2904332.1
Non-intuitively, this refers the user to another document, that says EM-only, but includes OpsCenter:
Critical Patch Update (CPU) Program Oct 2022 Patch Availability Document (EM-only) (Doc ID 2888514.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=298046759019691&id=2888514.1&_adf.ctrl-state=1b5ay5nont_659
A "Bundle Patch" was created, which must be downloaded, and applied according to the instructions:
Ops Center UI/Other patches for Oct CPU 2022 Patch 34611523 or later
https://support.oracle.com/epmos/faces/ui/patch/PatchDetail.jspx?patchId=34611523
These are the major bugs which have been resolved:
Bug 33952830 CVE-2021-23450: DOJO UPDATE TO AT LEAST 1.17.0
This must be only applied after the April and July releases.
OS Upgrade to Solaris 11.4 SRU 48
This may seem counter intuitive, but there is a bug in SRU 48, which disconnects OpsCenter agent from the OpsCenter Proxy Controller, which reflects in a down agent in OpsCenter Enterprise Controller... and before you can fix this bug, the operating system must be upgraded to crash the agent, then the fix can be applied.
Fixing the OpsCenter OS Agent
With application of Oracle Solaris 11.4 SRU 48, the agent fails to connect to the management station.
A good article on the topic is:
Ops Center 12.4: CDOM Agents fail to start after a Solaris upgrade to 11.4 SRU 48 ( Doc ID 2892465.1 )
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2892465.1
At the root cause, there are a variety of bugs identified with SRU48.
Bug 34525568 : OpsCenter 12.4 CDOM Agents fail to start on Solaris SRU11.4.48 due to XMPP
Bug 34560282 : Ops Center Agent won't start after upgrade to 11.4SRU48 with S7 having Global zone
Bug 33876279 Local connections should skip TLS, SASL handshake
The patch to correct this problem, with SRU 48+, can only be acquired from the OpsCenter team, and is not generally available through the Oracle Patch Management system
A Service Request must be filed, specifically asking for the patch:
Patch 34525568
