Network Management

Network Management Basics: SNMP

Abstract:
From the dawning days of The Internet, the network grew from hosts on a wire, to hosts on a wire joined by a bridge to extend electrical signals, to a logical group of hosts on wires being defined as a network and joined to other networks via routers. Throughout these periods, there was always a need for a way to manage the infrastructure, and SNMP is The Internet Standard. The SNMP Internet Standard is a critical piece of total management business requirements.


The Network:
Every device on The Internet has a physical Hardware Address, to facilitate communications on it's own wire, and a logical Internet Protocol (IP) Address, to facilitate communications to other locations, provided through Routers. Someone on that network has to provide the logical IP Addresses, this person is normally some kind of network administrator. This person has some kind of responsibility to manage the network.


The Creation:
Networks were traditionally circuit switched, driven by a telephone company. In 1969, Steve Crocker developed a system to track agreed upon standards, called RFC's (Request for Comments), to facilitate interconnection of networks. The worlds first operational packet switching network came into existence, known as ARPANET (the Advanced Research Projects Agency Network) in 1977.

Ping:
As The Internet started to grow, basic diagnosis utilities were needed. Mike Muuss created a utility called Ping in December 1983. The most important function of this tool was the use of the ICMP Echo Request (type 8) network packet to another IP Address and the observation of the returned value.

The Manager may send an Echo Request or Ping to a remote device's logical IP Address to see if there is connectivity. If there is no connectivity, no packet is returned, or sometimes an Router in the path may return a message such as "Host Unreachable" or "TTL Exceeded" (packet time-to-live.) The manager may receive additional information such as the time it took for the packet to make the round trip.

Traceroute:
As networks continued to get more complex, the management requirements grew. Traceroute was born, attributed to Van Jacobson in 1987. Now, the manager could send a packet to an agent and receive a path of each router which the packet would traverse, bundling in the round trip times.

The Problem:
Such tools like "ping" and "traceroute" were critical for an individual manager to understand network connectivity - but neither provided in-depth information about the target agent device. A "ping" not being returned did not necessarily mean that the agent or target device is "down". A “ping” returning does not necessarily mean that the agent did not go down a few minutes earlier. A "traceroute" response to another location does not necessarily mean there is a problem with the agent or target device. These tools did not do much to allow a manager to understand history of a device or the intermediate network devices.
SNMPv1:
In 1988, SNMP (now referred to as Version 1) was born, through a variety of published RFC's. SNMP retained many of the advantages of ICMP and Traceroute (light-weight, avoided use of heavy TCP protocol), but brought to the world:

• programmable name for a device agent
• programmable location field for a device agent
• a description of the hardware and firmware on the device agent
• last-reboot counter of the device agent
• configuration, fault, and performance knowledge of interfaces (Interface Table)
• other physical hardware devices connected on the network (ARP Table)
• other neighboring logical devices connected on the network (Routing Table)
• passwords (called community strings) for basic protection
• framework for vendors to extend the management capabilities

This information is held in the MIB (Management Information Base) of the device - a database of information that each device holds regarding the health of the hardware, firmware, operating system, and applications.)


SNMPv1 was made up of RFC 1065, 1066, 1067. Updates included 1155, 1156, 1157. RFC 1213 (called MIB-1) was later updated 1156 (called MIB-2.)

SNMPv2:
In 1993, SNMP Version 2 was created through RFC's 1141-1452. Security was updated, but not widely adopted. Introduced was an efficient way to transfer information (GetBulkRequest) - which was readily adopted, to alleviate concerns of the protocol being "overly chatty".

SNMPv2c:
In 1996, SNMPv2c (Community-Based Simple Network Management Protocol Version 2) was introduced in RFC 1901-1908. The most important added the capability was to encrypt the password (community string) in transit, alleviating the concerns of the protocol being "insecure".

SNMPv3:
In December 2002, SNMPv3 was released, comprised of RFC's 3411-3418. In 2004, the IETF (Internet Engineering Task Force) designated SNMPv3 as STD0062 or a Full Internet Standard. Practically speaking, SNMPv3 adds encryption of the payload, to completely secure the protocol.

Modern Computing:
Today, nearly every modern equipment vendor, who instruments their internet equipment for management, bundles SNMP in their standard packaging - since SNMPv3 is The Internet Standard. This means that most equipment that plugs into a network via ethernet or wireless can be managed in an "agentless" manor (i.e. without loading any special additional components.)

Most Internet Infrastructure (i.e. computers, servers, routers, switches, etc.) allow for the following basic capabilities (sometimes using an internet standard, sometimes using vendor extension):

• Interface Configuration (administratively up, down; interface capacity)
• Interface Fault Status (Up, Down, Testing, Last-Change Time-stamp))
• Interface Performance Statistics (packets, bytes, errors, etc.)
• SNMP Agent Last-Reboot Timestamp
• Memory and/or Buffer Usage; Buffer Allocation Errors
• Flash and/or Disk Capacity and Usage
• Running Processes
• Installed Software
• CPU Usage
• Alert to a Manager when an Agent detects a problem

Customer Benefit:
Since SNMPv3 is The IETF Internet Standard, most equipment on a network can be reasonably managed without ever adding software to an end device. This means a service provider can provide greater insight into the health and performance of a customer estate with proper management software, especially historical trends when data is captured and stored in a database.

Difficulties:
SNMP is only a piece of the puzzle for managing a network.

• Business Processes
  A customer must know what business services are traversing a device to understand the impact of an outage or what business processes are at risk when assets in the estate are performing poorly.
• Security / End-of-Life Management
  A customer must know the version of the hardware and firmware is in the estate in order to understand when a security vulnerability or end-of-life equipment may place their business at risk.
• Logistics / Asset Management
  A customer must know what assets make up their network estate and where the assets are located in order to understand where impacts originate during faults or where security risks exist.
• Configuration Management
  A customer must know how to update the firmware on managed devices in the estate when defects in the software may be impacting business processes or creating security risks due to vulnerabilities.
• Performance Management
  A customer must know what "normal" operation of their estate is, collecting this data over time, in order to predict when faults will arise, so impacts to business processes are minimized.
• Fault Management
  A customer must know when faults occurred in the past, where they occured, when they occurred, what the problem was, and what the solution was - in order to understand the business impacts and create a strategy to mitigate future similar business impacts.

SNMP is a single skill, which can be leveraged to manage any number of device vendor, types, and model numbers. Network Management requires an expertise in all of the above areas, in addition to understanding SNMP.

This open up a prime opportunity for service providers with experience to assist customers since customers may only have experience with a particular device vendor/model/type or not have experience in SNMP.

Shut Down EMC Ionix (Voyence) NCM Port

Every try to shut down EMC Ionix (formerly Voyence) NCM (Network Configuration Manager) related tcp port services, by disabling /etc/init.d scripts, to find that there are still sockets being listened to?

The Problem

It was noted, on an NCM or Voyence platform, that a required port was still being listened to.

sun9999/root# netstat -anf inet | grep 1029
*.1029 *.* 0 0 49152 0 LISTEN

Verify the Culprit

Was it really a part of EMC Ionix NCM or Voyence?

sun9999/root# telnet localhost 1029
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Welcome to EMC Proxy
Copyright (c) 2011 EMC Corporation

User Access Verification
Enter user name:
^]
telnet> quit
Connection to localhost closed.

Well, it appears that EMC is definitely at the root cause.

Not a Start/Stop Script?

Since all the start/stop scripts were disabled from starting up, what else could be the cause?

Under modern UNIX systems, there is a service management facility.

Track Down the Service

Check the port against the registered services file.

sun9999/root# grep telnetproxy /etc/services
telnetproxy 1029/tcp # telnetproxy

Check Against Service Management Facility

EMC appeared nice enough to name the service consistently across the infrastructure

sun9999/root# inetadm | grep telnetproxy
enabled onlinne svc:/network/telnetproxy/tcp:default

sun9999/root# svcs -a | grep telnetproxy
enabled 18:22:21 svc:/network/telnetproxy/tcp:default

Where is the Executable for the Service?

The inet service can be interrogated to reveal the executable being run.

sun9999/root# inetadm -l svc:/network/telnetproxy/tcp:default
SCOPE NAME=VALUE
name="telnetproxy"
endpoint_type="stream"
proto="tcp"
isrpc=FALSE
wait=FALSE
exec="/usr/sbin/in.telnetproxy"
user="root"
default bind_addr=""
default bind_fail_max=-1
default bind_fail_interval=-1
default max_con_rate=-1
default max_copies=-1
default con_rate_offline=-1
default failrate_cnt=40
default failrate_interval=60
default inherit_env=TRUE
default tcp_trace=FALSE
default tcp_wrappers=FALSE
default connection_backlog=10

sun9999/root# ls -al /usr/sbin/in.telnetproxy
-rwxr-xr-x 1 root voyence 1151 Feb 7 18:18 /usr/sbin/in.telnetproxy

EMC was kind enough to name the group of the file, to correctly identify the origin. It is safe to shut down this service.

sun9999/root# svcs svc:/network/telnetproxy/tcp:default
STATE STIME FMRI
online Feb_07 svc:/network/telnetproxy/tcp:default

sun9999/root# svcadm disable svc:/network/telnetproxy/tcp:default

sun9999/root# svcs svc:/network/telnetproxy/tcp:default
STATE STIME FMRI
disabled 18:22:21 svc:/network/telnetproxy/tcp:default

Verify the Telnet Proxy Disable

Check for the tcp port via netstat, to verify that disabling the service did the job.

sun9999/root# netstat -anf inet |grep 1029
sun9999/root#

Solaris 10: SSH and Forwarding HTTP

Abstract:
When Sun first produced systems, the common way for users to move around a network and to distribute workload was to leverage the Berkeley "r" tools, such as "rsh", "rlogin", "rexec", etc. under Solaris. As academics became professional, security concerns over passwords being passed in the clear were raised and SSH was born. SSH was built with a compatible superset to "rsh", but this was later removed with the second version of the protocol. This document discusses the implementation of SSH under Solaris.

Global Configurations

SSH uses several global configuration files, one for the client, and another for the server. Each of these config files document the default compiler flags under Solaris. The "ssh" client global configuration file can be tailored on a per-user basis while the "sshd" server global configuration file is managed at the global level.

SSH Server Daemon

Under Solaris 10, related OS's, and above - SSHD is started through the services infrastructure.

sunserver/user$ svcs ssh
STATE STIME FMRI
online Aug_17 svc:/network/ssh:default

There are built-in compiled defaults and global defaults which are reviewed, upon startup, and connection.

Start a Session with X and HTTP Forwarding

For demonstration purposes, there may be the need to temporarily open an X Console (to install an Oracle Database) and forward HTTP ports (to test an application) on a platform in a DMZ. The sample command may look like this:

sunclient/user$ ssh user@sunserver -b 10.1.2.3 \
-L 58080:127.0.0.1:58080 -L 8080:127.0.0.1:8080 -g

Since the ports to be forwarded are over 1024, there is no requirement for special "root" permissions. The proxied HTTPD connections can be observed.

sunclient/user$ netstat -an grep 8080
 *.58080 *.* 0 0 49152 0 LISTEN
 *.8080 *.* 0 0 49152 0 LISTEN

To perform a basic test of the forwarded HTTP port, the classic "telnet" can be used on the command line, but the connection is closed.

sunclient/user$ telnet localhost 58080
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection to localhost closed by foreign host.

Note, the error on the remote side.

channel 5: open failed: administratively prohibited: open failed

This is a configuration issue.

Global SSHD Configuration

Under Solaris 10, forwarding agent is disabled as a compile flag, and is documented in the global configuration file. If one makes a connection via SSH, and proxies a port - an error message will be produced upon the first connection attempt to the proxied port.

To allow for the port forwarding, edit the configuration file "/etc/ssh/sshd_config".

AllowTcpForwarding yes
GatewayPorts yes
X11Forwarding yes

You will need to restart the "sshd" service, the administrative message disappears.

sunserver/root# svcadm restart ssh

Your port HTTP and X Windows Port Forwarding will now work for ad-hoc tasks.

Sun V100: Installation of Solaris 11 Express

Abstract:
The SPARC platform had existed for decades in the telecommunications environment. It was built upon open standards for instruction set as well as boot proms and moved to 64 bit arcrhitecture long before most processors on the market. The open nature and advanced 64 bit architecture of the platform makes older units a perfect inexpensive test bed to train with advanced modern day operating systems. The 64 bit SPARC V100 has the ability to install Solaris 11 Express.



LOM: Basics

When a new machine is received, it normally comes with an OS installed. After the power-up, it would normally boot directy into the pre-configured operating system, bypassing the Lights Out Management and OpenBoot prom.

To simplify the installation from a console cable, we will start from the V100 from the ground-up, with the LOM through toggling off (down) the power button in the back of the server.

Mar 8 11:39:59 v100ex11 poweroff: initiated by user777 on /dev/console
bootadm: /boot/solaris/bin/extract_boot_filelist is not owned by 101, skipping
syncing file systems... done
lom>
LOM event: +0h28m56s host power off

Command supported by the OpenBoot prom are:

lom>help
The following commands are supported:
alarmon
alarmoff
check
console
environment
faulton
faultoff
help
poweron
poweroff
reset
shutdown
show
version
set
break
bootmode
loghistory
showlogs
consolehistory
chist
date
showdate
logout
userpassword
useradd
userdel
userperm
usershow
lom>

During the powering up of the server, the IDE drive can be pulled, in order to keep the system from booting onto the pre-installed operating system. Powering on the unit can be done from the LOM.

lom>poweron

lom>
LOM event: +0h33m58s host power on
Sun Fire V100 (UltraSPARC-IIe 548MHz), No Keyboard
OpenBoot 4.0, 2048 MB memory installed, Serial #66241418.
Ethernet address 0:3:ba:f2:c3:8a, Host ID: 83f2c38a.

OpenBoot: Basics

After the OK prompt for the OpenBoot is seen, the IDE disk can be plugged back in. The system will be in a funny state, which will be resolved with a future boot.

There are a variety of basic commands which can be run from the OpenBoot PROM. It is more advanced than BIOS in a PC - it is an entire programming and debugging environment driven by Forth.

ok help

Enter 'help command-name' or 'help category-name' for more help
(Use ONLY the first word of a category description)
Examples: help select -or- help line

 Main categories are:

Breakpoints (debugging)
Repeated loops
Defining new commands
Numeric output
Radix (number base conversions)
Arithmetic
Memory access
Line editor
System and boot configuration parameters
Select I/O devices
Floppy eject
Power on reset
Diag (diagnostic routines)
Resume execution
File download and boot
nvramrc (making new commands permanent)

The OpenBoot includes some parameters in a non-volitile RAM.

ok printenv

output-device ttya ttya
input-device ttya ttya
load-base 16384 16384
auto-boot-retry? false false
boot-command boot boot
auto-boot? true true
watchdog-reboot? false false
diag-file
diag-device disk2 net
boot-file
boot-device /pci@1f,0/ide@d/disk@2,0 disk net
local-mac-address? false false
net-timeout 0 0
ansi-terminal? true true
screen-#columns 80 80
screen-#rows 34 34
silent-mode? false false
use-nvramrc? false false
nvramrc
security-mode none No default
security-password No default
security-#badlogins 0 No default
oem-logo No default
oem-logo? false false
oem-banner No default
oem-banner? false false
hardware-revision No default
last-hardware-update No default
diag-switch? false false

By default, this machine is set to automatically boot, which is the factory default.

The boot occurs from the hard disk on this machine, while the factory default is to have the machine attempt to boot from the network.

To see all of your devices which were recognized by the OpenBoot:

ok devalias

disk /pci@1f,0/ide@d/disk@2,0 
rtc /pci@1f,0/isa@7/rtc@0,70 
usb /pci@1f,0/usb@a 
flash /pci@1f,0/isa@7/flashprom@1f,0 
lom /pci@1f,0/isa@7/SUNW,lomh@0,8010 
i2c-nvram /pci@1f,0/pmu@3/i2c@0,0/i2c-nvram@0,aa 
net1 /pci@1f,0/ethernet@5 
dload1 /pci@1f,0/ethernet@5:, 
dload /pci@1f,0/ethernet@c:, 
net0 /pci@1f,0/ethernet@c 
net /pci@1f,0/ethernet@c 
cdrom /pci@1f,0/ide@d/cdrom@3,0:f 
disk3 /pci@1f,0/ide@d/disk@3,0 
disk2 /pci@1f,0/ide@d/disk@2,0 
disk1 /pci@1f,0/ide@d/disk@1,0 
disk0 /pci@1f,0/ide@d/disk@0,0 
ide /pci@1f,0/ide@d 
floppy /pci@1f,0/isa@7/dma/floppy 
ttyb /pci@1f,0/isa@7/serial@0,2e8 
ttya /pci@1f,0/isa@7/serial@0,3f8 

OpenBoot: Stop Automatic Boot

We will set the machine to not automatically boot on power-up to easiy adjust the environment after power cycles.

ok setenv auto-boot? false
auto-boot? = false

Now, the next step is to power-off the system and then poweron from the LOM, to boot the system fresh to an OK prompt:

ok power-off
lom>
LOM event: +1h10m46s host power off
lom>poweron
lom>
LOM event: +1h11m39s host power on
Sun Fire V100 (UltraSPARC-IIe 548MHz), No Keyboard
OpenBoot 4.0, 2048 MB memory installed, Serial #66241418.
Ethernet address 0:3:ba:f2:c3:8a, Host ID: 83f2c38a.
ok

OpenBoot: Boot Solaris 11 Express

The Installation of Solaris 11 Express can be done via a CD-ROM from the OpenBoot.

ok boot cdrom
Boot device: /pci@1f,0/ide@d/cdrom@3,0:f File and args:

The cursor will swap between characters "", "/", "-", etc., indicating the system is working. The CDROM boot banner will appear, once the OS is boot-strapped.

Solaris 11 Express: Install From CD-ROM

Once the OS is boot-strapped, the install routine prompts for information.

Boot device: /pci@1f,0/ide@d/cdrom@3,0:f File and args:
SunOS Release 5.11 Version snv_151a 64-bit
Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved.
WARNING: invalid vector intr: number 0x7de, pil 0x0
Hostname: solaris
Remounting root read/write
Probing for device nodes ...
Preparing text install image for use
Done mounting text install image
USB keyboard

1. Albanian 25. Latin-American
2. Arabic 26. Lithuanian
3. Belarusian 27. Latvian
4. Belgian 28. Macedonian
5. Brazilian 29. Malta_UK
6. Bulgarian 30. Malta_US
7. Canadian-Bilingual 31. Norwegian
8. Croatian 32. Polish
9. Czech 33. Portuguese
10. Danish 34. Romanian
11. Dutch 35. Russian
12. Dvorak 36. Serbia-And-Montenegro
13. Estonian 37. Slovak
14. Finnish 38. Slovenian
15. French 39. Spanish
16. French-Canadian 40. Swedish
17. Hungarian 41. Swiss-French
18. German 42. Swiss-German
19. Greek 43. Traditional-Chinese
20. Icelandic 44. TurkishF
21. Italian 45. TurkishQ
22. Japanese-type6 46. UK-English
23. Japanese 47. US-English
24. Korean
To select the keyboard layout,
enter a number [default 47]: 47

1. Arabic 12. Hungarian
2. Catalan 13. Indonesian
3. Chinese - Simplified 14. Italian
4. Chinese - Traditional 15. Japanese
5. Czech 16. Korean
6. Dutch 17. Polish
7. English 18. Portuguese - Brazil
8. French 19. Russian
9. German 20. Slovak
10. Greek 21. Spanish
11. Hebrew 22. Swedish
To select the language you wish to use,

enter a number [default is 7]: 7

User selected: English
Configuring devices.

Using HyperTerminal with "Auto" emulation, does not help the installer, since it will think that "xterm" is the terminal type available.

If performing an installation from a "tip" command in an "xterm", no change is required, and the installation screens look wonderful.

Welcome to the Oracle Solaris snv_151a installation menu
1 Install Oracle Solaris
2 Install Additional Drivers
3 Shell
4 Terminal type (currently xterm)
5 Reboot
Please enter a number [1]: 4

Indicate the type of terminal being used, such as:
dtterm CDE terminal emulator
xterm xterm
vt100 DEC VT100
Enter terminal type [xterm]: vt100

Welcome to the Oracle Solaris snv_151a installation menu
1 Install Oracle Solaris
2 Install Additional Drivers
3 Shell
4 Terminal type (currently vt100)
5 Reboot
Please enter a number [1]: 1

The Welcome Spash Screen provides the basic terminal keystroke instructions.

 Welcome to Oracle Solaris
 Thanks for choosing to install Oracle Solaris! This installer enables you
 to install the Oracle Solaris Operating System (OS) on SPARC or x86
 systems.

 The installation log will be at /tmp/install_log.

 How to navigate through this installer:
 - Use the function keys listed at the bottom of each screen to move from
 screen to screen and to perform other operations.
 - Use the up/down arrow keys to change the selection or to move between
 input fields.
 - If your keyboard does not have function keys, or they do not respond,
 press ESC; the legend at the bottom of the screen will change to show
 the ESC keys for navigation and other functions.

F2_Continue F6_Help F9_Quit

Select the disk at the next screen with F2, to perform the install to the base disk.

Microsoft Windows, unfortunately, does not provide a real terminal emulator that understand the VT100 character set, when you select VT100, so some grahical characters do not emulate correctly.

 Disks
 Where should Oracle Solaris be installed?
 Recommended size: 4.7GB Minimum size: 2.7GB

 Type Size(GB) Boot Device Manufacturer Notes
 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
 ATA 74.5 + c2t2d0 unknown

 The following slices were found on the disk.

Slice # Size(GB) Slice # Size(GB)
qqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqq
rpool 0 74.5 Unused 5 0.0
Unused 1 0.0 Unused 6 0.0
Unused 3 0.0 Unused 7 0.0
Unused 4 0.0 backup 2 74.5

F2_Continue F3_Back F6_Help F9_Quit

Continue with the default slicing through F2.

 Solaris Slices: 74.5GB ATA Boot

Oracle Solaris can be installed on the whole disk or a slice on the disk.
The following slices were found on the disk.

Slice # Size(GB) Slice # Size(GB)
qqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqq
rpool 0 74.5 Unused 5 0.0
Unused 1 0.0 Unused 6 0.0
Unused 3 0.0 Unused 7 0.0
Unused 4 0.0 backup 2 74.5

 Use the whole disk
 Use a slice on the disk

F2_Continue F3_Back F6_Help F9_Quit

Move the cursor to "Use the whole disk" and Continue with F2 to the network configuration.

 Network

 Enter a name for this computer that identifies it on the network. It must
 be at least two characters. It can contain letters, numbers, and minus
 signs (-).

 Computer Name: sol11v100

 Select how the wired ethernet network connection is configured.

 Automatically Automatically configure the connection
 None Do not configure the network at this time

F2_Continue F3_Back F6_Help F9_Quit

Type a name for the computer operating system instance name, such as "sol11v100".

Move the cursor to "Automatically" to configure an IP address via DHCP on boot.
Continue with F2.

 Time Zone: Regions

 Select the region that contains your time zone.

 Regions
 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
 UTC/GMT
 Africa
 Americas
 Antarctica
 Arctic Ocean
 Asia
 Atlantic Ocean
 Australia
 Europe
 Indian Ocean
 Pacific Ocean

F2_Continue F3_Back F6_Help F9_Quit

Timezone should be selected, in this case we move the cursor to Americas and Continue with F2.

 Time Zone: Locations

 Select the location that contains your time zone.
 Locations
 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
 ^ St Barthelemy
 x St Kitts & Nevis
 x St Lucia
 x St Martin (French part)
 x St Pierre & Miquelon
 x St Vincent
 x Suriname
 x Trinidad & Tobago
 x Turks & Caicos Is
 x United States
 x Uruguay
 x Venezuela
 x Virgin Islands (UK)
 q Virgin Islands (US)

F2_Continue F3_Back F6_Help F9_Quit

Move your cursor to the appropriate Timezone Location, in this case United States was selected by moving the cursor off the bottom of the screen, and Continue via F2 was done.

 Time Zone

 Select your time zone.
 Time Zones
 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
 q Eastern Time
 x Eastern Time - Michigan - most locations
 x Eastern Time - Kentucky - Louisville area
 x Eastern Time - Kentucky - Wayne County
 x Eastern Time - Indiana - most locations
 x Eastern Time - Indiana - Daviess, Dubois, Knox & Martin Counties
 x Eastern Time - Indiana - Pulaski County
 x Eastern Time - Indiana - Crawford County
 x Eastern Time - Indiana - Pike County
 x Eastern Time - Indiana - Switzerland County
 x Central Time
 x Central Time - Indiana - Perry County
 x Central Time - Indiana - Starke County
 v Central Time - Michigan - Dickinson, Gogebic, Iron & Menominee Counti

F2_Continue F3_Back F6_Help F9_Quit

Move the cursor to the appropriate Time Zone and Continue with F2.

 Date and Time

 Edit the date and time as necessary.
 The time is in 24 hour format.

 Year: 2011 (YYYY)
 Month: 03 (1-12)
 Day: 08 (1-31)
 Hour: 13 (0-23)
 Minute: 57 (0-59)

F2_Continue F3_Back F6_Help F9_Quit

Select your Date and Time, Continue with F2.

 Users

 Define a root password for the system and user account for yourself.

 System Root Password
 Root password: *****
 Confirm password: *****

 Create a user account
 Your real name: Administrator
 Username: admin
 User password: *****
 Confirm password: *****

F2_Continue F3_Back F6_Help F9_Quit

The installation is secure, by default. The Root password is required as well as a user account, so you can assume the privileges associated with "root" at a later point in time.

Complete the prompts and Continue with F2.

There is a brief pause after F2, so do not hit the key a second or third time.

 Installation Summary

 Review the settings below before installing. Go back (F3) to make changes.

 Software: Oracle Solaris 11 Express snv_151a SPARC
 Disk: 74.5GB ATA
 Slice 0: 74.5GB rpool1
 Time Zone: US/Eastern
 Language: *The following can be changed when logging in.
 Default language: English
 Users:
 Username: admin
 Network:
 Computer name: sol11v100
 Network Configuration: Automatic

F2_Install F3_Back F6_Help F9_Quit

The Installation Summary is provided, after a brief pause. Continue with F2 to install.

 Installing Oracle Solaris

 Preparing disk for Oracle Solaris installation
[ (5%) ]

 Building cpio file lists
[ (6%) ]

 Transferring Contents
[ (98%) ]
F9_Quit

The installation begins with various phases, some of which were captured and copied into the installation window above.

Once the installation is complete, the option to reboot is provided.

 Installation Complete

 The installation of Oracle Solaris has completed successfully.
 Reboot to start the newly installed software or Quit if you wish to
 perform additional tasks before rebooting.

 The installation log is available at /tmp/install_log. After reboot it
 can be found at /var/sadm/system/logs/install_log.

F4_View Log F8_Reboot F9_Quit

Reboot the system via F8 would normally be done.

Since Microsoft Windows HyperTerm is broken, the F8 and F9 does not work.

A straight power down via the rocker switch can be done and a "powerup" from LOM to restart the system.

Mar 8 19:26:50 solaris power: WARNING: Power off requested from power button or
SC, powering down the system!
Shutdown started. Tue Mar 8 19:26:53 GMT 2011
Changing to init state 5 - please wait
showmount: solaris: RPC: Program not registered
bootadm: /media/Oracle_Solaris_Text_SPARC/ filesystem is read-only, skipping archives update
svc.startd: The system is coming down. Please wait.
svc.startd: 74 system services are now being stopped.
Mar 8 19:27:12 solaris syslogd: going down on signal 15
svc.startd: Killing user processes.
umount: /.cdrom busy
Mar 8 19:27:29 The system is down. Shutdown took 25 seconds.
syncing file systems... done
lom>
LOM event: +3h17m53s host power off

lom>poweron
lom>
LOM event: +3h19m46s host power on

Solaris 11 Express: Booting From Disk

Since the OpenBoot was set to not automatically boot, the ok prompt is available, to boot from disk or cdrom.

Since the installation was complete, the boot from the default disk can be done.

Sun Fire V100 (UltraSPARC-IIe 548MHz), No Keyboard
OpenBoot 4.0, 2048 MB memory installed, Serial #66241418.
Ethernet address 0:3:ba:f2:c3:8a, Host ID: 83f2c38a.
ok boot disk
Boot device: /pci@1f,0/ide@d/disk@2,0 File and args:
SunOS Release 5.11 Version snv_151a 64-bit
Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved.
/
Loading smf(5) service descriptions: 5/178
-
Loading smf(5) service descriptions: 178/178
_
WARNING: invalid vector intr: number 0x7de, pil 0x0
/
Hostname: sol11v100
-
Configuring devices.

Loading smf(5) service descriptions: 7/7
-
sol11v100 console login:

After watching the rotating bar, the OS banner page appears, the services are instantiated through SMF, and the console login prompt from the serial cable.

Login on the console cable and find the IP Address.

sol11v100 console login: msadmin
Password:
Oracle Corporation SunOS 5.11 snv_151a November 2010

msadmin@sol11v100:~$ ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
 inet 127.0.0.1 netmask ff000000
dmfe0: flags=1004843 mtu 1500 index 4
 inet 253.60.174.73 netmask ffffff00 broadcast 253.60.174.255
dmfe1: flags=1004803 mtu 1500 index 3
 inet 0.0.0.0 netmask ff000000
lo0: flags=2002000849 mtu 8252 index 1
 inet6 ::1/128
dmfe0: flags=20002004841 mtu 1500 index 4
 inet6 fe80::203:baff:fef2:c38a/10
dmfe1: flags=20002004801 mtu 1500 index 3
 inet6 fe80::203:baff:fef2:c38b/10
dmfe1:1: flags=20002000800 mtu 1500 index 3
 inet6 ::/0
dmfe1:2: flags=20002000800 mtu 1500 index 3
 inet6 ::/10

Solaris 11 Express: Enabling Telnet

Insecure operating systems like Microsoft Windows XP do not provide a simple SSH client from the Command prompt. Insecure Telnet can be enabled to facilitate access with the root login and verified using Solaris services.

admin@sol11v100:~$ svcs telnet
STATE STIME FMRI
disabled 14:40:05 svc:/network/telnet:default

admin@sol11v100:~$ su root
Password:
Mar 8 14:59:40 sol11v100 su: 'su root' succeeded for msadmin on /dev/console

msadmin@sol11v100:~# svcadm enable telnet

msadmin@sol11v100:~# svcs telnet
STATE STIME FMRI
online 15:02:33 svc:/network/telnet:default

The platform is now ready for basic configuration without a console cable.

Additional configuation steps can be done from the console, to set up tools like GNOME through Headless X configuration, and VNC, to get a graphical windowing environment.

Network Management Connection

In the arena of Network Management, Solaris 11 Express it the premier operating system for security as well as stability. If it has to to run and be available, it runs under Solaris.

Orgad Kimchi at Sun, now Oracle, blogged on VReality an overview of Oracle VM Server for SPARC, previously called Sun Logical Domains or LDoms. In particular, he discussed Version 1.3 with Dynamic Resource Management or DRM. The allocation of CPU threads or resources according to pre-defined polices was the target.

Orgad posted a PDF which was formatted reasonably well, but the fonts made certain sections difficult to read in the PDF that he included. I copied the PDF contents into this blog, re-formated it (while trying to keep as close to the original style as possible), adjusted some typographical errors, and included it in this blog. While the blog is not the optimal format to hold this content in, I left some feedback on his original content suggesting some reformatting suggestions.

---

Oracle VM Server for SPARC (LDoms) Dynamic Resource Management

ABSTRACT:

In this entry, I will demonstrate how to use the new feature of Oracle VM Server for SPARC (previously called Sun Logical Domains or LDoms) version 1.3 Dynamic Resource Management (a.k.a DRM) for allocating CPUs resources based on workload and pre defined polices.

Introduction to Oracle VM Server for SPARC:

Oracle VM Server for SPARC is a virtualization and partitioning solution supported on Oracle Solaris CoolThreads technology-based servers powered by UltraSPARC T1, T2, and T2 Plus processors with Chip Multi-threading Technology (CMT).

This technology allows the creation of multiple virtual systems on a single physical system. Each virtual system is called a logical domain (LDom) and runs a unique and distinct copy of the Solaris operating system.

Introduction to Dynamic Resource Management:

With this feature, we can define policies to control an upper and lower threshold for virtual CPU utilization on an LDom. If an LDom needs more capacity and other LDoms on the same physical server have spare capacity, the system can automatically add to or remove CPUs from domains - as per the defined policies.

The main goal of dynamic resource management (DRM) is to provide the LDoms resource allocation flexibility in order to allocate resources to the LDom during peak time without human intervention.

Architecture layout :


Prerequisites:

We need to define the control domain and three logical domains. Refer to the Logical Domains 1.3 Administration Guide (http://docs.sun.com/app/docs/doc/821-0406) for a complete procedure on how to install Oracle VM Server for SPARC.

Dynamic Resource Management configuration:

We will define a total of three polices (policy1, policy2 ,policy3), one for each domain (ldg1,ldg2 ,ldg3), each policy will define under what conditions virtual CPUs can be automatically added to and removed from a logical domain.

A policy is managed by using the commands: ldm add-policy, ldm set-policy, and ldm remove-policy commands.

The following ldm add-policy command creates the policy to be used on the ldg1 logical domain.

# ldm add-policy util-lower=25 util-upper=75 vcpu-min=4 vcpu-max=8 attack=1 decay=1 priority=1 name=policy1 ldg1 

The following policy does the following:

■ Specifies that the lower and upper limits at which to perform policy analysis are 25 percent
and 75 percent by setting the util-lower and util-upper properties, respectively.

■ Specifies that the minimum and maximum number of virtual CPUs is 4 and 8 by setting
the vcpu-min and vcpu-max properties, respectively.

■ Specifies that the maximum number of virtual CPUs to be added during any one resource
control cycle is 1 by setting the attack property.

■ Specifies that the maximum number of virtual CPUs to be removed during any one resource
control cycle is 1 by setting the decay property.

■ Specifies that the priority of this policy is 1 by setting the priority property. A priority of 1
means that this policy will be enforced even if another policy can take effect.

■ Specifies that the name of the policy file is policy1 by setting the name property.

■ Uses the default values for those properties that are not specified, such as enable (off) and
sample-rate (10 sec).

This is the second policy for the second LDom (ldg2)

# ldm add-policy util-lower=25 util-upper=75 vcpu-min=8 vcpu-max=16 attack=1 decay=1 priority=2 name=policy2 ldg2

This is the third policy for the third LDom (ldg3)

# ldm add-policy util-lower=25 util-upper=75 vcpu-min=8 vcpu-max=16 attack=1 decay=1 priority=3 name=policy3 ldg3

Now we need to enable the policies:

# ldm set-policy enable=yes name=policy1 ldg1
# ldm set-policy enable=yes name=policy2 ldg2
# ldm set-policy enable=yes name=policy3 ldg3

The following example shows how the configuration looks on the control domain. You can verify
the policies have been created by using the "ldm ls -o res" subcommand.

# ldm ls -o res
NAME
primary
------------------------------------------------------------------------------
NAME
ldg1

POLICY
STATUS PRI MIN MAX LO UP BEGIN END RATE EM ATK DK NAME
 on 1 4 8 25 75 00:00:00 23:59:59 10 5 1 1 policy1
WEIGHTED MEAN UTILIZATION
4.2%
------------------------------------------------------------------------------
NAME
ldg2

POLICY
STATUS PRI MIN MAX LO UP BEGIN END RATE EM ATK DK NAME
 on 2 8 16 25 75 00:00:00 23:59:59 10 5 1 1 policy2
WEIGHTED MEAN UTILIZATION
0.1%
------------------------------------------------------------------------------
NAME
ldg3

POLICY
STATUS PRI MIN MAX LO UP BEGIN END RATE EM ATK DK NAME
 on 3 8 16 25 75 00:00:00 23:59:59 10 5 1 1 policy3
WEIGHTED MEAN UTILIZATION
0.0%

The following example shows how a policy, called policy1, can be changed in order to add more
CPUs to a machine called ldg1

# ldm set-policy name=policy1 vcpu-max=16 ldg1

The following example shows how we can remove a policy, called policy1

# ldm remove-policy name=policy1 ldg1

Now, let's check how dynamic resource management works :
In order stress the CPU of your system, you can get the spinners loading tool from BigAdmin (see http://www.sun.com/bigadmin/software/nspin/nspin.tar.gz .)

We will monitor the system before and during the workload.

Connect to the console of the first guest domain (ldg1)

# telnet localhost 5000

Verify the number and CPUs load using the mpstat command

# mpstat

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
 0 0 0 4 215 7 20 0 0 0 0 11 1 0 0 99
 1 0 0 3 21 6 19 0 0 0 0 9 1 0 0 99
 2 0 0 3 21 6 19 0 0 0 0 11 1 0 0 99
 3 0 0 3 21 6 19 0 0 0 0 9 1 0 0 99

We can see that the LDom is underutilized (idl =99) and that we have 4 CPUs (0-3)
Let's start the workload using the nspins command and monitor the effect on the system utilization and the total number of CPUs :

# nspins -n 8 &
# mpstat 10

Now give it ~40 seconds. or so to run

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
 0 0 0 52 201 0 2 8 0 0 0 1 100 0 0 0
 1 0 0 4 20 4 12 13 0 0 0 6 100 0 0 0
 2 0 0 2 31 11 23 18 0 0 0 13 100 0 0 0
 3 0 0 3 21 5 11 12 0 1 0 38 100 0 0 0
 4 0 0 2 16 1 6 10 0 0 0 1 100 0 0 0
 5 0 0 2 23 2 13 13 0 0 0 2 100 0 0 0
 6 0 0 1 17 2 8 10 0 1 0 2 100 0 0 0
 7 0 0 0 12 1 4 9 0 0 0 1 100 0 0 0

We can see that all the machine's CPUs are utilized (idl=0) and the total number of CPUs are increased to 8 (0-7) In order to see the CPUs diminished effect we can stop the workload and monitor the LDom again.

# pkill nspins
# mpstat 10
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
 0 0 0 4 215 7 20 0 0 0 0 11 1 0 0 99
 1 0 0 3 21 6 19 0 0 0 0 9 1 0 0 99
 2 0 0 3 21 6 19 0 0 0 0 11 1 0 0 99
 3 0 0 3 21 6 19 0 0 0 0 9 1 0 0 99
 4 1 0 3 21 4 12 10 0 0 0 4 91 0 0 9
 5 1 0 3 15 2 7 9 0 0 0 7 91 0 0 9
 6 0 0 2 15 2 7 9 0 0 0 2 91 0 0 9

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
 0 0 0 4 215 7 20 0 0 0 0 11 1 0 0 99
 1 0 0 3 21 6 19 0 0 0 0 9 1 0 0 99
 2 0 0 3 21 6 19 0 0 0 0 11 1 0 0 99
 3 0 0 3 21 6 19 0 0 0 0 9 1 0 0 99
 4 1 0 3 20 4 12 10 0 0 0 4 89 0 0 10
 5 1 0 5 15 2 7 9 0 0 0 7 89 0 0 11

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
 0 0 0 4 215 7 20 0 0 0 0 11 1 0 0 99
 1 0 0 3 21 6 19 0 0 0 0 9 1 0 0 99
 2 0 0 3 21 6 19 0 0 0 0 11 1 0 0 99
 3 0 0 3 21 6 19 0 0 0 0 9 1 0 0 99
 4 1 0 3 20 4 12 10 0 0 0 5 88 0 0 12

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
 0 0 0 4 215 7 20 0 0 0 0 11 1 0 0 99
 1 0 0 3 21 6 19 0 0 0 0 9 1 0 0 99
 2 0 0 3 21 6 19 0 0 0 0 11 1 0 0 99
 3 0 0 3 21 6 19 0 0 0 0 9 1 0 0 99

We see from the mpstat output that the total number of CPUs has decreased by 1 in a cycle from 8 to 4

Conclusion:

Oracle VM Server for SPARC Dynamic Resource Management provides the system administrator the flexibility to have better dynamic resource allocation based on system utilization. In this blog entry, I demonstrated how to set up Dynamic Resource Management and how to monitor this feature during CPU utilization peak time.

About the Author:

Orgad Kimchi joined Sun in September 2007. He is currently working in the Independent Software Vendors (ISV) Engineering organization helping software vendors adopt Sun technology and improve performance on Sun hardware and software. Orgad’s blog can be found at http://blogs.sun.com/vreality.