Evaluation of ears890

Saturday, July 27, 1996Review of ears890

System performance tuning is more than just tweaking some parameters. System performance tuning requires a thorough knowledge of the hardware, operating system, and application. One must know how each of these items interact with each other. This task in itself can be a full time job. This report is designed to show everyone how the box is running now and what we can do to improve performance. You should understand that some of the changes that might be recommended here will not always be for the good of the user is the short term but will benefit him/her in the long run.

I have broken this document into five specific areas of system performance tuning. These areas are System & Processes, Networking, Disk Performance, Paging & Swapping, and Kernel Tuning. Each of these areas will provide information on how performance may be affected on any Unix platform and descriptions of the system as it stands right now. This document should be read thoroughly so that you may understand the recommendations at the end of this report. Failure to understand how the box (the host name is ears890) is operating will result in serious injury.

I have spent two weeks monitoring these main parts of system performance tuning for ears890. I have covered some areas in more detail than others. Because of the nature of the beast, I am continuing on going efforts to closely monitor the box. Once the recommended changes have been approved, I will establish a time line to complete some of the tasks at hand.

Another area that has not been mentioned in this document is security. This area alone requires a thorough investigation. There are so many system essential files that can be overwritten by everyday users. In fact, any database user has the ability to wipe out another user's data or even remove an entire file system. The first thing that should be done before any changes are performed is to lock down and secure ears890. This task can be done by inspecting every file system and file for proper permissions, disabling root accessed, creating group permissions, and securing the network.System & Processes

The Host

Machine Type: HP9000 Series 800, model T500

Machine Name: ears890

Memory: 1.2 GB MEM

Disk Space: 95GB

Virtual memory = 650 active vm 228 free mem 285

Hardware Devices

Software Subsystems Installed

Processes Management

When a user logs into the system over the network, the process begins by opening a socket and the getty takes control of the port. The process is forked and a login is passed to the port. The user will log in and a shell is spawned. The shell runs the profile which in turn runs the /u1/uv/bin/uv command (this starts the Universe database for that user). This is the short hand version of a typical user login. Many things are happening here and many things could go wrong during a thrashing system. Giving the database user access to the Unix shell could be dangerous. The system is starting many processes just to get into the database. This causes a big problem for the machine because of the high volume of users. the host ears890 typically has 500 concurrent users. All of these users are using the database (or they seem to be.) This in itself can cause a slow response time from the system do to the heavy I/O requests from the databases. But the system also has erroneous system users that stay idle for more than 15 minutes and may even have a user on more than once.

These users were idle for more than 15 minutes and were logged on more than once:

xhvan Xiem H Van,Los Angeles CA,818-304-2601, in 2 times

sshmc Homer M Carter,JVWTC,740-6774, in 1 times

halee H. Alexander Lee,Operations,, in 73 times

blbaker Bonnie L Baker,Portland,503-222-5335, in 1 times

srpiccia Stephen R Picciano,Baltimore MD,410-653-7041, in 5 times

mjchilde Melanie J Childers,Houston,713-242-7878, in 1 times

llvillar L L VILLARREAL,Portland OR,, in 1 times

trbishop T R Bishop,Richmond VA,, in 3 times

emaslami Eshaq M Aslami,Operations,, in 9 times

vsjones Vanvaris S Jones,Dept of Ed,, in 2 times

bemiller Beth E Miller,ATLMED,770-300-7054, in 1 times

dmcrisci Doreen M Crisci,Support,404-250-4050, in 15 times

mplawren Michael P Lawrence,Atlanta,936-1125, in 1 times

bdrogers Bonnie D Rogers,Chamblee-Tucker,936-1052, in 2 times

snhypoli Shakisha N Hypolite,HOUSTON,713-560-1795, in 1 times

mbwiener Mark B Wiener,Raleigh NC,, in 2 times

mlmichel Manfred L Michel,Operations,404-740-5369, in 3 times

dlcallow Dee L Calloway,JVWTC,770-740-5366,, in 5 times

These users represent 128 login processes that were just sitting idle taking up precious systems resources. At the time the specified sample was made (which is consistent throughout the days of the week), there were 640 users that were logged on the box. By eliminating 128 of these idle processes, we could ideally gain up to 20% of the system's resources back.

The system also has over 2000 process running at a given time. This kind of usage has surprised many support representative at Cubs, VMark, and HP (as well as myself).

Networking with ears890

Lan Devices

lan0 or network 147.146.116

lan1 or network 147.146.1

lan2 or network 147.146.22

Gateways & Routes

Every subnet within the equifax environment has a router attached at address 200. This router contains all information about the other networks. Right now ears890 has a default route to a host earssr1at ip address

147.146.116.100. This router is known as the SR1 router. Unfortunately this router doesn't know about other networks or is not configured to do so ... this poses a big problem in trying to setup communications with other boxes.

ears890 is configured to use static routes to get to specific areas on the network from ears890. Static routes serve their purpose and are very useful for keeping traffic low. Another reason to use static routes is to keep the box more secured. Any other host on another network can not get to your box unless there is a route from your box to that network (this originating network). Once a static route table grows to large, the box becomes a router and more cpu time is dedicated to the resolution of ip addresses instead of running the application.

Currently there are routes to:

Routing tables

Destination Gateway Flags Refs Use Interface

localhost localhost UH 0 72 lo0

172.16 147.146.22.200 UG 0 18015 lan2

default earssr1 UG 102 12251578 lan0

147.146.41 147.146.1.250 UG 0 219641 lan1

147.146.1 147.146.1.101 U 0 4155 lan1

147.146.3 147.146.22.200 UG 1 1968 lan2

147.146.19 147.146.1.81 UG 0 2847 lan1

135.68.159.110 netblazer UG 0 228 lan0

147.146.116 ears890 U 538 99430560 lan0

147.146.22 ears890_a U 1 2402553 lan2

147.146.7 147.146.1.81 UG 0 31043 lan1

netblazer 147.146.116.200

ears890 147.146.116.1

ears890_a 147.146.22.13

default 147.146.116.100

what is 135.68.159.110 ???

Disk Performance

Drives & Controllers

The disk drive location is very critical to the performance of the database. A very busy database should be located on a disk that can produce the most I/O. This task may be achieved by making logical drives speed across two or more controllers. One should take care as to not spread the data too far because the CPU could spend more time routing the data to the proper I/O than processing data. Small databases should be located on the same disk and controller as to make the most quick and efficient data access. One example says spread the data across disks & controllers while the other says keep everything localized. Depending on the type of application, the administrator should pick the environment that is best suited for the user.

The Universe Cubs product has very high process and disk I/O usage caused by data retrievals and spooler activities. The database on ears890 is also duplicated about 44 times. This makes monitoring system performance very difficult because of all the different activities used by every instance of the Cubs product. The network traffic is minimal by the actual Cubs product. Most of the activities on the system are I/O based. Because of this I/O activity, careful attention should be placed on I/O kernel parameters.

ears890 has very confusing disk drive configurations. The disk drive mapping from controllers to disk drive, to file system to database has no pattern and could use some improvement. Careful attention will be paid to disk performance because this is most likely the second of two culprits for cause very slow response (the first one is the number of processes).

Logical volumes are used on this system is stead of raw file systems. The use of logical file systems will be slower than the use of the physical volumes. The typical scenario used in the ears890 host might involve taking two 2GB drives and combining them into 1 volume. This volume can then be split up into logical volumes. The host usually has two to four 2GB drives combined into one volume. This one volume is then split into a 2GB or 4GB logical volume with mirroring.

Disk Space

The system is beginning to bulge from the lack of disk space. Universe recommends and a general rule of thumb is to have the available free disk space at about 20% of the total disk space available. Looking at the file system usage below, one can see that some file systems are within tolerance and others are way beyond the acceptable level. Although on an average the used disk space is right at 80% of the total available disk space.

Breakdown by File system

File system kbytes used avail capacity Mounted on

/dev/vg00/lvol1 683243 346513 268405 56% /

/dev/vg30/lvol1 4115277 1418713 2285036 38% /fs24

/dev/vg29/lvol1 4115277 149551 3554198 4% /fs23

/dev/vg28/lvol1 4115277 143555 3560194 4% /fs22

/dev/vg01/lvol1 1917323 1357007 368583 79% /usr

/dev/vg03/lvol1 3834667 3420611 299015 92% /u1

/dev/vg04/lvol1 4115277 3813230 178588 96% /u2

/dev/vg14/lvol1 4115277 2925088 778661 79% /u3

/dev/vg02/lvol1 3834667 3427362 23838 99% /u4

/dev/vg05/lvol1 3834667 3271521 448105 88% /u5

/dev/vg06/lvol1 3834667 3341321 109879 97% /u6

/dev/vg07/lvol1 3834667 3011372 439828 87% /u7

/dev/vg11/lvol1 3834667 3145562 305638 91% /u8

/dev/vg12/lvol1 4115277 3732956 258862 94% /u9

/dev/vg15/lvol1 4115277 3158527 545222 85% /u10

/dev/vg08/lvol1 4115277 2683890 1019859 72% /u11

/dev/vg13/lvol1 2057629 1562177 289689 84% /u12

/dev/vg17/lvol1 2057629 1551407 300459 84% /u13

/dev/vg16/lvol1 4115277 3109214 594535 84% /u14

/dev/vg18/lvol1 4115277 2919248 784501 79% /u15

/dev/vg19/lvol1 2057629 1257224 594642 68% /u16

/dev/vg22/lvol1 2057629 1382932 468934 75% /u17

/dev/vg24/lvol1 4115277 2195823 1507926 59% /u19

/dev/vg25/lvol1 4115277 3233787 469962 87% /u20

/dev/vg27/lvol1 2057629 1333213 518653 72% /u21

/dev/vg23/lvol1 2057629 1472957 378909 80% /cbi.reports2

/dev/vg10/lvol1 3834667 3601711 117915 97% /cbi.reports

/dev/vg09/lvol1 3834667 2836273 883353 76% /test.filesys2

/dev/dsk/c1d0s2 546660 546660 0 100% /cdrom

Paging & Swapping

How Paging Works

When a program is compiled, the compiler generates address known as virtual addresses for the code. These virtual addresses must be mapped into memory to a physical address for the compiled code to execute. The mapping between virtual memory and physical memory for a process is stored in tables in physical memory. The memory management unit (MMU) manages the interface between blocks of virtual address space and physical memory locations. The MMU uses the translation lookaside buffer (TLB) hardware component in the processor to cache this data. When the program executes, the code and data is stored in the processor registers for referencing. While the program is executing, the program could get to a point where part of the code is not loaded into memory. When this occurs the processor looks in cache. If the code is not present there, the processor looks to the translation tables in RAM to find the mappings between virtual and physical memory. If the code is not in RAM, then the code might have been paged to disk.

When a processes executes, the kernel maintains a proc structure which is a data structure that points to a virtual address space. The virtual address space is a link list of several Per-process regions or pregions. These pregions are pointers to spcific areas within a process. A program is compiled into distinct logical stacks that get loaded when executed. Each logical stack or region is directly mapped to a specific area within physical memory. These stacks which are pointed to by the pregions are:

Each of these regions usually references a virtual node (vnode) which is an interface for reading or writing pages of data between memory and disk. vnodes are supersets of inodes that the MMU stores to keep track of swapped or paged memory. vnodes are categorized by file type of hfs, nfs, dux, cdfs, and swap.

Again. a compiled program has virtual address space and contains information about the size of the code and data regions. Based on this information, a process is created and the kernel sets up its data structures and the process

starts executing its instructions from user mode. When a process tries to access an address that is not currently in main memory, a page fault occurs. The kernel switches from user mode to kernel mode and tries to resolve the

page fault by locating the pregion containing the sought after virtual address. The kernel then looks for free pages within main memory. If no free pages are available, the systems swaps or pages out selected used pages to

make room. The kernel then pages in the required page from disk to memory. Once the page is in place the kernel exists back to user mode and continues execution.

How Swapping Works

The vhand daemon handles the page out requests and monitors free pages to keep the number of free pages above the threshold. vhand decides when to start paging by determining how much free memory is available. Two factors are used to determine the amount of free memory:

pgpslim

This is the upper limit of when vhand starts to page. this figure should represent about 70% of memory is used.

desfree

This is the upper limit used by swapper. this limit should represent about 85%-90% of free memory is used.

The swapper is a little different from paging in that an entire process may be pre-empted temporarily and swapped out to disk. Since the swapper deactivated the process, the request for new free pages is reduced. Once the memory threshold is above desfree, the swapper will reactivate the process and swap it back into memory.

Thrashing can occur when the system is low on memory and the system spends more time fulfilling requests to page and swap out processes than actually processing the code. Systems performance will degrade rapidly once this starts occurring. Thrashing may occur by very high disk activity or from very high graphic data going from CPU to the video output. The vhand skew (vhandsk) is the amount of time pages can remain in memory before being paged out.

Disk buffer cache is essential for a well running universe database. This controls the amount of memory allocated to cache. This can be useful when several processes use the same routine. Instead of seeking to physical I/O, the process goes to memory. Of course, the more that is dedicated to cache, then less that is available for the user processes.

nbuf, bufpages kernel parameter

The number of seconds until an unused disk buffer updates to disk is set by nautop.

nautoup kernel parameter

If the cache read hit rate is less than 90% or the write hit rate is less than 65% then can increase the nbuf parameter and increase performance. The physical block transfers should be about the limit of the disk subsystem.

Look for the I/O queue. This is the number of processes waiting I/O. This umber can be non-zero.

Use the sar -b or iostat to retrieve this information

What VMark Universe Reconmends

The number of file opens used by a universe process can be configured within Universe or the kernel. The number of igets (inode opens) should not exceed 40 per second. Use the sar -a to find this information. The universe file

handling parameters are: rotating file pool, explicit BASIC OPEN statements, exceeding the TRANS cache buffer, and implicit opens in the temporary work directory.

The paging faults are 4 times the amount of paging requests on ears890. The page in usually is about the same as the page faults and there are no page outs or swap outs on the box.

Universe recommends the following changes:

Paging on ears890

What we know about the system is this:

  1. ears890 usually has about 200MB of free memory
  2. there are 2k-4k page faults (this is a lot)
  3. there are 2k-4k of pages in
  4. there are no pages out
  5. disk I/O is nearly at 100%
  6. cpu utilization is at 100% of which 75% is system mode - only 25% is dedicated to the application
  7. system calls by interrupts are 20k-40k per second
  8. there is no swapping in or out

Analyzing this information will tell you a few things. There is plenty of memory available for system use. This fact is confirmed by the free available memory and the fact that there are no page outs. But why are there paging faults which cause the paging in? It is obvious that the process is not able to complete its operation before it is interrupted by one of two things. A process may be interrupted because it requires more memory or it has been pre-empted by another process. We know that there is enough memory. This leaves the theory that the process is constantly getting interrupted by other processes with equal or higher priority. This fact is not so hard to conceive because of the number of processes that run on the box. Each process is entitled to its share of the CPU and therefore gets processed in while the original process is thrown to disk or is sitting in cache. This event is known as context switching. And because of the switching to disk or cache, the CPU is busy doing these swapping tasks and doing less users requests like processing database queries. Increasing the buffer cache might relieve some of the disk I/O in this instance.

Kernel Tuning

What is a Kernel

The kernel is the main program that controls the computer's physical and logical components. It is one of the key factors to system tuning and performance. The fastest system in the world that is not tuned properly for a specific application would run like a 286 and would be totally unusable. The kernel has three main parts:

File Subsystem

Accesses file data using a buffering mechanism that regulates data flow between kernel and storage devices. The buffering mechanism interacts with block I/O device drivers to initiate / control data transfer to and from kernel.

Process Control Subsystem

Performs memory management. If there is not enough physical memory, the kernel swaps processes in and out between main memory and swap space. The swapper schedules the allocation of memory. The scheduler allocates the CPU to the processes.

Hardware Control

Handles I/O and interrupts through device drivers.

Once the unix kernel is found on disk, it is loaded into memory. The kernel then builds the first process, process 0 or swapper. It is responsible for scheduling. Process 0 creates the init process (process 1) and the pager process (process 2). The pager is responsible for maintaining free memory pages. All other processes are descendants of init by either fork() or exec(). The VMark Universe product hits all the main areas of the kernel for tuning purposes. Here are some of the settings currently on ears890.

kernel parameters system default VMark recommendation currently on ears890 need to change to
shmseg 12 15 16
shmmax 671,088,608 360 671,088,608
shmmni 100 concurrent users + 2 1,024
semmns 128 51 128
msgmnb 32,768 32k 32,768
msgmax 32,768 2,048 32,768
msgseg 7,168 2,048 7,168
msgssz 8 32 8 32
nbuf 0 some setting 0 1,024
maxfiles 60 N/A 80
maxswapchunks 256 N/A 700
maxuprc 25 N/A 512
maxusers 32 N/A 2,048
maxvgs 10 N/A 32
ncallout 64+nproc N/A (64+nproc+36*32)
npty 60 N/A 2,048
nstlbe 0 N/A 1,024
semmap semmni + 2 N/A 10 1,026
nproc 20+ 8*maxusers N/A 20+ 8*maxusers

/* default configurable parameter values */

tunable acctresume default 4;

tunable acctsuspend default 2;

tunable bootspinlocks default 64;

tunable bufpages default 0;

tunable check_alive_period default 4;

tunable dmmax default 2048;

tunable dmmin default 32;

tunable dmshm default 2048;

tunable dmtext default 2048;

tunable dskless_fsbufs default "(SERVING_ARRAY_SIZE)";

tunable dskless_node default 0;

tunable dst default 1;

tunable eqmemsize default 10;

tunable file_pad default 10;

tunable fs_async default 0;

tunable iomemsize default "(10 * NBPG)";

tunable maxdsiz default 0x04000000;

tunable maxssiz default 0x00800000;

tunable maxswapchunks default 256;

tunable maxtsiz default 0x04000000;

tunable maxuprc default 25;

tunable minswapchunks default 1;

tunable msgmap default "(2 + MSGTQL)";

tunable msgmax default 32768;

tunable msgmnb default 32768;

tunable msgmni default 50;

tunable msgseg default 7168;

tunable msgssz default 8;

tunable msgtql default 256;

tunable nbuf default 0;

tunable ncallout default "(64 + NPROC)";

tunable netisr_priority default -1;

tunable netmemmax default 0;

tunable nni default 2;

tunable nfile default "(16 * (NPROC+16+MAXUSERS) / 10 + 32 + 2*NPTY)";

tunable nflocks default 200;

tunable ngcsp default "(8 * NUM_CNODES)";

tunable ninode default "((NPROC + 16 + MAXUSERS) +32)";

tunable no_lvm_disks default 0;

tunable nproc default "(20 + 8 * MAXUSERS)";

tunable no_lvm_disks default 0;

tunable nproc default "(20 + 8 * MAXUSERS)";

tunable npty default 60;

tunable nstlbe default 0;

tunable nswapdev default 10;

tunable nswapfs default 10;

tunable ntext default "(24 + MAXUSERS + 20)";

tunable num_cnodes default "((5*SERVER_NODE) + (DSKLESS_NODE))";

tunable pfail_enabled default 1;

tunable public_shlibs default 1;

tunable reboot_option default 1;

tunable rec_inst_kernel default 0;

tunable retry_alive_period default 21;

tunable scroll_lines default 100;

tunable selftest_period default 120;

tunable semaem default 16384;

tunable semmap default "(SEMMNI + 2)";

tunable semmni default 64;

tunable semmns default 128;

tunable semmnu default 30;

tunable semume default 10;

tunable semvmx default 32767;

tunable server_node default 1;

tunable serving_array_size

default "(SERVER_NODE * NUM_CNODES * MAXUSERS + 2 * MAXUSERS)";

tunable shmmax default 0x4000000;

tunable shmmni default 100;

tunable shmseg default 12;

tunable swapmem_on default 1;

tunable swchunk default 2048;

tunable timeslice default "(HZ/10)";

tunable timezone default 420;

tunable unlockable_mem default 0;

tunable using_array_size default "(NPROC)";

tunable maxfiles_lim default 1024;

tunable maxfiles default 60;

/* Tunable parameter itebuflines has been renamed scroll_lines. This

* line will cause assignments to itebuflines to be made to scroll_lines.

*/

#define itebuflines scroll_lines

/* default BX.25 configurable parameter values */

tunable BX25SMAX default 128; /* maximum number of concurrent connections */

tunable BX25SBUFS default 256; /* buffer descriptors of session level */

tunable BX25SBYTES default 264192; /* bufr space available to l3 and session */

tunable BX25LINKS default 64; /* maximum number of X.25 synchronous links */

tunable BX25BUFS default 1032; /* buffer descriptors of packet level */

tunable BX25BYTES default 196608;/* buffer space available to l2 and l3 */

tunable BX25HLPROT default 2; /* number of higher level protocols */

/* default DataKit configurable parameter value */

tunable DELAYCLOSETIMER default 500;

/* default DataKit configurable parameter value */

tunable DELAYCLOSETIMER default 500;

tunable NUM_DKHS default 0; /* if changed to 1, then space.h includes dkspace.h

*/

/* default streams configurable parameter values */

tunable NSTREVENT default 50;

tunable STRMSGSZ default 4096;

tunable STRCTLSZ default 1024;

tunable NSTRPUSH default 16;

/* default netware parameter values */

tunable ipx_max_sockets default 100;

tunable spx_max_sockets default 100;

tunable spx_max_connections default 30;

tunable nvt_connections default 60;

tunable ncp_max_engines default 50;

tunable ncp_max_clients default 250;

/* default LVM parameter */

tunable maxvgs default 10;

~

Use the following commands to help evaluate some kernel information.

/etc/lssf `/etc/devnm / | cut -d" " -f1`

Look in the following directory for more information.

/etc/conf/S800

/etc/conf/gen/S800.SAM

Action Items

Processes

  1. Reduce the login processes by going straight to uv as opposed to ksh
  2. Limit the amount of concurrent users or offset the load by adding more processors

Networking

  1. Remove static routes and let the Network Backbone Routers do the work
  2. Eliminate other network heavy traffic during this "reconfiguration phase"

Disk Performance

  1. Move the heavy hit databases to another file system that has the least amount of disk I/O
  2. Re-partition the logical volumes so that the physical drives span more than one controller
  3. Invoke more buffer cache in memory (bufpages). This will reduce the number of disk I/O hits

Paging

  1. Reduce the amount of page faults by decreasing the amount of processes.

Kernel

  1. Reduce number of processes per user
  2. Reduce the number of ptys (to control a limited number of users)
  3. Reduce maxusers
  4. Increase bufpages for more memory buffer cache
  5. Increase msgssz to recommended size
  6. Reduce ncallout, shmmni & semmap to recommended size

Goals for the Unix Team

Mark's Goals

  • My goals are to attack the problems of the host ears890 by the specific area that has been outlined in this report. Taking one step at a time will ensure the stability of the host to users while the box goes is in the transition period.
  • After the first evaluation of the system's resources as it exists today, I will put a plan into effect for the future. Outlining the possibilities for the next 6 months and the next 5 years.
  • Create an effective team environment that cna plan an coordinate task efficiently between departments.

    Dave's Goals

    1. He wants to control root access from operations. I would tend to agree in this area. System security is a very serious manner and the first thing I want to do is restrict the root password from everyone.
    2. He wants system maintenance to be completed.
    3. Desires 1 person concentrating on HW/SW and 1 person concentrating Universe.
    4. He wants to control the programmer's environment in the sense of environment control.
    5. He sees himself communicating / interfacing with ops, clients, & programmers as his daily task.
    6. He acknowledges the need for a heavy weight Unix administrator
    7. He wants to upgrade the OS to 10.x and Universe to 8.3.3.2 from 7.3.3.1

    Dave's Major Issues

    1. SNA problems on reboot
    2. 24 hour on-call support since February
    3. enp (vax server) problems with transfer & base dir on root file system
    4. Dave is very comfortable with his physical environment. He prefers the vt100 emulation on the ibm3151 terminal. He uses his PC for mail and word processing. He is very comfortable with win31 but needs pctcp working. He uses an aol account at home.
    Configure NFS

    (Not Using SAM ... in HP-UX 7.0)

    /etc/netnfsrc change NFS_SERVER=1

    /etc/inetd.conf rm rpc

    /etc/inetd -c

    /etc/metnfsrc nfs and mountd should be running

    Peforming memory diags:

    ps -ef | grep MEMLOGP

    kill -STOP PID

    kill -CONT PID

    /etc/init S Single User mode

    <cntl> x continue install

    Install from Server

    /etc/update to load NE-INSTAL-700 fileset to boot server

    /etc/boottab contaons install:HPS700:

    system things

    files

    /hp-ux kernel built here

    /etc/ttytype tty entries example: 3151 tty01 #mstolz - model 3151 found in /usr/lib/terminfo

    /etc/inittab for getty entries:

    04:2:respawn:/etc/getty tty04 H #mstolz 04=i.d. 2=init state H=Hardwire

    /etc/conf/dfile device driver info

    /usr/adm/rc.log logging

    /etc/newconfig/Ignition/SystemInfo for explorer

    /etc/rc calls /etc/netlinkrc which calls /etc/src.sh for hostname

    vi /etc/issue contains hostname + sw version

    /etc/newconfig/README

    /etc/master

    commands

    cstm or xstm (X Support Tools Manager)

    mv /etc/copyright /etc/copyright.ORIG

    rm "cat copyright" from /etc/profile

    telinit q reread inittab file for changes

    ifconfig lan0 inet 11.0.0.1

    mkfontdir

    cp newfont.snf $HOME/myfonts

    mkfontdir $HOME/myfonts

    xset fp rehash

    tee

    cut -c1 file cuts 1st char in every line of file

    92135 swap space of bighp

    96204 swap space of nowhere

    Login

    $HOME/.login

    umask 002 will mask the 2 positions in "other" permissions.

    owner-group-other

    rwx rwx rwx

    0 0 XX (2)

    files that are created can only be read by "other."

    biff [y or n] yes or no for notification of mail arriving.

    stty intr ^c use <cntl> c to interupt

    stty echoe destructive backspace

    /etc/security - disables root login, only devices listed in file have access

    fuser - files open by user

    what "file" - file stats

    strip "file" strips the debug info

    nroff -man Tcl.1 -Tlj | lp to print a man page

    ulimit is set in mtune

    nm "file" or file core will show you information in that file

    pstat pid shows the memory stat us

    mount -F hsfs -o ro /dev/dsk/cott6d0s0 -d Solaris-2.0 PKS

    HP-UX has 9.01 VUE 3.0 Motif 2.1

    Quatum PD425S drive cdrom scsi=3 on nowhere,dursban; scsi=2 on sevin