Notes for Week 4

The system log file is /var/log/sys.log, and security logging is done to /var/log/auth.log. Recent entries can be examined using
tail -n100 /var/log/sys.log
grep can be used to search the log file for recurring events of interest, such as reboots:
grep /var/log/sys.log -ie 'syslogd.*restart'
users logging in as root, or su-ing to become root:
grep /var/log/auth.log -ie 'root'
or hardware errors:
grep /var/log/sys.log -ie 'device-file-name'
where "(device file name)" is, for example, sr0.
The Xorg log file is /var/log/Xorg.(display-number).log.
Windows log files can be examined using the "Event Viewer" (under "Administrative Tools" in the Control Panel). The files themselves are C:\Windows\*.log.

The kernel is essentially a large C program. The main program begins by interrogating the CPU, RAM, the I/O devices, the PCI bus, the Plug and Play BIOS, etc. The root partition is then mounted read-only (so that it can be checked without corrupting it) and the init program is started. init stays running as long as the system is up, and is the ultimate "ancestor" (parent, grandparent, etc.), of every process that runs.
The following output of the pstree -npl command illustrates this:
```
init(1)-+-udevd(1055)
        |-syslogd(1265)
        |-klogd(1288)
        |-cupsd(1439)
        |-ntpd(1481)
        |-dbus-daemon(1509)
        |-sshd(1526)
        |-master(1618)-+-pickup(1620)
        |              `-qmgr(1621)
        |-fcron(1646)
        |-gpm(1672)
        |-bash(1678)---bash(1700)---xinit(1716)-+-X(1717)
        |                                       `-sh(1721)---wmaker(1722)---wmaker(1723)-+-xosview(1726)
        |                                                                                |-xclock(1727)
        |                                                                                `-xterm(1730)---luit(1731)---bash(1732)---pstree(2827)
        |-agetty(1679)
        |-agetty(1680)
        |-agetty(1681)
        |-agetty(1682)
        `-agetty(1683)
```
The numbers in parentheses are pids (process identifiers); every process that is running has a unique pid whose maximum value is a function of RAM size (after the pids increment to the maximum, they start over again at the lowest unused number).
Some of the processes started by init are daemons: programs that provide services to the operating system, it's users, or network clients. Their names typically end with a "d" (although not always), and are pronounced, for instance, "k"-"log"-"d".
- udevd handles dynamic creation and destruction of device files as hardware is attached and detached from the system;
- syslogd and klogd handle system and kernel logging (of error and status messages);
- cupsd is the print spooler daemon;
- ntpd is the network time protocol daemon, which keeps the system clock synchronized to standard clocks on the Internet;
- dbus-daemon is the message bus daemon (used for inter-process communication);
- sshd is the secure shell server daemon;
- master, pickup and qmgr are processes associated with the postfix mail transport agent (see week 9);
- fcron processes scheduled jobs (see week 9);
- gpm is the general purpose mouse handler (does the auto-copy and scroll-wheel pasting we discussed the first week).
The monitor and keyboard directly attached to the PC which runs Linux is the console. By entering Alt-F1 through Alt-F6, the console "operator" (as distinguished from any user which might telnet to Linux to use the system remotely) can switch between the virtual consoles. The agetty processes listed above issue a login prompt to each of the virtual consoles; when a user enters a login name, the agetty program exits and passes that name to the login program, which asks for and validates the password, and then starts a shell (in this case bash), which supplies the command prompt. When the user executes the logout command, the login process exits and init starts another agetty.
The startx command is really a script which in turn runs the xinit program, which in turn runs X. X is the display "server", or driver, which switches the console to graphical mode and controls the display until X-Windows is exited. While X is running, the consoles are unavailable through the Alt-F# keys. X starts the window manager (in this case, wmaker, or WindowMaker), which controls drawing on the graphical display (windows, title bars, buttons, scroll bars, icons, etc.). There are a number of window managers available. Client programs running under the window manager include xosview, xclock and xterm (which requires an associated luit process for Unicode support). And of course, xterm emulates a terminal (telnet) session, and so starts a bash shell under which the pstree command whose output we have been analyzing was run.

The corresponding output of the ps aux command is

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   2164   588 ?        Ss   13:56   0:00 init [3]  
root         2  0.0  0.0      0     0 ?        S    13:56   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    13:56   0:00 [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S<   13:56   0:00 [kworker/0:0H]
root         7  0.0  0.0      0     0 ?        S    13:56   0:01 [rcu_sched]
root         8  0.0  0.0      0     0 ?        S    13:56   0:00 [rcu_bh]
root         9  0.0  0.0      0     0 ?        S    13:56   0:00 [migration/0]
root        10  0.0  0.0      0     0 ?        S<   13:56   0:00 [khelper]
root        11  0.0  0.0      0     0 ?        S    13:56   0:00 [kdevtmpfs]
root        12  0.0  0.0      0     0 ?        S<   13:56   0:00 [netns]
root        13  0.0  0.0      0     0 ?        S    13:56   0:00 [kworker/u2:1]
root       311  0.0  0.0      0     0 ?        S<   13:56   0:00 [writeback]
root       313  0.0  0.0      0     0 ?        S<   13:56   0:00 [bioset]
root       315  0.0  0.0      0     0 ?        S<   13:56   0:00 [kblockd]
root       382  0.0  0.0      0     0 ?        S<   13:56   0:00 [ata_sff]
root       393  0.0  0.0      0     0 ?        S    13:56   0:00 [khubd]
root       404  0.0  0.0      0     0 ?        S<   13:56   0:00 [md]
root       526  0.0  0.0      0     0 ?        S    13:56   0:00 [kswapd0]
root       593  0.0  0.0      0     0 ?        S    13:56   0:00 [fsnotify_mark]
root       610  0.0  0.0      0     0 ?        S<   13:56   0:00 [crypto]
root       746  0.0  0.0      0     0 ?        S    13:56   0:00 [scsi_eh_0]
root       749  0.0  0.0      0     0 ?        S    13:56   0:00 [scsi_eh_1]
root       753  0.0  0.0      0     0 ?        S    13:56   0:00 [kworker/u2:3]
root       797  0.0  0.0      0     0 ?        S    13:56   0:00 [kworker/0:2]
root       812  0.0  0.0      0     0 ?        S<   13:56   0:00 [kpsmoused]
root       852  0.0  0.0      0     0 ?        S<   13:56   0:00 [raid5wq]
root       892  0.0  0.0      0     0 ?        S<   13:56   0:00 [deferwq]
root       901  0.0  0.0      0     0 ?        S<   13:56   0:00 [kworker/0:1H]
root      1055  0.0  0.1  10048  1208 ?        Ss   13:56   0:00 /lib/udev/udevd --daemon
root      1265  0.0  0.0   2220   604 ?        Ss   13:56   0:00 /sbin/syslogd -m 0
root      1288  0.0  0.1   3044  1460 ?        Ss   13:56   0:00 /sbin/klogd
root      1439  0.0  0.2   6272  2252 ?        Ss   13:56   0:00 /usr/sbin/cupsd -C /etc/cups/cupsd.conf
ntp       1481  0.0  0.1   5348  1820 ?        Ss   13:56   0:00 /usr/sbin/ntpd -g -u ntp:ntp
message+  1509  0.0  0.0   2908   360 ?        Ss   13:56   0:00 /usr/bin/dbus-daemon --system
root      1526  0.0  0.1   5064  1064 ?        Ss   13:56   0:00 /usr/sbin/sshd
root      1618  0.0  0.1   7928  1368 ?        Ss   13:56   0:00 /usr/libexec/postfix/master -w
postfix   1620  0.0  0.1   7996  1636 ?        S    13:56   0:00 pickup -l -t unix -u
postfix   1621  0.0  0.1   8052  1648 ?        S    13:56   0:00 qmgr -l -t unix -u
root      1646  0.0  0.0   3076   624 ?        Ss   13:56   0:00 /usr/sbin/fcron
root      1672  0.0  0.0   2508   388 ?        Ss   13:56   0:00 /usr/sbin/gpm -m /dev/psaux -t ps2
ken       1678  0.0  0.1   5988  1836 tty1     Ss   13:56   0:00 -bash
root      1679  0.0  0.0   4268   696 tty2     Ss+  13:56   0:00 /sbin/agetty tty2 9600
root      1680  0.0  0.0   4268   700 tty3     Ss+  13:56   0:00 /sbin/agetty tty3 9600
root      1681  0.0  0.0   4268   692 tty4     Ss+  13:56   0:00 /sbin/agetty tty4 9600
root      1682  0.0  0.0   4268   700 tty5     Ss+  13:56   0:00 /sbin/agetty tty5 9600
root      1683  0.0  0.0   4268   692 tty6     Ss+  13:56   0:00 /sbin/agetty tty6 9600
ken       1700  0.0  0.1   5988  1148 tty1     S+   13:56   0:00 -bash
ken       1716  0.0  0.0   3572   684 tty1     S+   13:56   0:00 xinit /home/ken/.xinitrc -- /usr/bin/X :0 -auth /home/ken/.serverauth.1700
root      1717  0.7  1.6  62572 16564 tty7     Ss+  13:56   0:18 /usr/bin/X :0 -auth /home/ken/.serverauth.1700
ken       1721  0.0  0.1   5812  1264 tty1     S    13:56   0:00 sh /home/ken/.xinitrc
ken       1722  0.0  0.1  10660  1264 tty1     S    13:56   0:00 /usr/local/bin/wmaker
ken       1723  0.0  0.5  12588  5440 tty1     S    13:56   0:00 /usr/local/bin/wmaker --for-real
ken       1726  0.4  0.1   5324  2024 tty1     S    13:56   0:10 xosview
ken       1727  0.0  0.3  10472  3720 tty1     S    13:56   0:00 xclock -strftime %a, %b %e - %l:%M %P -g 172x32+0-0 -digital -update 1
ken       1730  0.0  0.8  15380  8708 ?        Ss   13:56   0:00 xterm
ken       1731  0.0  0.0   4632   692 pts/0    Ss+  13:56   0:00 /usr/bin/luit
ken       1732  0.0  0.1   6136  1944 pts/1    Ss   13:56   0:00 bash
root      2782  0.0  0.0      0     0 ?        S    14:31   0:00 [kworker/0:0]
root      2826  0.0  0.0      0     0 ?        S    14:36   0:00 [kworker/0:1]
ken       2828  0.0  0.0   5240   976 pts/1    R+   14:36   0:00 ps aux

except that the process 2827 executing the pstree command in the previous output has been replaced by the process 2828 executing the ps command whose output appears above.

The columns are interpreted as follows:

USER = the login account under which the process is running;
PID = the process ID;
%CPU = the CPU time used by the process divided by the time the process has been running;
%MEM = the RAM used by the process divided by the total RAM available;
VSZ = the total virtual memory size of the process in Kb;
RSS = the size in Kb of the resident set (portion currently residing in RAM);
TTY = where text output goes (ttyn = virtual console, pts/n = xterm window)
STAT = process state
R = runnable (on the run queue)
S = sleeping
T = traced or stopped (for instance, by Crtl-Z, which you should never use!)
W = paging
Z = a zombie process, which has terminated but its parent has not cleaned it up properly; init will do this when the parent process exits
START = clock time when process started
TIME = CPU time used to date
COMMAND = process command line

Some of the process names are self-explanatory, while others are not so. For example,

[kdevtmpfs] handles the /dev pseudo-filesystem;
[khubd] handles the internal USB hub;
[kswapd0] handles writing of modified virtual memory pages to swap partition 0;

The daemons listed in brackets are actually not separate programs, but are part of the kernel itself. Those ending with "/0" or "/1" denote processes attached to specific CPUs.

Note that there is no agetty for tty1; this is the virtual console in which all of the processes in the current interactive login session are running. Also note that X runs as root, although it was started by ken; X must run as root in order to control the display.

In Windows you can invoke the task manager using the keys Ctrl-Alt-Delete, but it does not show kernel processes, amd provides precious little information.

Information about processes running on the system is kept in the /proc directory. It and /sys are really pseudo-filesystems: they only exist in memory, providing a filesystem interface to kernel data. Each process has its own directory whose name is /proc/pid (ie., the wmaker process in the example above would be located in the directory /proc/3407). In that directory is (among other things):
- the command line used to start the process (cat /proc/pid/cmdline);
- the current working directory of the process (ls -l /proc/pid/cwd);
- the process' environment variables (cat /proc/pid/environ | tr "\000" "\n");
  The stroke ("|") indicates a pipe: the output (stdout) of one command becomes the input (stdin) of the next. We will go into pipes in more detail in week 8.
  The tr command is used here to translate NUL characters to newline characters (strings are terminated by NULs in C; newline is the end of line character in UNIX; Windows uses "\r\n", or "carriage return/line feed" for its newline sequence).
- the program which is running (ls -l /proc/pid/exe);
- the files which the process currently has open (ls -l /proc/pid/fd);
  ("fd" is short for "file descriptor".) Note that all processes have the following three files open:
  - 0 - stdin: the default source of input (the keyboard, when running interactively from the command line);
  - 1 - stdout: the default destination of output (the display, when running interactively from the command line); and
  - 2 - stderr: the default destination for error messages (the display, when running interactively from the command line).
- limits on resource utilization by the process (cat /proc/pid/limits);
- the complete map of virtual memory utilization by the process (cat /proc/pid/maps);
- the root of the file system accessible to the process (ls -l /proc/pid/root) (see the chroot command);
- the process status (cat /proc/pid/status) and kernel wait point (cat /proc/pid/wchan); most of this information can be teased out of ps.
In addition, information about the hardware and operating system can be obtained by:
- cat /proc/asound/* (information about ALSA devices);
- cat /proc/cmdline (the kernel startup parameters);
- cat /proc/cpuinfo (detailed information about the cpu chip(s));
- cat /proc/diskstats (see /usr/doc/linux-3.13/iostats.txt);
- cat /proc/scsi/scsi;
- cat /proc/interrupts (also iomem and ioports; device resource information);
- cat /proc/locks (compare with fuser output);
- cat /proc/meminfo (see /usr/doc/linux-3.13/filesystems/proc.txt);
- cat /proc/mounts | column -t (currently mounted filesystems, with options);
  "column -t" formats the output of the cat command to make it more readable.
- cat /proc/stat (see /usr/doc/linux-3.13/filesystems/proc.txt);
- cat /proc/uptime (see /usr/doc/linux-3.13/filesystems/proc.txt);
- cat /proc/version (current running kernel version);
- cat /sys/bus/usb/devices/device/* (particularly bMaxPower and speed).
You can also find information on your pci devices in /proc/bus/pci/devices, but the lspci command is much more helpful.
Study the proc man page in section 5 for more information.

In general, stay out of /proc/sys - unless you really know what you're doing, it is extremely easy to shoot yourself in the foot by changing parameters on the running kernel. That said, there are two parameters that I like to change at startup:
echo 1 > /proc/sys/vm/dirty_background_ratio
echo 1 > /proc/sys/vm/dirty_ratio
">" indicates a redirection of stdout to a file.

These cause flush to run more frequently, which I find spreads disk i/o out, reducing the times when the system seems to pause while buffers are being flushed. Your mileage may vary... See /usr/doc/linux-3.13/sysctl/vm.txt for more information.

kill can be used to send signals to processes (see the signal man page in section 7 for the meanings of the various signals). Many daemons will re-load their configuration files when they receive a SIGHUP.

EXERCISES for Week 4:
1. What is the most recent precision computed for ntpd?
2. What is the fully qualified filename for the parent program of xclock?
3. What is the program on your system which has used the largest amount of CPU time since it began?
4. How much cache memory does your cpu chip have?

Please send comments or suggestions to the author.