PAR(1) PAR(1) NAME par - process activity reporter / truss-like system call tracer SYNOPSIS par [report-options] [collection-options] [cmd args ...] par [report-options] [collection-options] [-p pid] [-p ...] par [report-options] [collection-options] [-t time] par [report-options] DESCRIPTION par is a system utility program that reports on system call and scheduling activity for one or more processes. par can be used to trace the activity of a single process, a related group of processes, or the system as a whole. See the EXAMPLES section near the end for some examples on how par is commonly used. When tracing system calls, par(1) prints a report showing all system calls made by the subject processes complete with arguments and return values. In this mode, par(1) also reports all signals delivered to the subject processes. When tracing scheduler actions, par(1) reports all scheduling events taking place in the system during the measurement period. The report shows each time a process is put on a run queue, started on a processor, and descheduled from a processor. All scheduling events are timestamped and, when available, include the reason for the action. two ways: padc can be run separately and the output saved in a file (to be fed to par as a separate operation), or padc can be invoked by par to perform the data collection and reporting in one step. par can generate different reports from data collected by padc depending on the reporting options that are specified. The ability to generate different reports from a single set of data is one reason that it is often desirable to run the data collection as a separate step. There are three things that need to be specified on the par command line: what information to report, what data should be collected, and what objects are to be monitored. par can be run without displaying any information (collection-only) or without collecting any event data (report-only). Objects to be monitored may be running processes or commands that are started up specifically for the purpose of collecting event data. If an object is specified for monitoring (either a command to launch or an existing process), but no data collection or reporting options are specified, then par defaults to collecting and reporting system call and signal data. Data Collection Options These options should only be supplied when event data is to be collected by running a command or by tracing an already running process or set of processes. -s Collect system call and signal data for the cmd or those processes specified via the -p option. If neither are specified, system call and signal data for all processes that you have permissions to access will be collected. -r Collect scheduler activity data for the cmd or those processes specified via the -p option. If neither are specified, scheduler activity data for the entire system will be collected. -k Collect disk i/o activity data. -i Inherit tracing to forked children of object processes. -O file Write raw event data to the specified file. -C Collect CXFS actitity events. Reporting Options Note that various options are meaningful only when the event data includes relevant information. For example, requesting a report on system call activity is useless if no system call events are collected (with the -s option) or none are present in a file of previously collected data. -S Print a summary of system calls and signal counts. -SS Print both the summary of system call activity and a trace of each system call and signal action. -Q Print a summary of scheduling work. -QQ Print both the summary of scheduling work and a trace of each scheduler operation. -QQQ In addition to the detailed scheduling trace, print the contents of the global run queue after each scheduler operation. -n syscall Show records for the specified system call, where the system call is specified by name or number. This option may be specified multiple times. Specifying this option automatically enables detailed system call reporting. -e syscall Exclude the specified system call from the report. This option may be specified multiple times. Specifying this option automatically enables detailed system call reporting. Other options that control the format and content of reports are: -A Show system call parameter data (e.g. from a read call) in character format. Non printable characters are output in hex. Normally, par selects ASCII or binary format for data according to the data content. -a len Set the maximum number of bytes printed in character format for data (e.g. from a read call) to len. This value defaults to 30 bytes. The larger of the value for this option and the -b option is used to inform padc, if appropriate, how much data to collect (see the -I option of padc). The maximum value for this option is 4096 bytes. -B Show system call parameter data (e.g. from a read call) in hex binary format. Normally, par selects ASCII or binary format for data according to the data content. -b len Set the maximum number of bytes printed in binary format for data (e.g. from a read call) to len. This value defaults to 16. The maximum value for this option is 4096 bytes. -c Do not print CPU numbers in detailed trace reports. -d Show each system call as two entries: one for when the system call is begun and a second when the system call completes. Normally par displays system calls as a single line, showing input arguments, output arguments and return values. The time displayed is the time of the start of the system call. This compaction is done unless the duration of the system call exceeds a nominal threshold (25 microseconds by default). With the -d option system calls are always displayed as beginning and ending operations. -l Show system call output in a long format that includes each process name and the CPU on which it is run. By default par will use this format whenever it is needed to avoid confusion; e.g. when multiple processes might appear in the report. Otherwise, par uses a more compact format that does not show the process name or CPU number. This option is only useful when a detailed report is requested; e.g. -QQ and/or -SS. -o file Print all output (including errors) to file. This is useful when monitoring a program that itself does output. -P pid List activity only for the process specified by pid. Note that this is markedly different from the -p pid option that requests that the named pid be traced. Thus one could request that processes 100 and 101 be traced, but only report system calls for process 101. This option is typically specified when padc has been used to collect data on a number of processes - often either by collecting for all processes on the system or all processes descended from a specified process. -R Print event times as raw times in terms of the absolute valued system real-time clock values when they occured. -T Print thread and process IDs. This is especially useful for pthread applications which may have more then one thread of execution operating under the same process ID. -u Print event times as milliseconds and microseconds since the last displayed event. Object Specification -p pid Trace the process specified by pid. If the -i flag is specified then any child processes created by pid will also be traced. Multiple -p options may be given to trace multiple processes. In this mode, padc(1) is automatically invoked by par. -t time Terminate the trace after time seconds. Primarily useful when tracing the system as a whole. [command arguments ...] Run the specified command with tracing enabled. If the -i option is specified, any child processes that are created by command will also be traced. In this mode, padc(1) is automatically invoked by par. nothing If no specification of an object is given, all specified activity will be traced for the system as a whole. Note that only the superuser can trace the system as a whole. In this mode, padc(1) is automatically invoked by par. If no data collection options are specified and no object is specified, par will read standard input as output from padc and report the data according to the reporting options selected. INTERPRETING THE REPORTS par generates several different reports. Summary reports, requested with the -S and -Q options, are straightforward and are not described here. Other reports provide a detailed listing of the event data; they are composed of lines of the form: <time>mS[<cpu>] <name>(<pid>): ... with the following explanations: <time> The time of the event in milliseconds relative to the start of data collection. If the -u option is supplied, <time> will be followed by the number of microseconds since the last event (enclosed in parenthesis). <cpu> The CPU number the event was generated on. This is displayed if a long listing is requested with the -l option or if there is more than one CPU in the system that data is collected on. The -c option can be used to disable display of the CPU number. <name> The name of the process (as displayed by ps(1)). This is only displayed for a long listing. <pid> The PID of the process. This is only displayed for a long listing. The remaining information that par prints depends on the type of event that is being reported. For system calls each line is of the form: ... : <syscall>(<arg1>, <arg2>, ..., <argN>) = <result> with the following information: <syscall> The system call name. If the system call being displayed is split into 2 events, the event marking the end of the system call will have END- prepended to the name. See below for some help in decoding system call names. par attempts to print an entire system call - input arguments, output arguments, and error return on a single line. It does not do this if the -d option is given or if another event needs to be reported between the start and end of a call. <argN> The system call arguments. Various amounts of decoding of arguments is done. Some system calls have complex arguments that have both input and output values. If an entire system call is printed on one single line, these input/output arguments have the words IN: or OUT: printed before the decoding of the argument. Some complex indirect parameters are displayed symbolically using their C structure definition. Note that not all indirect parameter values are available; some are not returned by the operation system while others may not be copied out because doing so would exceed the maximum amount of indirect data to included in an event (see the -I option for padc). <result> The error status or return value of the system call. For system calls that simply return success or failure, par prints OK for success, and the error value for failure. System calls that return values have those values printed. Since par's information comes straight from the operating system at the system call level, some calls that par presents may not seem to correspond to the calls that the application made. This is because some system calls are implemented in runtime libraries on top of more primitive system calls. Some notable examples of this are: waitsys is the underlying system call for all wait-like calls. Its arguments are the same as waitid(2) except that it takes as a fifth argument a pointer to a struct rusage. ?xstat These stat calls are the same as the application entry points except that the first argument is a version number. sigaction is used to implement all type signal routines. It takes one additional parameter than the application entry point - the address of the library handler that all signals funnel through. sigreturn is used to return a process from its signal handler to the previous context. sigpoll is used to implement sigwaitrt(3) and sigtimedwait(3). ERESTART is returned when a system call should be automatically restarted after being interrupted by a signal (see sigaction). This error is never actually returned to the user but par reports the re-invocation of a system call as an error. EXAMPLES par ls / Display a system call trace and summary for the command 'ls /'. (par supplies the implicit -sSS options because a command to launch was specified without any reporting or collection options.): apache% par ls / MISER de hosts mnt par.out tmp var RTMON debug hw ns proc tmp_mnt TESTS dev lib opt proj unix bin doouf lib32 out.1 rtmon.out unix.benf build etc lib64 output.1 sbin unix.orig build11 ficus miser par stand usr 0mS[ 1] was sent signal SIGUSR1 0mS[ 3] received signal SIGUSR1 (handler 0x10002560) 0mS[ 3] END-pause() errno = 4 (Interrupted function call) 1mS[ 3] sigreturn(0x7fff2b40) OK 1mS[ 3] execve(./ls, 0x7fff2f6c, 0x7fff2f78) 262mS[ 3] END-execve() errno = 2 (No such file or directory) 262mS[ 3] execve(/usr/sbin/ls, 0x7fff2f6c, 0x7fff2f78) errno = 2 (No such file or directory) 263mS[ 3] execve(/usr/bsd/ls, 0x7fff2f6c, 0x7fff2f78) errno = 2 (No such file or directory) 264mS[ 3] execve(/sbin/ls, 0x7fff2f6c, 0x7fff2f78) 274mS[ 3] END-execve() OK 274mS[ 3] open(/lib32/rld, O_RDONLY, 04) = 3 275mS[ 3] read(3, <7f 45 4c 46 01 02 01 00 00 00 00 00 00 00 00 00>..., 512) = 512 276mS[ 3] elfmap(3, 0x7fff2d54, 2) = 0xfb60000 276mS[ 3] close(3) OK 279mS[ 3] getpagesize() = 16384 279mS[ 3] sysinfo(_MIPS_SI_PROCESSORS, 0x7fff2dc0, 257) = 43 281mS[ 3] open(/dev/zero, O_RDONLY, 0) = 3 282mS[ 3] mmap(0xfbd4000, 16384, PROT_WRITE|PROT_READ, MAP_PRIVATE, 3, 0) = 0xfbd4000 282mS[ 3] close(3) OK ... Note that output from the command is intermixed with the system call report; the -o option can be used to direct the report to a file separately from any output generated by the command. The report about the command receiving a SIGUSR1 signal is expected; this is done as part of the normal procedure for starting up a program with tracing. Finally, note that many system call parameters are displayed symbolically and that the indirect value of many parameters is displayed; e.g. ``/lib32/rld'' and ``/dev/zero'' for open. par -rsSSQQ -O ls.padc ls / Report on system calls and scheduling activities for the command 'ls /', and also record the raw event data in the file ls.padc. par -o outfile -n open -n close ls Trace only the open and close system calls. Write the resulting output to outfile. Note that it is not necessary to specify -SS options since they are implied by the -n option. Also, the -s option is not required because system calls are the default data to collect when a command is specified. par -o outfile -i -t 30 -p 1 Trace all processes started directly by process 1 (which is the init process, the ancestor of all user processes) for thirty seconds, and store the report in the file outfile. Note that the -i option will cause only processes newly created by init to be traced; i.e. it does not mark all existing child processes for tracing. LIMITATIONS To reduce system load, when collecting system call event data, system calls executed by padc(1) and rtmond(1) are not recorded. This can lead to some inexplicable gaps when tracing complete system activity. The process name associated with an event may be misleading. This is because a process's name may change between the time an event is generated and the time the event collection process (rtmond) checks for the name. For example, a process may generate events then exit before rtmond is able to query the system for the process name. In this case the events will show up as being associated with a process without a name. A user must have the CAP_PROC_MGT capability to monitor setuid processes or processes with capabilities, and the CAP_DAC_READ_SEARCH capability to monitor a process owned by another user. Under Trusted IRIX, the CAP_MAC_READ capability is also required to monitor processes that are not dominated by the user's MAC label. See capabilities(4) and dominance(5) for more information. SEE ALSO padc(1), rtmond(1), capabilities(4), dominance(5). Page 8