NAME pmie - inference engine for performance metrics SYNOPSIS pmie [-bCdefVvWxz] [-A align] [-a archive] [-c filename] [-h host] [-l logfile] [-n pmnsfile] [-O offset] [-S starttime] [-T endtime] [-t interval] [-Z timezone] [filename ...] DESCRIPTION pmie accepts a collection of arithmetic, logical, and rule expressions to be evaluated at specified frequencies. The base data for the expressions consists of performance metrics values delivered in real-time from any host running the Performance Metrics Collection Daemon (PMCD), or using historical data from Performance Co-Pilot (PCP) archive logs. As well as computing arithmetic and logical values, pmie can execute actions (popup alarms, write system log messages, and launch programs) in response to specified conditions. Such actions are extremely useful in detecting, monitoring and correcting performance related problems. The expressions to be evaluated are read from configuration files specified by one or more filename arguments. In the absence of any filename, expressions are read from standard input. A description of the command line options specific to pmie follows: -a archive is the base name of a PCP archive log written by pmlogger(1). Multiple instances of the -a flag may appear on the command line to specify a set of archives. In this case, it is required that only one archive be present for any one host. Also, any explicit host names occurring in a pmie expression must match the host name recorded in one of the archive labels. In the case of multiple archives, timestamps recorded in the archives are used to ensure temporal consistency. -b Output will be line buffered and standard output is attached to standard error. This is most useful for background execution in conjunction with the -l option. The -b option is always used for pmie instances launched from pmie_check(1). -C Parse the configuration file(s) and exit before performing any evaluations. Any errors in the configuration file are reported. -c An alternative to specifying filename at the end of the command line. -d Normally pmie would be launched as a non-interactive process to monitor and manage the performance of one or more hosts. Given the -d flag however, execution is interactive and the user is presented with a menu of options. Interactive mode is useful mainly for debugging new expressions. -e When used with -V, -v or -W, this option forces timestamps to be reported with each expression. The timestamps are in ctime(3) format, enclosed in parenthesis and appear after the expression name and before the expression value, e.g. expr_1 (Tue Feb 6 19:55:10 2001): 12 1 PMIE(1) PMIE(1) -f If the -l option is specified and there is no -a option (ie. real- time monitoring) then pmie is run as a daemon in the background (in all other cases foreground is the default). The -f option forces pmie to be run in the foreground, independent of any other options. -h By default performance data is fetched from the local host (in real-time mode) or the host for the first named archive on the command line (in archive mode). The host argument overrides this default. It does not override hosts explicitly named in the expressions being evaluated. -l Standard error is sent to logfile. -n An alternative Performance Metrics Name Space (PMNS) is loaded from the file pmnsfile. -t The interval argument follows the syntax described in PCPIntro(1), and in the simplest form may be an unsigned integer (the implied units in this case are seconds). The value is used to determine the sample interval for expressions that do not explicitly set their sample interval using the pmie variable delta described below. The default is 10.0 seconds. -v Unless one of the verbose options -V, -v or -W appears on the command line, expressions are evaluated silently, the only output is as a result of any actions being executed. In the verbose mode, specified using the -v flag, the value of each expression is printed as it is evaluated. The values are in canonical units; bytes in the dimension of ``space'', seconds in the dimension of ``time'' and events in the dimension of ``count''. See pmLookupDesc(3) for details of the supported dimension and scaling mechanisms for performance metrics. The verbose mode is useful in monitoring the value of given expressions, evaluating derived performance metrics, passing these values on to other tools for further processing and in debugging new expressions. -V This option has the same effect as the -v option, except that the name of the host and instance (if applicable) are printed as well as expression values. -W This option has the same effect as the -V option described above, except that for boolean expressions, only those names and values that make the expression true are printed. These are the same names and values accessible to rule actions as the %h, %i and %v bindings, as described below. -x Execute in domain agent mode. This mode is used within the Performance Co-Pilot product to derive values for summary metrics, see pmdasummary(1). Only restricted functionality is available in this mode (expressions with actions may not be used). -Z Change the reporting timezone to timezone in the format of the environment variable TZ as described in environ(5). -z Change the reporting timezone to the timezone of the host that is the source of the performance metrics, as identified via either the -h option or the first named archive (as described above for the -a option). The -S, -T, -O, and -A options may be used to define a time window to restrict the samples retrieved, set an initial origin within the time window, or specify a ``natural'' alignment of the sample times; refer to PCPIntro(1) for a complete description of these options. Output from pmie is directed to standard output and standard error as follows: stdout Expression values printed in the verbose -v mode and the output of print actions. stderr Error and warning messages for any syntactic or semantic problems during expression parsing, and any semantic or performance metrics availability problems during expression evaluation. EXAMPLES The following example expressions demonstrate some of the capabilities of the inference engine. The directory $PCP_DEMOS_DIR/pmie contains a number of other annotated examples of pmie expressions. The variable delta controls expression evaluation frequency. Specify that subsequent expressions be evaluated once a second, until further notice: delta = 1 sec; If total syscall rate exceeds 5000 per second per CPU, then display an alarm notifier: kernel.all.syscall / hinv.ncpu > 5000 count/sec -> alarm "high syscall rate"; If the high syscall rate is sustained for 10 consecutive samples, then launch top(1) in an xwsh(1G) window to monitor processes, but do this at most once every 5 minutes: all_sample ( kernel.all.syscall @0..9 > 5000 count/sec * hinv.ncpu ) -> shell 5 min "xwsh -e 'top'"; The following rules are evaluated once every 20 seconds: delta = 20 sec; If any disk is performing more than 60 I/Os per second, then print a message identifying the busy disk to standard output and launch dkvis(1): some_inst ( disk.dev.total > 60 count/sec ) -> print "disk %i busy " & shell 5 min "dkvis"; Refine the preceding rule to apply only between the hours of 9am and 5pm, and to require 3 of 4 consecutive samples to exceed the threshold before executing the action: $hour >= 9 && $hour <= 17 && some_inst ( 75 %_sample ( disk.dev.total @0..3 > 60 count/sec ) ) -> print "disk %i busy "; The following rules are evaluated once every 10 minutes: delta = 10 min; If either the / or the /usr filesystem is more than 95% full, display an alarm popup, but not if it has already been displayed during the last 4 hours: filesys.free #'/dev/root' / filesys.capacity #'/dev/root' < 0.05 -> alarm 4 hour "root filesystem (almost) full"; filesys.free #'/dev/usr' / filesys.capacity #'/dev/usr' < 0.05 -> alarm 4 hour "/usr filesystem (almost) full"; The following rule requires a machine that supports the PCP environment metrics. If the machine environment temperature rises more than 2 degrees over a 10 minute interval, write an entry in the system log: environ.temp @0 - environ.temp @1 > 2 -> alarm "temperature rising fast" & syslog "machine room temperature rise alarm"; And last, something interesting if you have performance problems with your Oracle database: db = "oracle.ptg1"; host = ":moomba.melbourne.sgi.com"; lru = "#'cache buffers lru chain'"; gets = "$db.latch.gets $host $lru"; total = "$db.latch.gets $host $lru + $db.latch.misses $host $lru + $db.latch.immisses $host $lru"; $total > 100 && $gets / $total < 0.2 -> alarm "high lru latch contention"; QUICK START The pmie specification language is powerful and large. To expedite rapid development of pmie rules, the pmieconf(1) tool provides a facility for generating a pmie configuration file from a set of generalized pmie rules. The supplied set of rules covers a wide range of performance scenarios. The pmrules(1) tool provides a GUI-based facility for generating pmie rules from parametrized templates. The supplied templates cover a wide range of performance scenarios. The development efforts of the PCP engineering team are focused on pmieconf rather than pmrules, and thus pmieconf is the recommended tool for quickly deploying useful pmie rules. The Performance Co-Pilot User's and Administrator's Guide provides a detailed tutorial-style chapter covering pmie. The PCP Tutorial from the pcp.man.tutorial subsystem includes a pmie tutorial. Access the URLs file:$PCP_DOC_DIR/Tutorial/pmie.html and file:$PCP_DOC_DIR/Tutorial/lab-pmie.html from your web browser. EXPRESSION SYNTAX This description is terse and informal. For a more comprehensive description see the Performance Co-Pilot User's and Administrator's Guide. A pmie specification is a sequence of semicolon terminated expressions. Basic operators are modeled on the arithmetic, relational and Boolean operators of the C programming language. Precedence rules are as expected, although the use of parentheses is encouraged to enhance readability and remove ambiguity. Operands are performance metric names (see pmns(4)) and the normal literal constants. Operands involving performance metrics may produce sets of values, as a result of enumeration in the dimensions of hosts, instances and time. Special qualifiers may appear after a performance metric name to define the enumeration in each dimension. For example, kernel.percpu.cpu.user :foo :bar #cpu0 @0..2 defines 6 values corresponding to the time spent executing in user mode on CPU 0 on the hosts ``foo'' and ``bar'' over the last 3 consecutive samples. The default interpretation in the absence of : (host), # (instance) and @ (time) qualifiers is all instances at the most recent sample time for the default source of PCP performance metrics. Host and instance names that do not follow the rules for variables in programming languages, ie. alphabetic optionally followed by alphanumerics, should be enclosed in single quotes. Expression evaluation follows the law of ``least surprises''. Where performance metrics have the semantics of a counter, pmie will automatically convert to a rate based upon consecutive samples and the time interval between these samples. All expressions are evaluated in double precision, and where appropriate, automatically scaled into canonical units of ``bytes'', ``seconds'' and ``counts''. A rule is a special form of expression that specifies a condition or logical expression, a special operator (->) and actions to be performed when the condition is found to be true. The following table summarizes the basic pmie operators: ______________________________________________________________ Operators Explanation ______________________________________________________________ + - * / Arithmetic < <= == >= > != Relational (value comparison) ! && || Boolean -> Rule rising Boolean, false to true transition falling Boolean, true to false transition rate Explicit rate conversion (rarely required) ______________________________________________________________ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Aggregate operators may be used to aggregate or summarize along one dimension of a set-valued expression. The following aggregate operators map from a logical expression to a logical expression of lower dimension. ___________________________________________________________________ Operators Type Explanation ___________________________________________________________________ some_inst some_host some_sample Trueifatleastoneset memberistrueinthe associateddimension Existential ___________________________________________________________________ all_inst all_host all_sample Trueifallsetmembers aretrueinthe associateddimension Universal ___________________________________________________________________ N%_inst N%_host N%_sample TrueifatleastN percentofsetmembers aretrueinthe associated dimension Percentile ___________________________________________________________________ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The following instantial operators may be used filter or limit a a set- valued logical expression, based on regular expression matching of instance names. The logical expression must be a set involving the dimension of instances, and the regular expression is of the form used by egrep(1) or the Extended Regular Expressions of regcomp(3G). _________________________________________________________ Operators Explanation _________________________________________________________ For each value of the logical expression that is ``true'', the result is ``true'' if the associated instance name matches the regular expression. Otherwise the result is ``false''. match_inst _________________________________________________________ For each value of the logical expression that is ``true'', the result is ``true'' if the associated instance name does not match the regular expression. Otherwise the result is ``false''. nomatch_inst _________________________________________________________ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For example, the expression below will be ``true'' for disks attached to controllers 2 or 3 performing more than 20 operations per second: match_inst "^dks[23]d" disk.dev.total > 20; The following aggregate operators map from an arithmetic expression to an arithmetic expression of lower dimension. _________________________________________________________________ Operators Type Explanation _________________________________________________________________ min_inst min_host min_sample Minimumvalueacrossall setmembersinthe associateddimension Extrema _________________________________________________________________ max_inst max_host max_sample Maximumvalueacrossall setmembersinthe associateddimension Extrema _________________________________________________________________ sum_inst sum_host sum_sample Sumofvaluesacrossall setmembersinthe associateddimension Aggregate _________________________________________________________________ avg_inst avg_host avg_sample Averagevalueacrossall setmembersinthe associateddimension Aggregate _________________________________________________________________ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The aggregate operators count_inst, count_host and count_sample map from a logical expression to an arithmetic expression of lower dimension by counting the number of set members for which the expression is true in the associated dimension. For action rules, the following actions are defined: ____________________________________________________ Operators Explanation ____________________________________________________ print Display on standard output shell Execute with sh(1) alarm Raise a visible alarm with xconfirm(1) syslog Append to /var/adm/SYSLOG ____________________________________________________ | | | | | | | | | | | | | | | | | | | | | Multiple actions may be separated by the & and | operators to specify respectively sequential execution (both actions are executed) and alternate execution (the second action will only be executed if the execution of the first action returns a non-zero error status. Arguments to actions are an optional suppression time, and then one or more expressions (a string is an expression in this context). Strings appearing as arguments to an action may include the following special selectors that will be replaced at the time the action is executed. %h Host(s) that make the left-most top-level expression in the condition true. %i Instance(s) that make the left-most top-level expression in the condition true. %v Values(s) from the left-most top-level expression in the condition subject to the host and instance assignments that make the condition true. Note that expansion of the special selectors is done by repeating the whole argument once for each unique binding to any of the qualifying special selectors. For example if a rule were true for the host mumble with instances grunt and snort, and for host fumble the instance puff makes the rule true, then the action ... -> shell myscript "Warning: %h-%i busy "; will execute myscript with the argument string "Warning: mumble-grunt busy Warning: mumble-snort busy Warning: fumble-puff busy". By comparison, if the action ... -> shell myscript "'Warning! busy:" " %i@%h" "'"; were executed under the same circumstances, then myscript would be executed with the argument string '"Warning! busy: grunt@mumble snort@mumble puff@fumble"'. The semantics of the expansion of the special selectors leads to a common usage, where one argument is a constant (contains no special selectors) the second argument contains the desired special selectors with minimal separator characters, and an optional third argument provides a constant postscript (e.g. to terminate any argument quoting from the first argument). If necessary post-processing (eg. in myscript) can provide the necessary enumeration over each unique expansion of the string containing just the special selectors. For complex conditions, the bindings to these selectors is not obvious. It is strongly recommended that pmie be used in the debugging mode (specify the -W command line option in particular) during rule development. SCALE FACTORS Scale factors may be appended to arithmetic expressions and force linear scaling of the value to canonical units. Simple scale factors are constructed from the keywords: nanosecond, nanosec, nsec, microsecond, microsec, usec, millisecond, millisec, msec, second, sec, minute, min, hour, byte, Kbyte, Mbyte, Gbyte, Tbyte, count, Kcount, Mcount, Gcount and Tcount, and the operator /, for example ``Kbytes / hour''. MACROS Macros are defined using expressions of the form: name = constexpr; Where name follows the normal rules for variables in programming languages, ie. alphabetic optionally followed by alphanumerics. constexpr must be a constant expression, either a string (enclosed in double quotes) or an arithmetic expression optionally followed by a scale factor. Macros are expanded when their name, prefixed by a dollar ($) appears in an expression, and macros may be nested within a constexpr string. The following reserved macro names are understood. minute Current minute of the hour. hour Current hour of the day, in the range 0 to 23. day Current day of the month, in the range 1 to 31. month Current month of the year, in the range 0 (January) to 11 (December). year Current year. day_of_week Current day of the week, in the range 0 (Sunday) to 6 (Saturday). delta Sample interval in effect for this expression. Dates and times are presented in the reporting time zone (see description of -Z and -z command line options above). AUTOMATIC RESTART It is often useful for pmie processes to be started and stopped when the local host is booted or shutdown, or when they have been detected as no longer running (when they have unexpectedly exited for some reason). Refer to pmie_check(1) for details on automating this process. FILES $PCP_DEMOS_DIR/pmie/* annotated example rules $PCP_VAR_DIR/pmns/* default PMNS specification files $PCP_TMP_DIR/pmie pmie maintains files in this directory to identify the running pmie instances and to export runtime information about each instance - this data forms the basis of the pmcd.pmie performance metrics $PCP_VAR_DIR/config/pmie/control the default set of pmie instances to start at boot time - refer to pmie_check(1) for details $PCP_VAR_DIR/config/pmie/* the predefined alarm action scripts (email, log, popup and syslog), the example action script (sample)and the concurrent action control file (control.master, see also pmrules(1)). /usr/pcp/lib/pmie-common common shell procedures for the predefined alarm action scripts /etc/config/pmie chkconfig(1M) control flag, to control launching of pmie from /etc/init.d/pmie - see also pmie_check(1) BUGS The lexical scanner and parser will attempt to recover after an error in the input expressions. Parsing resumes after skipping input up to the next semi-colon (;), however during this skipping process the scanner is ignorant of comments and strings, so an embedded semi-colon may cause parsing to resume at an unexpected place. This behavior is largely benign, as until the initial syntax error is corrected, pmie will not attempt any expression evaluation. PCP ENVIRONMENT Environment variables with the prefix PCP_ are used to parameterize the file and directory names used by PCP. On each installation, the file /etc/pcp.conf contains the local values for these variables. The $PCP_CONF variable may be used to specify an alternative configuration file, as described in pcp.conf(4). SEE ALSO PCPIntro(1), pmcd(1), pmdumplog(1), pmieconf(1), pmie_check(1), pminfo(1), pmlogger(1), pmval(1), PMAPI(3), pcp.conf(4) and pcp.env(4). TUTORIAL AND USER GUIDE Relevant information is also available from the on-line PCP Tutorial. Provided the pcp.man.tutorial subsystem from the PCP images has been installed, access the URLs file:$PCP_DOC_DIR/Tutorial/pmie.html and file:$PCP_DOC_DIR/Tutorial/lab-pmie.html from your web browser. For a more complete description of the pmie language, refer to the Performance Co-Pilot Users and Administrators Guide. This is distributed in insight(1) format as part of the pcp.books subsystem, or in HTML format from: http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?\ db=bks&fname=/SGI_Admin/PCP_UAG/ CAVEAT The example rules in $PCP_DEMOS_DIR/pmie are part of the pcp.sw.demo subsystem and are not installed by default, they must be explicitly installed using inst(1) or swmgr(1). Page 11