Thursday, June 23, 2011

Monitoring User/Application Activity with psacct

One of the big advantages of using psacct on your server is that it provides excellent logging for activities of applications and users.  When you are running scripts one of the important aspects of that script is how much resources it may be using and are there any resource limitations that may exist with the application.  In addition, there may be times when you run a script as a user.  In other words, you create a user with specific rights, maybe even using visudo.  You will likely use this to reduce the security risks of a user who must issue a command with root privileges.
Install Process Accounting
yum install psacct
Start Process Accounting
/etc/init.d/psacct start
Starting process accounting:                               [  OK  ]
Connect Time
The connect time in hours is based on logins and logouts.  The ac command provides a total.
ac
total      256.64
Accounting By Day
The system’s default login accounting file is /var/log/wtmp.
ac -d
Nov  1  total        8.78
Nov  3  total        6.15
Nov  6  total       13.68
Nov  7  total       15.50
Nov  8  total       10.54
Nov  9  total       10.56
Nov 10  total        8.41
Nov 13  total        8.66

—cut—
Today   total       19.33
Time Totals for Users
ac -p
root                                 5.83
mike                               251.29
total      257.13

Commands of Users
You can search out the commands of users with the lastcomm command which prints out the previously executed commands.
Explanation of Output:
Process  Flag    Username    Terminal   Time
ping     S       mike        pts/5      0.00 secs Thu Nov 30 13:39
Flags:
S -     executed as super-user
F -     executed after  but not following exec
D -     terminated with core file
X -     terminated with signal SIGTERM
lastcomm mike
bash                  X     mike     __         0.04 secs Thu Nov 30 12:49
bash                  X     mike     __         0.04 secs Thu Nov 30 12:57
ping                  S     mike     pts/5      0.00 secs Thu Nov 30 13:39
nmap                        mike     pts/4      4.01 secs Thu Nov 30 13:38
ping                  S     mike     pts/5      0.00 secs Thu Nov 30 13:37
ping                  S     mike     pts/5      0.00 secs Thu Nov 30 13:34
ping                  S     mike     pts/5      0.00 secs Thu Nov 30 13:34
bash                  X     mike     __         0.03 secs Thu Nov 30 13:28
bash                   F    mike     pts/6      0.00 secs Thu Nov 30 13:28
consoletype             mike     pts/6      0.00 secs Thu Nov 30 13:28
bash                   F        mike     pts/6      0.00 secs Thu Nov 30 13:28
id                                  mike     pts/6      0.00 secs Thu Nov 30 13:28
grep                            mike     pts/6      0.00 secs Thu Nov 30 13:28
bash               F        mike     pts/6      0.00 secs Thu Nov 30 13:28
grep                        mike     pts/6      0.00 secs Thu Nov 30 13:28
bash               F        mike     pts/6      0.00 secs Thu Nov 30 13:28
egrep                       mike     pts/6      0.00 secs Thu Nov 30 13:28
bash               F        mike     pts/6      0.00 secs Thu Nov 30 13:28
dircolors                   mike     pts/6      0.00 secs Thu Nov 30 13:28
bash               F        mike     pts/6      0.00 secs Thu Nov 30 13:28

Search Logs for Commands
Using the lastcomm command you will be able to view each use of an individual command.
lastcomm grep
grep                    mike     pts/6      0.00 secs Thu Nov 30 13:28
grep                    mike     pts/6      0.00 secs Thu Nov 30 13:28
grep                    mike     pts/5      0.00 secs Thu Nov 30 12:57
grep                    mike     pts/5      0.00 secs Thu Nov 30 12:57

lastcomm grep
grep                    mike     pts/7      0.00 secs Wed Mar 14 06:07
grep                    mike     pts/7      0.00 secs Wed Mar 14 06:07
grep                    root     pts/7      0.00 secs Wed Mar 14 05:39
grep                    root     pts/7      0.00 secs Wed Mar 14 05:39
grep                    root     pts/2      0.00 secs Wed Mar 14 05:16

Print Summary
The sa command will print a summary of commands that were executed.  It will also condense the information into a summary file called savacct which contains the number of times that the command was executed.  The useracct file keeps a summary of the commands by user.
Output Fields
cpu   -  sum of system and user time in cpu minutes
re    -  actual time in minutes
k     -  cpu-time averaged core usage, in 1k units
k*sec -  cpu storage integral (kilo-core seconds)
u     -  user cpu time in cpu minutes
s     -  system time in cpu minutes
/usr/sbin/sa
Print User Information
Use the -u option to provide information on individual users.
/usr/sbin/sa -u
/usr/sbin/sa -u
root       0.00 cpu      598k mem accton
root       0.00 cpu     1081k mem initlog
root       0.00 cpu      920k mem initlog
root       0.00 cpu     1172k mem touch
root       0.00 cpu     1402k mem psacct
mike       0.01 cpu     7282k mem kdeinit          *
mike       0.00 cpu     6232k mem gnome-panel      *
mike       0.02 cpu     4848k mem gnome-terminal

Display Number of Processes
An increase in these fields indicates a problem.  This prints the number of processes and the number of CPU minutes.  If these numbers continue to increase it is time to look into what is happening.
/usr/sbin/sa -m
195         220.31re           0.09cp     2220k
mike                                  65          198.37re           0.08cp     2135k
root                                  88          21.86re           0.00cp     1084k
postgres                              40          0.09re           0.00cp     4879k
smmsp                                 2           0.00re           0.00cp     1827k

Display All Names
This option will show each of the programs on your server so you may evaluate, real time, memory usage and which programs are running.

sa -a
221      83.36re       0.01cp     1414k
1       0.01re       0.00cp     1471k   rpmq
7       0.33re       0.00cp     2465k   sendmail*
1      40.78re       0.00cp     1844k   sshd
37       0.00re       0.00cp      964k   bash*
32       0.00re       0.00cp      604k   tmpwatch
27       0.00re       0.00cp     4984k   postmaster*
26       0.00re       0.00cp     1116k   df
15       0.00re       0.00cp      959k   id
11       0.00re       0.00cp      709k   egrep
8       0.00re       0.00cp      636k   sa
7       0.00re       0.00cp      817k   grep
6       0.00re       0.00cp      562k   ac
5       0.01re       0.00cp      789k   awk
3       0.41re       0.00cp     1219k   crond*
3       0.40re       0.00cp      674k   run-parts
3       0.00re       0.00cp      774k   dircolors
3       0.00re       0.00cp      673k   consoletype
2      40.98re       0.00cp     1344k   bash
2       0.14re       0.00cp     1628k   sshd*
2       0.00re       0.00cp      914k   logrotate

Sort by Percentage
This will provide the highest percentage users first.

sa -c
232  100.00%      83.36re  100.00%       0.01cp  100.00%     1482k
14    6.03%      41.09re   49.29%       0.01cp   58.82%      957k   ***other*
7    3.02%       0.33re    0.39%       0.00cp   41.18%     2465k   sendmail*
37   15.95%       0.00re    0.00%       0.00cp    0.00%      964k   bash*
32   13.79%       0.00re    0.00%       0.00cp    0.00%     4984k   postmaster*
32   13.79%       0.00re    0.00%       0.00cp    0.00%      604k   tmpwatch
31   13.36%       0.00re    0.00%       0.00cp    0.00%     1122k   df
15    6.47%       0.00re    0.00%       0.00cp    0.00%      959k   id
11    4.74%       0.00re    0.00%       0.00cp    0.00%      709k   egrep

No comments:

Post a Comment