Gathering System Statistics for Linux using SAR
You know how it happens, you are working on a killer bash script and all of the sudden your manager is standing at your cube asking you to look at slow performing server. It happens all of the time, but luckily we have many tools at our disposal and one such tool is Sar.
If you are needing I/O, CPU stats and other data from today or several days back, then sar is the tool to use for Linux. If you are needing graphing and alerting then Sar is not the tool you are wanting, for that you need to look at Nagios or other monitoring tools. But, for an administrator needing data to troubleshoot and gauge what the server is doing, then Sar is the tool. Sar is part of the sysstat package and can be installed in Fedora or RHEL, it i also available on almost all distributions.
$ sudo yum install sysstat
Once installed, it will be enabled by default. Sar will log seven days of statistics by default and compress after 10 days. If you want to log more than that, you can edit /etc/sysconfig/sysstat and change the HISTORY option. You can also set compression for the logs after so many days. History and compression settings come in handy for managing log rotations.
# sysstat-10.0.3 configuration file.
# How long to keep log files (in days)./etc/cron.d/0hourly
# If value is greater than 28, then log files are kept in
# multiple directories, one for each month.
HISTORY=7
# Compress (using gzip or bzip2) sa and sar files older than (in days):
COMPRESSAFTER=10
Once sysstat is configured and enabled, it will collect statistics about your system every ten minutes and store them in a logfile under /var/log/sa via a cron job in /etc/cron.d/sysstat. There is also a daily cron job that will run right before midnight and rotate out the day’s statistics, this is in /etc/cron.d/0hourly. By default, the logfiles will be date-stamped with the current day of the month, so the logs will rotate automatically and overwrite the log from a month ago.
By typing sar with no parameters it will display the current day’s CPU statistics, if you have just installed it you will need to wait sometime for stats to be gathered.
$ sar
07:00:01 AM CPU %user %nice %system %iowait %steal %idle
07:10:01 AM all 0.60 0.02 0.76 3.17 0.00 95.45
07:20:01 AM all 0.03 0.03 0.54 3.90 0.00 95.51
07:30:02 AM all 2.75 0.00 1.72 6.99 0.00 88.53
07:40:01 AM all 0.02 0.00 0.07 0.02 0.00 99.89
07:50:01 AM all 0.02 0.00 0.08 0.02 0.00 99.88
08:00:01 AM all 0.13 0.01 0.18 0.35 0.00 99.34
08:10:01 AM all 0.07 0.00 0.13 0.03 0.00 99.76
08:20:01 AM all 0.01 0.00 0.06 0.02 0.00 99.91
08:30:01 AM all 0.01 0.00 0.06 0.02 0.00 99.91
08:40:01 AM all 0.01 0.00 0.05 0.02 0.00 99.92
08:50:01 AM all 0.01 0.00 0.06 0.03 0.00 99.89
09:00:01 AM all 0.01 0.00 0.05 0.04 0.00 99.90
Using the -r option sar will display RAM statistics.
$ sar -r
07:10:01 AM 402912 1096168 73.12 45204 618888 1143864 25.34 492336 486420
07:20:01 AM 310780 1188300 79.27 92672 628116 1126108 24.95 524772 509792
07:30:02 AM 216740 1282340 85.54 87036 697792 1172404 25.97 494168 631360
07:40:01 AM 237720 1261360 84.14 87068 697820 1134920 25.14 474132 630852
07:50:01 AM 237844 1261236 84.13 87116 697824 1134920 25.14 474140 630896
Using -b option you can get disk I/O data from the past
$ sar -b
07:00:01 AM tps rtps wtps bread/s bwrtn/s
07:10:01 AM 15.73 13.77 1.97 1701.20 22.60
07:20:01 AM 44.80 38.04 6.76 367.22 123.17
07:30:02 AM 26.66 21.74 4.92 1188.71 992.72
07:40:01 AM 0.19 0.01 0.18 0.08 2.49
07:50:01 AM 0.20 0.00 0.20 0.00 2.67
08:00:01 AM 1.63 0.77 0.86 49.86 9.96
08:10:01 AM 0.27 0.02 0.25 0.48 3.15
08:20:01 AM 0.19 0.00 0.19 0.00 2.56
Retrieving Older Data
Using -s (start) and -e (end) options you can retrieve data from past days.If you want to get information a few days in the past. For example if you want data from 12:00 to 12:30 your syntax would be:
$ sar -s 12:00:00 -e 12:30:00
12:00:01 PM CPU %user %nice %system %iowait %steal %idle
12:10:01 PM all 0.05 0.00 0.10 0.02 0.00 99.83
12:20:01 PM all 0.02 0.00 0.06 0.03 0.00 99.89
Average: all 0.03 0.00 0.08 0.02 0.00 99.86
You also can add all of the normal sar options when pulling from past logfiles, so you could run the same command and add the -r argument to get RAM statistics:
$ sar -s 17:00:00 -e 17:30:00 -f /var/log/sysstat/sa01 -r
This just covers a few of the options available with Sar. Be sure to use and add this powerful tool to your bag of tricks in getting your systems to run to peak performance and find those trouble areas.
Note that the default interval for sa1 is one second. If you put a crontab entry to run sa1 each 10 minutes without specifying the interval, you get what the cpu, etc. was for one second out of that 10:
In the crontab entry, you should not be limiting the interval to 1 second. Sar uses the same system resources no matter how long the interval is. It reads kernel values, sleeps, reads the values again and records/prints the difference value. 1 second, 10 seconds, 1200 seconds are the same as far as sar’s resource usage. 99.99% of sar’s usage is sleep, which is what the kernel does anyway when it’s not doing anything. Note below that the first sar sample of only a second showed an average cpu of 3%. The longer samples, averaging over a longer period, show that 6% is probably more of an accurate average, at this time. The web pages I’ve seen so far feed each other with this 1 second sample thing, almost like someone is afraid sar might bog the system down. It won’t. The same two sets of kernel reads happens no matter what the interval is:
time sar 1 1; time sar 10 1; time sar 100 1
Linux 2.6.18-194.el5 (blahblah) 10/07/14
12:04:51 CPU %user %nice %system %iowait %steal %idle
12:04:52 all 3.00 0.00 0.75 0.00 0.00 96.25
Average: all 3.00 0.00 0.75 0.00 0.00 96.25
sar 1 1 0.00s user 0.00s system 0% cpu 1.005 total
Linux 2.6.18-194.el5 (blahblah) 10/07/14
12:04:52 CPU %user %nice %system %iowait %steal %idle
12:05:02 all 6.21 0.00 0.93 0.20 0.00 92.67
Average: all 6.21 0.00 0.93 0.20 0.00 92.67
sar 10 1 0.00s user 0.00s system 0% cpu 10.005 total
Linux 2.6.18-194.el5 (blahblah) 10/07/14
12:05:02 CPU %user %nice %system %iowait %steal %idle
12:06:42 all 6.32 0.00 0.97 0.24 0.00 92.47
Average: all 6.32 0.00 0.97 0.24 0.00 92.47
sar 100 1 0.00s user 0.00s system 0% cpu 1:40.01 total
From the man page example it shows each hour having 3 20 minute samples. This provides accurate averaging and small sa## files. A 1 second interval each 10 minutes is 1/600th of the information available.
EXAMPLES
To create a daily record of sar activities, place the following entry
in your root or adm crontab file:
0 8-18 * * 1-5 /usr/lib/sa/sa1 1200 3 &
Thanks for the nice tutorial.
Can you please share the way to get stats for multiple days in a single file so that it can be imported in ksar for analysis