Unveiling the Power of 6 Linux Troubleshooting Commands
![]() |
Unveiling the Power of 6 Linux Troubleshooting Commands |
Unveiling the Power of 6 Linux Troubleshooting Commands - This Tutorial introduces some tools that go beyond df -h and du -sh. Tools that will enable you to troubleshoot a variety of issues with file systems and storage.
The tutorial includes examples demonstrating the practical application of each command, such as listing open files, finding processes using specific file systems, terminating processes with fuser, using chroot for troubleshooting, and monitoring I/O activities with iostat, iotop, and vmstat.
6 Linux Troubleshooting Commands
Explore six crucial Linux troubleshooting commands beyond df -h and du -sh. From lsof to vmstat, discover tools for effective system diagnostics and problem resolution.
Each command is explained with practical examples, empowering you to enhance your Linux troubleshooting skills and ensure seamless system maintenance.
Dive into this comprehensive guide to elevate your understanding of critical commands and optimize the performance of your Linux environment.
1. lsof
List open files with lsof.
When invoked without options, lsof will list all open files. You can see the command (init in this case), its PID (1) and the user (root) has openend the root directory and /sbin/init. The FD (file descriptor) columns shows that / is both the root directory (rtd) and current working directory (cwd) for the /sbin/init command. The FD column displays rtd for root directory, cwd for current directory and txt for text (both including data and code).
root@debian7:~# lsof | head -4 COMMAND PID TID USER FD TYPE DEVICE SIZE/OFF NODE NAME init 1 root cwd DIR 254,0 4096 2 / init 1 root rtd DIR 254,0 4096 2 / init 1 root txt REG 254,0 36992 130856 /sbin/
Other options in the FD column besides w for writing, are r for reading and u for both reading and writing. You can look at open files for a process id by typing lsof -p PID. For init this would look like this:
lsof -p 1
The screenshot below shows basic use of lsof to prove that vi keeps a .swp file open (even when stopped in background) on our freshly mounted file system.
[root@RHEL65 ~]# df -h | grep sdb /dev/sdb1 541M 17M 497M 4% /srv/project33 [root@RHEL65 ~]# vi /srv/project33/busyfile.txt [1]+ Stopped vi /srv/project33/busyfile.txt [root@RHEL65 ~]# lsof /srv/* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME vi 3243 root 3u REG 8,17 4096 12 /srv/project33/.busyfile.txt.swp
Here we see that rsyslog has a couple of log files open for writing (the FD column).
root@debian7:~# lsof /var/log/* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 2013 root 1w REG 254,0 454297 1308187 /var/log/syslog rsyslogd 2013 root 2w REG 254,0 419328 1308189 /var/log/kern.log rsyslogd 2013 root 5w REG 254,0 116725 1308200 /var/log/debug rsyslogd 2013 root 6w REG 254,0 309847 1308201 /var/log/messages rsyslogd 2013 root 7w REG 254,0 17591 1308188 /var/log/daemon.log rsyslogd 2013 root 8w REG 254,0 101768 1308186 /var/log/auth.log
You can specify a specific user with lsof -u. This example shows the current working directory for a couple of command line programs.
[paul@RHEL65 ~]$ lsof -u paul | grep home bash 3302 paul cwd DIR 253,0 4096 788024 /home/paul lsof 3329 paul cwd DIR 253,0 4096 788024 /home/paul grep 3330 paul cwd DIR 253,0 4096 788024 /home/paul lsof 3331 paul cwd DIR 253,0 4096 788024 /home/paul
The -u switch of lsof also supports the ^ character meaning 'not'. To see all open files, but not those open by root:
lsof -u^root
2. fuser
The fuser command will display the 'user' of a file system.
In this example we still have a vi process in background and we use fuser to find the process id of the process using this file system.
[root@RHEL65 ~]# jobs [1]+ Stopped vi /srv/project33/busyfile.txt [root@RHEL65 ~]# fuser -m /srv/project33/ /srv/project33/: 3243
Adding the -u switch will also display the user name.
[root@RHEL65 ~]# fuser -m -u /srv/project33/ /srv/project33/: 3243(root)
You can quickly kill all processes that are using a specific file (or directory) with the -k switch.
[root@RHEL65 ~]# fuser -m -k -u /srv/project33/ /srv/project33/: 3243(root) [1]+ Killed vi /srv/project33/busyfile.txt [root@RHEL65 ~]# fuser -m -u /srv/project33/ [root@RHEL65 ~]#
This example shows all processes that are using the current directory (bash and vi in this case).
root@debian7:~/test42# vi file42 [1]+ Stopped vi file42 root@debian7:~/test42# fuser -v . USER PID ACCESS COMMAND /root/test42: root 2909 ..c.. bash root 3113 ..c.. vi
This example shows that the vi command actually accesses /usr/bin/vim.basic as an executable file.
root@debian7:~/test42# fuser -v $(which vi) USER PID ACCESS COMMAND /usr/bin/vim.basic: root 3113 ...e. vi
The last example shows how to find the process that is accessing a specific file.
[root@RHEL65 ~]# vi /srv/project33/busyfile.txt [1]+ Stopped vi /srv/project33/busyfile.txt [root@RHEL65 ~]# fuser -v -m /srv/project33/busyfile.txt USER PID ACCESS COMMAND /srv/project33/busyfile.txt: root 13938 F.... vi [root@RHEL65 ~]# ps -fp 13938 UID PID PPID C STIME TTY TIME CMD root 13938 3110 0 15:47 pts/0 00:00:00 vi /srv/project33/busyfile.txt
3. chroot
The chroot command creates a shell with an alternate root directory. It effectively hides anything outside of this directory.
In the example below we assume that our system refuses to start (maybe because there is a problem with /etc/fstab or the mounting of the root file system).
We start a live system (booted from cd/dvd/usb) to troubleshoot our server. The live system will not use our main hard disk as root device
root@livecd:~# df -h | grep root rootfs 186M 11M 175M 6% / /dev/loop0 807M 807M 0 100% /lib/live/mount/rootfs/filesystem.squashfs root@livecd:~# mount | grep root /dev/loop0 on /lib/live/mount/rootfs/filesystem.squashfs type squashfs (ro)
We create some test file on the current rootfs.
root@livecd:~# touch /file42 root@livecd:~# mkdir /dir42 root@livecd:~# ls / bin dir42 home lib64 opt run srv usr boot etc initrd.img media proc sbin sys var dev file42 lib mnt root selinux tmp vmlinuz
First we mount the root file system from the disk (which is on lvm so we use /dev/mapper instead of /dev/sda5).
root@livecd:~# mount /dev/mapper/packer--debian--7-root /mnt
We are now ready to chroot into the rootfs on disk.
root@livecd:~# cd /mnt root@livecd:/mnt# chroot /mnt root@livecd:/# ls / bin dev initrd.img lost+found opt run srv usr vmlinuz boot etc lib media proc sbin sys vagrant data home lib64 mnt root selinux tmp var
Our test files (file42 and dir42) are not visible because they are out of the chrooted environment.
Note that the hostname of the chrooted environment is identical to the existing hostname.
To exit the chrooted file system:
root@livecd:/# exit exit root@livecd:~# ls / bin dir42 home lib64 opt run srv usr boot etc initrd.img media proc sbin sys var dev file42 lib mnt root selinux tmp vmlinuz
4. iostat
iostat reports IO statitics every given period of time. It also includes a small cpu usage summary. This example shows iostat running every ten seconds with /dev/sdc and /dev/sde showing a lot of write activity.
[root@RHEL65 ~]# iostat 10 3 Linux 2.6.32-431.el6.x86_64 (RHEL65) 06/16/2014 _x86_64_ (1 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 5.81 0.00 3.15 0.18 0.00 90.85 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 42.08 1204.10 1634.88 1743708 2367530 sdb 1.20 7.69 45.78 11134 66292 sdc 0.92 5.30 45.82 7672 66348 sdd 0.91 5.29 45.78 7656 66292 sde 1.04 6.28 91.49 9100 132496 sdf 0.70 3.40 91.46 4918 132440 sdg 0.69 3.40 91.46 4918 132440 dm-0 191.68 1045.78 1362.30 1514434 1972808 dm-1 49.26 150.54 243.55 218000 352696 avg-cpu: %user %nice %system %iowait %steal %idle 56.11 0.00 16.83 0.10 0.00 26.95 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 257.01 10185.97 76.95 101656 768 sdb 0.00 0.00 0.00 0 0 sdc 3.81 1.60 2953.11 16 29472 sdd 0.00 0.00 0.00 0 0 sde 4.91 1.60 4813.63 16 48040 sdf 0.00 0.00 0.00 0 0 sdg 0.00 0.00 0.00 0 0 dm-0 283.77 10185.97 76.95 101656 768 dm-1 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 67.65 0.00 31.11 0.11 0.00 1.13 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 466.86 26961.09 178.28 238336 1576 sdb 0.00 0.00 0.00 0 0 sdc 31.45 0.90 24997.29 8 220976 sdd 0.00 0.00 0.00 0 0 sde 0.34 0.00 5.43 0 48 sdf 0.00 0.00 0.00 0 0 sdg 0.00 0.00 0.00 0 0 dm-0 503.62 26938.46 178.28 238136 1576 dm-1 2.83 22.62 0.00 200 0 [root@RHEL65 ~]#
Other options are to specify the disks you want to monitor (every 5 seconds here):
iostat sdd sde sdf 5
Or to show statistics per partition:
iostat -p sde -p sdf 5
5. iotop
iotop works like the top command but orders processes by input/output instead of by CPU.
By default iotop will show all processes. This example uses iotop -o to only display processes with actual I/O.
[root@RHEL65 ~]# iotop -o Total DISK READ: 8.63 M/s | Total DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 15000 be/4 root 2.43 M/s 0.00 B/s 0.00 % 14.60 % tar cjf /srv/di... 25000 be/4 root 6.20 M/s 0.00 B/s 0.00 % 6.15 % tar czf /srv/di... 24988 be/4 root 0.00 B/s 7.21 M/s 0.00 % 0.00 % gzip 25003 be/4 root 0.00 B/s 1591.19 K/s 0.00 % 0.00 % gzip 25004 be/4 root 0.00 B/s 193.51 K/s 0.00 % 0.00 % bzip2
Use the -b switch to create a log of iotop output (instead of the default interactive view).
[root@RHEL65 ~]# iotop -bod 10 Total DISK READ: 12.82 M/s | Total DISK WRITE: 5.69 M/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 25153 be/4 root 2.05 M/s 0.00 B/s 0.00 % 7.81 % tar cjf /srv/di... 25152 be/4 root 10.77 M/s 0.00 B/s 0.00 % 2.94 % tar czf /srv/di... 25144 be/4 root 408.54 B/s 0.00 B/s 0.00 % 0.05 % python /usr/sbi... 12516 be/3 root 0.00 B/s 1491.33 K/s 0.00 % 0.04 % [jbd2/sdc1-8] 12522 be/3 root 0.00 B/s 45.48 K/s 0.00 % 0.01 % [jbd2/sde1-8] 25158 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [flush-8:64] 25155 be/4 root 0.00 B/s 493.12 K/s 0.00 % 0.00 % bzip2 25156 be/4 root 0.00 B/s 2.81 M/s 0.00 % 0.00 % gzip 25159 be/4 root 0.00 B/s 528.63 K/s 0.00 % 0.00 % [flush-8:32]
This is an example of iotop to track disk I/O every ten seconds for one user named vagrant (and only one process of this user, but this can be omitted). The -a switch accumulates I/O over time.
[root@RHEL65 ~]# iotop -q -a -u vagrant -b -p 5216 -d 10 -n 10 Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 5216 be/4 vagrant 0.00 B 0.00 B 0.00 % 0.00 % gzip Total DISK READ: 818.22 B/s | Total DISK WRITE: 20.78 M/s 5216 be/4 vagrant 0.00 B 213.89 M 0.00 % 0.00 % gzip Total DISK READ: 2045.95 B/s | Total DISK WRITE: 23.16 M/s 5216 be/4 vagrant 0.00 B 430.70 M 0.00 % 0.00 % gzip Total DISK READ: 1227.50 B/s | Total DISK WRITE: 22.37 M/s 5216 be/4 vagrant 0.00 B 642.02 M 0.00 % 0.00 % gzip Total DISK READ: 818.35 B/s | Total DISK WRITE: 16.44 M/s 5216 be/4 vagrant 0.00 B 834.09 M 0.00 % 0.00 % gzip Total DISK READ: 6.95 M/s | Total DISK WRITE: 8.74 M/s 5216 be/4 vagrant 0.00 B 920.69 M 0.00 % 0.00 % gzip Total DISK READ: 21.71 M/s | Total DISK WRITE: 11.99 M/s
6. vmstat
While vmstat is mainly a memory monitoring tool, it is worth mentioning here for its reporting on summary I/O data for block devices and swap space.
This example shows some disk activity (underneath the -----io---- column), without swapping.
[root@RHEL65 ~]# vmstat 5 10 procs ----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 5420 9092 14020 340876 7 12 235 252 77 100 2 1 98 0 0 2 0 5420 6104 13840 338176 0 0 7401 7812 747 1887 38 12 50 0 0 2 0 5420 10136 13696 336012 0 0 11334 14 1725 4036 76 24 0 0 0 0 0 5420 14160 13404 341552 0 0 10161 9914 1174 1924 67 15 18 0 0 0 0 5420 14300 13420 341564 0 0 0 16 28 18 0 0 100 0 0 0 0 5420 14300 13420 341564 0 0 0 0 22 16 0 0 100 0 0 ... [root@RHEL65 ~]#
You can benefit from vmstat's ability to display memory in kilobytes, megabytes or even kibibytes and mebibytes using -S (followed by k K m or M).
[root@RHEL65 ~]# vmstat -SM 5 10 procs ----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 5 14 11 334 0 0 259 255 79 107 2 1 97 0 0 0 0 5 14 11 334 0 0 0 2 21 18 0 0 100 0 0 0 0 5 15 11 334 0 0 6 0 35 31 0 0 100 0 0 2 0 5 6 11 336 0 0 17100 7814 1378 2945 48 21 31 0 0 2 0 5 6 11 336 0 0 13193 14 1662 3343 78 22 0 0 0 2 0 5 13 11 330 0 0 11656 9781 1419 2642 82 18 0 0 0 2 0 5 9 11 334 0 0 10705 2716 1504 2657 81 19 0 0 0 1 0 5 14 11 336 0 0 6467 3788 765 1384 43 9 48 0 0 0 0 5 14 11 336 0 0 0 13 28 24 0 0 100 0 0 0 0 5 14 11 336 0 0 0 0 20 15 0 0 100 0 0 [root@RHEL65 ~]#
Closing statement
In conclusion, mastering these Linux troubleshooting commands empowers users to diagnose and resolve a variety of system issues efficiently, contributing to the overall stability and performance of a Linux environment.
Post a Comment for "Unveiling the Power of 6 Linux Troubleshooting Commands"
Post a Comment