Chapter 8 Intermediate Shell Tools II

1.1 Sorting Your Output

Use the sort utility. You can sort one or more files by putting the file names on the command line.

[maxwell@MaxwellDBA sample]$ sort a.txt b.txt c.txt d.txt
It can be handly to have your output in sorted order, and handier still not have to add sorting code to every program you write.
Use the sort utility
With no filenames on the command,sort will read from standard input you can pipe the output from a previous command into sort.
You can sort one or more files by putting the file names on the command line
[maxwell@MaxwellDBA sample]$ ls -ltr
total 16
-rw-rw-r-- 1 maxwell maxwell  21 Jul 28 16:09 a.txt
-rw-rw-r-- 1 maxwell maxwell  77 Jul 28 16:10 b.txt
-rw-rw-r-- 1 maxwell maxwell 127 Jul 28 16:10 c.txt
-rw-rw-r-- 1 maxwell maxwell 129 Jul 28 16:12 d.txt
[maxwell@MaxwellDBA sample]$ 

With no filenames on the command, sort will read from standard input so you can pipe the output from a previous command into sort.

[maxwell@MaxwellDBA sample]$ somecommands | sort

• sort -r 

to reverse the order of the sort.

• sort -f

to "fold" lower- and uppercase characters together.

• sort --ignore-case

[maxwell@MaxwellDBA sample]$ sort -r a.txt
Use the sort utility
[maxwell@MaxwellDBA sample]$ sort -r a.txt b.txt c.txt d.txt
You can sort one or more files by putting the file names on the command line
With no filenames on the command,sort will read from standard input you can pipe the output from a previous command into sort.
Use the sort utility
It can be handly to have your output in sorted order, and handier still not have to add sorting code to every program you write.
[maxwell@MaxwellDBA sample]$ sort -f a.txt b.txt c.txt d.txt
It can be handly to have your output in sorted order, and handier still not have to add sorting code to every program you write.
Use the sort utility
With no filenames on the command,sort will read from standard input you can pipe the output from a previous command into sort.
You can sort one or more files by putting the file names on the command line
[maxwell@MaxwellDBA sample]$ 
[maxwell@MaxwellDBA sample]$ sort --ignore-case a.txt b.txt c.txt d.txt
It can be handly to have your output in sorted order, and handier still not have to add sorting code to every program you write.
Use the sort utility
With no filenames on the command,sort will read from standard input you can pipe the output from a previous command into sort.
You can sort one or more files by putting the file names on the command line
[maxwell@MaxwellDBA sample]$ 

1.2 Sorting Numbers

[maxwell@MaxwellDBA sample]$ sort somedata
2
200
21
250
[maxwell@MaxwellDBA sample]$ sort -n somedata
2
21
200
250
[maxwell@MaxwellDBA sample]$

cut -d':' -f7 /etc/passwd isolates the shell from the /etc/passwd file. Then we have to do an initial sort so that uniq will work. uniq -c counts consecutive, duplicate lines, which is why we need the pre-sort. Then sort -rn gives us a reverse, numerical sort, with the most popular shell at the top.

[maxwell@MaxwellDBA sample]$ cut -d':' -f7 /etc/passwd | sort | uniq -c | sort -rn
     27 /sbin/nologin
      4 /bin/bash
      1 /sbin/shutdown
      1 /sbin/halt
      1 /bin/sync
[maxwell@MaxwellDBA sample]$ cut -d':' -f7 /etc/passwd | sort -u
/bin/bash
/bin/sync
/sbin/halt
/sbin/nologin
/sbin/shutdown
[maxwell@MaxwellDBA sample]$ 

1.3 Sorting IP Addresses

To sort the entire address as you would expect(POSIX syntax):

[maxwell@MaxwellDBA sample]$ sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n ipaddr.list
10.0.0.2
10.0.0.5
10.0.0.20
118.178.233.224
192.168.0.1
192.168.0.2
192.168.0.4
192.168.0.12
[maxwell@MaxwellDBA sample]$ 

The -t option indicates the character to use as a separator between fields (in our case, a period) so that we can also specify which fields to sort first.

In this case, -k 1,1n means “start sorting at the beginning of field one (1) and (,) stop sorting at the end of field one (1) and do a numerical sort (n).

[maxwell@MaxwellDBA sample]$ sort -t. -k4n ipaddr.list
192.168.0.1
10.0.0.2
192.168.0.2
192.168.0.4
10.0.0.5
192.168.0.12
10.0.0.20
118.178.233.224
[maxwell@MaxwellDBA sample]$

1.4 Cutting Out Parts of Your Output

Use the cut command with the -c option to take particular columns: Note that our example 'ps' command only works with certain systems

[maxwell@MaxwellDBA sample]$ ps -l | cut -c12-15
   P
2004
2005
2005
[maxwell@MaxwellDBA sample]$ ps -elf | cut -c58-
IME TTY          TIME CMD
n28 ?        00:00:45 /usr/lib/systemd/systemd --system --deserialize 21
n28 ?        00:00:01 [kthreadd]
n28 ?        00:00:00 [rcu_gp]
n28 ?        00:00:00 [rcu_par_gp]
n28 ?        00:00:00 [kworker/0:0H-kblockd]
n28 ?        00:00:00 [mm_percpu_wq]
n28 ?        00:00:07 [ksoftirqd/0]
n28 ?        00:14:11 [rcu_sched]
n28 ?        00:00:00 [migration/0]
n28 ?        00:00:01 [watchdog/0]
n28 ?        00:00:00 [cpuhp/0]
n28 ?        00:00:00 [cpuhp/1]
n28 ?        00:00:01 [watchdog/1]

Using cut to print out fields rather than columns is possible, though more limited than other choices such as awk. The default delimiter between fields is the Tab character, but you can specify a different delimiter with the -d option. Here is an example of a cut command using fields:

[maxwell@MaxwellDBA sample]$ cut -d'#' -f2 < ipaddr.list
10.0.0.2
118.178.233.224
192.168.0.2
192.168.0.1
192.168.0.4
10.0.0.5
192.168.0.12
10.0.0.20

[maxwell@MaxwellDBA sample]$ cut -d'[' -f2 delimited_data | cut -d']' -f1
l1
l2
l3
[maxwell@MaxwellDBA sample]$ cat delimited_data
Line [l1].
Line [l2].
Line [l3].
[maxwell@MaxwellDBA sample]$ 

1.5 Removing Duplicate Lines

somesequence | sort -u

 If you aren't running sort, just pipe the output into uniq.

$ somesequence > myfile
$ uniq myfile

1.6 Compressing Files

$ tar cf tarball_name.tar directory_of_files
$ gzip tarball_name.tar

[maxwell@MaxwellDBA sample]$ ls -ltr
total 28
-rw-rw-r-- 1 maxwell maxwell  21 Jul 28 16:09 a.txt
-rw-rw-r-- 1 maxwell maxwell  77 Jul 28 16:10 b.txt
-rw-rw-r-- 1 maxwell maxwell 127 Jul 28 16:10 c.txt
-rw-rw-r-- 1 maxwell maxwell 129 Jul 28 16:12 d.txt
-rw-rw-r-- 1 maxwell maxwell  13 Jul 28 16:22 somedata
-rw-rw-r-- 1 maxwell maxwell  93 Jul 28 16:32 ipaddr.list
-rw-rw-r-- 1 maxwell maxwell  33 Jul 28 16:45 delimited_data
[maxwell@MaxwellDBA sample]$ tar cf tarball_somedata.tar somedata
[maxwell@MaxwellDBA sample]$ ls -ltr
total 40
-rw-rw-r-- 1 maxwell maxwell    21 Jul 28 16:09 a.txt
-rw-rw-r-- 1 maxwell maxwell    77 Jul 28 16:10 b.txt
-rw-rw-r-- 1 maxwell maxwell   127 Jul 28 16:10 c.txt
-rw-rw-r-- 1 maxwell maxwell   129 Jul 28 16:12 d.txt
-rw-rw-r-- 1 maxwell maxwell    13 Jul 28 16:22 somedata
-rw-rw-r-- 1 maxwell maxwell    93 Jul 28 16:32 ipaddr.list
-rw-rw-r-- 1 maxwell maxwell    33 Jul 28 16:45 delimited_data
-rw-rw-r-- 1 maxwell maxwell 10240 Jul 29 08:12 tarball_somedata.tar
[maxwell@MaxwellDBA sample]$ gzip tarball_somedata.tar
[maxwell@MaxwellDBA sample]$ ls -ltr
total 32
-rw-rw-r-- 1 maxwell maxwell  21 Jul 28 16:09 a.txt
-rw-rw-r-- 1 maxwell maxwell  77 Jul 28 16:10 b.txt
-rw-rw-r-- 1 maxwell maxwell 127 Jul 28 16:10 c.txt
-rw-rw-r-- 1 maxwell maxwell 129 Jul 28 16:12 d.txt
-rw-rw-r-- 1 maxwell maxwell  13 Jul 28 16:22 somedata
-rw-rw-r-- 1 maxwell maxwell  93 Jul 28 16:32 ipaddr.list
-rw-rw-r-- 1 maxwell maxwell  33 Jul 28 16:45 delimited_data
-rw-rw-r-- 1 maxwell maxwell 157 Jul 29 08:12 tarball_somedata.tar.gz
[maxwell@MaxwellDBA sample]$ 

1.7 Uncompressing Files

 1.8 Checking a tar Archive for Unique Directories

Use an awk script to parse off the directory names from the tar archieve's table of contents, then use sort -u to leave you with just the unique directory names:

tar tf some.tar | awk -F/ '{print $1}' | sort -u

1.9 Translating Characters

Use the tr command to translate one character to another.

[maxwell@MaxwellDBA sample]$ tr ";" ',' after.txt
[maxwell@MaxwellDBA sample]$ ls -ltr
total 40
-rw-rw-r-- 1 maxwell maxwell  21 Jul 28 16:09 a.txt
-rw-rw-r-- 1 maxwell maxwell  77 Jul 28 16:10 b.txt
-rw-rw-r-- 1 maxwell maxwell 127 Jul 28 16:10 c.txt
-rw-rw-r-- 1 maxwell maxwell 129 Jul 28 16:12 d.txt
-rw-rw-r-- 1 maxwell maxwell  13 Jul 28 16:22 somedata
-rw-rw-r-- 1 maxwell maxwell  93 Jul 28 16:32 ipaddr.list
-rw-rw-r-- 1 maxwell maxwell  33 Jul 28 16:45 delimited_data
-rw-rw-r-- 1 maxwell maxwell  45 Jul 29 08:15 tarball_somedata.tar.gz
-rw-rw-r-- 1 maxwell maxwell 342 Jul 30 20:32 before.txt
-rw-rw-r-- 1 maxwell maxwell 342 Jul 30 20:33 after.txt
[maxwell@MaxwellDBA sample]$ cat after.txt
In its simplest form, a tr command replaces occurrences of the first (and only) character of, the first argument with the first (and only) character of the second argument.
In the example solution, we redirected input from the file named before and sent the
output into the file named after and we translated all occurrences of a semicolon,
[maxwell@MaxwellDBA sample]$ cat before.txt
In its simplest form; a tr command replaces occurrences of the first (and only) character of; the first argument with the first (and only) character of the second argument.
In the example solution; we redirected input from the file named before and sent the
output into the file named after and we translated all occurrences of a semicolon;
[maxwell@MaxwellDBA sample]$ 

The tr command can do more that one translation at a time by putting the several characters to be translated in the first argument and their corresponding resultant characters in the second argument.

 tr ';:.!?' ',' commas.all

1.10 Converting Uppercase to Lowercase

You can translate all uppercase characters(A-Z) to lowercase(a-z) using the tr command and specifying a range of characters.as in:

tr 'A-Z' 'a-z' af.ter

There is also special syntax in tr for specifying this sort of range for upper- and lowercase conversions:

tr '[:upper:]' '[:lower:]' af.ter

Here’s a very simplistic encoding of a text message using a simple substitution cypher that offsets each character by 13 places (i.e., ROT13). An interesting characteristic of ROT13 is that the same process is used to both encipher and decipher the text:

[maxwell@MaxwellDBA tmp]$ cat /tmp/joke
Q: Why did the chicken cross the road?
A: To get to the other side.
[maxwell@MaxwellDBA tmp]$ tr 'A-Za-z' 'N-ZA-Mn-za-m' < /tmp/joke
D: Jul qvq gur puvpxra pebff gur ebnq?
N: Gb trg gb gur bgure fvqr.
[maxwell@MaxwellDBA tmp]$ tr 'A-Za-z' 'N-ZA-Mn-za-m' < /tmp/joke | tr 'A-Za-z' 'N-ZA-Mn-za-m'
Q: Why did the chicken cross the road?
A: To get to the other side.
[maxwell@MaxwellDBA tmp]$ 

1.11 Converting DOS Files to Linux Format

Use the -d option on tr to delete the character(s) in the supplied list. For example, to delete all DOS carriage returns (\r), use the command:

tr -d '\r' file.txt

 1.13 Counting Lines,Words,or Characters in a File

[maxwell@MaxwellDBA sample]$ wc before.txt
  3  58 342 before.txt
[maxwell@MaxwellDBA sample]$ # Line only
[maxwell@MaxwellDBA sample]$ wc -l before.txt
3 before.txt
[maxwell@MaxwellDBA sample]$ # Words only
[maxwell@MaxwellDBA sample]$ wc -w before.txt
58 before.txt
[maxwell@MaxwellDBA sample]$ # Characters only
[maxwell@MaxwellDBA sample]$ wc -c before.txt
342 before.txt
[maxwell@MaxwellDBA sample]$ ls -l before.txt
-rw-rw-r-- 1 maxwell maxwell 

 1.14 Rewrapping Paragraphs

Use the fmt command, optionally with a goal and maximum line length:

$ fmt mangled_text
$ fmt 55 60 mangled_text