If we look at the command line tools for processing files, we see that there is a separate tool for each task. For example, we do not have a tool called headortail that can take either the first or last lines of a file. We have separate tools for each of the tasks.

While this might seem like extra work, it actually enables the user to be more precise about what he/she is doing, as well as be more expressive. It also improves the readability and comprehension of commands; the user doesn't have to read lots of command line arguments to figure out what's going on.

The command line tools we look at also inter-operate really well because they compliment each other. For example, consider some of the pipelines you wrote in lab and how you can use cut to get a field from structure data, then you can use grep to isolate some set of those fields, and finally you can use wc to count how many fields remain.

Finally, the ability to handle text streams is the basis of the pipeline and what enables small and simple Unix commands to be "glued" together to form more complex and interesting Unix operations.

This philosophy leads to the development of well formed Unix command line tools that have the three properties:

1. They can take input from the terminal through a pipeline or by reading an input file provided as an argument
2. They write all their output back to the terminal such that it can be read as input by another command.
3. They do not write error information to standard output in a way that can interfere with a pipeline.

This process of taking input from the terminal and writing output to the terminal is the notion of handling text streams through the pipeline. In this lecture, we will look at this process in more detail.

Given the description of the standard file descriptors, we can better understand our pipelines with respect to the standard file descriptors.

Head writes to stdout--. .---the stdout of head is the stdin of cat
                       | |
                       v v

   head -3 BAD_FILENAME | cat BeatArmy.txt - GoNavy.txt  

                       \_/
                        |
                A pipe just connects the stdout of 
                one command to the stdin of another

The pipe (|) is a semantic construct for the shell to connect the standard output of one program to the standard input of another program, thus piping the output to input.

The fact that input is connected to output in a pipeline actually necessitates stderr because if an error was to occur along the pipeline, you would not want that error to propagate as input to the next program in the pipeline especially when the pipeline can proceed despite the error. There needs to be a mechanism to report the error to the terminal outside the pipeline, and that mechanism is standard error.

As an example, consider the the case where head is provided a bad file name.

#> head -3 BAD_FILENAME| cat BeatArmy.txt - GoNavy.txt  
head: BAD_FILENAME: No such file or directory          <--- Written to stderr not piped to cat
Go Navy!                                 
Beat Army!

Here, head has an error BAD_FILENAME doesn't exist, so head prints an error message to stderr and does not write anything to stdout, and thus, cat only prints the contents of the two files to stdout. If there was no stderr, then head could only report the error to stdout and thus it would interfere with the pipeline;

head: BAD_FILENAME: No such file or directory        

is not part of the first 3 lines of any file.

In addition to piping the standard file streams, you can also redirect them to a file on the filesystem. The redirect symbols is > and <. Consider a dummy command below:

cmd < input_file > output_file 2> error_file

This would mean that cmd (a fill in for a well formed Unix command) will read input from the file input_file, all output that would normally go to stdout is now written to output_file, and any error messages will be written to error_file. Note that 2 and the > together (2>) indicates to redirect file descriptor 2, which maps to stderr (see above).

You can also use redirects interspersed in a pipeline like below.

cmd < input_file | cmd 2> error_file |  cmd > output_file

However, you cannot mix two redirects for the same standard stream, like so:

cat input_file > output_file | head 

This command will result in nothing being printed to the screen via head and all redirected to output_file. This is because the > and < redirects always take precedence over a pipe, and the last > or < in a sequence takes the most precedence. For example:

cat input_file > out1 > out2 | head

will write the contents of the input file to the out2 file and not to out1.

Output redirects will automatically truncate the file being redirected to. That is, it will essentially erase the file and create a new one. There are situations where, instead, you want to append to the end of the file, such as cumulating log files. You can do such output redirects with >> symbols, double greater-then signs. For example,

cat input_file > out
cat input_file >> out

will produce two copies of the input file concatenated together in the output file, out.

Unix also provides a number of device files for getting information:

• /dev/zero : Provide zero bytes. If you read from /dev/zero you only get zero. For example the following writes 20 zero bytes to a file:

head -c 20 /dev/zero > zero-20-byte-file.dat

• /dev/urandom : Provides random bytes. If you read from /dev/urandom you get a random byte. For example the following writes a random 20 bytes to a file:

head -c 20 /dev/urandom > random-20-byte-file.dat

The files you find in /dev are not really files, but actually devices provided by the Operating System. A device generally connects to some input or output component of the OS. The three devices above (null, zero, and urandom) are special functions of the OS to provide the user with a null space (null), a consistent zero base (zero) , and a source of random entropy (urandom).

If we take a closer look at the /dev directory you see that there is actually quite a lot going on here.

alarm            hidraw0          network_throughput  ram9      tty13  tty35  tty57      ttyS2       vboxusb/
ashmem           hidraw1          null                random    tty14  tty36  tty58      ttyS20      vcs
autofs           hpet             oldmem              rfkill    tty15  tty37  tty59      ttyS21      vcs1
binder           input/           parport0            rtc@      tty16  tty38  tty6       ttyS22      vcs2
block/           kmsg             port                rtc0      tty17  tty39  tty60      ttyS23      vcs3
bsg/             kvm              ppp                 sda       tty18  tty4   tty61      ttyS24      vcs4
btrfs-control    lirc0            psaux               sda1      tty19  tty40  tty62      ttyS25      vcs5
bus/             log=             ptmx                sda2      tty2   tty41  tty63      ttyS26      vcs6
cdrom@           loop0            pts/                sg0       tty20  tty42  tty7       ttyS27      vcsa
cdrw@            loop1            ram0                sg1       tty21  tty43  tty8       ttyS28      vcsa1
char/            loop2            ram1                shm@      tty22  tty44  tty9       ttyS29      vcsa2
console          loop3            ram10               snapshot  tty23  tty45  ttyprintk  ttyS3       vcsa3
core@            loop4            ram11               snd/      tty24  tty46  ttyS0      ttyS30      vcsa4
cpu/             loop5            ram12               sr0       tty25  tty47  ttyS1      ttyS31      vcsa5
cpu_dma_latency  loop6            ram13               stderr@   tty26  tty48  ttyS10     ttyS4       vcsa6
disk/            loop7            ram14               stdin@    tty27  tty49  ttyS11     ttyS5       vga_arbiter
dri/             loop-control     ram15               stdout@   tty28  tty5   ttyS12     ttyS6       vhost-net
dvd@             lp0              ram2                tpm0      tty29  tty50  ttyS13     ttyS7       watchdog
dvdrw@           mapper/          ram3                tty       tty3   tty51  ttyS14     ttyS8       watchdog0
ecryptfs         mcelog           ram4                tty0      tty30  tty52  ttyS15     ttyS9       zero
fb0              mei              ram5                tty1      tty31  tty53  ttyS16     uinput
fd@              mem              ram6                tty10     tty32  tty54  ttyS17     urandom
full             net/             ram7                tty11     tty33  tty55  ttyS18     vboxdrv
fuse             network_latency  ram8                tty12     tty34  tty56  ttyS19     vboxnetctl

You will learn more about /dev's in your OS class, but for now you should know that this is a way to connect the user-space with the kernel-space through the file system. It is incredibly powerful and useful, beyond just sending stuff to /dev/null.

What each of the files are is the input to some OS process. For example, each of the tty information is a terminal that is open on the computer. The ram refer to what is currently in the computer's memory. The dvd and cdrom, that is the file that you write and read to when connecting with the cd/dvd-rom. And the items under disk, that a way to get to the disk drives.