All I/O, including files, pipes, and sockets, are handled by the kernel via a mechanism called the file descriptor. A file descriptor is a small unsigned integer, an index into a file-descriptor table maintained by the kernel and used by the kernel to reference open files and I/O streams. Each process inherits its own file-descriptor table from its parent. The first three file descriptors are 0, 1, and 2. File descriptor 0 is standard input (stdin), 1 is standard output (stdout), and 2 is standard error (stderr). When you open a file, the next available descriptor is 3, and it will be assigned to the new file.
One of the first things a UNIX programmer learns is that every running program starts with three files already opened:
|Descriptive Name||Short Name||File Number||Description|
|Standard In||stdin||0||Input from the keyboard|
|Standard Out||stdout||1||Output to the console|
|Standard Error||stderr||2||Error output to the console|
This raises the question of what an open file represents. The value returned by an
open call is termed afile descriptor and is essentially an index into an array of open files kept by the kernel.
In short, the file descriptor is the gateway into the kernel’s abstractions of underlying hardware. An overall view of the abstraction for physical-devices is shown in Figure 1.3, “Abstraction”.
Starting at the lowest level, the operating system requires a programmer to create a device driver to be able to communicate with a hardware device. This device driver is written to an API provided by the kernel just like in Example 1.2, “Abstraction in
include/linux/virtio.h”; the device driver will provide a range of functions which are called by the kernel in response to various requirements. In the simplified example above, we can see the drivers provide a
write function that will be called in response to the analogous operations on the file descriptor. The device driver knows how to convert these generic requests into specific requests or commands for a particular device.
To provide the abstraction to user-space, the kernel provides a file-interface via what is generically termed a device layer. Physical devices on the host are represented by a file in a special file system such as
/dev. In UNIX-like systems, so-called device-nodes have what are termed a major and a minornumber, which allow the kernel to associate particular nodes with their underlying driver. These can be identified via
ls as illustrated in Example 1.3, “Example of major and minor numbers”.
$ ls -l /dev/null /dev/zero /dev/tty crw-rw-rw- 1 root root 1, 3 Aug 26 13:12 /dev/null crw-rw-rw- 1 root root 5, 0 Sep 2 15:06 /dev/tty crw-rw-rw- 1 root root 1, 5 Aug 26 13:12 /dev/zero
This brings us to the file descriptor, which is the handle user-space uses to talk to the underlying device. In a broad sense, what happens when a file is
opened is that the kernel is using the path information to map the file descriptor with something that provides an appropriate
write, etc., API. When this
open is for a device (
/dev/sr0 above), the major and minor number of the opened device node provides the information the kernel needs to find the correct device driver and complete the mapping. The kernel will then know how to route further calls such as
read to the underlying functions provided by the device driver.
A non-device file operates similarly, although there are more layers in between. The abstraction here is the mount point; mounting a file system has the dual purpose of setting up a mapping so the file system knows the underlying device that provides the storage and the kernel knows that files opened under that mount-point should be directed to the file system driver. Like device drivers, file systems are written to a particular generic file system API provided by the kernel.
There are indeed many other layers that complicate the picture in real-life. For example, the kernel will go to great efforts to cache as much data from disks as possible in otherwise-free memory; this provides many speed advantages. It will also try to organise device access in the most efficient ways possible; for example trying to order disk-access to ensure data stored physically close together is retrieved together, even if the requests did not arrive in sequential order. Further, many devices are of a more generic class such as USB or SCSI devices which provide their own abstraction layers to write to. Thus, rather than writing directly to devices, file systems will go through these many layers. Understanding the kernel is to understand how these many APIs interrelate and coexist.
The shell is the gateway to interacting with the operating system. Be it
csh or any of the many other shells, they all fundamentally have only one major task — to allow you to execute programs (you will begin to understand how the shell actually does this when we talk about some of the internals of the operating system later).
But shells do much more than allow you to simply execute a program. They have powerful abilities to redirect files, allow you to execute multiple programs simultaneously and script complete programs. These all come back to the everything is a file idiom.
Often we do not want the standard file descriptors mentioned in the section called “File Descriptors” to point to their default places. For example, you may wish to capture all the output of a program into a file on disk or, alternatively, have it read its commands from a file you prepared earlier. Another useful task might like to pass the output of one program to the input of another. With the operating system, the shell facilitates all this and more.
|Redirect to a file||
||Take all output from standard out and place it into
|Read from a file||<
||Copy all data from the file to the standard input of the program||
||Take everything from standard out of
The implementation of
ls | more is just another example of the power of abstraction. What fundamentally happens here is that instead of associating the file descriptor for the standard-output with some sort of underlying device (such as the console, for output to the terminal), the descriptor is pointed to an in-memory buffer provided by the kernel commonly termed a
pipe. The trick here is that another process can associate its standard input with the other side of this same buffer and effectively consume the output of the other process. This is illustrated in Figure 1.4, “A pipe in action”.
Writes to the pipe are stored by the kernel until a corresponding read from the other side drains the buffer. This is a very powerful concept and is one of the fundamental forms of inter-process communication or IPC in UNIX-like operating systems. The pipe allows more than just a data transfer; it can act as a signaling channel. If a process
reads an empty pipe, it will by default block or be put into hibernation until there is some data available (this is discussed in much greater depth in Chapter 5, The Process). Thus two processes may use a pipe to communicate that some action has been taken just by writing a byte of data; rather than the actual data being important, the mere presence of any data in the pipe can signal a message. Say for example one process requests that another print a file — something that will take some time. The two processes may set up a pipe between themselves where the requesting process does a
read on the empty pipe; being empty, that call blocks and the process does not continue. Once the print is done, the other process can write a message into the pipe, which effectively wakes up the requesting process and signals the work is done.
Allowing processes to pass data between each other like this springs another common UNIX idiom of small tools doing one particular thing. Chaining these small tools gives flexibility that a single monolithic tool often can not.
This will list all open file descriptors:
mohammedrafi@NOC-RAFI:~$ ls -la /proc/$$/fd
dr-x—— 2 mohammedrafi mohammedrafi 0 Jul 6 09:24 .
dr-xr-xr-x 9 mohammedrafi mohammedrafi 0 Jul 6 09:24 ..
lrwx—— 1 mohammedrafi mohammedrafi 64 Jul 6 09:24 0 -> /dev/pts/4
lrwx—— 1 mohammedrafi mohammedrafi 64 Jul 6 09:24 1 -> /dev/pts/4
lrwx—— 1 mohammedrafi mohammedrafi 64 Jul 6 09:24 2 -> /dev/pts/4
lrwx—— 1 mohammedrafi mohammedrafi 64 Jul 6 12:05 255 -> /dev/pts/4
The gcc is a C language compiler program. If an error is encountered during compilation, then it will be redirected to error_file . The > character is used for a success result and 2> is used for error results redirection. We can use error_file for debugging purposes:
$ find . –name “*.sh” > success_file 2> /dev/null
In the preceding example, we are redirecting output or success results to success_file and errors to /dev/null ./dev/null is used to destroy the data, which we do not want to be shown on screen.
$ find . –name “*.sh” &> log.txt
The preceding command will redirect both output and error to log.txt .
$ find . –name “*.sh”> log.tx 2>&1
The preceding command will redirect result to log.txt and send errors to where the output is going, such as log.txt .
$ echo “File needs an argument” 1>&2
The preceding command will send a standard output to the standard error. This will merge the output with the standard error.
The summary of all I/O redirection commands will be as follows:
< sample.txt The command will take input from sample.txt
> sample.txt The success result will be stored in sample.txt
>> sample.txt The successive outputs will be appended to sample.txt
2> sample.txt The error results will be stored in sample.txt
2>> sample.txt The successive error output will be appended to sample.txt
&> sample.txt This will store success and errors, such as in sample.txt
>& sample.txt This will store success and errors, such as in sample.txt
(same as above)
2>&1 This will redirect an error to where output is going
1>&2 This will redirects output to where error is going
>| This overrides no clobber when redirecting the output
<> filename This uses the file as both standard input and output if a device
file (from /dev)
cat xyz > success_file
2> error_file This stores success and failure in different files