Piping: A simple means of data transfer between commands and programs

When we were discussing about Data processing, we saw that a data processing machine consists of a number of data processors working in unison (in a series or in parallel) to generate a meaningful output. To see such a data processing in action, we can take a look at the idea of piping in Linux (or Unix). A pipe reminds me of the cylindrical tubes that are used to transfer water or oil from one place to another. So a pipe is something which doesn’t modify the data (or entity) passed through it, rather it is just for transfer. In other words, the input to a pipe is the same as the output.

When we discussed about commands, we saw that there was a time, when the users had to enter the name of a command on the terminal of a computer to get things done. So to list the contents of a directory, the command ‘ls’ is used. To change the current directory, the command ‘cd’ was used. To count the number of words, characters or lines, the command ‘wc’ was used. All these commands were meant for a specific purpose. But what if I want to get another work done, that is not possible with a particular command, should i program a new command. What if I am not a programmer, it will be really difficult to create a new command. So there must exist a simpler option. And here comes the idea of piping. As we saw earlier, pipe is used to transfer an object or simply it gives out what is fed into it. So what if the output of one command is fed as an input to the other command. So we need a mechanism to transfer the contents of one command as an input to the other command. And for this purpose, piping is used in many Linux (Unix) based variants.

So a pipe transfers the output of one command as an input to the other command.

Take for example, I want to know the number of entries in a directory. As we saw earlier, ‘ls’ is used to see the contents of a directory and ‘wc’ is used to count the number of lines. What if we combine these two commands using a pipe, we get our work done. We feed the output of ls as an input to wc using a pipe.

In Linux, the symbol | is used for the purpose of piping

ls | wc -l

The option -l is used to get only the number of lines displayed. Thus we are able to get the number of entries in a directory without creating a new command. The best way to reuse the existing resources to get things done.

The idea of piping helped in a way that the users need not learn a large number of commands, but a very few number of commands. With this limited number of commands, one can get all the things done required to get their daily jobs done.

Compare this to the present day state of softwares and applications. A large number of softwares doing the same thing but with different user interfaces. What’s the result? A regular user is baffled to make a choice of a software ending up reading multiple review sites to make a choice. Sometimes, it is good to look at what people did in the past when they had a crunch of resources. Those lessons are helpful in making good design decisions.