More text processing

====================

``sort``

--------

Let's say we have a file which lists a few of the stalwarts of the open source community and a few details about them, like their "other" name, their homepage address, and what they are well known for or their claim to fame. 

::

  Richard Stallman%rms%GNU Project

  Eric Raymond%ESR%Jargon File

  Ian Murdock% %Debian

  Lawrence Lessig% %Creative Commons

  Linus Torvalds% %Linux Kernel

  Guido van Rossum%BDFL%Python

  Larry Wall% %Perl

The sort command enables us to do this in a flash! Just running the sort command with the file name as a parameter sorts the lines of the file alphabetically and prints the output on the terminal. 

::

  $ sort stalwarts.txt 

  Eric Raymond%ESR%Jargon File

  Guido van Rossum%BDFL%Python

  Ian Murdock% %Debian

  Larry Wall% %Perl

  Lawrence Lessig% %Creative Commons

  Linus Torvalds% %Linux Kernel

  Richard Stallman%rms%GNU Project

If you wish to sort them reverse alphabetically, you just need to pass the ``-r`` option. Now, you might want to sort the lines, based on each person's claim to fame or their "other" name. What do we do in that case? 

Below is an example that sorts the file based on "other" names. 

::

  $ sort -t % -k 2,2  stalwarts.txt

  Ian Murdock% %Debian

  Larry Wall% %Perl

  Lawrence Lessig% %Creative Commons

  Linus Torvalds% %Linux Kernel

  Guido van Rossum%BDFL%Python

  Eric Raymond%ESR%Jargon File

  Richard Stallman%rms%GNU Project

Sort command assumes white space to be the default delimiter for columns in each line. The ``-t`` option specifies the delimiting character, which is ``%`` in this case. 

The ``-k`` option starts a key at position 2 and ends it at 2, essentially telling the sort command that it should sort based on the 2nd column, which is the other name. ``sort`` also supports conflict resolution using multiple columns for sorting. You can see that the first three lines have nothing in the "other" names column. We could resolve the conflict by sorting based on the project names (the 3rd column). 

::

  $ sort -t % -k 2,2 -k 3,3  stalwarts.txt

  Lawrence Lessig% %Creative Commons

  Ian Murdock% %Debian

  Linus Torvalds% %Linux Kernel

  Larry Wall% %Perl

  Guido van Rossum%BDFL%Python

  Eric Raymond%ESR%Jargon File

  Richard Stallman%rms%GNU Project

``sort`` also has a lot of other options like ignoring case differences, month sort(JAN<FEB<...), merging already sorted files. ``man sort`` would give you a lot of information. 

``uniq``

--------

Suppose we have a list of items, say books, and we wish to obtain a list which names of all the books only once, without any duplicates. We use the ``uniq`` command to achieve this. 

::

  Programming Pearls

  The C Programming Language

  The Mythical Man Month: Essays on Software Engineering 

  Programming Pearls

  The C Programming Language

  Structure and Interpretation of Computer Programs

  Programming Pearls

  Compilers: Principles, Techniques, and Tools

  The C Programming Language

  The Art of UNIX Programming

  Programming Pearls

  The Art of Computer Programming

  Introduction to Algorithms

  The Art of UNIX Programming

  The Pragmatic Programmer: From Journeyman to Master

  Programming Pearls

  Unix Power Tools

  The Art of UNIX Programming

Let us try and get rid of the duplicate lines from this file using the ``uniq`` command. 

::

  $ uniq items.txt 

  Programming Pearls

  The C Programming Language

  The Mythical Man Month: Essays on Software Engineering 

  Programming Pearls

  The C Programming Language

  Structure and Interpretation of Computer Programs

  Programming Pearls

  Compilers: Principles, Techniques, and Tools

  The C Programming Language

  The Art of UNIX Programming

  Programming Pearls

  The Art of Computer Programming

  Introduction to Algorithms

  The Art of UNIX Programming

  The Pragmatic Programmer: From Journeyman to Master

  Programming Pearls

  Unix Power Tools

  The Art of UNIX Programming

Nothing happens! Why? The ``uniq`` command removes duplicate lines only when they are next to each other. So, we get a sorted file from the original file and work with that file, henceforth. 

::

  $ sort items.txt > items-sorted.txt

  $ uniq items-sorted.txt

  Compilers: Principles, Techniques, and Tools

  Introduction to Algorithms

  Programming Pearls

  Structure and Interpretation of Computer Programs

  The Art of Computer Programming

  The Art of UNIX Programming

  The C Programming Language

  The Mythical Man Month: Essays on Software Engineering 

  The Pragmatic Programmer: From Journeyman to Master

  Unix Power Tools

``uniq -u`` command gives the lines which are unique and do not have any duplicates in the file. ``uniq -d`` outputs only those lines which have duplicates. The ``-c`` option displays the number of times each line occurs in the file. 

::

  $ uniq -u items-sorted.txt 

  Compilers: Principles, Techniques, and Tools

  Introduction to Algorithms

  Structure and Interpretation of Computer Programs

  The Art of Computer Programming

  The Mythical Man Month: Essays on Software Engineering 

  The Pragmatic Programmer: From Journeyman to Master

  Unix Power Tools

  $ uniq -dc items-sorted.txt      

  5 Programming Pearls

  3 The Art of UNIX Programming

  3 The C Programming Language

``join``

--------

Now suppose we had the file ``stalwarts1.txt``, which lists the home pages of all the people listed in ``stalwarts.txt``.

::

  Richard Stallman%http://www.stallman.org

  Eric Raymond%http://www.catb.org/~esr/

  Ian Murdock%http://ianmurdock.com/

  Lawrence Lessig%http://lessig.org

  Linus Torvalds%http://torvalds-family.blogspot.com/

  Guido van Rossum%http://www.python.org/~guido/

  Larry Wall%http://www.wall.org/~larry/

It would be nice to have a single file with the information in both the files. To achieve this we use the ``join`` command. 

::

  $ join stalwarts.txt stalwarts1.txt -t %

  Richard Stallman%rms%GNU Project%http://www.stallman.org

  Eric Raymond%ESR%Jargon File%http://www.catb.org/~esr/

  Ian Murdock% %Debian%http://ianmurdock.com/

  Lawrence Lessig% %Creative Commons%http://lessig.org

  Linus Torvalds% %Linux Kernel%http://torvalds-family.blogspot.com/

  Guido van Rossum%BDFL%Python%http://www.python.org/~guido/

  Larry Wall% %Perl%http://www.wall.org/~larry/

The ``join`` command joins the two files, based on the common field present in both the files, which is the name, in this case. 

The ``-t`` option again specifies the delimiting character. Unless that is specified, join assumes that the fields are separated by spaces. 

Note that, for ``join`` to work, the common field should be in the same order in both the files. If this is not so, you could use ``sort``, to sort the files on the common field and then join the files. In the above example, we have the common field to be the first column in both the files. If this is not the case we could use the ``-1`` and ``-2`` options to specify the field to be used for joining the files. 

::

  $ join -2 2 stalwarts.txt stalwarts2.txt -t %

  Richard Stallman%rms%GNU Project%http://www.stallman.org

  Eric Raymond%ESR%Jargon File%http://www.catb.org/~esr/

  Ian Murdock% %Debian%http://ianmurdock.com/

  Lawrence Lessig% %Creative Commons%http://lessig.org

  Linus Torvalds% %Linux Kernel%http://torvalds-family.blogspot.com/

  Guido van Rossum%BDFL%Python%http://www.python.org/~guido/

  Larry Wall% %Perl%http://www.wall.org/~larry/

Generating a word frequency list

================================

Now, let us use the tools we have learnt to use, to generate a word frequency list of a text file. We shall use the free text of Alice in Wonderland.

The basic steps to achieve this task would be -

1. Eliminate the punctuation and spaces from the document. 

2. Generate a list of words.

3. Count the words.

We first use ``grep`` and some elementary ``regex`` to eliminate the non-alpha-characters. 

::

  $ grep "[A-Za-z]*" alice-in-wonderland.txt

This outputs all the lines which has any alphabetic characters on it. This isn't of much use, since we haven't done anything with the code. We only require the alphabetic characters, without any of the other junk. ``man grep`` shows us the ``-o`` option for outputting only the text which matches the regular expression.

::

  $ grep "[A-Za-z]*" -o alice-in-wonderland.txt

Not very surprisingly, we have all the words, spit out in the form of a list! Now that we have a list of words, it is quite simple to count the occurrences of the words. You would've realized that we can make use of ``sort`` and ``uniq`` commands. We pipe the output from the ``grep`` to the ``sort`` and then pipe it's output to ``uniq``.

::

  $ grep "[A-Za-z]*" -o alice-in-wonderland.txt | sort | uniq -c 

Notice that you get the list of all words in the document in the alphabetical order, with it's frequency written next to it. But, you might have observed that Capitalized words and lower case words are being counted as different words. We therefore, replace all the Upper case characters with lower case ones, using the ``tr`` command. 

::

  $ grep  "[A-Za-z]*" -o alice-in-wonderland.txt | tr 'A-Z' 'a-z' | sort | uniq -c 

Now, it would also be nice to have the list ordered in the decreasing order of the frequency of the appearance of the words. We sort the output of the ``uniq`` command with ``-n`` and ``-r`` options, to get the desired output. 

::

  $ grep  "[A-Za-z]*" -o alice-in-wonderland.txt | tr 'A-Z' 'a-z' | sort | uniq -c | sort -nr