--- a/ult/session4.rst Tue Sep 07 11:07:31 2010 +0530
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,228 +0,0 @@
-More text processing
-====================
-
-``sort``
---------
-Let's say we have a file which lists a few of the stalwarts of the open source community and a few details about them, like their "other" name, their homepage address, and what they are well known for or their claim to fame.
-
-::
-
- Richard Stallman%rms%GNU Project
- Eric Raymond%ESR%Jargon File
- Ian Murdock% %Debian
- Lawrence Lessig% %Creative Commons
- Linus Torvalds% %Linux Kernel
- Guido van Rossum%BDFL%Python
- Larry Wall% %Perl
-
-
-The sort command enables us to do this in a flash! Just running the sort command with the file name as a parameter sorts the lines of the file alphabetically and prints the output on the terminal.
-::
-
- $ sort stalwarts.txt
- Eric Raymond%ESR%Jargon File
- Guido van Rossum%BDFL%Python
- Ian Murdock% %Debian
- Larry Wall% %Perl
- Lawrence Lessig% %Creative Commons
- Linus Torvalds% %Linux Kernel
- Richard Stallman%rms%GNU Project
-
-If you wish to sort them reverse alphabetically, you just need to pass the ``-r`` option. Now, you might want to sort the lines, based on each person's claim to fame or their "other" name. What do we do in that case?
-
-Below is an example that sorts the file based on "other" names.
-::
-
- $ sort -t % -k 2,2 stalwarts.txt
-
- Ian Murdock% %Debian
- Larry Wall% %Perl
- Lawrence Lessig% %Creative Commons
- Linus Torvalds% %Linux Kernel
- Guido van Rossum%BDFL%Python
- Eric Raymond%ESR%Jargon File
- Richard Stallman%rms%GNU Project
-
-Sort command assumes white space to be the default delimiter for columns in each line. The ``-t`` option specifies the delimiting character, which is ``%`` in this case.
-
-The ``-k`` option starts a key at position 2 and ends it at 2, essentially telling the sort command that it should sort based on the 2nd column, which is the other name. ``sort`` also supports conflict resolution using multiple columns for sorting. You can see that the first three lines have nothing in the "other" names column. We could resolve the conflict by sorting based on the project names (the 3rd column).
-
-::
-
- $ sort -t % -k 2,2 -k 3,3 stalwarts.txt
-
- Lawrence Lessig% %Creative Commons
- Ian Murdock% %Debian
- Linus Torvalds% %Linux Kernel
- Larry Wall% %Perl
- Guido van Rossum%BDFL%Python
- Eric Raymond%ESR%Jargon File
- Richard Stallman%rms%GNU Project
-
-``sort`` also has a lot of other options like ignoring case differences, month sort(JAN<FEB<...), merging already sorted files. ``man sort`` would give you a lot of information.
-
-
-``uniq``
---------
-
-Suppose we have a list of items, say books, and we wish to obtain a list which names of all the books only once, without any duplicates. We use the ``uniq`` command to achieve this.
-
-::
-
- Programming Pearls
- The C Programming Language
- The Mythical Man Month: Essays on Software Engineering
- Programming Pearls
- The C Programming Language
- Structure and Interpretation of Computer Programs
- Programming Pearls
- Compilers: Principles, Techniques, and Tools
- The C Programming Language
- The Art of UNIX Programming
- Programming Pearls
- The Art of Computer Programming
- Introduction to Algorithms
- The Art of UNIX Programming
- The Pragmatic Programmer: From Journeyman to Master
- Programming Pearls
- Unix Power Tools
- The Art of UNIX Programming
-
-Let us try and get rid of the duplicate lines from this file using the ``uniq`` command.
-
-::
-
- $ uniq items.txt
- Programming Pearls
- The C Programming Language
- The Mythical Man Month: Essays on Software Engineering
- Programming Pearls
- The C Programming Language
- Structure and Interpretation of Computer Programs
- Programming Pearls
- Compilers: Principles, Techniques, and Tools
- The C Programming Language
- The Art of UNIX Programming
- Programming Pearls
- The Art of Computer Programming
- Introduction to Algorithms
- The Art of UNIX Programming
- The Pragmatic Programmer: From Journeyman to Master
- Programming Pearls
- Unix Power Tools
- The Art of UNIX Programming
-
-Nothing happens! Why? The ``uniq`` command removes duplicate lines only when they are next to each other. So, we get a sorted file from the original file and work with that file, henceforth.
-
-::
-
- $ sort items.txt > items-sorted.txt
- $ uniq items-sorted.txt
- Compilers: Principles, Techniques, and Tools
- Introduction to Algorithms
- Programming Pearls
- Structure and Interpretation of Computer Programs
- The Art of Computer Programming
- The Art of UNIX Programming
- The C Programming Language
- The Mythical Man Month: Essays on Software Engineering
- The Pragmatic Programmer: From Journeyman to Master
- Unix Power Tools
-
-``uniq -u`` command gives the lines which are unique and do not have any duplicates in the file. ``uniq -d`` outputs only those lines which have duplicates. The ``-c`` option displays the number of times each line occurs in the file.
-::
-
- $ uniq -u items-sorted.txt
- Compilers: Principles, Techniques, and Tools
- Introduction to Algorithms
- Structure and Interpretation of Computer Programs
- The Art of Computer Programming
- The Mythical Man Month: Essays on Software Engineering
- The Pragmatic Programmer: From Journeyman to Master
- Unix Power Tools
-
- $ uniq -dc items-sorted.txt
- 5 Programming Pearls
- 3 The Art of UNIX Programming
- 3 The C Programming Language
-
-
-``join``
---------
-
-Now suppose we had the file ``stalwarts1.txt``, which lists the home pages of all the people listed in ``stalwarts.txt``.
-::
-
- Richard Stallman%http://www.stallman.org
- Eric Raymond%http://www.catb.org/~esr/
- Ian Murdock%http://ianmurdock.com/
- Lawrence Lessig%http://lessig.org
- Linus Torvalds%http://torvalds-family.blogspot.com/
- Guido van Rossum%http://www.python.org/~guido/
- Larry Wall%http://www.wall.org/~larry/
-
-It would be nice to have a single file with the information in both the files. To achieve this we use the ``join`` command.
-::
-
- $ join stalwarts.txt stalwarts1.txt -t %
- Richard Stallman%rms%GNU Project%http://www.stallman.org
- Eric Raymond%ESR%Jargon File%http://www.catb.org/~esr/
- Ian Murdock% %Debian%http://ianmurdock.com/
- Lawrence Lessig% %Creative Commons%http://lessig.org
- Linus Torvalds% %Linux Kernel%http://torvalds-family.blogspot.com/
- Guido van Rossum%BDFL%Python%http://www.python.org/~guido/
- Larry Wall% %Perl%http://www.wall.org/~larry/
-
-The ``join`` command joins the two files, based on the common field present in both the files, which is the name, in this case.
-
-The ``-t`` option again specifies the delimiting character. Unless that is specified, join assumes that the fields are separated by spaces.
-
-Note that, for ``join`` to work, the common field should be in the same order in both the files. If this is not so, you could use ``sort``, to sort the files on the common field and then join the files. In the above example, we have the common field to be the first column in both the files. If this is not the case we could use the ``-1`` and ``-2`` options to specify the field to be used for joining the files.
-::
-
- $ join -2 2 stalwarts.txt stalwarts2.txt -t %
- Richard Stallman%rms%GNU Project%http://www.stallman.org
- Eric Raymond%ESR%Jargon File%http://www.catb.org/~esr/
- Ian Murdock% %Debian%http://ianmurdock.com/
- Lawrence Lessig% %Creative Commons%http://lessig.org
- Linus Torvalds% %Linux Kernel%http://torvalds-family.blogspot.com/
- Guido van Rossum%BDFL%Python%http://www.python.org/~guido/
- Larry Wall% %Perl%http://www.wall.org/~larry/
-
-
-Generating a word frequency list
-================================
-
-Now, let us use the tools we have learnt to use, to generate a word frequency list of a text file. We shall use the free text of Alice in Wonderland.
-
-The basic steps to achieve this task would be -
-
-1. Eliminate the punctuation and spaces from the document.
-2. Generate a list of words.
-3. Count the words.
-
-We first use ``grep`` and some elementary ``regex`` to eliminate the non-alpha-characters.
-::
-
- $ grep "[A-Za-z]*" alice-in-wonderland.txt
-
-This outputs all the lines which has any alphabetic characters on it. This isn't of much use, since we haven't done anything with the code. We only require the alphabetic characters, without any of the other junk. ``man grep`` shows us the ``-o`` option for outputting only the text which matches the regular expression.
-::
-
- $ grep "[A-Za-z]*" -o alice-in-wonderland.txt
-
-Not very surprisingly, we have all the words, spit out in the form of a list! Now that we have a list of words, it is quite simple to count the occurrences of the words. You would've realized that we can make use of ``sort`` and ``uniq`` commands. We pipe the output from the ``grep`` to the ``sort`` and then pipe it's output to ``uniq``.
-::
-
- $ grep "[A-Za-z]*" -o alice-in-wonderland.txt | sort | uniq -c
-
-Notice that you get the list of all words in the document in the alphabetical order, with it's frequency written next to it. But, you might have observed that Capitalized words and lower case words are being counted as different words. We therefore, replace all the Upper case characters with lower case ones, using the ``tr`` command.
-::
-
- $ grep "[A-Za-z]*" -o alice-in-wonderland.txt | tr 'A-Z' 'a-z' | sort | uniq -c
-
-Now, it would also be nice to have the list ordered in the decreasing order of the frequency of the appearance of the words. We sort the output of the ``uniq`` command with ``-n`` and ``-r`` options, to get the desired output.
-::
-
- $ grep "[A-Za-z]*" -o alice-in-wonderland.txt | tr 'A-Z' 'a-z' | sort | uniq -c | sort -nr
-