1 Introducing Linux

     2 =================

     3 

     4 GNU/Linux is an  operating system that uses the Linux Kernel. It is similar to the Unix operating system. It is an open source operating system which basically means you can view and change the code.  

     5 

     6 The Linux Kernel written by Linus Torvalds in 1991. Although written only for x86 architecture , it was ported to many other architectures.The whole operating system contains the kernel and several other system and application software contributed by many different projects. A major contributor has been the GNU project. GNU project was started by Richard Stallman in 1983. Richard Stallman wrote the GNU General Public License which gave the first impetus to the free software movement leading up do development of the family of Linux operating systems that we see today . 

     7 

     8 Design and Implications

     9 ------------------------

    10 

    11 Linux is a modular operating system, deriving much of its basic design from principles established in Unix earlier. The kernel  manages the systems resources like process control, networking, peripherals and file system access. Application Software written on top of it gives higher level functionality. 

    12 

    13 Reasons for Using Linux

    14 -----------------------

    15 - Linux is free:

    16 

    17 As in "free beer". Linux can be downloaded in its entirety from the Internet completely for free. No registration fees, no costs per user, free updates, and freely available source code in case you want to change the behavior of your system.

    18 

    19 - Linux can be deployed easily on clusters for parallel and distributed computing 

    20 

    21 There are many distributions of Linux meant for clusters. One of the popular ones is Rocks Cluster Distribution.   

    22 

    23 - Linux was made to keep on running:

    24 

    25 As with UNIX, a Linux system expects to run without rebooting all the time. That is why a lot of tasks are being executed at night or scheduled automatically for other times, resulting in higher availability during busier periods and a more balanced use of the hardware. This property allows for Linux to be applicable to environments where people do not have the time or the possibility to control their systems constantly.

    26 

    27 - Linux is secure and versatile:

    28 

    29 The security model used in Linux is based on the UNIX idea of security, which is known to be robust and of proven quality. 

    30 

    31 - Linux contains a tools for scientific computing

    32 

    33 Linux contains many tools like latex for reading and writing scientific text. It also contains many softwares like scilab , python and fortran used for scientific computing needs. 

    34 

    35 

    36 

    37 Getting Started

    38 ================

    39 

    40 Logging in, activating the user interface and logging out

    41 ----------------------------------------------------------

    42 Linux supports multiple users on a machine. Each user must log in with his or her username and password.

    43 

    44 In order to work on a Linux system directly, one needs to provide a username and password. You always need to authenticate to the system. After booting , you will see a login screen/prompt asking for username and password , enter the username and password , if it is correct you will be logged in . One can logout by typing logout on the prompt or navigating to logout button if using Graphical User Interface . 

    45 

    46 

    47 When you see the login screen again, asking to enter username and password, logout was successful.

    48 

    49 

    50 Basic Commands

    51 ===============

    52 

    53 What files do I have on my computer?

    54 -------------------------------------

    55 

    56 All content in Linux  is kept on data structure called files.We can list those files to know what all is there.

    57 *ls* lists the files in the current working directory. A directory that is not the current working directory can be specified and ls will list the files there.::

    58 

    59 

    60 	$ ls

    61 	jeeves.rst psmith.html blandings.html Music

    62 

    63 How do I move around the file system?

    64 -------------------------------------

    65 

    66 This stands for "change directory". When one wants to change the directory .

    67 

    68        $cd Music 

    69 

    70 One dot '.' represents the current directory while two dots '..' represent the parent directory.

    71 

    72 “ cd -” will return you to the previous directory.

    73 

    74 You can also use cd [absolute path] or cd [relative path] (see below):

    75 

    76 Absolute paths:

    77 

    78 Absolute Path is the path of the directory from root i.e / . / is the top most level in file system.

    79 

    80 For example to get to /var/www you would type::

    81 

    82 	$cd /var/www

    83 

    84 This is an absolute path because you start at the top of the hierarchy and go downwards from there (it doesn't matter where in the filesystem you were when you typed the command).

    85 

    86 Relative paths:

    87 

    88    Releative Path is path in relation to your current location . 

    89 

    90     For example if you are in Music directory and want to get to Rock directory inside Music, you type::

    91 

    92 	Music$ cd Rock

    93 

    94 Please note that there is no / using the above cd command. Using a / would cause this to be an absolute path, working from the top of the hierarchy downward.

    95 

    96 Linux is multiuser system so *who* all are using my system now?

    97 --------------------------------------------------------

    98 

    99 The standard Unix command *who* displays a list of users who are currently logged into a computer.::

   100 

   101 

   102 

   103 	$who

   104 	user       tty7         2009-09-08 10:50 (:0)

   105 	harry      pts/0        2009-09-08 11:25 (:0.0)

   106 	dumbledore pts/1        2009-09-08 18:11 (potter.xyz.in)

   107 

   108 The columns represent user, current terminal , date and time of login and the host from which he is logged in respectively. 

   109 The command can be invoked with the arguments *am i* or *am I* (so it is invoked as *who am i* or * who am I*), showing information about the current terminal only (see the *-m* option below, of which this invocation is equivalent).

   110 

   111 

   112 

   113 How do I organize my files?

   114 ---------------------------

   115 

   116 This command is used to make a new directory. Normal usage is as straightforward as follows::

   117 

   118 	$mkdir name_of_directory

   119 

   120 Where *name_of_directory* is the name of the directory one wants to create. When typed as above (ie. normal usage), the new directory would be created within the current directory. On Unix, multiple directories can be specified, and *mkdir* will try to create all of them.

   121 

   122 

   123 

   124 

   125 Where am I now on the filesystem?

   126 --------------

   127 pwd is a Linux / Unix command which prints the current working directory. If you wish to know the full path of the  directory in which you are in from the Linux console, then the pwd command will come to your rescue. pwd stands for Print Working Directory.

   128 

   129 Usage of pwd command::

   130 

   131       $ cd Examples

   132       $ pwd

   133       /home/user/Examples

   134 

   135 

   136 

   137  I wish some commads were a bit smarter ? 

   138 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   139 

   140 The standard commands in Linux have a lot of options also called flags to change or provide some additional functionality to the command For example ::

   141       

   142        $ls -l 

   143        

   144        * *ls with flag -l* displays the result in long format, displaying Unix file types, permissions, number of hard links, owner, group, size, date, and filename ::

   145 

   146        $ls ­a 

   147        * *ls with flag -a*  lists all files including hidden files

   148 

   149 

   150 Similarly, mkdir with -p option automatically creates parent directory even if it does not exist.::

   151 

   152 	   $mkdir -p this/path/never/existed/earlier/

   153 	    

   154 

   155 

   156 

   157 Getting Help

   158 ============

   159 

   160 How do I find what a command does?

   161 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   162 

   163 A short index of explanations for commands is available using the *whatis* command, like in the examples below::

   164 

   165 	$whatis ls

   166 	ls (1) 		 - list directory contents

   167 

   168 This displays short information about a command, and the first section in the collection of man pages that contains an appropriate page.

   169 

   170 More extensive Documentation

   171 ----------------------------

   172 

   173 Man pages (short for "manual pages") are the extensive documentation that comes preinstalled with almost all substantial Unix and Unix-like operating systems. The Unix command used to display them is *man*. Each page is a self-contained document.

   174 

   175 To read a manual page for a Unix command, one can use::

   176 

   177 	$ man <command_name>

   178 

   179 To see the manual on man itself do::

   180 

   181 	$man man

   182 

   183 The previous example will take you to the "Manual" page entry about manual pages!

   184 

   185 Looking at man pages is a very good way to actually check flags and other help related to a command. 

   186 

   187 --help

   188 -------

   189 

   190 Most GNU commands support the --help, which gives a short explanation about how to use the command and a list of available options. Below is the output of this option with the *mkdir* command::

   191 

   192 	$ mkdir --help

   193 	

   194 	Usage: mkdir [OPTION]... DIRECTORY...

   195         Create the DIRECTORY(ies), if they do not already exist.

   196 

   197         Mandatory arguments to long options are mandatory for short options too.

   198           -m, --mode=MODE   set file mode (as in chmod), not a=rwx - umask

   199           -p, --parents     no error if existing, make parent directories as needed

   200           -v, --verbose     print a message for each created directory

   201           -Z, --context=CTX  set the SELinux security context of each created

   202                             directory to CTX

   203           --help     display this help and exit

   204           --version  output version information and exit

   205 

   206         Report mkdir bugs to bug-coreutils@gnu.org

   207         GNU coreutils home page: <http://www.gnu.org/software/coreutils/>

   208         General help using GNU software: <http://www.gnu.org/gethelp/>

   209         Report mkdir translation bugs to <http://translationproject.org/team/>

   210 

   211 

   212 

   213 Working with Files

   214 ===================

   215 

   216 Copying Files

   217 -------------

   218 

   219 *cp* is the command to copy a file from one place to another including different file system(#change? ellaborate). The original file remains unchanged, and the new file may have the same or a different name.

   220 

   221 Usage

   222 ~~~~~

   223 

   224 To copy a file to another file ::

   225 

   226 	$ cp  SourceFile TargetFile

   227 

   228 To copy a file to a directory::

   229 

   230 	$ cp  SourceFile  TargetDirectory

   231  

   232 To copy a directory to a directory::

   233 

   234 	$ cp  -r SourceDirectory  TargetDirectory

   235 

   236 In case target Directory has a file of the same name::

   237    	

   238 	TargetDirectory$ls

   239 	jeeves.rst psmith.html

   240 

   241 	SourceDirectory$ls

   242 	jeeves.rst index.html

   243 	

   244 	$cp -i 	jeeves.rst TargetDirectory/

   245 	cp: overwrite 'TargetDirectory/jeeves.rst'? 

   246 

   247 -i option is for interactive usage.

   248 

   249 

   250 Flags

   251 ~~~~~

   252 *-i* (interactive) – prompts you with the name of a file to be overwritten. This occurs if the TargetDirectory or TargetFile parameter contains a file with the same name as a file specified in the SourceFile or SourceDirectory parameter. If you enter y or the locale's equivalent of y, the cp command continues. Any other answer prevents the cp command from overwriting the file.

   253 

   254 

   255 *-r* (recursive) – copy directories (recursively copying all the contents)

   256 

   257 

   258 Moving Files

   259 ------------

   260 

   261 *mv* (short for move) is a Unix command that moves one or more files or directories from one place to another. The original file is deleted, and the new file may have the same or a different name.An interesting usage of mv is actualy to rename it by moving it in same directory under a different name. 

   262 

   263 

   264 

   265 Usage

   266 ~~~~~~~~

   267 

   268 

   269 To rename a file ::

   270 

   271    $ mv myfile mynewfilename  

   272 

   273 To move to a different directory ::

   274    $ mv myfile otherdir/     

   275 

   276 To move a directory ::

   277    

   278    $mv mydir otherdir

   279 

   280 

   281 Using -i to avoid overwrite(just like cp)::

   282    

   283    $mv -i mydir otherdir

   284    mv: overwrite `otherdir/mydir'?

   285 

   286 

   287 Removing files

   288 --------------

   289 

   290 *rm*  is used to delete files from a filesystem. 

   291 

   292 Here's example to remove a file named "foo" from a directory, here shown with the -i option::

   293 

   294   	$ rm -i foo

   295     	remove foo? y

   296 

   297 Options

   298 ~~~~~~~

   299 

   300 Common options that rm accepts include:

   301 

   302     * *-r*, which removes directories, removing the contents recursively beforehand (so as not to leave files without a directory to reside in) ("recursive")

   303     * *-i*, which asks for every deletion to be confirmed ("interactive")

   304     * *-f*, which ignores non-existent files and overrides any confirmation prompts ("force")

   305     * *-v*, which shows what is being removed as it happens ("verbose")

   306 

   307 *rm* is often aliased to "rm -i" so as to avoid accidental deletion of files. 

   308 

   309 *rm -rf* (variously, rm -rf /, rm -rf `*`, and others) is frequently used in jokes and anecdotes about Unix disasters. The rm -rf variant of the command, if run by a superuser on the root directory, would cause the contents of every writable mounted filesystem on the computer to be deleted.

   310 

   311 

   312 

   313 

   314 

   315 Permissions

   316 ~~~~~~~~~~~

   317 Linux is a proper multi-user environment. In a multi-user environment, security of user and system data is very important. Access should be given only to users who need to access the data. Since Linux is essentially a server OS, good and efficient file security is built right . The permissions are based on whether one is allowed to read, write or execute a file.

   318 

   319 Usually, on most filesystems, deleting a file requires write permission on the parent directory (and execute permission, in order to enter the directory in the first place). (Note that, confusingly for beginners, permissions on the file itself are irrelevant. However, GNU rm asks for confirmation if a write-protected file is to be deleted, unless the -f option is used.)

   320 

   321 To delete a directory (with rm -r), one must delete all of its contents recursively. This requires that one must have read and write and execute permission to that directory (if it's not empty) and all non-empty subdirectories recursively (if there are any).

   322 

   323 

   324 Is there a default organization system of files and Directories that Linux follows

   325 ==================================================================================

   326 

   327 In the File Hierarchy Standard (FHS) all files and directories appear under the root directory "/", even if they are stored on different physical devices. Note however that some of these directories may or may not be present on a Unix system depending on whether certain subsystems, such as the X Window System, are installed.

   328 

   329 The majority of these directories exist in all UNIX operating systems and are generally used in much the same way; however, the descriptions here are those used specifically for the FHS, and are not considered authoritative for platforms other thanmajor Linux distros.

   330 

   331 +---------------+------------------------------------------------+

   332 |   Directory   |             Description                        |

   333 +===============+================================================+

   334 | /             | Primary hierarchy root and root directory of   |

   335 |               | the entire file system hierarchy.              |

   336 +---------------+------------------------------------------------+

   337 | /bin/         | Essential command binaries that need to be     |

   338 |               | available in single user mode; for all users,  |

   339 |               | e.g., *cat*, *ls*, *cp*.                       |

   340 +---------------+------------------------------------------------+

   341 | /boot/        | Boot loader files, e.g., *kernels*, *initrd*;  |

   342 |               | often a separate partition.                    |

   343 +---------------+------------------------------------------------+

   344 | /dev/         | Essential devices, e.g., /dev/null             |

   345 +---------------+------------------------------------------------+

   346 | /etc/         | Host-specific system-wide configuration files  |

   347 |               | (the name comes from *et cetera*)              |

   348 +---------------+------------------------------------------------+

   349 | /home/        | User's home directories, containing saved      |

   350 |               | files, personal settings, etc.; often a        |

   351 |               | separate partition.                            |

   352 +---------------+------------------------------------------------+

   353 | /lib/         | Libraries essential for the binaries in        |

   354 |               | */bin/* and */sbin/*                           |

   355 +---------------+------------------------------------------------+

   356 | /media/       | Mount points for removable media such as       |

   357 |               | CD-ROMs, external hard disks, USB sticks, etc. |

   358 +---------------+------------------------------------------------+

   359 | /mnt/         | Temporarily mounted file systems               |

   360 +---------------+------------------------------------------------+

   361 | /opt/         | Optional application software packages         |

   362 +---------------+------------------------------------------------+

   363 | /proc/        | Virtual filesystem documenting kernel and      |

   364 |               | process status as text files; e.g., uptime,    |

   365 |               | network. In Linux, corresponds to a *Procfs*   |

   366 |               | mount.                                         |

   367 +---------------+------------------------------------------------+

   368 | /root/        | Home directory for the root user               |

   369 +---------------+------------------------------------------------+

   370 | /sbin/        | Essential system binaries; e.g., *init*,       |

   371 |               | *route*, *mount*.                              |

   372 +---------------+------------------------------------------------+

   373 | /srv/         | Site-specific data which is served by the      |

   374 |               | system.                                        |

   375 +---------------+------------------------------------------------+

   376 | /tmp/         | Temporary files. Often not preserved between   |

   377 |               | system reboots.                                |

   378 +---------------+------------------------------------------------+

   379 | /usr/         | Secondary hierarchy for read-only user data;   |

   380 |               | contains the majority of (multi-)user          |

   381 |               | utilities and applications.                    |

   382 +---------------+------------------------------------------------+

   383 | /var/         | Variable files - files whose content is        |

   384 |               | expected to continually change during normal   |

   385 |               | operation of the system - such as logs, spool  |

   386 |               | files, and temporary e-mail files.             |

   387 |               | Sometimes a separate partition.                |

   388 +---------------+------------------------------------------------+

   389 

   390 

   391 man hier

   392 ---------

   393 

   394 This is the manual page on the UNIX filesystem. The syntax for this is::

   395 

   396 	$ man hier

   397 

   398 

   399 

   400 

   401 

   402 

   403 Permissions and Ownership

   404 =========================

   405 

   406 drwxr-x--- 2 user group 4096 Dec 28 04:09 tmp

   407 -rw-r--r-- 1 user group 969 Dec 21 02:32 foo

   408 -rwxr-xr-x 1 user group 345 Sep 1 04:12 somefile

   409 

   410 d - The file is a directory.

   411 

   412 r - Read permission. The file may be read. In the case of a directory, this would mean the ability to list the contents of the directory.

   413 

   414 w - Write permission.For a directory, this defines whether you can make any changes to the contents

   415 of the directory. If write permission is not set then you will not be able to delete, rename or create a file.

   416 

   417 x - Execute permission. Whether the file may be executed. In the case of a directory, this attribute decides whether you have permission to enter,run a search through that directory or execute some program from that directory.

   418 

   419 The Permission are read in the following manner::

   420 

   421     | User | Group | Others |

   422     |------+-------+--------|

   423     | drwx | rwx   | rwx    |

   424 

   425 

   426 Problem: Given a file with only read permission, add execute permission to User, Group and Others.

   427 --------------------------------------------------------------------------------------------------

   428 

   429 Solution::

   430 	

   431 	$chmod a+x executablefile

   432    	$ls -l executablefile

   433 

   434 Thats it .

   435 

   436 

   437 chmod

   438 ------

   439 

   440 The *chmod* command changes the  file system modes of files and directories. The modes include permissions and special modes.

   441  

   442 Usage

   443 ~~~~~

   444 

   445 The *chmod* command options are specified like this:

   446 ::

   447 

   448 	$ chmod [options] mode[,mode] file1 [file2 ...]

   449 

   450 To view what the permissions currently are, type:

   451 ::

   452 

   453 	$ ls -l file

   454 

   455 Command line options

   456 ~~~~~~~~~~~~~~~~~~~~

   457 

   458 The *chmod* command has a number of command line options that affect its behavior. The most common options are:

   459 

   460     * -R: Changes the modes of directories and files recursively

   461 

   462     * -v: Verbose mode; lists all files as they are being processed

   463 

   464 Symbolic modes

   465 +++++++++++++++

   466 

   467 To the *chmod* utility, all permissions and special modes are represented by its mode parameter. One way to adjust the mode of files or directories is to specify a symbolic mode. The symbolic mode is composed of three components, which are combined to form a single string of text:

   468 ::

   469 

   470 	$ chmod [references][operator][modes] file1 ...

   471 

   472 The references (or classes) are used to distinguish the users to whom the permissions apply. If no references are specified it defaults to “all” but modifies only the permissions allowed by the umask. The references are represented by one or more of the following letters:

   473 

   474 +--------------+--------+---------------------------------------------+

   475 | Reference    | Class  |                Description                  |

   476 +==============+========+=============================================+

   477 |      u       | user   | the owner of the file                       |

   478 +--------------+--------+---------------------------------------------+

   479 |      g       | group  | users who are members of the file's group   |

   480 +--------------+--------+---------------------------------------------+

   481 |      o       | others | users who are not hte owner of the file or  |

   482 |              |        | members of the group                        |

   483 +--------------+--------+---------------------------------------------+

   484 |      a       | all    | all three of the above; is the same as *ugo*|

   485 +--------------+--------+---------------------------------------------+

   486 

   487 The *chmod* program uses an operator to specify how the modes of a file should be adjusted. The following operators are accepted:

   488 

   489 +--------------+------------------------------------------------------+

   490 | Operator     |                      Description                     |

   491 +==============+======================================================+

   492 | +            | adds the specified modes to the specified classes    |

   493 +--------------+------------------------------------------------------+

   494 | -            | removes the specified modes from the specified       |

   495 |              | classes                                              |

   496 +--------------+------------------------------------------------------+

   497 | =            | the modes specified are to be made the exact modes   |

   498 |              | for the specified classes                            |

   499 +--------------+------------------------------------------------------+

   500 

   501 The modes indicate which permissions are to be granted or taken away from the specified classes. There are three basic modes which correspond to the basic permissions:

   502 

   503 +-----+--------------+------------------------------------------------+

   504 |Mode |    Name      |                 Description                    |

   505 +=====+==============+================================================+

   506 | r   | read         | read a file or list a directory's contents     |

   507 +-----+--------------+------------------------------------------------+

   508 | w   | write        | write to a file or directory                   |   

   509 +-----+--------------+------------------------------------------------+

   510 | x   | execute      | execute a file or recurse a directory tree     |

   511 +-----+--------------+------------------------------------------------+

   512 

   513 Examples

   514 +++++++++

   515 

   516 Add the 'read' and 'write' permissions to the 'user' and 'group' classes of a directory:

   517 ::

   518 

   519 	$ chmod ug+rw mydir

   520 	$ ls -ld mydir

   521 	drw-rw----   2 starwars  yoda  96 Dec 8 12:53 mydir

   522 

   523 For a file, remove *write* permissions for all classes:

   524 ::

   525 

   526 	$ chmod a-w myfile

   527 	$ ls -l myfile

   528 	-r-xr-xr-x   2 starwars  yoda 96 Dec 8 12:53 myfile

   529 

   530 Octal numbers

   531 +++++++++++++

   532 

   533 The *chmod* command also accepts three and four-digit octal numbers representing modes. Using a three-digit octal number to set the modes of a file named myfile :

   534 ::

   535 

   536 	$ chmod 664 myfile

   537 	$ ls -l myfile

   538 	-rw-rw-r--  1   57 Jul  3 10:13  myfile

   539 

   540 Foe each one, you define the right like that :

   541 

   542     * a read right correspond to 4

   543     * a write right correspond to 2

   544     * an execute right correspond to 1

   545 

   546 You want the user to have all the rights? : 4 + 2 + 1 = 7

   547 

   548 you want the group to have read and write rights : 4 + 2 = 6

   549 

   550 

   551 Changing Ownership of Files

   552 ===========================

   553 

   554 

   555 chown

   556 ~~~~~

   557 The chown command is used to change the owner and group of files, directories and links.

   558 

   559 By default, the owner of a filesystem object is the user that created it. The group is a set of users that share the same access permissions (i.e., read, write and execute) for that object.

   560 

   561 The basic syntax for using chown to change owners is::

   562 

   563     $chown -v alice wonderland.txt

   564 

   565 -v - For verbose Mode

   566  

   567 alice - Username of new owner 

   568 

   569 

   570 

   571 

   572 

   573 Working with text

   574 =================

   575 

   576 How do I look into a file?

   577 ~~~~~~~~~~~~~~~~~~~~~~~~~~

   578 

   579 more

   580 -----

   581 

   582 In computing, *more* is a command to view  contents of a text file one screen at a time 

   583 

   584 Usage

   585 ~~~~~

   586 

   587 The command-syntax is::

   588 

   589 	$ more [options] [file_name]

   590 

   591 Traversing the pages ::

   592 

   593 

   594      SPACE       Display next k lines of text.  Defaults to current screen

   595                  size.

   596 

   597 

   598      RETURN      Display next k lines of text.  Defaults to 1.  Argument

   599                  becomes new default.

   600 	       

   601      /pattern    Search for kth occurrence of regular expression.  Defaults to

   602                  1 .

   603 

   604 

   605 

   606 less

   607 -----

   608 

   609 *less*  is similar to *more* in the sense that it is used to view files , but has the extended capability of allowing both forward and backward navigation through the file. Unlike most Unix text editors/viewers, *less* does not need to read the entire file before starting, resulting in faster load times with large files.

   610 

   611 Usage

   612 ~~~~~~

   613 

   614 *less* can be invoked with options to change its behaviour, for example, the number of lines to display on the screen. A few options vary depending on the operating system. While *less* is displaying the file, various commands can be used to navigate through the file. These commands are based on those used by both *more* and *vi*. It is also possible to search for character patterns in the file.

   615 

   616 By default, *less* displays the contents of the file to the standard output (one screen at a time). If the file name argument is omitted, it displays the contents from standard input (usually the output of another command through a pipe). If the output is redirected to anything other than a terminal, for example a pipe to another command, less behaves like cat.

   617 

   618 The command-syntax is ::

   619 

   620 	$ less [options] file_name

   621 

   622 

   623 

   624 Frequently Used Commands

   625 ~~~~~~~~~~~~~~~~~~~~~~~~

   626 

   627     * [Arrows]/[Page Up]/[Page Down]/[Home]/[End]: Navigation.

   628 

   629     * [Space bar]: Next page.

   630 

   631     * b: Previous page.

   632 

   633     * ng: Jump to line number n. Default is the start of the file.

   634 

   635     * nG: Jump to line number n. Default is the end of the file.

   636 

   637     * /pattern: Search for pattern. Regular expressions can be used.

   638 

   639     * '^ or g: Go to start of file.

   640 

   641     * '$ or G: Go to end of file.

   642 

   643     * =: File information.

   644 

   645     * h: Help.

   646 

   647     * q: Quit.

   648 

   649 

   650 cat

   651 ---

   652 

   653 The *cat* command is a standard Unix program used to concatenate and display files. The name is from "catenate", a synonym of *concatenate*.

   654 

   655 The Single Unix Specification specifies the behavior that the contents of each of the files given in sequence as arguments will be written to the standard output in the same sequence, and mandates one option, -u, where each byte is printed as it is read.

   656 

   657 If the filename is specified as -, then *cat* will read from standard input at that point in the sequence. If no files are specified, *cat* will read from standard input entered.

   658 

   659 Usage ::

   660         $ cat foo boo

   661 	This is file foo

   662 	

   663 	This is file boo.

   664 

   665 Text Processing 

   666 ---------------

   667 

   668 

   669 How do look at part of a file?

   670 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   671 

   672 head

   673 -----

   674 

   675 *head* is a program on Unix and Unix-like systems used to display the first few lines of a text file or piped data. The command syntax is::

   676 

   677 	$ head [options] <file_name>

   678 

   679 By default, *head* will print the first 10 lines of its input to the standard output. The number of lines printed may be changed with a command line option. The following example shows the first 20 lines of somefile::

   680 

   681 	$ head -n 20 somefile

   682 

   683 

   684 

   685 

   686 tail

   687 ----

   688 

   689 *tail* is a program on Unix and Unix-like systems used to display the last few lines of a text file or piped data.

   690 

   691 The command-syntax is::

   692 

   693 	$ tail [options] <file_name>

   694 

   695 By default, *tail* will print the last 10 lines of its input to the standard output. With command line options the number of lines printed and the printing units (lines, blocks or bytes) may be changed. The following example shows the last 20 lines of somefile::

   696 

   697 	$ tail -n 20 somefile

   698 

   699 

   700 

   701 This example shows all lines of somefile from the second line onwards::

   702 

   703 	$ tail -n +2 somefile

   704 

   705 

   706 

   707 Monitoring a continously changing file(example: A log file) 

   708 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   709 

   710 *tail* has a special command line option *-f* (follow) that allows a file to be monitored. Instead of displaying the last few lines and exiting, tail displays the lines and then monitors the file. As new lines are added to the file by another process, tail updates the display. This is particularly useful for monitoring log files. The following command will display the last 10 lines of messages and append new lines to the display as new lines are added to messages::

   711 

   712 	$ tail -f /var/log/dmesg

   713 

   714 To interrupt tail while it is monitoring, break-in with *Ctrl+C*. This command can be run "in the background" with &, see job control.

   715 

   716 More serious Text Processing:

   717 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   718 

   719 Problem1. Get the names of people in the following file

   720 -----------------------------------------------------

   721 ::

   722 

   723 	Foot in Mouth:Bully:Fat:Peter

   724 	Rich:Simple:Peabrain:Lois

   725 	Self-concious:Wannabe:Adolescent:Meg

   726 	Dim-witted:Fat:evil-monkey:Chris

   727 	Matricidal:Over-Ambitious:Infant:Stewy

   728 	Anthropomorphic:Democrat:Sensible:Brian

   729 

   730 Soulution::

   731  	$cut -d : -f 4- file

   732 	

   733 	Peter

   734 	Lois

   735 	Meg

   736 	Chris

   737 	Stewy

   738 	Brian

   739 

   740 

   741 		

   742 

   743 

   744 

   745 cut

   746 ----

   747 

   748 In computing, *cut* is a Unix command line utility which is used to extract sections from each line of input — usually from a file.

   749 

   750 Extraction of line segments can typically be done by  a *delimiter (-d — the tab character by default)*. A range must be provided in which consists of one of N, N-M, N- (N to the end of the line), or -M (beginning of the line to M), where N and M are counted from 1 (there is no zeroth value). 

   751 

   752 

   753 Options

   754 -------

   755 ::

   756 

   757 	-b   select on the basis of bytes

   758 

   759         -c   select on the basis of characters

   760 			

   761 			

   762 	-f  select on  the basis of fields,also print any line that contains no delimiter character

   763 

   764 Problem2.Given two files of student name and marks merge them

   765 -------------------------------------------------------------

   766 

   767 File of students(students.txt) ::

   768      

   769      Hussain

   770      Dilbert

   771      Alex

   772      Raul

   773      Sven

   774 

   775 File of marks(marks.txt)::

   776      

   777 

   778      89

   779      98

   780      67

   781      78

   782      67

   783 

   784 Solution::

   785 

   786 	$paste students.txt marks.txt

   787 	Hussain	89

   788 	Dilbert	98

   789  	Alex	67

   790  	Raul	78

   791  	Sven	67

   792 

   793 

   794 	$paste -d : students.txt marks.txt

   795 	

   796 	Hussain:89

   797 	Dilbert:98

   798 	Alex:67

   799  	Raul:78

   800  	Sven:67

   801 

   802 	

   803 	$paste -s students.txt marks.txt

   804 

   805 	Hussain	  	 Dilbert	 Alex	 Raul	 Sven	

   806 	89		 98		 67	 78	 67

   807 

   808 

   809 

   810 

   811 paste

   812 ------

   813 

   814 *paste* is a Unix command line utility which is used to join files horizontally (parallel merging) by outputting lines consisting of the sequentially corresponding lines of each file specified, separated by tabs, to the standard output. 

   815 

   816 

   817 Some frequently used Text Processing tools

   818 ==========================================

   819 

   820 Given a file with names US presidents  . Sort the file  on the names . The contents of the file are mentioned below

   821 ::

   822       

   823        

   824        John%Kennedy

   825        Bill%Clinton

   826        Bill%Clinton

   827        Franklin%Rosevelt

   828        George%Bush 

   829        Abraham%Lincoln

   830        Abraham%Lincoln

   831        George%Washington

   832        George%Washington

   833        

   834 

   835 

   836 Solution:: 

   837 	

   838 	 $sort presidents.txt

   839 

   840        Abraham%Lincoln

   841        Abraham%Lincoln

   842        Bill%Clinton

   843        Bill%Clinton       

   844        Franklin%Rosevelt

   845        George%Bush 

   846        George%Washington

   847        George%Washington

   848        John%Kennedy

   849 

   850 

   851 Sort

   852 ====

   853 sort command with the file name as a parameter sorts the lines of the file alphabetically and prints the output on the terminal.	

   854 

   855 

   856 To sort the same file using the last names     

   857 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   858 

   859 ::

   860 

   861 	$sort -t % -k 2 presidents.txt

   862 	

   863 	George%Bush 

   864         Bill%Clinton

   865         Bill%Clinton       

   866         John%Kennedy

   867         Abraham%Lincoln

   868         Abraham%Lincoln

   869         Franklin%Rosevelt

   870         George%Washington

   871         George%Washington

   872 

   873 

   874 

   875 	-t - specify the delimiter 

   876 	-k - specify the column 

   877 

   878 There are other options like -r to sort on reverse order etc ..

   879 

   880 

   881 

   882 Problem:Given a file remove all the duplicate lines in it ?

   883 ===========================================================

   884 

   885 File ::

   886      

   887        Abraham%Lincoln

   888        Abraham%Lincoln

   889        Bill%Clinton

   890        Bill%Clinton       

   891        Franklin%Rosevelt

   892        George%Bush 

   893        George%Washington

   894        George%Washington

   895        John%Kennedy

   896 

   897 Solution::

   898       

   899       $uniq -c sortpresidents.txt

   900 

   901 

   902       2 Abraham%Lincoln       

   903       2 Bill%Clinton

   904       1 Franklin%Rosevelt

   905       1 George%Bush 

   906       2 George%Washington

   907       1 John%Kennedy

   908 

   909 

   910 uniq

   911 ====

   912 

   913 

   914 ``uniq`` command removes duplicate lines next to each other.

   915 

   916 -c - also gives the no. of repetitions .

   917 

   918 

   919 Redirection and Piping

   920 =======================

   921 

   922 In computing, *redirection* is a function common to most command-line interpreters, including the various Unix shells that can redirect standard streams to user-specified locations.

   923 

   924 

   925 

   926 Problem: Get a sorted list of all xml file in directory ?

   927 

   928 Soulition::

   929 	$ls | grep '.xml$' > xmllist

   930 

   931 

   932 This problem introduced a lot of concepts we shall look at them one by one .

   933 

   934 

   935 Redirecting standard input and standard output

   936 -----------------------------------------------

   937 

   938 Redirection refers to redirecting standard Input or output to a command or file .

   939 

   940 Ex::

   941 

   942 	$ command > file1

   943 

   944 This command executes *command1*, placing the output in file1. Note that this will truncate any existing data in *file1*. To append output to the end of the file, use the >> operator like this

   945      	$ command >> file1

   946 	

   947 Ex2::

   948 	

   949 

   950 

   951 	$ command < file1

   952 

   953 executes *command1*, using *file1* as the source of input (as opposed to the keyboard).::

   954 

   955 	$ command1 < infile > outfile

   956 

   957 combines the two capabilities: *command1* reads from *infile* and writes to *outfile*

   958 

   959 

   960 In this example we see redirection in the end ::

   961 

   962    $(some commands) > xmllist 

   963 

   964 

   965 

   966 

   967 Piping

   968 -------

   969 

   970 Programs can be run together such that one program reads the output from another with no need for an explicit intermediate file:

   971 A pipeline of two programs run on a text terminal::

   972 

   973 	$ command1 | command2

   974 

   975 executes *command1*, using its output as the input for *command2* (commonly called piping, since the '|' character is known as a "pipe").

   976 

   977 This is equivalent to using two redirects and a temporary file ::

   978 

   979 	$ command1 > tempfile

   980 	$ command2 < tempfile

   981 	$ rm tempfile

   982 

   983 In the above example we used piping in the following manner ::

   984 

   985 	$ echo

   986 

   987 This runs the ftp client with input user, press return, then pass.

   988 

   989 Redirecting to and from the standard file handles

   990 --------------------------------------------------

   991 

   992 In Unix shells derived from the original Bourne shell, the first two actions can be further modified by placing a number (the file descriptor) immediately before the character; this will affect which stream is used for the redirection. The Unix standard I/O streams are:

   993 

   994 +------------+-------------+------------------------+

   995 |   Handle   |    Name     |      Description       |

   996 +============+=============+========================+

   997 | 0          |   stdin     |    Standard input      |

   998 +------------+-------------+------------------------+

   999 | 1          |   stdout    |    Standard output     |

  1000 +------------+-------------+------------------------+

  1001 | 2          |   stderr    |    Standard error      |

  1002 +------------+-------------+------------------------+

  1003 

  1004 For example:

  1005 ::

  1006 

  1007 	$ command1 2> file1

  1008 

  1009 executes *command1*, directing the standard error stream to *file1*. 

  1010 

  1011 

  1012 

  1013 	$ find / -name .profile 2>&1 | less

  1014 

  1015 

  1016 

  1017 Chained pipelines

  1018 ------------------

  1019 

  1020 The redirection and piping tokens can be chained together to create complex commands. For example:

  1021 ::

  1022 

  1023 	$ ls | grep '\.sh' | sort > shlist

  1024 

  1025 lists the contents of the current directory, where this output is filtered to only contain lines which contain *.sh*, sort this resultant output lexicographically, and place the final output in *shlist*. This type of construction is used very commonly in shell scripts and batch files.

  1026 

  1027 Redirect to multiple outputs

  1028 -----------------------------

  1029 

  1030 The standard command *tee* can redirect output from a command to several destinations.

  1031 ::

  1032 

  1033 	$ ls -lrt | tee xyz

  1034 

  1035 This directs the file list output to both standard output as well as to the file *xyz*.

  1036 

  1037 

  1038 

  1039 Shell Meta Characters

  1040 ======================

  1041 

  1042 Unix recognizes certain special characters, called "meta characters," as command directives. The shell meta characters are recognized anywhere they appear in the command line, even if they are not surrounded by blank space. For that reason, it is safest to only use the characters A-Z, a-z, 0-9, and the period, dash, and underscore characters when naming files and directories on Unix. If your file or directory has a shell meta character in the name, you will find it difficult to use the name in a shell command.

  1043 

  1044 The shell meta characters include:

  1045 

  1046 \ / < > ! $ % ^ & * | { } [ ] " ' ` ~ ; 

  1047 

  1048 

  1049 As an example,

  1050 ::

  1051 

  1052 	$ ls file.*

  1053 

  1054 run on a directory containing the files file, file.c, file.lst, and myfile would list the files file.c and file.lst. However,::

  1055 

  1056 	$ ls file.?

  1057 

  1058 run on the same directory would only list file.c because the ? only matches one character, no more, no less. This can save you a great deal of typing time. For example, if there is a file called california_cornish_hens_with_wild_rice and no other files whose names begin with 'c', you could view the file without typing the whole name by typing this::

  1059 

  1060 	$ more c*

  1061 

  1062 because the c* matches that long file name.

  1063 

  1064 filenames containing metacharacters can pose many problems and should never be intentionally created.

  1065 

  1066 

  1067 

  1068 

  1069 grep

  1070 -----

  1071 

  1072 *grep* is a command line text search utility originally written for Unix. The name is taken from the first letters in *global / regular expression / print*, a series of instructions for the *ed* text editor. The *grep* command searches files or standard input globally for lines matching a given regular expression, and prints them to the program's standard output.

  1073 

  1074 Usage

  1075 ~~~~~~

  1076 

  1077 This is an example of a common *grep* usage:

  1078 ::

  1079 

  1080 	$ grep apple fruitlist.txt

  1081 

  1082 In this case, *grep* prints all lines containing 'apple' from the file *fruitlist.txt*, regardless of word boundaries; therefore lines containing 'pineapple' or 'apples' are also printed. The *grep* command is case sensitive by default, so this example's output does not include lines containing 'Apple' (with a capital A) unless they also contain 'apple'.

  1083 

  1084 Like most Unix commands, *grep* accepts command line arguments to change this and many other behaviors. For example:

  1085 ::

  1086 

  1087 	$ grep -i apple fruitlist.txt

  1088 

  1089 This prints all lines containing 'apple' regardless of capitalization. The '-i' argument tells *grep* to be case insensitive, or to ignore case.

  1090 

  1091 To print all lines containing 'apple' as a word ('pineapple' and 'apples' will not match):

  1092 ::

  1093 

  1094 	$ grep -w apple fruitlist.txt

  1095 

  1096 Regular expressions can be used to match more complicated queries.

  1097 

  1098 Variations

  1099 +++++++++++

  1100 

  1101 There are countless implementations and derivatives of *grep* available for many operating systems. Early variants of *grep* included *egrep* and *fgrep*. The former applies an extended regular expression syntax that was added to Unix after Ken Thompson's original regular expression implementation. The latter searches for any of a list of 'fixed' strings using the Aho-Corasick algorithm. These variants are embodied in most modern *grep* implementations as command-line switches (and standardized as -E and -F in POSIX). In such combined implementations, *grep* may also behave differently depending on the name by which it is invoked, allowing *fgrep*, *egrep*, and *grep* to be links to the same program.

  1102 

  1103 *pcregrep* is an implementation of *grep* that uses Perl regular expression syntax.

  1104 

  1105 Other commands contain the word 'grep' to indicate that they search (usually for regular expression matches). The *pgrep* utility, for instance, displays the processes whose names match a given regular expression.

  1106 

  1107 

  1108 Elementary Regex

  1109 =================

  1110 

  1111 In computing, regular expressions provide a concise and flexible means for identifying strings of text of interest, such as particular characters, words, or patterns of characters. A regular expression (often shortened to regex or regexp) is written in a formal language that can be interpreted by a regular expression processor, a program that either serves as a parser generator or examines text and identifies parts that match the provided specification.

  1112 

  1113 Regular expressions are used by many text editors, utilities, and programming languages to search and manipulate text based on patterns. For example, Perl, Ruby and Tcl have a powerful regular expression engine built directly into their syntax. Several utilities provided by Unix distributions—including the editor *ed* and the filter *grep* — were the first to popularize the concept of regular expressions.

  1114 

  1115 

  1116 Regular Expressions are a feature of UNIX. They describe a pattern to match, a sequence of characters, not words, within a line of text. Here is a quick summary of the special characters used in the grep tool and their meaning: 

  1117 

  1118 * ^ (Caret)        =    match expression at the start of a line, as in ^A.

  1119 * $ (Question)     =    match expression at the end of a line, as in A$.

  1120 * \ (Back Slash)   =    turn off the special meaning of the next character, as in \^.

  1121 * [ ] (Brackets)   =    match any one of the enclosed characters, as in [aeiou].

  1122                       Use Hyphen "-" for a range, as in [0-9].

  1123 * [^ ]             =    match any one character except those enclosed in [ ], as in [^0-9].

  1124 * . (Period)       =    match a single character of any value, except end of line.

  1125 * * (Asterisk)     =    match zero or more of the preceding character or expression.

  1126 * \{x,y\}          =    match x to y occurrences of the preceding.

  1127 * \{x\}            =    match exactly x occurrences of the preceding.

  1128 * \{x,\}           =    match x or more occurrences of the preceding.

  1129 

  1130 

  1131 

  1132 Here are some examples using grep:

  1133 

  1134 *    grep smug files         {search files for lines with 'smug'}