statistics.txt
changeset 62 3acc3a02fc4c
parent 59 b62177acce71
equal deleted inserted replaced
61:38ef280b1408 62:3acc3a02fc4c
     1 Hello and welcome to the tutorial on handling large data files and processing them.
     1 Hello and welcome to the tutorial on handling large data files and processing them.
     2 
     2 
     3 Till now we have covered:
     3 Up until now we have covered:
     4 * How to create plots.
     4 * How to create plots.
     5 * How to read data from files and process it.
     5 * How to read data from files and process it.
     6 
     6 
     7 In this session, we will use these concepts and some new ones, to solve a problem/exercise. 
     7 In this tutorial, we shall use these concepts and some new ones, to solve a problem/exercise. 
     8 
     8 
     9 We have a file named sslc.txt. 
     9 We have a file named sslc.txt on our desktop.
    10 It contains record of students and their performance in one of the State Secondary Board Examination. It has 180, 000 lines of record. We are going to read it and process this data.
    10 It contains record of students and their performance in one of the State Secondary Board Examination. It has 180, 000 lines of record. We are going to read it and process this data.
    11 We can see the content of file by opening with any text editor.
    11 We can see the content of file by double clicking on it. It might take some time to open since it is quite a large file.
    12 Please don't edit the data.
    12 Please don't edit the data.
    13 This file has a particular structure. Each line in the file is a set of 11 fields separated by semi-colons
    13 This file has a particular structure. Each line in the file is a set of 11 fields separated by semi-colons
       
    14 Consider a sample line from this file.
    14 A;015163;JOSEPH RAJ S;083;042;47;AA;72;244;;;
    15 A;015163;JOSEPH RAJ S;083;042;47;AA;72;244;;;
    15 The following are the fields in any given line.
    16 The following are the fields in any given line.
    16 * Region Code which is 'A'
    17 * Region Code which is 'A'
    17 * Roll Number 015163
    18 * Roll Number 015163
    18 * Name JOSEPH RAJ S
    19 * Name JOSEPH RAJ S
    36 File Reading - which we have already looked at.
    37 File Reading - which we have already looked at.
    37 parsing  - which we have looked at partially.
    38 parsing  - which we have looked at partially.
    38 Dictionaries - we shall be introducing the concept of dictionaries here.
    39 Dictionaries - we shall be introducing the concept of dictionaries here.
    39 And finally plotting - which we have been doing all along.
    40 And finally plotting - which we have been doing all along.
    40 
    41 
       
    42 Since this file is on our Desktop, let's navigate by typing 
       
    43 
       
    44 cd Desktop
       
    45 
       
    46 Let's get started, by opening the IPython prompt by typing, 
       
    47 
       
    48 ipython -pylab
       
    49 
    41 Let's first start off with dictionaries.
    50 Let's first start off with dictionaries.
    42 
    51 
    43 We earlier used lists briefly. Back then we just created lists and appended items into them. 
    52 We earlier used lists briefly. Back then we just created lists and appended items into them. 
    44 x = [1, 4, 2, 7, 6]
    53 x = [1, 4, 2, 7, 6]
    45 In order to access any element in a list, we use its index number. Index starts from 0.
    54 In order to access any element in a list, we use its index number. Index starts from 0.
    46 For eg. x[0] will give 1 and x[3] will give 7.
    55 For eg. x[0] gives 1 and x[3] gives 7.
    47 
    56 
    48 But, using integer indexes isn't always convenient. For example, consider a telephone directory. We give it a name and it should return a corresponding number. A list is not well suited for such problems. Python's dictionaries are better, for such problems. Dictionaries are just key-value pairs. For example:
    57 But, using integer indexes isn't always convenient. For example, consider a telephone directory. We give it a name and it should return a corresponding number. A list is not well suited for such problems. Python's dictionaries are better, for such problems. Dictionaries are just key-value pairs. For example:
    49 
    58 
    50 d = {'png' : 'image',
    59 d = {'png' : 'image',
    51       'txt' : 'text', 
    60       'txt' : 'text', 
    52       'py' : 'python'} 
    61       'py' : 'python'} 
       
    62 
       
    63 And that is how we create a dictionary. Dictionaries are created by typing the key-value pairs within flower brackets.
    53 
    64 
    54 d
    65 d
    55 
    66 
    56 d is a dictionary. The first element in the pair is called the `key' and the second is called the `value'. The key always has to be a string while the value can be of any type.
    67 d is a dictionary. The first element in the pair is called the `key' and the second is called the `value'. The key always has to be a string while the value can be of any type.
    57 
    68 
    67 True
    78 True
    68 
    79 
    69 'jpg' in d
    80 'jpg' in d
    70 False
    81 False
    71 
    82 
       
    83 
       
    84 
    72 Please note that keys, and not values, are searched. 
    85 Please note that keys, and not values, are searched. 
    73 'In a telephone directory one can search for a number based on a name, but not for a name based on a number'
    86 'In a telephone directory one can search for a number based on a name, but not for a name based on a number'
    74 
    87 
    75 to obtain the list of all keys in a dictionary, type
    88 to obtain the list of all keys in a dictionary, type
    76 d.keys()
    89 d.keys()
    79 Similarly,
    92 Similarly,
    80 d.values()
    93 d.values()
    81 ['python', 'text', 'image']
    94 ['python', 'text', 'image']
    82 is used to obtain the list of all values in a dictionary
    95 is used to obtain the list of all values in a dictionary
    83 
    96 
    84 Let's now see what the dictionary contains
    97 one more thing to note about dictionaries, in this case for d, 
    85 d 
       
    86 
    98 
    87 Please observe that dictionaries do not preserve the order in which the items were entered. The order of the elements in a dictionary should not be relied upon.
    99 d  
       
   100 
       
   101 is that dictionaries do not preserve the order in which the items were entered. The order of the elements in a dictionary should not be relied upon.
    88 
   102 
    89 ------------------------------------------------------------------------------------------------------------------
   103 ------------------------------------------------------------------------------------------------------------------
    90 
   104 
    91 Parsing and string processing
   105 Parsing and string processing
    92 
   106 
    93 As we saw previously we will be dealing with lines with content of the form
   107 As we saw previously we shall be dealing with lines with content of the form
    94 A;015162;JENIL T P;081;060;77;41;74;333;P;;
   108 A;015162;JENIL T P;081;060;77;41;74;333;P;;
    95 Here ';' is delimiter, that is ';' is used to separate the fields.
   109 Here ';' is delimiter, that is ';' is used to separate the fields.
    96 
   110 
    97 We shall create one string variable to see how can we process it to get the desired output.
   111 We shall create one string variable to see how can we process it to get the desired output.
    98 
   112 
   106 a
   120 a
   107 
   121 
   108 is list containing all the fields separately.
   122 is list containing all the fields separately.
   109 
   123 
   110 a[0] is the region code, a[1] the roll no., a[2] the name and so on.
   124 a[0] is the region code, a[1] the roll no., a[2] the name and so on.
   111 Similarly, a[6] will give us the science marks of that particular region.
   125 Similarly, a[6] gives us the science marks of that particular region.
   112 
   126 
   113 So we create a dictionary of all the regions with number of students having more than 90 marks.
   127 So we create a dictionary of all the regions with number of students having more than 90 marks.
   114 
   128 
   115 ------------------------------------------------------------------------------------------------------------------
   129 ------------------------------------------------------------------------------------------------------------------
   116 
   130 
   145        	  if yes we add 1 to the value of dictionary for that region by
   159        	  if yes we add 1 to the value of dictionary for that region by
   146        	  science[region_code] += 1
   160        	  science[region_code] += 1
   147 
   161 
   148     Hit return twice to exit the for loop
   162     Hit return twice to exit the for loop
   149 
   163 
   150 by end of this loop we will have our desired output in the dictionary 'science'
   164 by end of this loop we shall have our desired output in the dictionary 'science'
   151 we can check the values by
   165 we can check the values by
   152 science
   166 science
   153 
   167 
   154 now to create a pie chart we use
   168 now to create a pie chart we use
   155 
   169