31  * Doing simple statistical operations in Python  

    33  * Doing statistical operations in Python  

    32  * Applying these to real world problems 

    34    * Summing set of numbers

    35    * Finding there mean

    36    * Finding there Median

    37    * Finding there Standard Deviation 

    38    

    48 We will first start with the most necessary statistical operation i.e

    54 For this tutorial We will use data file that is at the a path

    49 finding mean.

    55 ``/home/fossee/sslc2.txt``.  It contains record of students and their

    50 

    56 performance in one of the State Secondary Board Examination. It has

    51 We have a list of ages of a random group of people ::

    57 180,000 lines of record. We are going to read it and process this

    52    

    58 data.  We can see the content of file by double clicking on it. It

    53    age_list = [4,45,23,34,34,38,65,42,32,7]

    59 might take some time to open since it is quite a large file.  Please

    54 

    60 don't edit the data.  This file has a particular structure.

    55 One way of getting the mean could be getting sum of all the ages and

    56 dividing by the number of people in the group. ::

    57 

    58     sum_age_list = sum(age_list)

    59 

    60 sum function gives us the sum of the elements. Note that the

    61 ``sum_age_list`` variable is an integer and the number of people or

    62 length of the list is also an integer. We will need to convert one of

    63 them to a float before carrying out the division. ::

    64 

    65     mean_using_sum = float(sum_age_list)/len(age_list)

    66 

    67 This obviously gives the mean age but there is a simpler way to do

    68 this in Python - using the mean function::

    69 

    70        mean(age_list)

    71 

    72 Mean can be used in more ways in case of 2 dimensional lists.  Take a

    73 two dimensional list ::

    74      

    75      two_dimension=[[1,5,6,8],[1,3,4,5]]

    76 

    77 The mean function by default gives the mean of the flattened sequence.

    78 A Flattened sequence means a list obtained by concatenating all the

    79 smaller lists into a large long list. In this case, the list obtained

    80 by writing the two lists one after the other. ::

    81 

    82     mean(two_dimension)

    83     flattened_seq=[1,5,6,8,1,3,4,5]

    84     mean(flattened_seq)

    85 

    86 As you can see both the results are same. ``mean`` function can also

    87 give us the mean of each column, or the mean of corresponding elements

    88 in the smaller lists. ::

    89    

    90    mean(two_dimension, 0)

    91    array([ 1. ,  4. ,  5. ,  6.5])

    92 

    93 we pass an extra argument 0 in that case.

    94 

    95 If we use an argument 1, we obtain the mean along the rows. ::

    96    

    97    mean(two_dimension, 1)

    98    array([ 5.  ,  3.25])

    99 

   100 We can see more option of mean using ::

   101    

   102    mean?

   103 

   104 Similarly we can calculate median and stanard deviation of a list

   105 using the functions median and std::

   106       

   107       median(age_list)

   108       std(age_list)

   109 

   110 Median and std can also be calculated for two dimensional arrays along

   111 columns and rows just like mean.

   112 

   113 For example ::

   114        

   115        median(two_dimension, 0)

   116        std(two_dimension, 1)

   117 

   118 This gives us the median along the colums and standard devition along

   119 the rows.

   120        

   121 Now lets apply this to a real world example 

   122     

   123 We will a data file that is at the a path ``/home/fossee/sslc2.txt``.

   124 It contains record of students and their performance in one of the

   125 State Secondary Board Examination. It has 180, 000 lines of record. We

   126 are going to read it and process this data.  We can see the content of

   127 file by double clicking on it. It might take some time to open since

   128 it is quite a large file.  Please don't edit the data.  This file has

   129 a particular structure.

   150 Now lets try and find the mean of English marks of all students.

    84 Lets try and load this data as an array and then run various function on

    85 it.

   152 For this we do. ::

    87 To get the data as an array we do. ::

   153 

    88    

   154      L=loadtxt('/home/fossee/sslc2.txt',usecols=(3,),delimiter=';')

    89      L=loadtxt('/home/amit/sslc2.txt',usecols=(3,4,5,6,7,),delimiter=';')

   156      mean(L)

    91      

   159 the kind of character are the fields of data seperated by. 

    94 the kind of character are the fields of data seperated by.  usecols

   160 usecols specifies  the columns to be used so (3,). The 'comma' is added

    95 specifies the columns to be used so (3,4,5,6,7) loads those

   161 because usecols is a sequence.

    96 colums. The 'comma' is added because usecols is a sequence.

   163 To get the median marks. ::

    98 As we can see L is an array. We can get the shape of this array using::

   165     median(L)

   100    L.shape

   101    (185667, 5)

   102 

   103 Lets start applying statistics operations on these. We will start with

   104 the most basic, summing. How do you find the sum of marks of all

   105 subjects for the first student.

   106 

   107 As we know from our knowledge of accessing pieces of arrays. To acess

   108 the first row we will do ::

   167 Standard deviation. ::

   110    L[0,:]

   168 	

   169     std(L)

   172 Now lets try and and get the mean for all the subjects ::

   114     totalmarks=sum(L[0,:]) 

   115     totalmarks

   174      L=loadtxt('/home/fossee/sslc2.txt',usecols=(3,4,5,6,7),delimiter=';')

   117 To get the mean we can do ::

   175      mean(L,0)

   176      array([ 73.55452504,  53.79828941,  62.83342759,  50.69806158,  63.17056881])

   178 As we can see from the result mean(L,0). The resultant sequence  

   119    totalmarks/len(L[0,:])

   179 is the mean marks of all students that gave the exam for the five subjects.

   181 and ::

   121 or simply ::

   182     

   122 

   123    mean(L[0,:])

   124 

   125 But we have such a large data set calculating one by one the mean of

   126 each student is impossible. Is there a way to reduce the work.

   127 

   128 For this we will look into the documentation of mean by doing::

   129 

   130     mean?

   131 

   132 As we know L is a two dimensional array. We can calculate the mean

   133 across each of the axis of the array. The axis of rows is referred by

   134 number 0 and columns by 1. So to calculate mean accross all colums we

   135 will pass extra parameter 1 for the axis.::

   136 

   185     

   139 L here is the two dimensional array.

   186 is the average accumalative marks of individual students. Clearly, mean(L,0)

   187 was a row wise calcultaion while mean(L,1) was a column wise calculation.