17 .. Author              : Puneeth 

    19 .. Author              : Amit Sethi

    18    Internal Reviewer   : Anoop Jacob Thomas<anoop@fossee.in>

    20    Internal Reviewer   : Puneeth

    31  * Doing simple statistical operations in Python  

    33  * Doing statistical operations in Python  

    32  * Applying these to real world problems 

    34    * Summing set of numbers

    35    * Finding there mean

    36    * Finding there Median

    37    * Finding there Standard Deviation 

    38    

    56 We will first start with the most necessary statistical operation i.e

    54 For this tutorial We will use data file that is at the a path

    57 finding mean.

    55 ``/home/fossee/sslc2.txt``.  It contains record of students and their

    58 

    56 performance in one of the State Secondary Board Examination. It has

    59 We have a list of ages of a random group of people ::

    57 180,000 lines of record. We are going to read it and process this

    60    

    58 data.  We can see the content of file by double clicking on it. It

    61    age_list = [4,45,23,34,34,38,65,42,32,7]

    59 might take some time to open since it is quite a large file.  Please

    62 

    60 don't edit the data.  This file has a particular structure.

    63 One way of getting the mean could be getting sum of all the ages and

    64 dividing by the number of people in the group. ::

    65 

    66     sum_age_list = sum(age_list)

    67 

    68 sum function gives us the sum of the elements. Note that the

    69 ``sum_age_list`` variable is an integer and the number of people or

    70 length of the list is also an integer. We will need to convert one of

    71 them to a float before carrying out the division. ::

    72 

    73     mean_using_sum = float(sum_age_list)/len(age_list)

    74 

    75 This obviously gives the mean age but there is a simpler way to do

    76 this in Python - using the mean function::

    77 

    78        mean(age_list)

    79 

    80 Mean can be used in more ways in case of 2 dimensional lists.  Take a

    81 two dimensional list ::

    82      

    83      two_dimension=[[1,5,6,8],[1,3,4,5]]

    84 

    85 The mean function by default gives the mean of the flattened sequence.

    86 A Flattened sequence means a list obtained by concatenating all the

    87 smaller lists into a large long list. In this case, the list obtained

    88 by writing the two lists one after the other. ::

    89 

    90     mean(two_dimension)

    91     flattened_seq=[1,5,6,8,1,3,4,5]

    92     mean(flattened_seq)

    93 

    94 As you can see both the results are same. ``mean`` function can also

    95 give us the mean of each column, or the mean of corresponding elements

    96 in the smaller lists. ::

    97    

    98    mean(two_dimension, 0)

    99    array([ 1. ,  4. ,  5. ,  6.5])

   100 

   101 we pass an extra argument 0 in that case.

   102 

   103 If we use an argument 1, we obtain the mean along the rows. ::

   104    

   105    mean(two_dimension, 1)

   106    array([ 5.  ,  3.25])

   107 

   108 We can see more option of mean using ::

   109    

   110    mean?

   111 

   112 Similarly we can calculate median and stanard deviation of a list

   113 using the functions median and std::

   114       

   115       median(age_list)

   116       std(age_list)

   117 

   118 Median and std can also be calculated for two dimensional arrays along

   119 columns and rows just like mean.

   120 

   121 For example ::

   122        

   123        median(two_dimension, 0)

   124        std(two_dimension, 1)

   125 

   126 This gives us the median along the colums and standard devition along

   127 the rows.

   128        

   129 Now lets apply this to a real world example 

   130     

   131 We will a data file that is at the a path ``/home/fossee/sslc2.txt``.

   132 It contains record of students and their performance in one of the

   133 State Secondary Board Examination. It has 180, 000 lines of record. We

   134 are going to read it and process this data.  We can see the content of

   135 file by double clicking on it. It might take some time to open since

   136 it is quite a large file.  Please don't edit the data.  This file has

   137 a particular structure.

   158 Now lets try and find the mean of English marks of all students.

    84 Lets try and load this data as an array and then run various function on

    85 it.

   160 For this we do. ::

    87 To get the data as an array we do. ::

   161 

    88    

   162      L=loadtxt('/home/fossee/sslc2.txt',usecols=(3,),delimiter=';')

    89      L=loadtxt('/home/amit/sslc2.txt',usecols=(3,4,5,6,7,),delimiter=';')

   164      mean(L)

    91      

   167 the kind of character are the fields of data seperated by. 

    94 the kind of character are the fields of data seperated by.  usecols

   168 usecols specifies  the columns to be used so (3,). The 'comma' is added

    95 specifies the columns to be used so (3,4,5,6,7) loads those

   169 because usecols is a sequence.

    96 colums. The 'comma' is added because usecols is a sequence.

   171 To get the median marks. ::

    98 As we can see L is an array. We can get the shape of this array using::

   173     median(L)

   100    L.shape

   101    (185667, 5)

   102 

   103 Lets start applying statistics operations on these. We will start with

   104 the most basic, summing. How do you find the sum of marks of all

   105 subjects for the first student.

   106 

   107 As we know from our knowledge of accessing pieces of arrays. To acess

   108 the first row we will do ::

   175 Standard deviation. ::

   110    L[0,:]

   176 	

   177     std(L)

   180 Now lets try and and get the mean for all the subjects ::

   114     totalmarks=sum(L[0,:]) 

   115     totalmarks

   182      L=loadtxt('/home/fossee/sslc2.txt',usecols=(3,4,5,6,7),delimiter=';')

   117 To get the mean we can do ::

   183      mean(L,0)

   184      array([ 73.55452504,  53.79828941,  62.83342759,  50.69806158,  63.17056881])

   186 As we can see from the result mean(L,0). The resultant sequence  

   119    totalmarks/len(L[0,:])

   187 is the mean marks of all students that gave the exam for the five subjects.

   189 and ::

   121 or simply ::

   190     

   122 

   123    mean(L[0,:])

   124 

   125 But we have such a large data set calculating one by one the mean of

   126 each student is impossible. Is there a way to reduce the work.

   127 

   128 For this we will look into the documentation of mean by doing::

   129 

   130     mean?

   131 

   132 As we know L is a two dimensional array. We can calculate the mean

   133 across each of the axis of the array. The axis of rows is referred by

   134 number 0 and columns by 1. So to calculate mean accross all colums we

   135 will pass extra parameter 1 for the axis.::

   136 

   193     

   139 L here is the two dimensional array.

   194 is the average accumalative marks of individual students. Clearly, mean(L,0)

   195 was a row wise calcultaion while mean(L,1) was a column wise calculation.