|
1 Hello friends and welcome to the tutorial on statistics using Python |
|
2 |
|
3 {{{ Show the slide containing title }}} |
|
4 |
|
5 {{{ Show the slide containing the outline slide }}} |
|
6 |
|
7 In this tutorial, we shall learn |
|
8 * Doing simple statistical operations in Python |
|
9 * Applying these to real world problems |
|
10 |
|
11 You will need Ipython with pylab running on your computer |
|
12 to use this tutorial. |
|
13 |
|
14 Also you will need to know about loading data using loadtxt to be |
|
15 able to follow the real world application. |
|
16 |
|
17 We will first start with the most necessary statistical |
|
18 operation i.e finding mean. |
|
19 |
|
20 We have a list of ages of a random group of people :: |
|
21 |
|
22 age_list=[4,45,23,34,34,38,65,42,32,7] |
|
23 |
|
24 One way of getting the mean could be getting sum of |
|
25 all the elements and dividing by length of the list.:: |
|
26 |
|
27 sum_age_list =sum(age_list) |
|
28 |
|
29 sum function gives us the sum of the elements.:: |
|
30 |
|
31 mean_using_sum=float(sum_age_list)/len(age_list) |
|
32 |
|
33 This obviously gives the mean age but python has another |
|
34 method for getting the mean. This is the mean function:: |
|
35 |
|
36 mean(age_list) |
|
37 |
|
38 Mean can be used in more ways in case of 2 dimensional lists. |
|
39 Take a two dimensional list :: |
|
40 |
|
41 two_dimension=[[1,5,6,8],[1,3,4,5]] |
|
42 |
|
43 the mean function used in default manner will give the mean of the |
|
44 flattened sequence. Flattened sequence means the two lists taken |
|
45 as if it was a single list of elements :: |
|
46 |
|
47 mean(two_dimension) |
|
48 flattened_seq=[1,5,6,8,1,3,4,5] |
|
49 mean(flattened_seq) |
|
50 |
|
51 As you can see both the results are same. The other way is mean |
|
52 of each column.:: |
|
53 |
|
54 mean(two_dimension,0) |
|
55 array([ 1. , 4. , 5. , 6.5]) |
|
56 |
|
57 we pass an extra argument 0 in that case. |
|
58 |
|
59 In case of getting mean along the rows the argument is 1:: |
|
60 |
|
61 mean(two_dimension,1) |
|
62 array([ 5. , 3.25]) |
|
63 |
|
64 We can see more option of mean using :: |
|
65 |
|
66 mean? |
|
67 |
|
68 Similarly we can calculate median and stanard deviation of a list |
|
69 using the functions median and std:: |
|
70 |
|
71 median(age_list) |
|
72 std(age_list) |
|
73 |
|
74 Median and std can also be calculated for two dimensional arrays along columns and rows just like mean. |
|
75 |
|
76 For example :: |
|
77 |
|
78 median(two_dimension,0) |
|
79 std(two_dimension,1) |
|
80 |
|
81 This gives us the median along the colums and standard devition along the rows. |
|
82 |
|
83 Now lets apply this to a real world example |
|
84 |
|
85 We will a data file that is at the a path |
|
86 ``/home/fossee/sslc2.txt``.It contains record of students and their |
|
87 performance in one of the State Secondary Board Examination. It has |
|
88 180, 000 lines of record. We are going to read it and process this |
|
89 data. We can see the content of file by double clicking on it. It |
|
90 might take some time to open since it is quite a large file. Please |
|
91 don't edit the data. This file has a particular structure. |
|
92 |
|
93 We can do :: |
|
94 |
|
95 cat /home/fossee/sslc2.txt |
|
96 |
|
97 to check the contents of the file. |
|
98 |
|
99 Each line in the file is a set of 11 fields separated |
|
100 by semi-colons Consider a sample line from this file. |
|
101 A;015163;JOSEPH RAJ S;083;042;47;00;72;244;;; |
|
102 |
|
103 The following are the fields in any given line. |
|
104 * Region Code which is 'A' |
|
105 * Roll Number 015163 |
|
106 * Name JOSEPH RAJ S |
|
107 * Marks of 5 subjects: ** English 083 ** Hindi 042 ** Maths 47 ** |
|
108 Science AA (Absent) ** Social 72 |
|
109 * Total marks 244 |
|
110 * |
|
111 |
|
112 Now lets try and find the mean of English marks of all students. |
|
113 |
|
114 For this we do. :: |
|
115 |
|
116 L=loadtxt('/home/fossee/sslc2.txt',usecols=(3,),delimiter=';') |
|
117 L |
|
118 mean(L) |
|
119 |
|
120 loadtxt function loads data from an external file.Delimiter specifies |
|
121 the kind of character are the fields of data seperated by. |
|
122 usecols specifies the columns to be used so (3,). The 'comma' is added |
|
123 because usecols is a sequence. |
|
124 |
|
125 To get the median marks. :: |
|
126 |
|
127 median(L) |
|
128 |
|
129 Standard deviation. :: |
|
130 |
|
131 std(L) |
|
132 |
|
133 |
|
134 Now lets try and and get the mean for all the subjects :: |
|
135 |
|
136 L=loadtxt('/home/fossee/sslc2.txt',usecols=(3,4,5,6,7),delimiter=';') |
|
137 mean(L,0) |
|
138 array([ 73.55452504, 53.79828941, 62.83342759, 50.69806158, 63.17056881]) |
|
139 |
|
140 As we can see from the result mean(L,0). The resultant sequence |
|
141 is the mean marks of all students that gave the exam for the five subjects. |
|
142 |
|
143 and :: |
|
144 |
|
145 mean(L,1) |
|
146 |
|
147 |
|
148 is the average accumalative marks of individual students. Clearly, mean(L,0) |
|
149 was a row wise calcultaion while mean(L,1) was a column wise calculation. |
|
150 |
|
151 |
|
152 {{{ Show summary slide }}} |
|
153 |
|
154 This brings us to the end of the tutorial. |
|
155 we have learnt |
|
156 |
|
157 * How to do the standard statistical operations sum , mean |
|
158 median and standard deviation in Python. |
|
159 * Combine text loading and the statistical operation to solve |
|
160 real world problems. |
|
161 |
|
162 {{{ Show the "sponsored by FOSSEE" slide }}} |
|
163 |
|
164 |
|
165 This tutorial was created as a part of FOSSEE project, NME ICT, MHRD India |
|
166 |
|
167 Hope you have enjoyed and found it useful. |
|
168 Thankyou |
|
169 |
|
170 .. Author : Amit Sethi |
|
171 Internal Reviewer 1 : |
|
172 Internal Reviewer 2 : |
|
173 External Reviewer : |
|
174 |