author | Madhusudan.C.S <madhusudancs@gmail.com> |
Fri, 17 Sep 2010 18:17:05 +0530 | |
changeset 165 | 53df732199a1 |
parent 140 | bc023595e167 |
child 179 | 1d04b6c5ff44 |
child 194 | ca81c0a67c75 |
permissions | -rw-r--r-- |
133 | 1 |
Hello friends and welcome to the tutorial on Parsing Data |
2 |
||
3 |
{{{ Show the slide containing title }}} |
|
4 |
||
5 |
{{{ Show the slide containing the outline slide }}} |
|
6 |
||
7 |
In this tutorial, we shall learn |
|
8 |
||
9 |
* What is parsing data |
|
10 |
* the string operations required for parsing data |
|
11 |
* datatype conversion |
|
12 |
||
13 |
Lets us have a look at the problem |
|
14 |
||
15 |
{{{ Show the slide containing problem statement. }}} |
|
16 |
||
17 |
There is an input file containing huge no.of records. Each record corresponds |
|
18 |
to a student. |
|
19 |
||
20 |
{{{ show the slide explaining record structure }}} |
|
21 |
As you can see, each record consists of fields seperated by a ";". The first |
|
22 |
record is region code, then roll number, then name, marks of second language, |
|
23 |
first language, maths, science and social, total marks, pass/fail indicatd by P |
|
24 |
or F and finally W if with held and empty otherwise. |
|
25 |
||
26 |
Our job is to calculate the mean of all the maths marks in the region "B". |
|
27 |
||
28 |
#[Nishanth]: Please note that I am not telling anything about AA since they do |
|
29 |
not know about any if/else yet. |
|
30 |
||
31 |
||
32 |
Now what is parsing data. |
|
33 |
||
34 |
From the input file, we can see that there is data in the form of text. Hence |
|
35 |
parsing data is all about reading the data and converting it into a form which |
|
36 |
can be used for computations. In our case, that is numbers. |
|
37 |
||
38 |
We can clearly see that the problem involves reading files and tokenizing. |
|
39 |
||
137
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
40 |
Let us learn about tokenizing strings. Let us define a string first. Type |
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
41 |
:: |
133 | 42 |
|
43 |
line = "parse this string" |
|
44 |
||
137
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
45 |
We are now going to split this string on whitespace. |
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
46 |
:: |
133 | 47 |
|
48 |
line.split() |
|
49 |
||
50 |
As you can see, we get a list of strings. Which means, when split is called |
|
51 |
without any arguments, it splits on whitespace. In simple words, all the spaces |
|
52 |
are treated as one big space. |
|
53 |
||
54 |
split also can split on a string of our choice. This is acheived by passing |
|
137
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
55 |
that as an argument. But first lets define a sample record from the file. |
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
56 |
:: |
133 | 57 |
|
58 |
record = "A;015163;JOSEPH RAJ S;083;042;47;AA;72;244;;;" |
|
59 |
record.split(';') |
|
60 |
||
61 |
We can see that the string is split on ';' and we get each field seperately. |
|
62 |
We can also observe that an empty string appears in the list since there are |
|
63 |
two semi colons without anything in between. |
|
64 |
||
65 |
Hence split splits on whitespace if called without an argument and splits on |
|
66 |
the given argument if it is called with an argument. |
|
67 |
||
68 |
{{{ Pause here and try out the following exercises }}} |
|
69 |
||
70 |
%% 1 %% split the variable line using a space as argument. Is it same as |
|
71 |
splitting without an argument ? |
|
72 |
||
73 |
{{{ continue from paused state }}} |
|
74 |
||
75 |
We see that when we split on space, multiple whitespaces are not clubbed as one |
|
76 |
and there is an empty string everytime there are two consecutive spaces. |
|
77 |
||
78 |
Now that we know splitting a string, we can split the record and retreive each |
|
79 |
field seperately. But there is one problem. The region code "B" and a "B" |
|
80 |
surrounded by whitespace are treated as two different regions. We must find a |
|
81 |
way to remove all the whitespace around a string so that "B" and a "B" with |
|
82 |
white spaces are dealt as same. |
|
83 |
||
84 |
This is possible by using the =strip= method of strings. Let us define a |
|
137
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
85 |
string by typing |
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
86 |
:: |
133 | 87 |
|
88 |
unstripped = " B " |
|
89 |
unstripped.strip() |
|
90 |
||
91 |
We can see that strip removes all the whitespace around the sentence |
|
92 |
||
93 |
{{{ Pause here and try out the following exercises }}} |
|
94 |
||
95 |
%% 2 %% What happens to the white space inside the sentence when it is stripped |
|
96 |
||
97 |
{{{ continue from paused state }}} |
|
98 |
||
137
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
99 |
Type |
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
100 |
:: |
133 | 101 |
|
102 |
a_str = " white space " |
|
103 |
a_str.strip() |
|
104 |
||
105 |
We see that the whitespace inside the sentence is only removed and anything |
|
106 |
inside remains unaffected. |
|
107 |
||
108 |
By now we know enough to seperate fields from the record and to strip out any |
|
109 |
white space. The only road block we now have is conversion of string to float. |
|
110 |
||
111 |
The splitting and stripping operations are done on a string and their result is |
|
112 |
also a string. hence the marks that we have are still strings and mathematical |
|
113 |
operations are not possible. We must convert them into integers or floats |
|
114 |
||
115 |
We shall look at converting strings into floats. We define an float string |
|
137
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
116 |
first. Type |
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
117 |
:: |
133 | 118 |
|
119 |
mark_str = "1.25" |
|
120 |
mark = int(mark_str) |
|
140 | 121 |
type(mark_str) |
122 |
type(mark) |
|
133 | 123 |
|
124 |
We can see that string is converted to float. We can perform mathematical |
|
125 |
operations on them now. |
|
126 |
||
127 |
{{{ Pause here and try out the following exercises }}} |
|
128 |
||
129 |
%% 3 %% What happens if you do int("1.25") |
|
130 |
||
131 |
{{{ continue from paused state }}} |
|
132 |
||
133 |
It raises an error since converting a float string into integer directly is |
|
137
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
134 |
not possible. It involves an intermediate step of converting to float. |
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
135 |
:: |
133 | 136 |
|
137 |
dcml_str = "1.25" |
|
138 |
flt = float(dcml_str) |
|
139 |
flt |
|
140 |
number = int(flt) |
|
141 |
number |
|
142 |
||
143 |
Using =int= it is also possible to convert float into integers. |
|
144 |
||
145 |
Now that we have all the machinery required to parse the file, let us solve the |
|
146 |
problem. We first read the file line by line and parse each record. We see if |
|
137
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
147 |
the region code is B and store the marks accordingly. |
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
148 |
:: |
133 | 149 |
|
150 |
math_marks_B = [] # an empty list to store the marks |
|
151 |
for line in open("/home/fossee/sslc1.txt"): |
|
152 |
fields = line.split(";") |
|
153 |
||
154 |
region_code = fields[0] |
|
155 |
region_code_stripped = region_code.strip() |
|
156 |
||
157 |
math_mark_str = fields[5] |
|
158 |
math_mark = float(math_mark_str) |
|
159 |
||
160 |
if region_code == "AA": |
|
161 |
math_marks_B.append(math_mark) |
|
162 |
||
163 |
||
164 |
Now we have all the maths marks of region "B" in the list math_marks_B. |
|
137
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
165 |
To get the mean, we just have to sum the marks and divide by the length. |
fc545d07b0ff
added a newline before :: so that a colon does not appear in html
nishanth
parents:
134
diff
changeset
|
166 |
:: |
133 | 167 |
|
168 |
math_marks_mean = sum(math_marks_B) / len(math_marks_B) |
|
169 |
math_marks_mean |
|
170 |
||
171 |
{{{ Show summary slide }}} |
|
172 |
||
173 |
This brings us to the end of the tutorial. |
|
174 |
we have learnt |
|
134 | 175 |
|
133 | 176 |
* how to tokenize a string using various delimiters |
177 |
* how to get rid of extra white space around |
|
178 |
* how to convert from one type to another |
|
179 |
* how to parse input data and perform computations on it |
|
180 |
||
181 |
{{{ Show the "sponsored by FOSSEE" slide }}} |
|
182 |
||
183 |
#[Nishanth]: Will add this line after all of us fix on one. |
|
184 |
This tutorial was created as a part of FOSSEE project, NME ICT, MHRD India |
|
185 |
||
186 |
Hope you have enjoyed and found it useful. |
|
187 |
Thankyou |
|
188 |
||
189 |
.. Author : Nishanth |
|
190 |
Internal Reviewer 1 : |
|
191 |
Internal Reviewer 2 : |
|
192 |
External Reviewer : |