app/django/utils/stopwords.py
author Sverre Rabbelier <srabbelier@gmail.com>
Sat, 13 Dec 2008 13:28:01 +0000
changeset 736 c70d56182ce2
parent 54 03e267d67478
permissions -rw-r--r--
No override needed in notifications.py The only downside is that the ?s=0 parameter will still be appended but is not handled in list(). The right solution would be to update list() so that it does say "Message Sent" or such, to give the user an indication that their message was sent succesfully. Patch by: Sverre Rabbelier

# Performance note: I benchmarked this code using a set instead of
# a list for the stopwords and was surprised to find that the list
# performed /better/ than the set - maybe because it's only a small
# list.

stopwords = '''
i
a
an
are
as
at
be
by
for
from
how
in
is
it
of
on
or
that
the
this
to
was
what
when
where
'''.split()

def strip_stopwords(sentence):
    "Removes stopwords - also normalizes whitespace"
    words = sentence.split()
    sentence = []
    for word in words:
        if word.lower() not in stopwords:
            sentence.append(word)
    return u' '.join(sentence)