+
−Comment Example
+
−===============
+
−
+
−.. contents::
+
−
+
−Introduction
+
−------------
+
−
+
−This is an example of how to write WSGI middleware with WebOb.  The
+
−specific example adds a simple comment form to HTML web pages; any
+
−page served through the middleware that is HTML gets a comment form
+
−added to it, and shows any existing comments.
+
−
+
−Code
+
−----
+
−
+
−The finished code for this is available in
+
−`docs/comment-example-code/example.py
+
−<http://svn.pythonpaste.org/Paste/WebOb/trunk/docs/comment-example-code/example.py>`_
+
−-- you can run that file as a script to try it out.
+
−
+
−Instantiating Middleware
+
−------------------------
+
−
+
−Middleware of any complexity at all is usually best created as a
+
−class with its configuration as arguments to that class.
+
−
+
−Every middleware needs an application (``app``) that it wraps.  This
+
−middleware also needs a location to store the comments; we'll put them
+
−all in a single directory.
+
−
+
−.. code-block::
+
−
+
−    import os
+
−    
+
−    class Commenter(object):
+
−        def __init__(self, app, storage_dir):
+
−            self.app = app
+
−            self.storage_dir = storage_dir
+
−            if not os.path.exists(storage_dir):
+
−                os.makedirs(storage_dir)
+
−
+
−When you use this middleware, you'll use it like:
+
−
+
−.. code-block::
+
−
+
−    app = ... make the application ...
+
−    app = Commenter(app, storage_dir='./comments')
+
−
+
−For our application we'll use a simple static file server that is
+
−included with `Paste <http://pythonpaste.org>`_ (use ``easy_install
+
−Paste`` to install this).  The setup is all at the bottom of
+
−``example.py``, and looks like this:
+
−
+
−.. code-block::
+
−
+
−    if __name__ == '__main__':
+
−        import optparse
+
−        parser = optparse.OptionParser(
+
−            usage='%prog --port=PORT BASE_DIRECTORY'
+
−            )
+
−        parser.add_option(
+
−            '-p', '--port',
+
−            default='8080',
+
−            dest='port',
+
−            type='int',
+
−            help='Port to serve on (default 8080)')
+
−        parser.add_option(
+
−            '--comment-data',
+
−            default='./comments',
+
−            dest='comment_data',
+
−            help='Place to put comment data into (default ./comments/)')
+
−        options, args = parser.parse_args()
+
−        if not args:
+
−            parser.error('You must give a BASE_DIRECTORY')
+
−        base_dir = args[0]
+
−        from paste.urlparser import StaticURLParser
+
−        app = StaticURLParser(base_dir)
+
−        app = Commenter(app, options.comment_data)
+
−        from wsgiref.simple_server import make_server
+
−        httpd = make_server('localhost', options.port, app)
+
−        print 'Serving on http://localhost:%s' % options.port
+
−        try:
+
−            httpd.serve_forever()
+
−        except KeyboardInterrupt:
+
−            print '^C'
+
−
+
−I won't explain it here, but basically it takes some options, creates
+
−an application that serves static files
+
−(``StaticURLParser(base_dir)``), wraps it with ``Commenter(app,
+
−options.comment_data)`` then serves that.
+
−
+
−The Middleware
+
−--------------
+
−
+
−While we've created the class structure for the middleware, it doesn't
+
−actually do anything.  Here's a kind of minimal version of the
+
−middleware (using WebOb):
+
−
+
−.. code-block::
+
−
+
−    from webob import Request
+
−
+
−    class Commenter(object):
+
−
+
−        def __init__(self, app, storage_dir):
+
−            self.app = app
+
−            self.storage_dir = storage_dir
+
−            if not os.path.exists(storage_dir):
+
−                os.makedirs(storage_dir)
+
−
+
−        def __call__(self, environ, start_response):
+
−            req = Request(environ)
+
−            resp = req.get_response(self.app)
+
−            return resp(environ, start_response)
+
−
+
−This doesn't modify the response it any way.  You could write it like
+
−this without WebOb:
+
−
+
−.. code-block::
+
−
+
−    class Commenter(object):
+
−        ...
+
−        def __call__(self, environ, start_response):
+
−            return self.app(environ, start_response)
+
−
+
−But it won't be as convenient later.  First, lets create a little bit
+
−of infrastructure for our middleware.  We need to save and load
+
−per-url data (the comments themselves).  We'll keep them in pickles,
+
−where each url has a pickle named after the url (but double-quoted, so
+
−``http://localhost:8080/index.html`` becomes
+
−``http%3A%2F%2Flocalhost%3A8080%2Findex.html``).
+
−
+
−.. code-block::
+
−
+
−    from cPickle import load, dump
+
−
+
−    class Commenter(object):
+
−        ...
+
−
+
−        def get_data(self, url):
+
−            filename = self.url_filename(url)
+
−            if not os.path.exists(filename):
+
−                return []
+
−            else:
+
−                f = open(filename, 'rb')
+
−                data = load(f)
+
−                f.close()
+
−                return data
+
−
+
−        def save_data(self, url, data):
+
−            filename = self.url_filename(url)
+
−            f = open(filename, 'wb')
+
−            dump(data, f)
+
−            f.close()
+
−
+
−        def url_filename(self, url):
+
−            # Double-quoting makes the filename safe
+
−            return os.path.join(self.storage_dir, urllib.quote(url, ''))
+
−
+
−You can get the full request URL with ``req.url``, so to get the
+
−comment data with these methods you do ``data =
+
−self.get_data(req.url)``.
+
−
+
−Now we'll update the ``__call__`` method to filter *some* responses,
+
−and get the comment data for those.  We don't want to change responses
+
−that were error responses (anything but ``200``), nor do we want to
+
−filter responses that aren't HTML.  So we get:
+
−
+
−.. code-block::
+
−
+
−    class Commenter(object):
+
−        ...
+
−
+
−        def __call__(self, environ, start_response):
+
−            req = Request(environ)
+
−            resp = req.get_response(self.app)
+
−            if resp.content_type != 'text/html' or resp.status_int != 200:
+
−                return resp(environ, start_response)
+
−            data = self.get_data(req.url)
+
−            ... do stuff with data, update resp ...
+
−            return resp(environ, start_response)
+
−
+
−So far we're punting on actually adding the comments to the page.  We
+
−also haven't defined what ``data`` will hold.  Let's say it's a list
+
−of dictionaries, where each dictionary looks like ``{'name': 'John
+
−Doe', 'homepage': 'http://blog.johndoe.com', 'comments': 'Great
+
−site!'}``.
+
−
+
−We'll also need a simple method to add stuff to the page.  We'll use a
+
−regular expression to find the end of the page and put text in:
+
−
+
−.. code-block::
+
−
+
−    import re
+
−
+
−    class Commenter(object):
+
−        ...
+
−
+
−        _end_body_re = re.compile(r'</body.*?>', re.I|re.S)
+
−
+
−        def add_to_end(self, html, extra_html):
+
−            """
+
−            Adds extra_html to the end of the html page (before </body>)
+
−            """
+
−            match = self._end_body_re.search(html)
+
−            if not match:
+
−                return html + extra_html
+
−            else:
+
−                return html[:match.start()] + extra_html + html[match.start():]
+
−
+
−And then we'll use it like:
+
−
+
−.. code-block::
+
−
+
−    data = self.get_data(req.url)
+
−    body = resp.body
+
−    body = self.add_to_end(body, self.format_comments(data))
+
−    resp.body = body
+
−    return resp(environ, start_response)
+
−
+
−We get the body, update it, and put it back in the response.  This
+
−also updates ``Content-Length``.  Then we define:
+
−
+
−.. code-block::
+
−
+
−    from webob import html_escape
+
−
+
−    class Commenter(object):
+
−        ...
+
−
+
−        def format_comments(self, comments):
+
−            if not comments:
+
−                return ''
+
−            text = []
+
−            text.append('<hr>')
+
−            text.append('<h2><a name="comment-area"></a>Comments (%s):</h2>' % len(comments))
+
−            for comment in comments:
+
−                text.append('<h3><a href="%s">%s</a> at %s:</h3>' % (
+
−                    html_escape(comment['homepage']), html_escape(comment['name']), 
+
−                    time.strftime('%c', comment['time'])))
+
−                # Susceptible to XSS attacks!:
+
−                text.append(comment['comments'])
+
−            return ''.join(text)
+
−
+
−We put in a header (with an anchor we'll use later), and a section for
+
−each comment.  Note that ``html_escape`` is the same as ``cgi.escape``
+
−and just turns ``&`` into ``&amp;``, etc.  
+
−
+
−Because we put in some text without quoting it is susceptible to a
+
−`Cross-Site Scripting
+
−<http://en.wikipedia.org/wiki/Cross-site_scripting>`_ attack.  Fixing
+
−that is beyond the scope of this tutorial; you could quote it or clean
+
−it with something like `lxml.html.clean
+
−<http://codespeak.net/lxml/lxmlhtml.html#cleaning-up-html>`_.
+
−
+
−Accepting Comments
+
−------------------
+
−
+
−All of those pieces *display* comments, but still no one can actually
+
−make comments.  To handle this we'll take a little piece of the URL
+
−space for our own, everything under ``/.comments``, so when someone
+
−POSTs there it will add a comment.
+
−
+
−When the request comes in there are two parts to the path:
+
−``SCRIPT_NAME`` and ``PATH_INFO``.  Everything in ``SCRIPT_NAME`` has
+
−already been parsed, and everything in ``PATH_INFO`` has yet to be
+
−parsed.  That means that the URL *without* ``PATH_INFO`` is the path
+
−to the middleware; we can intercept anything else below
+
−``SCRIPT_NAME`` but nothing above it.  The name for the URL without
+
−``PATH_INFO`` is ``req.application_url``.  We have to capture it early
+
−to make sure it doesn't change (since the WSGI application we are
+
−wrapping may update ``SCRIPT_NAME`` and ``PATH_INFO``).
+
−
+
−So here's what this all looks like:
+
−
+
−.. code-block::
+
−
+
−    class Commenter(object):
+
−        ...
+
−
+
−        def __call__(self, environ, start_response):
+
−            req = Request(environ)
+
−            if req.path_info_peek() == '.comments':
+
−                return self.process_comment(req)(environ, start_response)
+
−            # This is the base path of *this* middleware:
+
−            base_url = req.application_url
+
−            resp = req.get_response(self.app)
+
−            if resp.content_type != 'text/html' or resp.status_int != 200:
+
−                # Not an HTML response, we don't want to
+
−                # do anything to it
+
−                return resp(environ, start_response)
+
−            # Make sure the content isn't gzipped:
+
−            resp.decode_content()
+
−            comments = self.get_data(req.url)
+
−            body = resp.body
+
−            body = self.add_to_end(body, self.format_comments(comments))
+
−            body = self.add_to_end(body, self.submit_form(base_url, req))
+
−            resp.body = body
+
−            return resp(environ, start_response)
+
−
+
−``base_url`` is the path where the middleware is located (if you run
+
−the example server, it will be ``http://localhost:PORT/``).  We use
+
−``req.path_info_peek()`` to look at the next segment of the URL --
+
−what comes after base_url.  If it is ``.comments`` then we handle it
+
−internally and don't pass the request on.
+
−
+
−We also put in a little guard, ``resp.decode_content()`` in case the
+
−application returns a gzipped response.
+
−
+
−Then we get the data, add the comments, add the *form* to make new
+
−comments, and return the result.
+
−
+
−submit_form
+
−~~~~~~~~~~~
+
−
+
−Here's what the form looks like:
+
−
+
−.. code-block::
+
−
+
−    class Commenter(object):
+
−        ...
+
−
+
−        def submit_form(self, base_path, req):
+
−            return '''<h2>Leave a comment:</h2>
+
−            <form action="%s/.comments" method="POST">
+
−             <input type="hidden" name="url" value="%s">
+
−             <table width="100%%">
+
−              <tr><td>Name:</td>
+
−                  <td><input type="text" name="name" style="width: 100%%"></td></tr>
+
−              <tr><td>URL:</td>
+
−                  <td><input type="text" name="homepage" style="width: 100%%"></td></tr>
+
−             </table>
+
−             Comments:<br>
+
−             <textarea name="comments" rows=10 style="width: 100%%"></textarea><br>
+
−             <input type="submit" value="Submit comment">
+
−            </form>
+
−            ''' % (base_path, html_escape(req.url))
+
−
+
−Nothing too exciting.  It submits a form with the keys ``url`` (the
+
−URL being commented on), ``name``, ``homepage``, and ``comments``.
+
−
+
−process_comment
+
−~~~~~~~~~~~~~~~
+
−
+
−If you look at the method call, what we do is call the method then
+
−treat the result as a WSGI application:
+
−
+
−.. code-block::
+
−
+
−    return self.process_comment(req)(environ, start_response)
+
−
+
−You could write this as:
+
−
+
−.. code-block::
+
−
+
−    response = self.process_comment(req)
+
−    return response(environ, start_response)
+
−
+
−A common pattern in WSGI middleware that *doesn't* use WebOb is to
+
−just do:
+
−
+
−.. code-block::
+
−
+
−    return self.process_comment(environ, start_response)
+
−
+
−But the WebOb style makes it easier to modify the response if you want
+
−to; modifying a traditional WSGI response/application output requires
+
−changing your logic flow considerably.
+
−
+
−Here's the actual processing code:
+
−
+
−.. code-block::
+
−
+
−    from webob import exc
+
−    from webob import Response
+
−
+
−    class Commenter(object):
+
−        ...
+
−
+
−        def process_comment(self, req):
+
−            try:
+
−                url = req.params['url']
+
−                name = req.params['name']
+
−                homepage = req.params['homepage']
+
−                comments = req.params['comments']
+
−            except KeyError, e:
+
−                resp = exc.HTTPBadRequest('Missing parameter: %s' % e)
+
−                return resp
+
−            data = self.get_data(url)
+
−            data.append(dict(
+
−                name=name,
+
−                homepage=homepage,
+
−                comments=comments,
+
−                time=time.gmtime()))
+
−            self.save_data(url, data)
+
−            resp = exc.HTTPSeeOther(location=url+'#comment-area')
+
−            return resp
+
−
+
−We either give a Bad Request response (if the form submission is
+
−somehow malformed), or a redirect back to the original page.
+
−
+
−The classes in ``webob.exc`` (like ``HTTPBadRequest`` and
+
−``HTTPSeeOther``) are Response subclasses that can be used to quickly
+
−create responses for these non-200 cases where the response body
+
−usually doesn't matter much.
+
−
+
−Conclusion
+
−----------
+
−
+
−This shows how to make response modifying middleware, which is
+
−probably the most difficult kind of middleware to write with WSGI --
+
−modifying the request is quite simple in comparison, as you simply
+
−update ``environ``.