WebOb Reference

+++++++++++++++

.. contents::

.. comment:

    >>> from dtopt import ELLIPSIS

Introduction

============

This document covers all the details of the Request and Response

objects.  It is written to be testable with `doctest

<http://python.org/doc/current/lib/module-doctest.html>`_ -- this

effects the flavor of the documentation, perhaps to its detriment.

But it also means you can feel confident that the documentation is

correct.

This is a somewhat different approach to reference documentation

compared to the extracted documentation for the `request

<class-webob.Request.html>`_ and `response

<class-webob.Response.html>`_.

Request

=======

The primary object in WebOb is ``webob.Request``, a wrapper around a

`WSGI environment <http://www.python.org/dev/peps/pep-0333/>`_.

The basic way you create a request object is simple enough:

.. code-block::

   >>> from webob import Request

   >>> environ = {}

   >>> req = Request(environ)

The request object *wraps* the environment; it has very little

internal state of its own.  Instead attributes you access read and

write to the environment dictionary.

You don't have to understand the details of WSGI to use this library;

this library handles those details for you.  You also don't have to

use this exclusively of other libraries.  If those other libraries

also keep their state in the environment, multiple wrappers can

coexist.  Examples of libraries that can coexist include

`paste.wsgiwrappers.Request

<http://pythonpaste.org/class-paste.wsgiwrappers.WSGIRequest.html>`_

(used by Pylons) and `yaro.Request

<http://lukearno.com/projects/yaro/>`_.

The WSGI environment has a number of required variables.  To make it

easier to test and play around with, the ``Request`` class has a

constructor that will fill in a minimal environment:

.. code-block::

   >>> req = Request.blank('/article?id=1')

   >>> from pprint import pprint

   >>> pprint(req.environ)

   {'HTTP_HOST': 'localhost:80',

    'PATH_INFO': '/article',

    'QUERY_STRING': 'id=1',

    'REQUEST_METHOD': 'GET',

    'SCRIPT_NAME': '',

    'SERVER_NAME': 'localhost',

    'SERVER_PORT': '80',

    'SERVER_PROTOCOL': 'HTTP/1.0',

    'wsgi.errors': <open file '<stderr>', mode 'w' at ...>,

    'wsgi.input': <cStringIO.StringI object at ...>,

    'wsgi.multiprocess': False,

    'wsgi.multithread': False,

    'wsgi.run_once': False,

    'wsgi.url_scheme': 'http',

    'wsgi.version': (1, 0)}

Request Body

------------

``req.body`` is a file-like object that gives the body of the request

(e.g., a POST form, the body of a PUT, etc).  It's kind of boring to

start, but you can set it to a string and that will be turned into a

file-like object.  You can read the entire body with

``req.body``.

.. code-block:: 

    >>> req.body_file

    <cStringIO.StringI object at ...>

    >>> req.body

    ''

    >>> req.body = 'test'

    >>> req.body_file

    <cStringIO.StringI object at ...>

    >>> req.body

    'test'

Method & URL

------------

All the normal parts of a request are also accessible through the

request object:

.. code-block::

    >>> req.method

    'GET'

    >>> req.scheme

    'http'

    >>> req.script_name  # The base of the URL

    ''

    >>> req.script_name = '/blog' # make it more interesting

    >>> req.path_info    # The yet-to-be-consumed part of the URL

    '/article'

    >>> req.content_type # Content-Type of the request body

    ''

    >>> print req.remote_user  # The authenticated user (there is none set)

    None

    >>> print req.remote_addr  # The remote IP

    None

    >>> req.host

    'localhost:80'

    >>> req.host_url

    'http://localhost'

    >>> req.application_url

    'http://localhost/blog'

    >>> req.path_url

    'http://localhost/blog/article'

    >>> req.url

    'http://localhost/blog/article?id=1'

    >>> req.path

    '/blog/article'

    >>> req.path_qs

    '/blog/article?id=1'

    >>> req.query_string

    'id=1'

You can make new URLs:

.. code-block::

    >>> req.relative_url('archive')

    'http://localhost/blog/archive'

For parsing the URLs, it is often useful to deal with just the next

path segment on PATH_INFO:

.. code-block::

    >>> req.path_info_peek() # Doesn't change request

    'article'

    >>> req.path_info_pop()  # Does change request!

    'article'

    >>> req.script_name

    '/blog/article'

    >>> req.path_info

    ''

Headers

-------

All request headers are available through a dictionary-like object

``req.headers``.  Keys are case-insensitive.

.. code-block::

    >>> req.headers['content-type'] = 'application/x-www-urlencoded'

    >>> req.headers

    {'Content-Length': '4', 'Content-Type': 'application/x-www-urlencoded', 'Host': 'localhost:80'}

    >>> req.environ['CONTENT_TYPE']

    'application/x-www-urlencoded'

Query & POST variables

----------------------

Requests can have variables in one of two locations: the query string

(``?id=1``), or in the body of the request (generally a POST form).

Note that even POST requests can have a query string, so both kinds of

variables can exist at the same time.  Also, a variable can show up

more than once, as in ``?check=a&check=b``.

For these variables WebOb uses a `MultiDict

<class-webob.multidict.MultiDict.html>`_, which is basically a

dictionary wrapper on a list of key/value pairs.  It looks like a

single-valued dictionary, but you can access all the values of a key

with ``.getall(key)`` (which always returns a list, possibly an empty

list).  You also get all key/value pairs when using ``.items()`` and

all values with ``.values()``.

Some examples:

.. code-block::

    >>> req = Request.blank('/test?check=a&check=b&name=Bob')

    >>> req.GET

    MultiDict([('check', 'a'), ('check', 'b'), ('name', 'Bob')])

    >>> req.GET['check']

    'b'

    >>> req.GET.getall('check')

    ['a', 'b']

    >>> req.GET.items()

    [('check', 'a'), ('check', 'b'), ('name', 'Bob')]

We'll have to create a request body and change the method to get

POST.  Until we do that, the variables are boring:

.. code-block::

    >>> req.POST

    <NoVars: Not a POST request>

    >>> req.POST.items()  # NoVars can be read like a dict, but not written

    []

    >>> req.method = 'POST'

    >>> req.body = 'name=Joe&email=joe@example.com'

    >>> req.POST

    MultiDict([('name', 'Joe'), ('email', 'joe@example.com')])

    >>> req.POST['name']

    'Joe'

Often you won't care where the variables come from.  (Even if you care

about the method, the location of the variables might not be

important.)  There is a dictionary called ``req.params`` that

contains variables from both sources:

.. code-block::

    >>> req.params

    NestedMultiDict([('check', 'a'), ('check', 'b'), ('name', 'Bob'), ('name', 'Joe'), ('email', 'joe@example.com')])

    >>> req.params['name']

    'Bob'

    >>> req.params.getall('name')

    ['Bob', 'Joe']

    >>> for name, value in req.params.items():

    ...     print '%s: %r' % (name, value)

    check: 'a'

    check: 'b'

    name: 'Bob'

    name: 'Joe'

    email: 'joe@example.com'

Unicode Variables

~~~~~~~~~~~~~~~~~

Submissions are non-unicode (``str``) strings, unless some character

set is indicated.  A client can indicate the character set with

``Content-Type: application/x-www-form-urlencoded; charset=utf8``, but

very few clients actually do this (sometimes XMLHttpRequest requests

will do this, as JSON is always UTF8 even when a page is served with a

different character set).  You can force a charset, which will effect

all the variables:

.. code-block::

    >>> req.charset = 'utf8'

    >>> req.GET

    UnicodeMultiDict([(u'check', u'a'), (u'check', u'b'), (u'name', u'Bob')])

If you always want ``str`` values, you can use ``req.str_GET``

and ``str_POST``.

Cookies

-------

Cookies are presented in a simple dictionary.  Like other variables,

they will be decoded into Unicode strings if you set the charset.

.. code-block::

    >>> req.headers['Cookie'] = 'test=value'

    >>> req.cookies

    UnicodeMultiDict([(u'test', u'value')])

    >>> req.charset = None

    >>> req.cookies

    {'test': 'value'}

Modifying the request

---------------------

The headers are all modifiable, as are other environmental variables

(like ``req.remote_user``, which maps to

``request.environ['REMOTE_USER']``).

If you want to copy the request you can use ``req.copy()``; this

copies the ``environ`` dictionary, and the request body from

``environ['wsgi.input']``.

The method ``req.remove_conditional_headers(remove_encoding=True)``

can be used to remove headers that might result in a ``304 Not

Modified`` response.  If you are writing some intermediary it can be

useful to avoid these headers.  Also if ``remove_encoding`` is true

(the default) then any ``Accept-Encoding`` header will be removed,

which can result in gzipped responses.

Header Getters

--------------

In addition to ``req.headers``, there are attributes for most of the

request headers defined by the HTTP 1.1 specification.  These

attributes often return parsed forms of the headers.

Accept-* headers

~~~~~~~~~~~~~~~~

There are several request headers that tell the server what the client

accepts.  These are ``accept`` (the Content-Type that is accepted),

``accept_charset`` (the charset accepted), ``accept_encoding``

(the Content-Encoding, like gzip, that is accepted), and

``accept_language`` (generally the preferred language of the client).

The objects returned support containment to test for acceptability.

E.g.:

.. code-block::

    >>> 'text/html' in req.accept

    True

Because no header means anything is potentially acceptable, this is

returning True.  We can set it to see more interesting behavior (the

example means that ``text/html`` is okay, but

``application/xhtml+xml`` is preferred):

.. code-block::

    >>> req.accept = 'text/html;q=0.5, application/xhtml+xml;q=1'

    >>> req.accept

    <MIMEAccept at ... Accept: text/html;q=0.5, application/xhtml+xml>

    >>> 'text/html' in req.accept

    True

There's three methods for different strategies of finding a match.

First, when you trust the server's preference over the client (a good

idea for Accept):

.. code-block::

    >>> req.accept.first_match(['text/html', 'application/xhtml+xml'])

    'text/html'

Because ``text/html`` is at least *somewhat* acceptible, it is

returned, even if the client says it prefers

``application/xhtml+xml``.  If we trust the client more:

.. code-block::

    >>> req.accept.best_match(['text/html', 'application/xhtml+xml'])

    'application/xhtml+xml'

If we just want to know everything the client prefers, in the order it

is preferred:

.. code-block::

    >>> req.accept.best_matches()

    ['application/xhtml+xml', 'text/html']

For languages you'll often have a "fallback" language.  E.g., if there's

nothing better then use ``en-US`` (and if ``en-US`` is okay, ignore

any less preferrable languages):

.. code-block::

    >>> req.accept_language = 'es, pt-BR'

    >>> req.accept_language.best_matches('en-US')

    ['es', 'pt-BR', 'en-US']

    >>> req.accept_language.best_matches('es')

    ['es']

Conditional Requests

~~~~~~~~~~~~~~~~~~~~

There a number of ways to make a conditional request.  A conditional

request is made when the client has a document, but it is not sure if

the document is up to date.  If it is not, it wants a new version.  If

the document is up to date then it doesn't want to waste the

bandwidth, and expects a ``304 Not Modified`` response.

ETags are generally the best technique for these kinds of requests;

this is an opaque string that indicates the identity of the object.

For instance, it's common to use the mtime (last modified) of the file,

plus the number of bytes, and maybe a hash of the filename (if there's

a possibility that the same URL could point to two different

server-side filenames based on other variables).  To test if a 304

response is appropriate, you can use:

.. code-block::

    >>> server_token = 'opaque-token'

    >>> server_token in req.if_none_match # You shouldn't return 304

    False

    >>> req.if_none_match = server_token

    >>> req.if_none_match

    <ETag opaque-token>

    >>> server_token in req.if_none_match # You *should* return 304

    True

For date-based comparisons If-Modified-Since is used:

.. code-block::

    >>> from webob import UTC

    >>> from datetime import datetime

    >>> req.if_modified_since = datetime(2006, 1, 1, 12, 0, tzinfo=UTC)

    >>> req.headers['If-Modified-Since']

    'Sun, 01 Jan 2006 12:00:00 GMT'

    >>> server_modified = datetime(2005, 1, 1, 12, 0, tzinfo=UTC)

    >>> req.if_modified_since and req.if_modified_since >= server_modified

    True

For range requests there are two important headers, If-Range (which is

form of conditional request) and Range (which requests a range).  If

the If-Range header fails to match then the full response (not a

range) should be returned:

.. code-block::

    >>> req.if_range

    <Empty If-Range>

    >>> req.if_range.match(etag='some-etag', last_modified=datetime(2005, 1, 1, 12, 0))

    True

    >>> req.if_range = 'opaque-etag'

    >>> req.if_range.match(etag='other-etag')

    False

    >>> req.if_range.match(etag='opaque-etag')

    True

You can also pass in a response object with:

.. code-block::

    >>> from webob import Response

    >>> res = Response(etag='opaque-etag')

    >>> req.if_range.match_response(res)

    True

To get the range information:

    >>> req.range = 'bytes=0-100'

    >>> req.range

    <Range ranges=(0, 99)>

    >>> cr = req.range.content_range(length=1000)

    >>> cr.start, cr.stop, cr.length

    (0, 99, 1000)

Note that the range headers use *inclusive* ranges (the last byte

indexed is included), where Python always uses a range where the last

index is excluded from the range.  The ``.stop`` index is in the

Python form.

Another kind of conditional request is a request (typically PUT) that

includes If-Match or If-Unmodified-Since.  In this case you are saying

"here is an update to a resource, but don't apply it if someone else

has done something since I last got the resource".  If-Match means "do

this if the current ETag matches the ETag I'm giving".

If-Unmodified-Since means "do this if the resource has remained

unchanged".

.. code-block::

    >>> server_token in req.if_match # No If-Match means everything is ok

    True

    >>> req.if_match = server_token

    >>> server_token in req.if_match # Still OK

    True

    >>> req.if_match = 'other-token'

    >>> # Not OK, should return 412 Precondition Failed:

    >>> server_token in req.if_match 

    False

For more on this kind of conditional request, see `Detecting the Lost

Update Problem Using Unreserved Checkout

<http://www.w3.org/1999/04/Editing/>`_.

Calling WSGI Applications

-------------------------

The request object can be used to make handy subrequests or test

requests against WSGI applications.  If you want to make subrequests,

you should copy the request (with ``req.copy()``) before sending it to

multiple applications, since applications might modify the request

when they are run.

There's two forms of the subrequest.  The more primitive form is

this:

.. code-block::

    >>> req = Request.blank('/')

    >>> def wsgi_app(environ, start_response):

    ...     start_response('200 OK', [('Content-type', 'text/plain')])

    ...     return ['Hi!']

    >>> req.call_application(wsgi_app)

    ('200 OK', [('Content-type', 'text/plain')], ['Hi!'])

Note it returns ``(status_string, header_list, app_iter)``.  If

``app_iter.close()`` exists, it is your responsibility to call it.

A handier response can be had with:

.. code-block::

    >>> res = req.get_response(wsgi_app)

    >>> res

    <Response ... 200 OK>

    >>> res.status

    '200 OK'

    >>> res.headers

    HeaderDict([('Content-type', 'text/plain')])

    >>> res.body

    'Hi!'

You can learn more about this response object in the Response_ section.

Thread-local Request Wrappers

-----------------------------

You can also give the ``Request`` object a function to get the

environment.  This can be used to make a single global request object

dynamic.  An example:

.. code-block::

    >>> import threading

    >>> import webob

    >>> environments = threading.local()

    >>> def get_environ():

    ...     return environments.environ

    >>> def set_thread_environ(environ):