thirdparty/google_appengine/google/appengine/api/urlfetch_stub.py
author Daniel Diniz <ajaksu@gmail.com>
Wed, 08 Jul 2009 10:40:46 +0200
changeset 2570 851640749319
parent 2413 d0b7dac5325c
child 2864 2e0b0af889be
permissions -rwxr-xr-x
Several Survey UI fixes. Fixes: Too narrow fieldsets in new question/option dialogs. Survey submit (on take view) and save/export/etc. buttons at weird places, instead of at bottom. Weird placement of radio buttons in Opera. Too narrow selects in IE. Broken images in edit view in IE and Opera. Reviewed by: Lennard de Rijk (Only on IE)
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     1
#!/usr/bin/env python
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     2
#
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     3
# Copyright 2007 Google Inc.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     4
#
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     5
# Licensed under the Apache License, Version 2.0 (the "License");
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     6
# you may not use this file except in compliance with the License.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     7
# You may obtain a copy of the License at
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     8
#
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     9
#     http://www.apache.org/licenses/LICENSE-2.0
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    10
#
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    11
# Unless required by applicable law or agreed to in writing, software
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    12
# distributed under the License is distributed on an "AS IS" BASIS,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    13
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    14
# See the License for the specific language governing permissions and
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    15
# limitations under the License.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    16
#
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    17
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    18
"""Stub version of the urlfetch API, based on httplib."""
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    19
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    20
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    21
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
    22
import gzip
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    23
import httplib
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    24
import logging
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    25
import socket
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
    26
import StringIO
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    27
import urllib
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    28
import urlparse
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    29
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    30
from google.appengine.api import apiproxy_stub
149
f2e327a7c5de Load ../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents: 109
diff changeset
    31
from google.appengine.api import urlfetch
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    32
from google.appengine.api import urlfetch_errors
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    33
from google.appengine.api import urlfetch_service_pb
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    34
from google.appengine.runtime import apiproxy_errors
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    35
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    36
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    37
MAX_RESPONSE_SIZE = 2 ** 24
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    38
149
f2e327a7c5de Load ../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents: 109
diff changeset
    39
MAX_REDIRECTS = urlfetch.MAX_REDIRECTS
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    40
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    41
REDIRECT_STATUSES = frozenset([
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    42
  httplib.MOVED_PERMANENTLY,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    43
  httplib.FOUND,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    44
  httplib.SEE_OTHER,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    45
  httplib.TEMPORARY_REDIRECT,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    46
])
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    47
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    48
PORTS_ALLOWED_IN_PRODUCTION = (
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    49
    None, '80', '443', '4443', '8080', '8081', '8082', '8083', '8084', '8085',
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    50
    '8086', '8087', '8088', '8089', '8188', '8444', '8990')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    51
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    52
_API_CALL_DEADLINE = 5.0
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    53
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    54
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    55
_UNTRUSTED_REQUEST_HEADERS = frozenset([
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    56
  'content-length',
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    57
  'host',
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    58
  'referer',
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    59
  'vary',
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    60
  'via',
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    61
  'x-forwarded-for',
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    62
])
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    63
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    64
class URLFetchServiceStub(apiproxy_stub.APIProxyStub):
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    65
  """Stub version of the urlfetch API to be used with apiproxy_stub_map."""
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    66
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    67
  def __init__(self, service_name='urlfetch'):
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    68
    """Initializer.
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    69
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    70
    Args:
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    71
      service_name: Service name expected for all calls.
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    72
    """
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
    73
    super(URLFetchServiceStub, self).__init__(service_name)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    74
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    75
  def _Dynamic_Fetch(self, request, response):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    76
    """Trivial implementation of URLFetchService::Fetch().
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    77
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    78
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    79
      request: the fetch to perform, a URLFetchRequest
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    80
      response: the fetch response, a URLFetchResponse
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    81
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    82
    (protocol, host, path, parameters, query, fragment) = urlparse.urlparse(request.url())
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    83
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
    84
    payload = None
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    85
    if request.method() == urlfetch_service_pb.URLFetchRequest.GET:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    86
      method = 'GET'
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    87
    elif request.method() == urlfetch_service_pb.URLFetchRequest.POST:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    88
      method = 'POST'
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    89
      payload = request.payload()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    90
    elif request.method() == urlfetch_service_pb.URLFetchRequest.HEAD:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    91
      method = 'HEAD'
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    92
    elif request.method() == urlfetch_service_pb.URLFetchRequest.PUT:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    93
      method = 'PUT'
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    94
      payload = request.payload()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    95
    elif request.method() == urlfetch_service_pb.URLFetchRequest.DELETE:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    96
      method = 'DELETE'
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    97
    else:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    98
      logging.error('Invalid method: %s', request.method())
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    99
      raise apiproxy_errors.ApplicationError(
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   100
        urlfetch_service_pb.URLFetchServiceError.UNSPECIFIED_ERROR)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   101
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   102
    if not (protocol == 'http' or protocol == 'https'):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   103
      logging.error('Invalid protocol: %s', protocol)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   104
      raise apiproxy_errors.ApplicationError(
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   105
        urlfetch_service_pb.URLFetchServiceError.INVALID_URL)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   106
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   107
    if not host:
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   108
      logging.error('Missing host.')
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   109
      raise apiproxy_errors.ApplicationError(
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   110
          urlfetch_service_pb.URLFetchServiceError.FETCH_ERROR)
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   111
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   112
    sanitized_headers = self._SanitizeHttpHeaders(_UNTRUSTED_REQUEST_HEADERS,
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   113
                                                  request.header_list())
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   114
    request.clear_header()
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   115
    request.header_list().extend(sanitized_headers)
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
   116
    deadline = _API_CALL_DEADLINE
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
   117
    if request.has_deadline():
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
   118
      deadline = request.deadline()
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   119
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   120
    self._RetrieveURL(request.url(), payload, method,
149
f2e327a7c5de Load ../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents: 109
diff changeset
   121
                      request.header_list(), response,
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
   122
                      follow_redirects=request.followredirects(),
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
   123
                      deadline=deadline)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   124
149
f2e327a7c5de Load ../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents: 109
diff changeset
   125
  def _RetrieveURL(self, url, payload, method, headers, response,
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
   126
                   follow_redirects=True, deadline=_API_CALL_DEADLINE):
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   127
    """Retrieves a URL.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   128
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   129
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   130
      url: String containing the URL to access.
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   131
      payload: Request payload to send, if any; None if no payload.
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   132
      method: HTTP method to use (e.g., 'GET')
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   133
      headers: List of additional header objects to use for the request.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   134
      response: Response object
149
f2e327a7c5de Load ../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents: 109
diff changeset
   135
      follow_redirects: optional setting (defaulting to True) for whether or not
f2e327a7c5de Load ../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents: 109
diff changeset
   136
        we should transparently follow redirects (up to MAX_REDIRECTS)
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
   137
      deadline: Number of seconds to wait for the urlfetch to finish.
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   138
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   139
    Raises:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   140
      Raises an apiproxy_errors.ApplicationError exception with FETCH_ERROR
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   141
      in cases where:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   142
        - MAX_REDIRECTS is exceeded
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   143
        - The protocol of the redirected URL is bad or missing.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   144
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   145
    last_protocol = ''
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   146
    last_host = ''
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   147
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   148
    for redirect_number in xrange(MAX_REDIRECTS + 1):
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   149
      parsed = urlparse.urlparse(url)
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   150
      protocol, host, path, parameters, query, fragment = parsed
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   151
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   152
      port = urllib.splitport(urllib.splituser(host)[1])[1]
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   153
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   154
      if port not in PORTS_ALLOWED_IN_PRODUCTION:
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   155
        logging.warning(
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   156
          'urlfetch received %s ; port %s is not allowed in production!' %
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   157
          (url, port))
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   158
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   159
      if protocol and not host:
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   160
        logging.error('Missing host on redirect; target url is %s' % url)
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   161
        raise apiproxy_errors.ApplicationError(
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   162
          urlfetch_service_pb.URLFetchServiceError.FETCH_ERROR)
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   163
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   164
      if not host and not protocol:
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   165
        host = last_host
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   166
        protocol = last_protocol
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   167
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   168
      adjusted_headers = {
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   169
          'User-Agent':
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   170
          'AppEngine-Google; (+http://code.google.com/appengine)',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   171
          'Referer': 'http://localhost/',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   172
          'Host': host,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   173
          'Accept-Encoding': 'gzip',
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   174
      }
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   175
      if payload is not None:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   176
        adjusted_headers['Content-Length'] = len(payload)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   177
      if method == 'POST' and payload:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   178
        adjusted_headers['Content-Type'] = 'application/x-www-form-urlencoded'
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   179
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   180
      for header in headers:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   181
        if header.key().title().lower() == 'user-agent':
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   182
          adjusted_headers['User-Agent'] = (
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   183
              '%s %s' %
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   184
              (header.value(), adjusted_headers['User-Agent']))
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   185
        else:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   186
          adjusted_headers[header.key().title()] = header.value()
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   187
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   188
      logging.debug('Making HTTP request: host = %s, '
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   189
                    'url = %s, payload = %s, headers = %s',
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   190
                    host, url, payload, adjusted_headers)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   191
      try:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   192
        if protocol == 'http':
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   193
          connection = httplib.HTTPConnection(host)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   194
        elif protocol == 'https':
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   195
          connection = httplib.HTTPSConnection(host)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   196
        else:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   197
          error_msg = 'Redirect specified invalid protocol: "%s"' % protocol
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   198
          logging.error(error_msg)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   199
          raise apiproxy_errors.ApplicationError(
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   200
              urlfetch_service_pb.URLFetchServiceError.FETCH_ERROR, error_msg)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   201
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   202
        last_protocol = protocol
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   203
        last_host = host
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   204
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   205
        if query != '':
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   206
          full_path = path + '?' + query
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   207
        else:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   208
          full_path = path
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   209
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   210
        orig_timeout = socket.getdefaulttimeout()
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   211
        try:
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
   212
          socket.setdefaulttimeout(deadline)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   213
          connection.request(method, full_path, payload, adjusted_headers)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   214
          http_response = connection.getresponse()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   215
          http_response_data = http_response.read()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   216
        finally:
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   217
          socket.setdefaulttimeout(orig_timeout)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   218
          connection.close()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   219
      except (httplib.error, socket.error, IOError), e:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   220
        raise apiproxy_errors.ApplicationError(
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   221
          urlfetch_service_pb.URLFetchServiceError.FETCH_ERROR, str(e))
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   222
149
f2e327a7c5de Load ../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents: 109
diff changeset
   223
      if http_response.status in REDIRECT_STATUSES and follow_redirects:
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   224
        url = http_response.getheader('Location', None)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   225
        if url is None:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   226
          error_msg = 'Redirecting response was missing "Location" header'
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   227
          logging.error(error_msg)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   228
          raise apiproxy_errors.ApplicationError(
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   229
              urlfetch_service_pb.URLFetchServiceError.FETCH_ERROR, error_msg)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   230
      else:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   231
        response.set_statuscode(http_response.status)
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   232
        if http_response.getheader('content-encoding') == 'gzip':
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   233
          gzip_stream = StringIO.StringIO(http_response_data)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   234
          gzip_file = gzip.GzipFile(fileobj=gzip_stream)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   235
          http_response_data = gzip_file.read()
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   236
        response.set_content(http_response_data[:MAX_RESPONSE_SIZE])
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   237
        for header_key, header_value in http_response.getheaders():
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   238
          if (header_key.lower() == 'content-encoding' and
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   239
              header_value == 'gzip'):
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   240
            continue
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   241
          if header_key.lower() == 'content-length':
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   242
            header_value = len(response.content())
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   243
          header_proto = response.add_header()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   244
          header_proto.set_key(header_key)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   245
          header_proto.set_value(header_value)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   246
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   247
        if len(http_response_data) > MAX_RESPONSE_SIZE:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   248
          response.set_contentwastruncated(True)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   249
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   250
        break
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   251
    else:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   252
      error_msg = 'Too many repeated redirects'
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   253
      logging.error(error_msg)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   254
      raise apiproxy_errors.ApplicationError(
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   255
          urlfetch_service_pb.URLFetchServiceError.FETCH_ERROR, error_msg)
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   256
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   257
  def _SanitizeHttpHeaders(self, untrusted_headers, headers):
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   258
    """Cleans "unsafe" headers from the HTTP request/response.
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   259
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   260
    Args:
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   261
      untrusted_headers: set of untrusted headers names
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   262
      headers: list of string pairs, first is header name and the second is header's value
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   263
    """
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
   264
    prohibited_headers = [h.key() for h in headers
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
   265
                          if h.key().lower() in untrusted_headers]
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
   266
    if prohibited_headers:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   267
      logging.warn('Stripped prohibited headers from URLFetch request: %s',
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
   268
                   prohibited_headers)
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
   269
    return (h for h in headers if h.key().lower() not in untrusted_headers)