thirdparty/google_appengine/google/appengine/tools/appcfg.py
author Pawel Solyga <Pawel.Solyga@gmail.com>
Fri, 19 Jun 2009 16:13:32 +0200
changeset 2413 d0b7dac5325c
parent 2309 be1b94099f2d
child 2864 2e0b0af889be
permissions -rwxr-xr-x
Update GAE to 1.2.3 in thirdparty folder.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     1
#!/usr/bin/env python
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     2
#
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     3
# Copyright 2007 Google Inc.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     4
#
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     5
# Licensed under the Apache License, Version 2.0 (the "License");
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     6
# you may not use this file except in compliance with the License.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     7
# You may obtain a copy of the License at
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     8
#
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
     9
#     http://www.apache.org/licenses/LICENSE-2.0
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    10
#
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    11
# Unless required by applicable law or agreed to in writing, software
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    12
# distributed under the License is distributed on an "AS IS" BASIS,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    13
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    14
# See the License for the specific language governing permissions and
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    15
# limitations under the License.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    16
#
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    17
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    18
"""Tool for deploying apps to an app server.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    19
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    20
Currently, the application only uploads new appversions. To do this, it first
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    21
walks the directory tree rooted at the path the user specifies, adding all the
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    22
files it finds to a list. It then uploads the application configuration
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    23
(app.yaml) to the server using HTTP, followed by uploading each of the files.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    24
It then commits the transaction with another request.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    25
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    26
The bulk of this work is handled by the AppVersionUpload class, which exposes
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    27
methods to add to the list of files, fetch a list of modified files, upload
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    28
files, and commit or rollback the transaction.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    29
"""
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    30
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    31
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
    32
import calendar
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    33
import datetime
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    34
import getpass
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    35
import logging
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    36
import mimetypes
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    37
import optparse
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    38
import os
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    39
import re
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    40
import sha
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    41
import sys
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    42
import tempfile
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    43
import time
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    44
import urllib2
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    45
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    46
import google
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
    47
import yaml
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
    48
from google.appengine.cron import groctimespecification
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    49
from google.appengine.api import appinfo
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
    50
from google.appengine.api import croninfo
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
    51
from google.appengine.api import queueinfo
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    52
from google.appengine.api import validation
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    53
from google.appengine.api import yaml_errors
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    54
from google.appengine.api import yaml_object
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    55
from google.appengine.datastore import datastore_index
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
    56
from google.appengine.tools import appengine_rpc
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
    57
from google.appengine.tools import bulkloader
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    58
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    59
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    60
MAX_FILES_TO_CLONE = 100
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
    61
LIST_DELIMITER = '\n'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
    62
TUPLE_DELIMITER = '|'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    63
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
    64
VERSION_FILE = '../VERSION'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    65
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    66
UPDATE_CHECK_TIMEOUT = 3
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    67
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
    68
NAG_FILE = '.appcfg_nag'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    69
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    70
MAX_LOG_LEVEL = 4
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    71
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    72
verbosity = 1
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    73
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    74
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
    75
appinfo.AppInfoExternal.ATTRIBUTES[appinfo.RUNTIME] = 'python'
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
    76
_api_versions = os.environ.get('GOOGLE_TEST_API_VERSIONS', '1')
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
    77
_options = validation.Options(*_api_versions.split(','))
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
    78
appinfo.AppInfoExternal.ATTRIBUTES[appinfo.API_VERSION] = _options
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
    79
del _api_versions, _options
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
    80
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
    81
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    82
def StatusUpdate(msg):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    83
  """Print a status message to stderr.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    84
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    85
  If 'verbosity' is greater than 0, print the message.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    86
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    87
  Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    88
    msg: The string to print.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    89
  """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    90
  if verbosity > 0:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    91
    print >>sys.stderr, msg
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    92
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    93
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    94
def GetMimeTypeIfStaticFile(config, filename):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    95
  """Looks up the mime type for 'filename'.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    96
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    97
  Uses the handlers in 'config' to determine if the file should
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    98
  be treated as a static file.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
    99
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   100
  Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   101
    config: The app.yaml object to check the filename against.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   102
    filename: The name of the file.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   103
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   104
  Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   105
    The mime type string.  For example, 'text/plain' or 'image/gif'.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   106
    None if this is not a static file.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   107
  """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   108
  for handler in config.handlers:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   109
    handler_type = handler.GetHandlerType()
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   110
    if handler_type in ('static_dir', 'static_files'):
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   111
      if handler_type == 'static_dir':
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   112
        regex = os.path.join(re.escape(handler.GetHandler()), '.*')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   113
      else:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   114
        regex = handler.upload
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   115
      if re.match(regex, filename):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   116
        if handler.mime_type is not None:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   117
          return handler.mime_type
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   118
        else:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   119
          guess = mimetypes.guess_type(filename)[0]
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   120
          if guess is None:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   121
            default = 'application/octet-stream'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   122
            print >>sys.stderr, ('Could not guess mimetype for %s.  Using %s.'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   123
                                 % (filename, default))
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   124
            return default
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   125
          return guess
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   126
  return None
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   127
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   128
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   129
def BuildClonePostBody(file_tuples):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   130
  """Build the post body for the /api/clone{files,blobs} urls.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   131
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   132
  Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   133
    file_tuples: A list of tuples.  Each tuple should contain the entries
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   134
      appropriate for the endpoint in question.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   135
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   136
  Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   137
    A string containing the properly delimited tuples.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   138
  """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   139
  file_list = []
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   140
  for tup in file_tuples:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   141
    path = tup[0]
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   142
    tup = tup[1:]
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   143
    file_list.append(TUPLE_DELIMITER.join([path] + list(tup)))
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   144
  return LIST_DELIMITER.join(file_list)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   145
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   146
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   147
class NagFile(validation.Validated):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   148
  """A validated YAML class to represent the user's nag preferences.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   149
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   150
  Attributes:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   151
    timestamp: The timestamp of the last nag.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   152
    opt_in: True if the user wants to check for updates on dev_appserver
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   153
      start.  False if not.  May be None if we have not asked the user yet.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   154
  """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   155
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   156
  ATTRIBUTES = {
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   157
      'timestamp': validation.TYPE_FLOAT,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   158
      'opt_in': validation.Optional(validation.TYPE_BOOL),
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   159
  }
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   160
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   161
  @staticmethod
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   162
  def Load(nag_file):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   163
    """Load a single NagFile object where one and only one is expected.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   164
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   165
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   166
      nag_file: A file-like object or string containing the yaml data to parse.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   167
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   168
    Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   169
      A NagFile instance.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   170
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   171
    return yaml_object.BuildSingleObject(NagFile, nag_file)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   172
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   173
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   174
def GetVersionObject(isfile=os.path.isfile, open_fn=open):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   175
  """Gets the version of the SDK by parsing the VERSION file.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   176
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   177
  Args:
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   178
    isfile: used for testing.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   179
    open_fn: Used for testing.
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   180
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   181
  Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   182
    A Yaml object or None if the VERSION file does not exist.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   183
  """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   184
  version_filename = os.path.join(os.path.dirname(google.__file__),
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   185
                                  VERSION_FILE)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   186
  if not isfile(version_filename):
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   187
    logging.error('Could not find version file at %s', version_filename)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   188
    return None
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   189
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   190
  version_fh = open_fn(version_filename, 'r')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   191
  try:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   192
    version = yaml.safe_load(version_fh)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   193
  finally:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   194
    version_fh.close()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   195
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   196
  return version
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   197
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   198
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   199
def RetryWithBackoff(initial_delay, backoff_factor, max_tries, callable_func):
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   200
  """Calls a function multiple times, backing off more and more each time.
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   201
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   202
  Args:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   203
    initial_delay: Initial delay after first try, in seconds.
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   204
    backoff_factor: Delay will be multiplied by this factor after each try.
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   205
    max_tries: Maximum number of tries.
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   206
    callable_func: The method to call, will pass no arguments.
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   207
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   208
  Returns:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   209
    True if the function succeded in one of its tries.
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   210
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   211
  Raises:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   212
    Whatever the function raises--an exception will immediately stop retries.
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   213
  """
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   214
  delay = initial_delay
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   215
  while not callable_func() and max_tries > 0:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   216
    StatusUpdate('Will check again in %s seconds.' % delay)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   217
    time.sleep(delay)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   218
    delay *= backoff_factor
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   219
    max_tries -= 1
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   220
  return max_tries > 0
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   221
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   222
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   223
class UpdateCheck(object):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   224
  """Determines if the local SDK is the latest version.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   225
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   226
  Nags the user when there are updates to the SDK.  As the SDK becomes
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   227
  more out of date, the language in the nagging gets stronger.  We
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   228
  store a little yaml file in the user's home directory so that we nag
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   229
  the user only once a week.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   230
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   231
  The yaml file has the following field:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   232
    'timestamp': Last time we nagged the user in seconds since the epoch.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   233
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   234
  Attributes:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   235
    server: An AbstractRpcServer instance used to check for the latest SDK.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   236
    config: The app's AppInfoExternal.  Needed to determine which api_version
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   237
      the app is using.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   238
  """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   239
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   240
  def __init__(self,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   241
               server,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   242
               config,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   243
               isdir=os.path.isdir,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   244
               isfile=os.path.isfile,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   245
               open_fn=open):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   246
    """Create a new UpdateCheck.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   247
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   248
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   249
      server: The AbstractRpcServer to use.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   250
      config: The yaml object that specifies the configuration of this
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   251
        application.
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   252
      isdir: Replacement for os.path.isdir (for testing).
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   253
      isfile: Replacement for os.path.isfile (for testing).
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   254
      open_fn: Replacement for the open builtin (for testing).
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   255
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   256
    self.server = server
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   257
    self.config = config
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   258
    self.isdir = isdir
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   259
    self.isfile = isfile
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   260
    self.open = open_fn
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   261
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   262
  @staticmethod
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   263
  def MakeNagFilename():
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   264
    """Returns the filename for the nag file for this user."""
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   265
    user_homedir = os.path.expanduser('~/')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   266
    if not os.path.isdir(user_homedir):
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   267
      drive, unused_tail = os.path.splitdrive(os.__file__)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   268
      if drive:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   269
        os.environ['HOMEDRIVE'] = drive
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   270
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   271
    return os.path.expanduser('~/' + NAG_FILE)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   272
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   273
  def _ParseVersionFile(self):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   274
    """Parse the local VERSION file.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   275
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   276
    Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   277
      A Yaml object or None if the file does not exist.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   278
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   279
    return GetVersionObject(isfile=self.isfile, open_fn=self.open)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   280
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   281
  def CheckSupportedVersion(self):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   282
    """Determines if the app's api_version is supported by the SDK.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   283
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   284
    Uses the api_version field from the AppInfoExternal to determine if
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   285
    the SDK supports that api_version.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   286
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   287
    Raises:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   288
      SystemExit if the api_version is not supported.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   289
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   290
    version = self._ParseVersionFile()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   291
    if version is None:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   292
      logging.error('Could not determine if the SDK supports the api_version '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   293
                    'requested in app.yaml.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   294
      return
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   295
    if self.config.api_version not in version['api_versions']:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   296
      logging.critical('The api_version specified in app.yaml (%s) is not '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   297
                       'supported by this release of the SDK.  The supported '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   298
                       'api_versions are %s.',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   299
                       self.config.api_version, version['api_versions'])
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   300
      sys.exit(1)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   301
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   302
  def CheckForUpdates(self):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   303
    """Queries the server for updates and nags the user if appropriate.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   304
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   305
    Queries the server for the latest SDK version at the same time reporting
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   306
    the local SDK version.  The server will respond with a yaml document
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   307
    containing the fields:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   308
      'release': The name of the release (e.g. 1.2).
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   309
      'timestamp': The time the release was created (YYYY-MM-DD HH:MM AM/PM TZ).
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   310
      'api_versions': A list of api_version strings (e.g. ['1', 'beta']).
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   311
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   312
    We will nag the user with increasing severity if:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   313
    - There is a new release.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   314
    - There is a new release with a new api_version.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   315
    - There is a new release that does not support the api_version named in
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   316
      self.config.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   317
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   318
    version = self._ParseVersionFile()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   319
    if version is None:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   320
      logging.info('Skipping update check')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   321
      return
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   322
    logging.info('Checking for updates to the SDK.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   323
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   324
    try:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   325
      response = self.server.Send('/api/updatecheck',
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   326
                                  timeout=UPDATE_CHECK_TIMEOUT,
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   327
                                  release=version['release'],
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   328
                                  timestamp=version['timestamp'],
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   329
                                  api_versions=version['api_versions'])
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   330
    except urllib2.URLError, e:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   331
      logging.info('Update check failed: %s', e)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   332
      return
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   333
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   334
    latest = yaml.safe_load(response)
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   335
    if latest['release'] == version['release']:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   336
      logging.info('The SDK is up to date.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   337
      return
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   338
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   339
    api_versions = latest['api_versions']
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   340
    if self.config.api_version not in api_versions:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   341
      self._Nag(
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   342
          'The api version you are using (%s) is obsolete!  You should\n'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   343
          'upgrade your SDK and test that your code works with the new\n'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   344
          'api version.' % self.config.api_version,
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   345
          latest, version, force=True)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   346
      return
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   347
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   348
    if self.config.api_version != api_versions[len(api_versions) - 1]:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   349
      self._Nag(
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   350
          'The api version you are using (%s) is deprecated. You should\n'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   351
          'upgrade your SDK to try the new functionality.' %
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   352
          self.config.api_version, latest, version)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   353
      return
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   354
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   355
    self._Nag('There is a new release of the SDK available.',
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   356
              latest, version)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   357
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   358
  def _ParseNagFile(self):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   359
    """Parses the nag file.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   360
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   361
    Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   362
      A NagFile if the file was present else None.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   363
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   364
    nag_filename = UpdateCheck.MakeNagFilename()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   365
    if self.isfile(nag_filename):
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   366
      fh = self.open(nag_filename, 'r')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   367
      try:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   368
        nag = NagFile.Load(fh)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   369
      finally:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   370
        fh.close()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   371
      return nag
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   372
    return None
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   373
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   374
  def _WriteNagFile(self, nag):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   375
    """Writes the NagFile to the user's nag file.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   376
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   377
    If the destination path does not exist, this method will log an error
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   378
    and fail silently.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   379
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   380
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   381
      nag: The NagFile to write.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   382
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   383
    nagfilename = UpdateCheck.MakeNagFilename()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   384
    try:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   385
      fh = self.open(nagfilename, 'w')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   386
      try:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   387
        fh.write(nag.ToYAML())
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   388
      finally:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   389
        fh.close()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   390
    except (OSError, IOError), e:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   391
      logging.error('Could not write nag file to %s. Error: %s', nagfilename, e)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   392
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   393
  def _Nag(self, msg, latest, version, force=False):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   394
    """Prints a nag message and updates the nag file's timestamp.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   395
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   396
    Because we don't want to nag the user everytime, we store a simple
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   397
    yaml document in the user's home directory.  If the timestamp in this
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   398
    doc is over a week old, we'll nag the user.  And when we nag the user,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   399
    we update the timestamp in this doc.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   400
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   401
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   402
      msg: The formatted message to print to the user.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   403
      latest: The yaml document received from the server.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   404
      version: The local yaml version document.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   405
      force: If True, always nag the user, ignoring the nag file.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   406
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   407
    nag = self._ParseNagFile()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   408
    if nag and not force:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   409
      last_nag = datetime.datetime.fromtimestamp(nag.timestamp)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   410
      if datetime.datetime.now() - last_nag < datetime.timedelta(weeks=1):
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   411
        logging.debug('Skipping nag message')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   412
        return
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   413
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   414
    if nag is None:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   415
      nag = NagFile()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   416
    nag.timestamp = time.time()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   417
    self._WriteNagFile(nag)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   418
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   419
    print '****************************************************************'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   420
    print msg
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   421
    print '-----------'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   422
    print 'Latest SDK:'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   423
    print yaml.dump(latest)
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   424
    print '-----------'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   425
    print 'Your SDK:'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   426
    print yaml.dump(version)
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   427
    print '-----------'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   428
    print 'Please visit http://code.google.com/appengine for the latest SDK'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   429
    print '****************************************************************'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   430
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   431
  def AllowedToCheckForUpdates(self, input_fn=raw_input):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   432
    """Determines if the user wants to check for updates.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   433
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   434
    On startup, the dev_appserver wants to check for updates to the SDK.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   435
    Because this action reports usage to Google when the user is not
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   436
    otherwise communicating with Google (e.g. pushing a new app version),
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   437
    the user must opt in.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   438
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   439
    If the user does not have a nag file, we will query the user and
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   440
    save the response in the nag file.  Subsequent calls to this function
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   441
    will re-use that response.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   442
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   443
    Args:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   444
      input_fn: used to collect user input. This is for testing only.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   445
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   446
    Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   447
      True if the user wants to check for updates.  False otherwise.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   448
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   449
    nag = self._ParseNagFile()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   450
    if nag is None:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   451
      nag = NagFile()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   452
      nag.timestamp = time.time()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   453
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   454
    if nag.opt_in is None:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   455
      answer = input_fn('Allow dev_appserver to check for updates on startup? '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   456
                        '(Y/n): ')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   457
      answer = answer.strip().lower()
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   458
      if answer == 'n' or answer == 'no':
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   459
        print ('dev_appserver will not check for updates on startup.  To '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   460
               'change this setting, edit %s' % UpdateCheck.MakeNagFilename())
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   461
        nag.opt_in = False
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   462
      else:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   463
        print ('dev_appserver will check for updates on startup.  To change '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   464
               'this setting, edit %s' % UpdateCheck.MakeNagFilename())
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   465
        nag.opt_in = True
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   466
      self._WriteNagFile(nag)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   467
    return nag.opt_in
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   468
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   469
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   470
class IndexDefinitionUpload(object):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   471
  """Provides facilities to upload index definitions to the hosting service."""
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   472
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   473
  def __init__(self, server, config, definitions):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   474
    """Creates a new DatastoreIndexUpload.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   475
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   476
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   477
      server: The RPC server to use.  Should be an instance of HttpRpcServer
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   478
        or TestRpcServer.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   479
      config: The AppInfoExternal object derived from the app.yaml file.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   480
      definitions: An IndexDefinitions object.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   481
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   482
    self.server = server
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   483
    self.config = config
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   484
    self.definitions = definitions
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   485
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   486
  def DoUpload(self):
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   487
    """Uploads the index definitions."""
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   488
    StatusUpdate('Uploading index definitions.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   489
    self.server.Send('/api/datastore/index/add',
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   490
                     app_id=self.config.application,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   491
                     version=self.config.version,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   492
                     payload=self.definitions.ToYAML())
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   493
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   494
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   495
class CronEntryUpload(object):
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   496
  """Provides facilities to upload cron entries to the hosting service."""
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   497
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   498
  def __init__(self, server, config, cron):
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   499
    """Creates a new CronEntryUpload.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   500
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   501
    Args:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   502
      server: The RPC server to use.  Should be an instance of a subclass of
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   503
      AbstractRpcServer
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   504
      config: The AppInfoExternal object derived from the app.yaml file.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   505
      cron: The CronInfoExternal object loaded from the cron.yaml file.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   506
    """
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   507
    self.server = server
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   508
    self.config = config
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   509
    self.cron = cron
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   510
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   511
  def DoUpload(self):
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   512
    """Uploads the cron entries."""
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   513
    StatusUpdate('Uploading cron entries.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   514
    self.server.Send('/api/datastore/cron/update',
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   515
                     app_id=self.config.application,
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   516
                     version=self.config.version,
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   517
                     payload=self.cron.ToYAML())
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   518
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   519
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   520
class QueueEntryUpload(object):
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   521
  """Provides facilities to upload task queue entries to the hosting service."""
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   522
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   523
  def __init__(self, server, config, queue):
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   524
    """Creates a new QueueEntryUpload.
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   525
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   526
    Args:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   527
      server: The RPC server to use.  Should be an instance of a subclass of
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   528
      AbstractRpcServer
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   529
      config: The AppInfoExternal object derived from the app.yaml file.
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   530
      queue: The QueueInfoExternal object loaded from the queue.yaml file.
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   531
    """
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   532
    self.server = server
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   533
    self.config = config
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   534
    self.queue = queue
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   535
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   536
  def DoUpload(self):
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   537
    """Uploads the task queue entries."""
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   538
    StatusUpdate('Uploading task queue entries.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   539
    self.server.Send('/api/queue/update',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   540
                     app_id=self.config.application,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   541
                     version=self.config.version,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   542
                     payload=self.queue.ToYAML())
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   543
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   544
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   545
class IndexOperation(object):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   546
  """Provide facilities for writing Index operation commands."""
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   547
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   548
  def __init__(self, server, config):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   549
    """Creates a new IndexOperation.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   550
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   551
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   552
      server: The RPC server to use.  Should be an instance of HttpRpcServer
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   553
        or TestRpcServer.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   554
      config: appinfo.AppInfoExternal configuration object.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   555
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   556
    self.server = server
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   557
    self.config = config
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   558
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   559
  def DoDiff(self, definitions):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   560
    """Retrieve diff file from the server.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   561
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   562
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   563
      definitions: datastore_index.IndexDefinitions as loaded from users
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   564
        index.yaml file.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   565
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   566
    Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   567
      A pair of datastore_index.IndexDefinitions objects.  The first record
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   568
      is the set of indexes that are present in the index.yaml file but missing
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   569
      from the server.  The second record is the set of indexes that are
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   570
      present on the server but missing from the index.yaml file (indicating
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   571
      that these indexes should probably be vacuumed).
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   572
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   573
    StatusUpdate('Fetching index definitions diff.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   574
    response = self.server.Send('/api/datastore/index/diff',
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   575
                                app_id=self.config.application,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   576
                                payload=definitions.ToYAML())
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   577
    return datastore_index.ParseMultipleIndexDefinitions(response)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   578
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   579
  def DoDelete(self, definitions):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   580
    """Delete indexes from the server.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   581
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   582
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   583
      definitions: Index definitions to delete from datastore.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   584
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   585
    Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   586
      A single datstore_index.IndexDefinitions containing indexes that were
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   587
      not deleted, probably because they were already removed.  This may
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   588
      be normal behavior as there is a potential race condition between fetching
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   589
      the index-diff and sending deletion confirmation through.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   590
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   591
    StatusUpdate('Deleting selected index definitions.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   592
    response = self.server.Send('/api/datastore/index/delete',
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   593
                                app_id=self.config.application,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   594
                                payload=definitions.ToYAML())
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   595
    return datastore_index.ParseIndexDefinitions(response)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   596
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   597
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   598
class VacuumIndexesOperation(IndexOperation):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   599
  """Provide facilities to request the deletion of datastore indexes."""
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   600
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   601
  def __init__(self, server, config, force,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   602
               confirmation_fn=raw_input):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   603
    """Creates a new VacuumIndexesOperation.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   604
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   605
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   606
      server: The RPC server to use.  Should be an instance of HttpRpcServer
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   607
        or TestRpcServer.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   608
      config: appinfo.AppInfoExternal configuration object.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   609
      force: True to force deletion of indexes, else False.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   610
      confirmation_fn: Function used for getting input form user.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   611
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   612
    super(VacuumIndexesOperation, self).__init__(server, config)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   613
    self.force = force
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   614
    self.confirmation_fn = confirmation_fn
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   615
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   616
  def GetConfirmation(self, index):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   617
    """Get confirmation from user to delete an index.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   618
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   619
    This method will enter an input loop until the user provides a
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   620
    response it is expecting.  Valid input is one of three responses:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   621
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   622
      y: Confirm deletion of index.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   623
      n: Do not delete index.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   624
      a: Delete all indexes without asking for further confirmation.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   625
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   626
    If the user enters nothing at all, the default action is to skip
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   627
    that index and do not delete.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   628
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   629
    If the user selects 'a', as a side effect, the 'force' flag is set.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   630
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   631
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   632
      index: Index to confirm.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   633
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   634
    Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   635
      True if user enters 'y' or 'a'.  False if user enter 'n'.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   636
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   637
    while True:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   638
      print 'This index is no longer defined in your index.yaml file.'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   639
      print
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   640
      print index.ToYAML()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   641
      print
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   642
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   643
      confirmation = self.confirmation_fn(
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   644
          'Are you sure you want to delete this index? (N/y/a): ')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   645
      confirmation = confirmation.strip().lower()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   646
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   647
      if confirmation == 'y':
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   648
        return True
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   649
      elif confirmation == 'n' or not confirmation:
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   650
        return False
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   651
      elif confirmation == 'a':
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   652
        self.force = True
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   653
        return True
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   654
      else:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   655
        print 'Did not understand your response.'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   656
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   657
  def DoVacuum(self, definitions):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   658
    """Vacuum indexes in datastore.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   659
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   660
    This method will query the server to determine which indexes are not
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   661
    being used according to the user's local index.yaml file.  Once it has
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   662
    made this determination, it confirms with the user which unused indexes
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   663
    should be deleted.  Once confirmation for each index is receives, it
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   664
    deletes those indexes.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   665
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   666
    Because another user may in theory delete the same indexes at the same
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   667
    time as the user, there is a potential race condition.  In this rare cases,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   668
    some of the indexes previously confirmed for deletion will not be found.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   669
    The user is notified which indexes these were.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   670
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   671
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   672
      definitions: datastore_index.IndexDefinitions as loaded from users
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   673
        index.yaml file.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   674
    """
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   675
    unused_new_indexes, notused_indexes = self.DoDiff(definitions)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   676
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   677
    deletions = datastore_index.IndexDefinitions(indexes=[])
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   678
    if notused_indexes.indexes is not None:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   679
      for index in notused_indexes.indexes:
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   680
        if self.force or self.GetConfirmation(index):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   681
          deletions.indexes.append(index)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   682
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   683
    if deletions.indexes:
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   684
      not_deleted = self.DoDelete(deletions)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   685
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   686
      if not_deleted.indexes:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   687
        not_deleted_count = len(not_deleted.indexes)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   688
        if not_deleted_count == 1:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   689
          warning_message = ('An index was not deleted.  Most likely this is '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   690
                             'because it no longer exists.\n\n')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   691
        else:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   692
          warning_message = ('%d indexes were not deleted.  Most likely this '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   693
                             'is because they no longer exist.\n\n'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   694
                             % not_deleted_count)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   695
        for index in not_deleted.indexes:
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   696
          warning_message += index.ToYAML()
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   697
        logging.warning(warning_message)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   698
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   699
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   700
class LogsRequester(object):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   701
  """Provide facilities to export request logs."""
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   702
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   703
  def __init__(self, server, config, output_file,
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   704
               num_days, append, severity, now, vhost, include_vhost):
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   705
    """Constructor.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   706
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   707
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   708
      server: The RPC server to use.  Should be an instance of HttpRpcServer
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   709
        or TestRpcServer.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   710
      config: appinfo.AppInfoExternal configuration object.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   711
      output_file: Output file name.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   712
      num_days: Number of days worth of logs to export; 0 for all available.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   713
      append: True if appending to an existing file.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   714
      severity: App log severity to request (0-4); None for no app logs.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   715
      now: POSIX timestamp used for calculating valid dates for num_days.
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
   716
      vhost: The virtual host of log messages to get. None for all hosts.
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   717
      include_vhost: If true, the virtual host is included in log messages.
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   718
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   719
    self.server = server
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   720
    self.config = config
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   721
    self.output_file = output_file
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   722
    self.append = append
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   723
    self.num_days = num_days
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   724
    self.severity = severity
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
   725
    self.vhost = vhost
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   726
    self.include_vhost = include_vhost
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   727
    self.version_id = self.config.version + '.1'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   728
    self.sentinel = None
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   729
    self.write_mode = 'w'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   730
    if self.append:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   731
      self.sentinel = FindSentinel(self.output_file)
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   732
      self.write_mode = 'a'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   733
    self.valid_dates = None
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   734
    if self.num_days:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   735
      patterns = []
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   736
      now = PacificTime(now)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   737
      for i in xrange(self.num_days):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   738
        then = time.gmtime(now - 24*3600 * i)
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   739
        patterns.append(re.escape(time.strftime('%d/%m/%Y', then)))
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   740
        patterns.append(re.escape(time.strftime('%d/%b/%Y', then)))
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   741
      self.valid_dates = re.compile(r'[^[]+\[(' + '|'.join(patterns) + r'):')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   742
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   743
  def DownloadLogs(self):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   744
    """Download the requested logs.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   745
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   746
    This will write the logs to the file designated by
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   747
    self.output_file, or to stdout if the filename is '-'.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   748
    Multiple roundtrips to the server may be made.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   749
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   750
    StatusUpdate('Downloading request logs for %s %s.' %
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   751
                 (self.config.application, self.version_id))
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   752
    tf = tempfile.TemporaryFile()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   753
    offset = None
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   754
    try:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   755
      while True:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   756
        try:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   757
          offset = self.RequestLogLines(tf, offset)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   758
          if not offset:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   759
            break
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   760
        except KeyboardInterrupt:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   761
          StatusUpdate('Keyboard interrupt; saving data downloaded so far.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   762
          break
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   763
      StatusUpdate('Copying request logs to %r.' % self.output_file)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   764
      if self.output_file == '-':
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   765
        of = sys.stdout
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   766
      else:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   767
        try:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   768
          of = open(self.output_file, self.write_mode)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   769
        except IOError, err:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   770
          StatusUpdate('Can\'t write %r: %s.' % (self.output_file, err))
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   771
          sys.exit(1)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   772
      try:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   773
        line_count = CopyReversedLines(tf, of)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   774
      finally:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   775
        of.flush()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   776
        if of is not sys.stdout:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   777
          of.close()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   778
    finally:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   779
      tf.close()
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   780
    StatusUpdate('Copied %d records.' % line_count)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   781
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   782
  def RequestLogLines(self, tf, offset):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   783
    """Make a single roundtrip to the server.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   784
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   785
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   786
      tf: Writable binary stream to which the log lines returned by
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   787
        the server are written, stripped of headers, and excluding
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   788
        lines skipped due to self.sentinel or self.valid_dates filtering.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   789
      offset: Offset string for a continued request; None for the first.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   790
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   791
    Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   792
      The offset string to be used for the next request, if another
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   793
      request should be issued; or None, if not.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   794
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   795
    logging.info('Request with offset %r.', offset)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   796
    kwds = {'app_id': self.config.application,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   797
            'version': self.version_id,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   798
            'limit': 100,
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   799
           }
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   800
    if offset:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   801
      kwds['offset'] = offset
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   802
    if self.severity is not None:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   803
      kwds['severity'] = str(self.severity)
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
   804
    if self.vhost is not None:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   805
      kwds['vhost'] = str(self.vhost)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   806
    if self.include_vhost is not None:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   807
      kwds['include_vhost'] = str(self.include_vhost)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   808
    response = self.server.Send('/api/request_logs', payload=None, **kwds)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   809
    response = response.replace('\r', '\0')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   810
    lines = response.splitlines()
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   811
    logging.info('Received %d bytes, %d records.', len(response), len(lines))
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   812
    offset = None
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   813
    if lines and lines[0].startswith('#'):
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   814
      match = re.match(r'^#\s*next_offset=(\S+)\s*$', lines[0])
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   815
      del lines[0]
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   816
      if match:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   817
        offset = match.group(1)
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   818
    if lines and lines[-1].startswith('#'):
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   819
      del lines[-1]
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   820
    valid_dates = self.valid_dates
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   821
    sentinel = self.sentinel
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   822
    len_sentinel = None
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   823
    if sentinel:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   824
      len_sentinel = len(sentinel)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   825
    for line in lines:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   826
      if ((sentinel and
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   827
           line.startswith(sentinel) and
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   828
           line[len_sentinel : len_sentinel+1] in ('', '\0')) or
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   829
          (valid_dates and not valid_dates.match(line))):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   830
        return None
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   831
      tf.write(line + '\n')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   832
    if not lines:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   833
      return None
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   834
    return offset
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   835
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   836
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   837
def PacificTime(now):
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   838
  """Helper to return the number of seconds between UTC and Pacific time.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   839
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   840
  This is needed to compute today's date in Pacific time (more
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   841
  specifically: Mountain View local time), which is how request logs
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   842
  are reported.  (Google servers always report times in Mountain View
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   843
  local time, regardless of where they are physically located.)
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   844
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   845
  This takes (post-2006) US DST into account.  Pacific time is either
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   846
  8 hours or 7 hours west of UTC, depending on whether DST is in
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   847
  effect.  Since 2007, US DST starts on the Second Sunday in March
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   848
  March, and ends on the first Sunday in November.  (Reference:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   849
  http://aa.usno.navy.mil/faq/docs/daylight_time.php.)
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   850
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   851
  Note that the server doesn't report its local time (the HTTP Date
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   852
  header uses UTC), and the client's local time is irrelevant.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   853
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   854
  Args:
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   855
    now: A posix timestamp giving current UTC time.
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   856
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   857
  Returns:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   858
    A pseudo-posix timestamp giving current Pacific time.  Passing
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   859
    this through time.gmtime() will produce a tuple in Pacific local
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   860
    time.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   861
  """
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   862
  now -= 8*3600
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   863
  if IsPacificDST(now):
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   864
    now += 3600
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   865
  return now
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   866
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   867
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   868
def IsPacificDST(now):
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   869
  """Helper for PacificTime to decide whether now is Pacific DST (PDT).
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   870
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   871
  Args:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   872
    now: A pseudo-posix timestamp giving current time in PST.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   873
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   874
  Returns:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   875
    True if now falls within the range of DST, False otherwise.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   876
  """
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   877
  DAY = 24*3600
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   878
  SUNDAY = 6
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   879
  pst = time.gmtime(now)
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   880
  year = pst[0]
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   881
  assert year >= 2007
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   882
  begin = calendar.timegm((year, 3, 8, 2, 0, 0, 0, 0, 0))
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   883
  while time.gmtime(begin).tm_wday != SUNDAY:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   884
    begin += DAY
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   885
  end = calendar.timegm((year, 11, 1, 2, 0, 0, 0, 0, 0))
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   886
  while time.gmtime(end).tm_wday != SUNDAY:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   887
    end += DAY
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   888
  return begin <= now < end
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   889
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   890
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   891
def CopyReversedLines(instream, outstream, blocksize=2**16):
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   892
  r"""Copy lines from input stream to output stream in reverse order.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   893
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   894
  As a special feature, null bytes in the input are turned into
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   895
  newlines followed by tabs in the output, but these 'sub-lines'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   896
  separated by null bytes are not reversed.  E.g. If the input is
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   897
  'A\0B\nC\0D\n', the output is 'C\n\tD\nA\n\tB\n'.
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   898
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   899
  Args:
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   900
    instream: A seekable stream open for reading in binary mode.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   901
    outstream: A stream open for writing; doesn't have to be seekable or binary.
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   902
    blocksize: Optional block size for buffering, for unit testing.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   903
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   904
  Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   905
    The number of lines copied.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   906
  """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   907
  line_count = 0
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   908
  instream.seek(0, 2)
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   909
  last_block = instream.tell() // blocksize
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   910
  spillover = ''
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   911
  for iblock in xrange(last_block + 1, -1, -1):
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   912
    instream.seek(iblock * blocksize)
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   913
    data = instream.read(blocksize)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   914
    lines = data.splitlines(True)
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   915
    lines[-1:] = ''.join(lines[-1:] + [spillover]).splitlines(True)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   916
    if lines and not lines[-1].endswith('\n'):
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   917
      lines[-1] += '\n'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   918
    lines.reverse()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   919
    if lines and iblock > 0:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   920
      spillover = lines.pop()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   921
    if lines:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   922
      line_count += len(lines)
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   923
      data = ''.join(lines).replace('\0', '\n\t')
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
   924
      outstream.write(data)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   925
  return line_count
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   926
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   927
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   928
def FindSentinel(filename, blocksize=2**16):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   929
  """Return the sentinel line from the output file.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   930
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   931
  Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   932
    filename: The filename of the output file.  (We'll read this file.)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   933
    blocksize: Optional block size for buffering, for unit testing.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   934
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   935
  Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   936
    The contents of the last line in the file that doesn't start with
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   937
    a tab, with its trailing newline stripped; or None if the file
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   938
    couldn't be opened or no such line could be found by inspecting
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   939
    the last 'blocksize' bytes of the file.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   940
  """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   941
  if filename == '-':
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   942
    StatusUpdate('Can\'t combine --append with output to stdout.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   943
    sys.exit(2)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   944
  try:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   945
    fp = open(filename, 'rb')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   946
  except IOError, err:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   947
    StatusUpdate('Append mode disabled: can\'t read %r: %s.' % (filename, err))
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   948
    return None
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   949
  try:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   950
    fp.seek(0, 2)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   951
    fp.seek(max(0, fp.tell() - blocksize))
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   952
    lines = fp.readlines()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   953
    del lines[:1]
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   954
    sentinel = None
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   955
    for line in lines:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   956
      if not line.startswith('\t'):
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   957
        sentinel = line
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   958
    if not sentinel:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   959
      StatusUpdate('Append mode disabled: can\'t find sentinel in %r.' %
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   960
                   filename)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   961
      return None
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
   962
    return sentinel.rstrip('\n')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   963
  finally:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   964
    fp.close()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   965
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   966
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   967
class AppVersionUpload(object):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   968
  """Provides facilities to upload a new appversion to the hosting service.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   969
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   970
  Attributes:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   971
    server: The AbstractRpcServer to use for the upload.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   972
    config: The AppInfoExternal object derived from the app.yaml file.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   973
    app_id: The application string from 'config'.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   974
    version: The version string from 'config'.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   975
    files: A dictionary of files to upload to the server, mapping path to
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   976
      hash of the file contents.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   977
    in_transaction: True iff a transaction with the server has started.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   978
      An AppVersionUpload can do only one transaction at a time.
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   979
    deployed: True iff the Deploy method has been called.
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   980
  """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   981
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   982
  def __init__(self, server, config):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   983
    """Creates a new AppVersionUpload.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   984
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   985
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   986
      server: The RPC server to use. Should be an instance of HttpRpcServer or
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   987
        TestRpcServer.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   988
      config: An AppInfoExternal object that specifies the configuration for
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   989
        this application.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   990
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   991
    self.server = server
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   992
    self.config = config
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   993
    self.app_id = self.config.application
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   994
    self.version = self.config.version
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   995
    self.files = {}
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   996
    self.in_transaction = False
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
   997
    self.deployed = False
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   998
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
   999
  def _Hash(self, content):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1000
    """Compute the hash of the content.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1001
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1002
    Args:
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1003
      content: The data to hash as a string.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1004
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1005
    Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1006
      The string representation of the hash.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1007
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1008
    h = sha.new(content).hexdigest()
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1009
    return '%s_%s_%s_%s_%s' % (h[0:8], h[8:16], h[16:24], h[24:32], h[32:40])
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1010
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1011
  def AddFile(self, path, file_handle):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1012
    """Adds the provided file to the list to be pushed to the server.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1013
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1014
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1015
      path: The path the file should be uploaded as.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1016
      file_handle: A stream containing data to upload.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1017
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1018
    assert not self.in_transaction, 'Already in a transaction.'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1019
    assert file_handle is not None
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1020
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1021
    reason = appinfo.ValidFilename(path)
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1022
    if reason:
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1023
      logging.error(reason)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1024
      return
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1025
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1026
    pos = file_handle.tell()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1027
    content_hash = self._Hash(file_handle.read())
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1028
    file_handle.seek(pos, 0)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1029
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1030
    self.files[path] = content_hash
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1031
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1032
  def Begin(self):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1033
    """Begins the transaction, returning a list of files that need uploading.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1034
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1035
    All calls to AddFile must be made before calling Begin().
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1036
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1037
    Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1038
      A list of pathnames for files that should be uploaded using UploadFile()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1039
      before Commit() can be called.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1040
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1041
    assert not self.in_transaction, 'Already in a transaction.'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1042
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1043
    StatusUpdate('Initiating update.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1044
    self.server.Send('/api/appversion/create', app_id=self.app_id,
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1045
                     version=self.version, payload=self.config.ToYAML())
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1046
    self.in_transaction = True
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1047
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1048
    files_to_clone = []
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1049
    blobs_to_clone = []
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1050
    for path, content_hash in self.files.iteritems():
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1051
      mime_type = GetMimeTypeIfStaticFile(self.config, path)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1052
      if mime_type is not None:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1053
        blobs_to_clone.append((path, content_hash, mime_type))
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1054
      else:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1055
        files_to_clone.append((path, content_hash))
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1056
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1057
    files_to_upload = {}
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1058
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1059
    def CloneFiles(url, files, file_type):
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1060
      """Sends files to the given url.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1061
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1062
      Args:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1063
        url: the server URL to use.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1064
        files: a list of files
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1065
        file_type: the type of the files
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1066
      """
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1067
      if not files:
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1068
        return
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1069
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1070
      StatusUpdate('Cloning %d %s file%s.' %
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1071
                   (len(files), file_type, len(files) != 1 and 's' or ''))
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1072
      for i in xrange(0, len(files), MAX_FILES_TO_CLONE):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1073
        if i > 0 and i % MAX_FILES_TO_CLONE == 0:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1074
          StatusUpdate('Cloned %d files.' % i)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1075
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1076
        chunk = files[i:min(len(files), i + MAX_FILES_TO_CLONE)]
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1077
        result = self.server.Send(url,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1078
                                  app_id=self.app_id, version=self.version,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1079
                                  payload=BuildClonePostBody(chunk))
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1080
        if result:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1081
          files_to_upload.update(dict(
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1082
              (f, self.files[f]) for f in result.split(LIST_DELIMITER)))
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1083
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1084
    CloneFiles('/api/appversion/cloneblobs', blobs_to_clone, 'static')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1085
    CloneFiles('/api/appversion/clonefiles', files_to_clone, 'application')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1086
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1087
    logging.info('Files to upload: ' + str(files_to_upload))
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1088
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1089
    self.files = files_to_upload
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1090
    return sorted(files_to_upload.iterkeys())
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1091
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1092
  def UploadFile(self, path, file_handle):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1093
    """Uploads a file to the hosting service.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1094
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1095
    Must only be called after Begin().
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1096
    The path provided must be one of those that were returned by Begin().
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1097
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1098
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1099
      path: The path the file is being uploaded as.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1100
      file_handle: A file-like object containing the data to upload.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1101
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1102
    Raises:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1103
      KeyError: The provided file is not amongst those to be uploaded.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1104
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1105
    assert self.in_transaction, 'Begin() must be called before UploadFile().'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1106
    if path not in self.files:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1107
      raise KeyError('File \'%s\' is not in the list of files to be uploaded.'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1108
                     % path)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1110
    del self.files[path]
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1111
    mime_type = GetMimeTypeIfStaticFile(self.config, path)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1112
    if mime_type is not None:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1113
      self.server.Send('/api/appversion/addblob', app_id=self.app_id,
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1114
                       version=self.version, path=path, content_type=mime_type,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1115
                       payload=file_handle.read())
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1116
    else:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1117
      self.server.Send('/api/appversion/addfile', app_id=self.app_id,
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1118
                       version=self.version, path=path,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1119
                       payload=file_handle.read())
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1120
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1121
  def Commit(self):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1122
    """Commits the transaction, making the new app version available.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1123
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1124
    All the files returned by Begin() must have been uploaded with UploadFile()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1125
    before Commit() can be called.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1126
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1127
    This tries the new 'deploy' method; if that fails it uses the old 'commit'.
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1128
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1129
    Raises:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1130
      Exception: Some required files were not uploaded.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1131
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1132
    assert self.in_transaction, 'Begin() must be called before Commit().'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1133
    if self.files:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1134
      raise Exception('Not all required files have been uploaded.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1135
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1136
    try:
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1137
      self.Deploy()
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1138
      if not RetryWithBackoff(1, 2, 8, self.IsReady):
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1139
        logging.warning('Version still not ready to serve, aborting.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1140
        raise Exception('Version not ready.')
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1141
      self.StartServing()
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1142
    except urllib2.HTTPError, e:
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1143
      if e.code != 404:
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1144
        raise
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1145
      StatusUpdate('Closing update.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1146
      self.server.Send('/api/appversion/commit', app_id=self.app_id,
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1147
                       version=self.version)
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1148
      self.in_transaction = False
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1149
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1150
  def Deploy(self):
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1151
    """Deploys the new app version but does not make it default.
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1152
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1153
    All the files returned by Begin() must have been uploaded with UploadFile()
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1154
    before Deploy() can be called.
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1155
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1156
    Raises:
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1157
      Exception: Some required files were not uploaded.
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1158
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1159
    assert self.in_transaction, 'Begin() must be called before Deploy().'
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1160
    if self.files:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1161
      raise Exception('Not all required files have been uploaded.')
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1162
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1163
    StatusUpdate('Deploying new version.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1164
    self.server.Send('/api/appversion/deploy', app_id=self.app_id,
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1165
                     version=self.version)
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1166
    self.deployed = True
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1167
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1168
  def IsReady(self):
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1169
    """Check if the new app version is ready to serve traffic.
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1170
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1171
    Raises:
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1172
      Exception: Deploy has not yet been called.
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1173
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1174
    Returns:
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1175
      True if the server returned the app is ready to serve.
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1176
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1177
    assert self.deployed, 'Deploy() must be called before IsReady().'
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1178
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1179
    StatusUpdate('Checking if new version is ready to serve.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1180
    result = self.server.Send('/api/appversion/isready', app_id=self.app_id,
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1181
                              version=self.version)
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1182
    return result == '1'
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1183
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1184
  def StartServing(self):
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1185
    """Start serving with the newly created version.
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1186
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1187
    Raises:
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1188
      Exception: Deploy has not yet been called.
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1189
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1190
    assert self.deployed, 'Deploy() must be called before IsReady().'
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1191
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1192
    StatusUpdate('Closing update: new version is ready to start serving.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1193
    self.server.Send('/api/appversion/startserving',
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1194
                     app_id=self.app_id, version=self.version)
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1195
    self.in_transaction = False
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1196
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1197
  def Rollback(self):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1198
    """Rolls back the transaction if one is in progress."""
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1199
    if not self.in_transaction:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1200
      return
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1201
    StatusUpdate('Rolling back the update.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1202
    self.server.Send('/api/appversion/rollback', app_id=self.app_id,
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1203
                     version=self.version)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1204
    self.in_transaction = False
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1205
    self.files = {}
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1206
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1207
  def DoUpload(self, paths, max_size, openfunc):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1208
    """Uploads a new appversion with the given config and files to the server.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1209
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1210
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1211
      paths: An iterator that yields the relative paths of the files to upload.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1212
      max_size: The maximum size file to upload.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1213
      openfunc: A function that takes a path and returns a file-like object.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1214
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1215
    logging.info('Reading app configuration.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1216
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1217
    path = ''
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1218
    try:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1219
      StatusUpdate('Scanning files on local disk.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1220
      num_files = 0
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1221
      for path in paths:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1222
        file_handle = openfunc(path)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1223
        try:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1224
          if self.config.skip_files.match(path):
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1225
            logging.info('Ignoring file \'%s\': File matches ignore regex.',
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1226
                         path)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1227
          else:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1228
            file_length = GetFileLength(file_handle)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1229
            if file_length > max_size:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1230
              logging.error('Ignoring file \'%s\': Too long '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1231
                            '(max %d bytes, file is %d bytes)',
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1232
                            path, max_size, file_length)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1233
            else:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1234
              logging.info('Processing file \'%s\'', path)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1235
              self.AddFile(path, file_handle)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1236
        finally:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1237
          file_handle.close()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1238
        num_files += 1
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1239
        if num_files % 500 == 0:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1240
          StatusUpdate('Scanned %d files.' % num_files)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1241
    except KeyboardInterrupt:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1242
      logging.info('User interrupted. Aborting.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1243
      raise
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1244
    except EnvironmentError, e:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1245
      logging.error('An error occurred processing file \'%s\': %s. Aborting.',
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1246
                    path, e)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1247
      raise
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1248
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1249
    try:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1250
      missing_files = self.Begin()
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1251
      if missing_files:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1252
        StatusUpdate('Uploading %d files.' % len(missing_files))
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1253
        num_files = 0
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1254
        for missing_file in missing_files:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1255
          logging.info('Uploading file \'%s\'' % missing_file)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1256
          file_handle = openfunc(missing_file)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1257
          try:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1258
            self.UploadFile(missing_file, file_handle)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1259
          finally:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1260
            file_handle.close()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1261
          num_files += 1
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1262
          if num_files % 500 == 0:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1263
            StatusUpdate('Uploaded %d files.' % num_files)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1264
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1265
      self.Commit()
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1266
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1267
    except KeyboardInterrupt:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1268
      logging.info('User interrupted. Aborting.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1269
      self.Rollback()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1270
      raise
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1271
    except:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1272
      logging.exception('An unexpected error occurred. Aborting.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1273
      self.Rollback()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1274
      raise
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1275
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1276
    logging.info('Done!')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1277
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1278
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1279
def FileIterator(base, separator=os.path.sep):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1280
  """Walks a directory tree, returning all the files. Follows symlinks.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1281
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1282
  Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1283
    base: The base path to search for files under.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1284
    separator: Path separator used by the running system's platform.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1285
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1286
  Yields:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1287
    Paths of files found, relative to base.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1288
  """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1289
  dirs = ['']
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1290
  while dirs:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1291
    current_dir = dirs.pop()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1292
    for entry in os.listdir(os.path.join(base, current_dir)):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1293
      name = os.path.join(current_dir, entry)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1294
      fullname = os.path.join(base, name)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1295
      if os.path.isfile(fullname):
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1296
        if separator == '\\':
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1297
          name = name.replace('\\', '/')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1298
        yield name
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1299
      elif os.path.isdir(fullname):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1300
        dirs.append(name)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1301
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1302
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1303
def GetFileLength(fh):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1304
  """Returns the length of the file represented by fh.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1305
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1306
  This function is capable of finding the length of any seekable stream,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1307
  unlike os.fstat, which only works on file streams.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1308
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1309
  Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1310
    fh: The stream to get the length of.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1311
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1312
  Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1313
    The length of the stream.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1314
  """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1315
  pos = fh.tell()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1316
  fh.seek(0, 2)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1317
  length = fh.tell()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1318
  fh.seek(pos, 0)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1319
  return length
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1320
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1321
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1322
def GetUserAgent(get_version=GetVersionObject,
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1323
                 get_platform=appengine_rpc.GetPlatformToken):
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1324
  """Determines the value of the 'User-agent' header to use for HTTP requests.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1325
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1326
  If the 'APPCFG_SDK_NAME' environment variable is present, that will be
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1327
  used as the first product token in the user-agent.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1328
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1329
  Args:
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1330
    get_version: Used for testing.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1331
    get_platform: Used for testing.
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1332
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1333
  Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1334
    String containing the 'user-agent' header value, which includes the SDK
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1335
    version, the platform information, and the version of Python;
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1336
    e.g., 'appcfg_py/1.0.1 Darwin/9.2.0 Python/2.5.2'.
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1337
  """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1338
  product_tokens = []
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1339
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1340
  sdk_name = os.environ.get('APPCFG_SDK_NAME')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1341
  if sdk_name:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1342
    product_tokens.append(sdk_name)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1343
  else:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1344
    version = get_version()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1345
    if version is None:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1346
      release = 'unknown'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1347
    else:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1348
      release = version['release']
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1349
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1350
    product_tokens.append('appcfg_py/%s' % release)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1351
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1352
  product_tokens.append(get_platform())
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1353
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1354
  python_version = '.'.join(str(i) for i in sys.version_info)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1355
  product_tokens.append('Python/%s' % python_version)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1356
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1357
  return ' '.join(product_tokens)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1358
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1359
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1360
def GetSourceName(get_version=GetVersionObject):
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1361
  """Gets the name of this source version."""
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1362
  version = get_version()
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1363
  if version is None:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1364
    release = 'unknown'
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1365
  else:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1366
    release = version['release']
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1367
  return 'Google-appcfg-%s' % (release,)
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1368
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1369
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1370
class AppCfgApp(object):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1371
  """Singleton class to wrap AppCfg tool functionality.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1372
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1373
  This class is responsible for parsing the command line and executing
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1374
  the desired action on behalf of the user.  Processing files and
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1375
  communicating with the server is handled by other classes.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1376
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1377
  Attributes:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1378
    actions: A dictionary mapping action names to Action objects.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1379
    action: The Action specified on the command line.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1380
    parser: An instance of optparse.OptionParser.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1381
    options: The command line options parsed by 'parser'.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1382
    argv: The original command line as a list.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1383
    args: The positional command line args left over after parsing the options.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1384
    raw_input_fn: Function used for getting raw user input, like email.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1385
    password_input_fn: Function used for getting user password.
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
  1386
    error_fh: Unexpected HTTPErrors are printed to this file handle.
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1387
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1388
  Attributes for testing:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1389
    parser_class: The class to use for parsing the command line.  Because
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1390
      OptionsParser will exit the program when there is a parse failure, it
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1391
      is nice to subclass OptionsParser and catch the error before exiting.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1392
  """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1393
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1394
  def __init__(self, argv, parser_class=optparse.OptionParser,
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1395
               rpc_server_class=appengine_rpc.HttpRpcServer,
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1396
               raw_input_fn=raw_input,
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
  1397
               password_input_fn=getpass.getpass,
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1398
               error_fh=sys.stderr,
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1399
               update_check_class=UpdateCheck):
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1400
    """Initializer.  Parses the cmdline and selects the Action to use.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1401
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1402
    Initializes all of the attributes described in the class docstring.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1403
    Prints help or error messages if there is an error parsing the cmdline.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1404
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1405
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1406
      argv: The list of arguments passed to this program.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1407
      parser_class: Options parser to use for this application.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1408
      rpc_server_class: RPC server class to use for this application.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1409
      raw_input_fn: Function used for getting user email.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1410
      password_input_fn: Function used for getting user password.
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
  1411
      error_fh: Unexpected HTTPErrors are printed to this file handle.
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1412
      update_check_class: UpdateCheck class (can be replaced for testing).
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1413
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1414
    self.parser_class = parser_class
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1415
    self.argv = argv
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1416
    self.rpc_server_class = rpc_server_class
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1417
    self.raw_input_fn = raw_input_fn
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1418
    self.password_input_fn = password_input_fn
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
  1419
    self.error_fh = error_fh
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1420
    self.update_check_class = update_check_class
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1421
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1422
    self.parser = self._GetOptionParser()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1423
    for action in self.actions.itervalues():
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1424
      action.options(self, self.parser)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1425
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1426
    self.options, self.args = self.parser.parse_args(argv[1:])
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1427
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1428
    if len(self.args) < 1:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1429
      self._PrintHelpAndExit()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1430
    if self.args[0] not in self.actions:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1431
      self.parser.error('Unknown action \'%s\'\n%s' %
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1432
                        (self.args[0], self.parser.get_description()))
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1433
    action_name = self.args.pop(0)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1434
    self.action = self.actions[action_name]
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1435
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1436
    self.parser, self.options = self._MakeSpecificParser(self.action)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1437
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1438
    if self.options.help:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1439
      self._PrintHelpAndExit()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1440
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1441
    if self.options.verbose == 2:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1442
      logging.getLogger().setLevel(logging.INFO)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1443
    elif self.options.verbose == 3:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1444
      logging.getLogger().setLevel(logging.DEBUG)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1445
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1446
    global verbosity
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1447
    verbosity = self.options.verbose
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1448
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
  1449
  def Run(self):
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1450
    """Executes the requested action.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1451
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1452
    Catches any HTTPErrors raised by the action and prints them to stderr.
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1453
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1454
    Returns:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1455
      1 on error, 0 if successful.
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1456
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1457
    try:
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1458
      self.action(self)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1459
    except urllib2.HTTPError, e:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1460
      body = e.read()
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1461
      print >>self.error_fh, ('Error %d: --- begin server output ---\n'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1462
                              '%s\n--- end server output ---' %
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1463
                              (e.code, body.rstrip('\n')))
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1464
      return 1
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1465
    except yaml_errors.EventListenerError, e:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1466
      print >>self.error_fh, ('Error parsing yaml file:\n%s' % e)
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1467
      return 1
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1468
    return 0
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1469
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1470
  def _GetActionDescriptions(self):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1471
    """Returns a formatted string containing the short_descs for all actions."""
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1472
    action_names = self.actions.keys()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1473
    action_names.sort()
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1474
    desc = ''
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1475
    for action_name in action_names:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1476
      desc += '  %s: %s\n' % (action_name, self.actions[action_name].short_desc)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1477
    return desc
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1478
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1479
  def _GetOptionParser(self):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1480
    """Creates an OptionParser with generic usage and description strings.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1481
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1482
    Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1483
      An OptionParser instance.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1484
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1485
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1486
    class Formatter(optparse.IndentedHelpFormatter):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1487
      """Custom help formatter that does not reformat the description."""
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1488
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1489
      def format_description(self, description):
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1490
        """Very simple formatter."""
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1491
        return description + '\n'
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1492
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1493
    desc = self._GetActionDescriptions()
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1494
    desc = ('Action must be one of:\n%s'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1495
            'Use \'help <action>\' for a detailed description.') % desc
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1496
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1497
    parser = self.parser_class(usage='%prog [options] <action>',
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1498
                               description=desc,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1499
                               formatter=Formatter(),
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1500
                               conflict_handler='resolve')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1501
    parser.add_option('-h', '--help', action='store_true',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1502
                      dest='help', help='Show the help message and exit.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1503
    parser.add_option('-q', '--quiet', action='store_const', const=0,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1504
                      dest='verbose', help='Print errors only.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1505
    parser.add_option('-v', '--verbose', action='store_const', const=2,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1506
                      dest='verbose', default=1,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1507
                      help='Print info level logs.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1508
    parser.add_option('--noisy', action='store_const', const=3,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1509
                      dest='verbose', help='Print all logs.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1510
    parser.add_option('-s', '--server', action='store', dest='server',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1511
                      default='appengine.google.com',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1512
                      metavar='SERVER', help='The server to connect to.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1513
    parser.add_option('--secure', action='store_true', dest='secure',
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1514
                      default=False,
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1515
                      help='Use SSL when communicating with the server.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1516
    parser.add_option('-e', '--email', action='store', dest='email',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1517
                      metavar='EMAIL', default=None,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1518
                      help='The username to use. Will prompt if omitted.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1519
    parser.add_option('-H', '--host', action='store', dest='host',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1520
                      metavar='HOST', default=None,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1521
                      help='Overrides the Host header sent with all RPCs.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1522
    parser.add_option('--no_cookies', action='store_false',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1523
                      dest='save_cookies', default=True,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1524
                      help='Do not save authentication cookies to local disk.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1525
    parser.add_option('--passin', action='store_true',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1526
                      dest='passin', default=False,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1527
                      help='Read the login password from stdin.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1528
    return parser
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1529
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1530
  def _MakeSpecificParser(self, action):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1531
    """Creates a new parser with documentation specific to 'action'.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1532
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1533
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1534
      action: An Action instance to be used when initializing the new parser.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1535
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1536
    Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1537
      A tuple containing:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1538
      parser: An instance of OptionsParser customized to 'action'.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1539
      options: The command line options after re-parsing.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1540
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1541
    parser = self._GetOptionParser()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1542
    parser.set_usage(action.usage)
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1543
    parser.set_description('%s\n%s' % (action.short_desc, action.long_desc))
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1544
    action.options(self, parser)
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1545
    options, unused_args = parser.parse_args(self.argv[1:])
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1546
    return parser, options
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1547
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1548
  def _PrintHelpAndExit(self, exit_code=2):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1549
    """Prints the parser's help message and exits the program.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1550
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1551
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1552
      exit_code: The integer code to pass to sys.exit().
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1553
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1554
    self.parser.print_help()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1555
    sys.exit(exit_code)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1556
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1557
  def _GetRpcServer(self):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1558
    """Returns an instance of an AbstractRpcServer.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1559
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1560
    Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1561
      A new AbstractRpcServer, on which RPC calls can be made.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1562
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1563
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1564
    def GetUserCredentials():
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1565
      """Prompts the user for a username and password."""
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1566
      email = self.options.email
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1567
      if email is None:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1568
        email = self.raw_input_fn('Email: ')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1569
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1570
      password_prompt = 'Password for %s: ' % email
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1571
      if self.options.passin:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1572
        password = self.raw_input_fn(password_prompt)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1573
      else:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1574
        password = self.password_input_fn(password_prompt)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1575
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1576
      return (email, password)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1577
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1578
    if self.options.host and self.options.host == 'localhost':
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1579
      email = self.options.email
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1580
      if email is None:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1581
        email = 'test@example.com'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1582
        logging.info('Using debug user %s.  Override with --email' % email)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1583
      server = self.rpc_server_class(
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1584
          self.options.server,
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1585
          lambda: (email, 'password'),
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1586
          GetUserAgent(),
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1587
          GetSourceName(),
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1588
          host_override=self.options.host,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1589
          save_cookies=self.options.save_cookies)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1590
      server.authenticated = True
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1591
      return server
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1592
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1593
    if self.options.passin:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1594
      auth_tries = 1
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1595
    else:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1596
      auth_tries = 3
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1597
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1598
    return self.rpc_server_class(self.options.server, GetUserCredentials,
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1599
                                 GetUserAgent(), GetSourceName(),
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1600
                                 host_override=self.options.host,
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1601
                                 save_cookies=self.options.save_cookies,
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1602
                                 auth_tries=auth_tries,
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1603
                                 account_type='HOSTED_OR_GOOGLE',
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1604
                                 secure=self.options.secure)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1605
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1606
  def _FindYaml(self, basepath, file_name):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1607
    """Find yaml files in application directory.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1608
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1609
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1610
      basepath: Base application directory.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1611
      file_name: Filename without extension to search for.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1612
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1613
    Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1614
      Path to located yaml file if one exists, else None.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1615
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1616
    if not os.path.isdir(basepath):
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1617
      self.parser.error('Not a directory: %s' % basepath)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1618
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1619
    for yaml_file in (file_name + '.yaml', file_name + '.yml'):
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1620
      yaml_path = os.path.join(basepath, yaml_file)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1621
      if os.path.isfile(yaml_path):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1622
        return yaml_path
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1623
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1624
    return None
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1625
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1626
  def _ParseAppYaml(self, basepath):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1627
    """Parses the app.yaml file.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1628
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1629
    Args:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1630
      basepath: the directory of the application.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1631
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1632
    Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1633
      An AppInfoExternal object.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1634
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1635
    appyaml_filename = self._FindYaml(basepath, 'app')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1636
    if appyaml_filename is None:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1637
      self.parser.error('Directory does not contain an app.yaml '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1638
                        'configuration file.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1639
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1640
    fh = open(appyaml_filename, 'r')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1641
    try:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1642
      appyaml = appinfo.LoadSingleAppInfo(fh)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1643
    finally:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1644
      fh.close()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1645
    return appyaml
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1646
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1647
  def _ParseYamlFile(self, basepath, basename, parser):
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1648
    """Parses the a yaml file.
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1649
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1650
    Args:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1651
      basepath: the directory of the application.
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1652
      basename: the base name of the file (with the '.yaml' stripped off).
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1653
      parser: the function or method used to parse the file.
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1654
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1655
    Returns:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1656
      A single parsed yaml file or None if the file does not exist.
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1657
    """
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1658
    file_name = self._FindYaml(basepath, basename)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1659
    if file_name is not None:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1660
      fh = open(file_name, 'r')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1661
      try:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1662
        defns = parser(fh)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1663
      finally:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1664
        fh.close()
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1665
      return defns
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1666
    return None
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1667
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1668
  def _ParseIndexYaml(self, basepath):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1669
    """Parses the index.yaml file.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1670
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1671
    Args:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1672
      basepath: the directory of the application.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1673
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1674
    Returns:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1675
      A single parsed yaml file or None if the file does not exist.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1676
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1677
    return self._ParseYamlFile(basepath, 'index',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1678
                               datastore_index.ParseIndexDefinitions)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1679
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1680
  def _ParseCronYaml(self, basepath):
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1681
    """Parses the cron.yaml file.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1682
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1683
    Args:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1684
      basepath: the directory of the application.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1685
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1686
    Returns:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1687
      A CronInfoExternal object or None if the file does not exist.
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1688
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1689
    return self._ParseYamlFile(basepath, 'cron', croninfo.LoadSingleCron)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1690
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1691
  def _ParseQueueYaml(self, basepath):
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1692
    """Parses the queue.yaml file.
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1693
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1694
    Args:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1695
      basepath: the directory of the application.
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1696
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1697
    Returns:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1698
      A CronInfoExternal object or None if the file does not exist.
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1699
    """
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1700
    return self._ParseYamlFile(basepath, 'queue', queueinfo.LoadSingleQueue)
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1701
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1702
  def Help(self):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1703
    """Prints help for a specific action.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1704
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1705
    Expects self.args[0] to contain the name of the action in question.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1706
    Exits the program after printing the help message.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1707
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1708
    if len(self.args) != 1 or self.args[0] not in self.actions:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1709
      self.parser.error('Expected a single action argument. Must be one of:\n' +
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1710
                        self._GetActionDescriptions())
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1711
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1712
    action = self.actions[self.args[0]]
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1713
    self.parser, unused_options = self._MakeSpecificParser(action)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1714
    self._PrintHelpAndExit(exit_code=0)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1715
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1716
  def Update(self):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1717
    """Updates and deploys a new appversion."""
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1718
    if len(self.args) != 1:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1719
      self.parser.error('Expected a single <directory> argument.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1720
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1721
    basepath = self.args[0]
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1722
    appyaml = self._ParseAppYaml(basepath)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1723
    rpc_server = self._GetRpcServer()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1724
1278
a7766286a7be Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 828
diff changeset
  1725
    updatecheck = self.update_check_class(rpc_server, appyaml)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1726
    updatecheck.CheckForUpdates()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1727
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1728
    appversion = AppVersionUpload(rpc_server, appyaml)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1729
    appversion.DoUpload(FileIterator(basepath), self.options.max_size,
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1730
                        lambda path: open(os.path.join(basepath, path), 'rb'))
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1731
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1732
    index_defs = self._ParseIndexYaml(basepath)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1733
    if index_defs:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1734
      index_upload = IndexDefinitionUpload(rpc_server, appyaml, index_defs)
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
  1735
      try:
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
  1736
        index_upload.DoUpload()
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
  1737
      except urllib2.HTTPError, e:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1738
        StatusUpdate('Error %d: --- begin server output ---\n'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1739
                     '%s\n--- end server output ---' %
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1740
                     (e.code, e.read().rstrip('\n')))
686
df109be0567c Load ../../google_appengine_1.1.7/ into
Sverre Rabbelier <srabbelier@gmail.com>
parents: 149
diff changeset
  1741
        print >> self.error_fh, (
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1742
            'Your app was updated, but there was an error updating your '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1743
            'indexes. Please retry later with appcfg.py update_indexes.')
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1744
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1745
    cron_entries = self._ParseCronYaml(basepath)
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1746
    if cron_entries:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1747
      cron_upload = CronEntryUpload(rpc_server, appyaml, cron_entries)
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1748
      cron_upload.DoUpload()
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1749
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1750
    queue_entries = self._ParseQueueYaml(basepath)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1751
    if queue_entries:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1752
      queue_upload = QueueEntryUpload(rpc_server, appyaml, queue_entries)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1753
      queue_upload.DoUpload()
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1754
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1755
  def _UpdateOptions(self, parser):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1756
    """Adds update-specific options to 'parser'.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1757
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1758
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1759
      parser: An instance of OptionsParser.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1760
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1761
    parser.add_option('-S', '--max_size', type='int', dest='max_size',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1762
                      default=10485760, metavar='SIZE',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1763
                      help='Maximum size of a file to upload.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1764
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1765
  def VacuumIndexes(self):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1766
    """Deletes unused indexes."""
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1767
    if len(self.args) != 1:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1768
      self.parser.error('Expected a single <directory> argument.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1769
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1770
    basepath = self.args[0]
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1771
    config = self._ParseAppYaml(basepath)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1772
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1773
    index_defs = self._ParseIndexYaml(basepath)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1774
    if index_defs is None:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1775
      index_defs = datastore_index.IndexDefinitions()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1776
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1777
    rpc_server = self._GetRpcServer()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1778
    vacuum = VacuumIndexesOperation(rpc_server,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1779
                                    config,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1780
                                    self.options.force_delete)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1781
    vacuum.DoVacuum(index_defs)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1782
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1783
  def _VacuumIndexesOptions(self, parser):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1784
    """Adds vacuum_indexes-specific options to 'parser'.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1785
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1786
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1787
      parser: An instance of OptionsParser.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1788
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1789
    parser.add_option('-f', '--force', action='store_true', dest='force_delete',
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1790
                      default=False,
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1791
                      help='Force deletion without being prompted.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1792
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1793
  def UpdateCron(self):
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1794
    """Updates any new or changed cron definitions."""
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1795
    if len(self.args) != 1:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1796
      self.parser.error('Expected a single <directory> argument.')
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1797
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1798
    basepath = self.args[0]
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1799
    appyaml = self._ParseAppYaml(basepath)
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1800
    rpc_server = self._GetRpcServer()
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1801
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1802
    cron_entries = self._ParseCronYaml(basepath)
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1803
    if cron_entries:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1804
      cron_upload = CronEntryUpload(rpc_server, appyaml, cron_entries)
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1805
      cron_upload.DoUpload()
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1806
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1807
  def UpdateIndexes(self):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1808
    """Updates indexes."""
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1809
    if len(self.args) != 1:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1810
      self.parser.error('Expected a single <directory> argument.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1811
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1812
    basepath = self.args[0]
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1813
    appyaml = self._ParseAppYaml(basepath)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1814
    rpc_server = self._GetRpcServer()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1815
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1816
    index_defs = self._ParseIndexYaml(basepath)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1817
    if index_defs:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1818
      index_upload = IndexDefinitionUpload(rpc_server, appyaml, index_defs)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1819
      index_upload.DoUpload()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1820
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1821
  def UpdateQueues(self):
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1822
    """Updates any new or changed task queue definitions."""
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1823
    if len(self.args) != 1:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1824
      self.parser.error('Expected a single <directory> argument.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1825
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1826
    basepath = self.args[0]
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1827
    appyaml = self._ParseAppYaml(basepath)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1828
    rpc_server = self._GetRpcServer()
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1829
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1830
    queue_entries = self._ParseQueueYaml(basepath)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1831
    if queue_entries:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1832
      queue_upload = QueueEntryUpload(rpc_server, appyaml, queue_entries)
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1833
      queue_upload.DoUpload()
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1834
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1835
  def Rollback(self):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1836
    """Does a rollback of any existing transaction for this app version."""
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1837
    if len(self.args) != 1:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1838
      self.parser.error('Expected a single <directory> argument.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1839
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1840
    basepath = self.args[0]
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1841
    appyaml = self._ParseAppYaml(basepath)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1842
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1843
    appversion = AppVersionUpload(self._GetRpcServer(), appyaml)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1844
    appversion.in_transaction = True
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1845
    appversion.Rollback()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1846
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1847
  def RequestLogs(self):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1848
    """Write request logs to a file."""
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1849
    if len(self.args) != 2:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1850
      self.parser.error(
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1851
          'Expected a <directory> argument and an <output_file> argument.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1852
    if (self.options.severity is not None and
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1853
        not 0 <= self.options.severity <= MAX_LOG_LEVEL):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1854
      self.parser.error(
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1855
          'Severity range is 0 (DEBUG) through %s (CRITICAL).' % MAX_LOG_LEVEL)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1856
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1857
    if self.options.num_days is None:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1858
      self.options.num_days = int(not self.options.append)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1859
    basepath = self.args[0]
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1860
    appyaml = self._ParseAppYaml(basepath)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1861
    rpc_server = self._GetRpcServer()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1862
    logs_requester = LogsRequester(rpc_server, appyaml, self.args[1],
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1863
                                   self.options.num_days,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1864
                                   self.options.append,
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1865
                                   self.options.severity,
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  1866
                                   time.time(),
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1867
                                   self.options.vhost,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1868
                                   self.options.include_vhost)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1869
    logs_requester.DownloadLogs()
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1870
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1871
  def _RequestLogsOptions(self, parser):
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1872
    """Adds request_logs-specific options to 'parser'.
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1873
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1874
    Args:
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1875
      parser: An instance of OptionsParser.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1876
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1877
    parser.add_option('-n', '--num_days', type='int', dest='num_days',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1878
                      action='store', default=None,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1879
                      help='Number of days worth of log data to get. '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1880
                      'The cut-off point is midnight UTC. '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1881
                      'Use 0 to get all available logs. '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1882
                      'Default is 1, unless --append is also given; '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1883
                      'then the default is 0.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1884
    parser.add_option('-a', '--append', dest='append',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1885
                      action='store_true', default=False,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1886
                      help='Append to existing file.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1887
    parser.add_option('--severity', type='int', dest='severity',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1888
                      action='store', default=None,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1889
                      help='Severity of app-level log messages to get. '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1890
                      'The range is 0 (DEBUG) through 4 (CRITICAL). '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1891
                      'If omitted, only request logs are returned.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1892
    parser.add_option('--vhost', type='string', dest='vhost',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1893
                      action='store', default=None,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1894
                      help='The virtual host of log messages to get. '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1895
                      'If omitted, all log messages are returned.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1896
    parser.add_option('--include_vhost', dest='include_vhost',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1897
                      action='store_true', default=False,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1898
                      help='Include virtual host in log messages.')
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1899
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1900
  def CronInfo(self, now=None, output=sys.stdout):
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1901
    """Displays information about cron definitions.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1902
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1903
    Args:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1904
      now: used for testing.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1905
      output: Used for testing.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1906
    """
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1907
    if len(self.args) != 1:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1908
      self.parser.error('Expected a single <directory> argument.')
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1909
    if now is None:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1910
      now = datetime.datetime.now()
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1911
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1912
    basepath = self.args[0]
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1913
    cron_entries = self._ParseCronYaml(basepath)
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1914
    if cron_entries and cron_entries.cron:
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1915
      for entry in cron_entries.cron:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1916
        description = entry.description
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1917
        if not description:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1918
          description = '<no description>'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1919
        print >>output, '\n%s:\nURL: %s\nSchedule: %s' % (description,
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  1920
                                                          entry.url,
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  1921
                                                          entry.schedule)
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1922
        schedule = groctimespecification.GrocTimeSpecification(entry.schedule)
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1923
        matches = schedule.GetMatches(now, self.options.num_runs)
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1924
        for match in matches:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1925
          print >>output, '%s, %s from now' % (
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1926
              match.strftime('%Y-%m-%d %H:%M:%S'), match - now)
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1927
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1928
  def _CronInfoOptions(self, parser):
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1929
    """Adds cron_info-specific options to 'parser'.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1930
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1931
    Args:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1932
      parser: An instance of OptionsParser.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  1933
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1934
    parser.add_option('-n', '--num_runs', type='int', dest='num_runs',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1935
                      action='store', default=5,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1936
                      help='Number of runs of each cron job to display'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1937
                      'Default is 5')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  1938
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  1939
  def _CheckRequiredLoadOptions(self):
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  1940
    """Checks that upload/download options are present."""
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1941
    for option in ['filename', 'kind', 'config_file']:
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1942
      if getattr(self.options, option) is None:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1943
        self.parser.error('Option \'%s\' is required.' % option)
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1944
    if not self.options.url:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1945
      self.parser.error('You must have google.appengine.ext.remote_api.handler '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1946
                        'assigned to an endpoint in app.yaml, or provide '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1947
                        'the url of the handler via the \'url\' option.')
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1948
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  1949
  def InferRemoteApiUrl(self, appyaml):
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1950
    """Uses app.yaml to determine the remote_api endpoint.
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1951
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1952
    Args:
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1953
      appyaml: A parsed app.yaml file.
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1954
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1955
    Returns:
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1956
      The url of the remote_api endpoint as a string, or None
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1957
    """
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1958
    handlers = appyaml.handlers
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1959
    handler_suffix = 'remote_api/handler.py'
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1960
    app_id = appyaml.application
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1961
    for handler in handlers:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1962
      if hasattr(handler, 'script') and handler.script:
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1963
        if handler.script.endswith(handler_suffix):
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1964
          server = self.options.server
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1965
          if server == 'appengine.google.com':
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1966
            return 'http://%s.appspot.com%s' % (app_id, handler.url)
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1967
          else:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1968
            return 'http://%s%s' % (server, handler.url)
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1969
    return None
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1970
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  1971
  def RunBulkloader(self, arg_dict):
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1972
    """Invokes the bulkloader with the given keyword arguments.
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1973
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1974
    Args:
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  1975
      arg_dict: Dictionary of arguments to pass to bulkloader.Run().
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1976
    """
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1977
    try:
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1978
      import sqlite3
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1979
    except ImportError:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1980
      logging.error('upload_data action requires SQLite3 and the python '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1981
                    'sqlite3 module (included in python since 2.5).')
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1982
      sys.exit(1)
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1983
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  1984
    sys.exit(bulkloader.Run(arg_dict))
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1985
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  1986
  def _SetupLoad(self):
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  1987
    """Performs common verification and set up for upload and download."""
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1988
    if len(self.args) != 1:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  1989
      self.parser.error('Expected <directory> argument.')
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1990
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1991
    basepath = self.args[0]
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1992
    appyaml = self._ParseAppYaml(basepath)
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1993
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1994
    self.options.app_id = appyaml.application
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1995
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1996
    if not self.options.url:
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  1997
      url = self.InferRemoteApiUrl(appyaml)
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1998
      if url is not None:
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  1999
        self.options.url = url
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2000
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2001
    self._CheckRequiredLoadOptions()
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2002
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2003
    if self.options.batch_size < 1:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2004
      self.parser.error('batch_size must be 1 or larger.')
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2005
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2006
    if verbosity == 1:
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2007
      logging.getLogger().setLevel(logging.INFO)
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2008
      self.options.debug = False
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2009
    else:
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2010
      logging.getLogger().setLevel(logging.DEBUG)
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2011
      self.options.debug = True
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2012
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2013
  def _MakeLoaderArgs(self):
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2014
    return dict([(arg_name, getattr(self.options, arg_name, None)) for
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2015
                 arg_name in (
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2016
                     'app_id',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2017
                     'url',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2018
                     'filename',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2019
                     'batch_size',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2020
                     'kind',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2021
                     'num_threads',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2022
                     'bandwidth_limit',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2023
                     'rps_limit',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2024
                     'http_limit',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2025
                     'db_filename',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2026
                     'config_file',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2027
                     'auth_domain',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2028
                     'has_header',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2029
                     'loader_opts',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2030
                     'log_file',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2031
                     'passin',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2032
                     'email',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2033
                     'debug',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2034
                     'exporter_opts',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2035
                     'result_db_filename',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2036
                     )])
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2037
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2038
  def PerformDownload(self, run_fn=None):
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2039
    """Performs a datastore download via the bulkloader.
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2040
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2041
    Args:
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2042
      run_fn: Function to invoke the bulkloader, used for testing.
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2043
    """
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2044
    if run_fn is None:
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2045
      run_fn = self.RunBulkloader
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2046
    self._SetupLoad()
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2047
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2048
    StatusUpdate('Downloading data records.')
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2049
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2050
    args = self._MakeLoaderArgs()
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2051
    args['download'] = True
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2052
    args['has_header'] = False
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2053
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2054
    run_fn(args)
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2055
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2056
  def PerformUpload(self, run_fn=None):
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2057
    """Performs a datastore upload via the bulkloader.
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2058
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2059
    Args:
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2060
      run_fn: Function to invoke the bulkloader, used for testing.
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2061
    """
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2062
    if run_fn is None:
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2063
      run_fn = self.RunBulkloader
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2064
    self._SetupLoad()
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2065
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2066
    StatusUpdate('Uploading data records.')
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2067
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2068
    args = self._MakeLoaderArgs()
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2069
    args['download'] = False
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2070
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2071
    run_fn(args)
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2072
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2073
  def _PerformLoadOptions(self, parser):
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2074
    """Adds options common to 'upload_data' and 'download_data'.
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2075
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2076
    Args:
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2077
      parser: An instance of OptionsParser.
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2078
    """
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2079
    parser.add_option('--filename', type='string', dest='filename',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2080
                      action='store',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2081
                      help='The name of the file containing the input data.'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2082
                      ' (Required)')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2083
    parser.add_option('--config_file', type='string', dest='config_file',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2084
                      action='store',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2085
                      help='Name of the configuration file. (Required)')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2086
    parser.add_option('--kind', type='string', dest='kind',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2087
                      action='store',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2088
                      help='The kind of the entities to store. (Required)')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2089
    parser.add_option('--url', type='string', dest='url',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2090
                      action='store',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2091
                      help='The location of the remote_api endpoint.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2092
    parser.add_option('--num_threads', type='int', dest='num_threads',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2093
                      action='store', default=10,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2094
                      help='Number of threads to upload records with.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2095
    parser.add_option('--batch_size', type='int', dest='batch_size',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2096
                      action='store', default=10,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2097
                      help='Number of records to post in each request.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2098
    parser.add_option('--bandwidth_limit', type='int', dest='bandwidth_limit',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2099
                      action='store', default=250000,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2100
                      help='The maximum bytes/second bandwidth for transfers.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2101
    parser.add_option('--rps_limit', type='int', dest='rps_limit',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2102
                      action='store', default=20,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2103
                      help='The maximum records/second for transfers.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2104
    parser.add_option('--http_limit', type='int', dest='http_limit',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2105
                      action='store', default=8,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2106
                      help='The maximum requests/second for transfers.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2107
    parser.add_option('--db_filename', type='string', dest='db_filename',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2108
                      action='store',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2109
                      help='Name of the progress database file.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2110
    parser.add_option('--auth_domain', type='string', dest='auth_domain',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2111
                      action='store', default='gmail.com',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2112
                      help='The name of the authorization domain to use.')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2113
    parser.add_option('--log_file', type='string', dest='log_file',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2114
                      help='File to write bulkloader logs.  If not supplied '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2115
                      'then a new log file will be created, named: '
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2116
                      'bulkloader-log-TIMESTAMP.')
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2117
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2118
  def _PerformUploadOptions(self, parser):
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2119
    """Adds 'upload_data' specific options to the 'parser' passed in.
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2120
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2121
    Args:
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2122
      parser: An instance of OptionsParser.
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2123
    """
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2124
    self._PerformLoadOptions(parser)
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2125
    parser.add_option('--has_header', dest='has_header',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2126
                      action='store_true', default=False,
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2127
                      help='Whether the first line of the input file should be'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2128
                      ' skipped')
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2129
    parser.add_option('--loader_opts', type='string', dest='loader_opts',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2130
                      help='A string to pass to the Loader.initialize method.')
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2131
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2132
  def _PerformDownloadOptions(self, parser):
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2133
    """Adds 'download_data' specific options to the 'parser' passed in.
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2134
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2135
    Args:
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2136
      parser: An instance of OptionsParser.
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2137
    """
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2138
    self._PerformLoadOptions(parser)
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2139
    parser.add_option('--exporter_opts', type='string', dest='exporter_opts',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2140
                      help='A string to pass to the Exporter.initialize method.'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2141
                     )
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2142
    parser.add_option('--result_db_filename', type='string',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2143
                      dest='result_db_filename',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2144
                      action='store',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2145
                      help='Database to write entities to for download.')
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2146
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2147
  class Action(object):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2148
    """Contains information about a command line action.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2149
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2150
    Attributes:
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2151
      function: The name of a function defined on AppCfg or its subclasses
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2152
        that will perform the appropriate action.
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2153
      usage: A command line usage string.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2154
      short_desc: A one-line description of the action.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2155
      long_desc: A detailed description of the action.  Whitespace and
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2156
        formatting will be preserved.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2157
      options: A function that will add extra options to a given OptionParser
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2158
        object.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2159
    """
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2160
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2161
    def __init__(self, function, usage, short_desc, long_desc='',
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2162
                 options=lambda obj, parser: None):
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2163
      """Initializer for the class attributes."""
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2164
      self.function = function
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2165
      self.usage = usage
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2166
      self.short_desc = short_desc
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2167
      self.long_desc = long_desc
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2168
      self.options = options
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2169
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2170
    def __call__(self, appcfg):
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2171
      """Invoke this Action on the specified AppCfg.
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2172
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2173
      This calls the function of the appropriate name on AppCfg, and
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2174
      respects polymophic overrides.
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2175
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2176
      Args:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2177
        appcfg: The appcfg to use.
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2178
      Returns:
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2179
        The result of the function call.
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2180
      """
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2181
      method = getattr(appcfg, self.function)
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2182
      return method()
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2183
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2184
  actions = {
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2185
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2186
      'help': Action(
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2187
          function='Help',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2188
          usage='%prog help <action>',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2189
          short_desc='Print help for a specific action.'),
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2190
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2191
      'update': Action(
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2192
          function='Update',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2193
          usage='%prog [options] update <directory>',
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2194
          options=_UpdateOptions,
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2195
          short_desc='Create or update an app version.',
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2196
          long_desc="""
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2197
Specify a directory that contains all of the files required by
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2198
the app, and appcfg.py will create/update the app version referenced
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2199
in the app.yaml file at the top level of that directory.  appcfg.py
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2200
will follow symlinks and recursively upload all files to the server.
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2201
Temporary or source control files (e.g. foo~, .svn/*) will be skipped."""),
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2202
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2203
      'update_cron': Action(
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2204
          function='UpdateCron',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2205
          usage='%prog [options] update_cron <directory>',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2206
          short_desc='Update application cron definitions.',
2172
ac7bd3b467ff Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 1278
diff changeset
  2207
          long_desc="""
ac7bd3b467ff Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 1278
diff changeset
  2208
The 'update_cron' command will update any new, removed or changed cron
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2209
definitions from the optional cron.yaml file."""),
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2210
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2211
      'update_indexes': Action(
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2212
          function='UpdateIndexes',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2213
          usage='%prog [options] update_indexes <directory>',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2214
          short_desc='Update application indexes.',
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2215
          long_desc="""
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2216
The 'update_indexes' command will add additional indexes which are not currently
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2217
in production as well as restart any indexes that were not completed."""),
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2218
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2219
      'update_queues': Action(
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2220
          function='UpdateQueues',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2221
          usage='%prog [options] update_queues <directory>',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2222
          short_desc='Update application task queue definitions.',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2223
          long_desc="""
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2224
The 'update_queue' command will update any new, removed or changed task queue
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2225
definitions from the optional queue.yaml file."""),
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2226
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2227
      'vacuum_indexes': Action(
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2228
          function='VacuumIndexes',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2229
          usage='%prog [options] vacuum_indexes <directory>',
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2230
          options=_VacuumIndexesOptions,
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2231
          short_desc='Delete unused indexes from application.',
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2232
          long_desc="""
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2233
The 'vacuum_indexes' command will help clean up indexes which are no longer
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2234
in use.  It does this by comparing the local index configuration with
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2235
indexes that are actually defined on the server.  If any indexes on the
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2236
server do not exist in the index configuration file, the user is given the
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2237
option to delete them."""),
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2238
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2239
      'rollback': Action(
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2240
          function='Rollback',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2241
          usage='%prog [options] rollback <directory>',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2242
          short_desc='Rollback an in-progress update.',
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2243
          long_desc="""
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2244
The 'update' command requires a server-side transaction.  Use 'rollback'
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2245
if you get an error message about another transaction being in progress
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2246
and you are sure that there is no such transaction."""),
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2247
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2248
      'request_logs': Action(
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2249
          function='RequestLogs',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2250
          usage='%prog [options] request_logs <directory> <output_file>',
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2251
          options=_RequestLogsOptions,
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2252
          short_desc='Write request logs in Apache common log format.',
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2253
          long_desc="""
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2254
The 'request_logs' command exports the request logs from your application
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2255
to a file.  It will write Apache common log format records ordered
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2256
chronologically.  If output file is '-' stdout will be written."""),
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2257
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2258
      'cron_info': Action(
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2259
          function='CronInfo',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2260
          usage='%prog [options] cron_info <directory>',
2172
ac7bd3b467ff Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 1278
diff changeset
  2261
          options=_CronInfoOptions,
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2262
          short_desc='Display information about cron jobs.',
2172
ac7bd3b467ff Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 1278
diff changeset
  2263
          long_desc="""
ac7bd3b467ff Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 1278
diff changeset
  2264
The 'cron_info' command will display the next 'number' runs (default 5) for
ac7bd3b467ff Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 1278
diff changeset
  2265
each cron job defined in the cron.yaml file."""),
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2266
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2267
      'upload_data': Action(
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2268
          function='PerformUpload',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2269
          usage='%prog [options] upload_data <directory>',
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2270
          options=_PerformUploadOptions,
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2271
          short_desc='Upload data records to datastore.',
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2272
          long_desc="""
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2273
The 'upload_data' command translates input records into datastore entities and
2273
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2274
uploads them into your application's datastore."""),
e4cb9c53db3e Load /Users/solydzajs/Desktop/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2172
diff changeset
  2275
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2276
      'download_data': Action(
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2277
          function='PerformDownload',
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2278
          usage='%prog [options] download_data <directory>',
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2279
          options=_PerformDownloadOptions,
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2280
          short_desc='Download entities from datastore.',
2309
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2281
          long_desc="""
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2282
The 'download_data' command downloads datastore entities and writes them to
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2283
file as CSV or developer defined format."""),
be1b94099f2d Update Google App Engine to 1.2.2 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2273
diff changeset
  2284
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2285
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2286
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2287
  }
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2288
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2289
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2290
def main(argv):
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2291
  logging.basicConfig(format=('%(asctime)s %(levelname)s %(filename)s:'
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2292
                              '%(lineno)s %(message)s '))
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2293
  try:
828
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2294
    result = AppCfgApp(argv).Run()
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2295
    if result:
f5fd65cc3bf3 Load /Users/solydzajs/Downloads/google_appengine into
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 686
diff changeset
  2296
      sys.exit(result)
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2297
  except KeyboardInterrupt:
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2298
    StatusUpdate('Interrupted.')
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2299
    sys.exit(1)
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2300
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2301
2413
d0b7dac5325c Update GAE to 1.2.3 in thirdparty folder.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents: 2309
diff changeset
  2302
if __name__ == '__main__':
109
620f9b141567 Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff changeset
  2303
  main(sys.argv)