author | Lennard de Rijk <ljvderijk@gmail.com> |
Fri, 17 Apr 2009 21:14:43 +0000 | |
changeset 2198 | 95ac403c6dd4 |
parent 109 | 620f9b141567 |
permissions | -rwxr-xr-x |
109
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
1 |
#!/usr/bin/env python |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
2 |
# |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
3 |
# Copyright 2007 Google Inc. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
4 |
# |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
5 |
# Licensed under the Apache License, Version 2.0 (the "License"); |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
6 |
# you may not use this file except in compliance with the License. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
7 |
# You may obtain a copy of the License at |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
8 |
# |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
9 |
# http://www.apache.org/licenses/LICENSE-2.0 |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
10 |
# |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
11 |
# Unless required by applicable law or agreed to in writing, software |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
12 |
# distributed under the License is distributed on an "AS IS" BASIS, |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
13 |
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
14 |
# See the License for the specific language governing permissions and |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
15 |
# limitations under the License. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
16 |
# |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
17 |
"""Imports CSV data over HTTP. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
18 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
19 |
Usage: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
20 |
%s [flags] |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
21 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
22 |
--debug Show debugging information. (Optional) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
23 |
--cookie=<string> Whole Cookie header to supply to the server, including |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
24 |
the parameter name (e.g., "ACSID=..."). (Optional) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
25 |
--url=<string> URL endpoint to post to for importing data. (Required) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
26 |
--batch_size=<int> Number of Entity objects to include in each post to |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
27 |
the URL endpoint. The more data per row/Entity, the |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
28 |
smaller the batch size should be. (Default 10) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
29 |
--filename=<path> Path to the CSV file to import. (Required) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
30 |
--kind=<string> Name of the Entity object kind to put in the datastore. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
31 |
(Required) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
32 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
33 |
The exit status will be 0 on success, non-zero on import failure. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
34 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
35 |
Works with the bulkload mix-in library for google.appengine.ext.bulkload. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
36 |
Please look there for documentation about how to setup the server side. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
37 |
""" |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
38 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
39 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
40 |
import StringIO |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
41 |
import httplib |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
42 |
import logging |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
43 |
import csv |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
44 |
import getopt |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
45 |
import socket |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
46 |
import sys |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
47 |
import urllib |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
48 |
import urlparse |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
49 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
50 |
from google.appengine.ext.bulkload import constants |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
51 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
52 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
53 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
54 |
class Error(Exception): |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
55 |
"""Base-class for exceptions in this module.""" |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
56 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
57 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
58 |
class PostError(Error): |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
59 |
"""An error has occured while trying to post data to the server.""" |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
60 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
61 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
62 |
class BadServerStatusError(PostError): |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
63 |
"""The server has returned an error while importing data.""" |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
64 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
65 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
66 |
def ContentGenerator(csv_file, |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
67 |
batch_size, |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
68 |
create_csv_reader=csv.reader, |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
69 |
create_csv_writer=csv.writer): |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
70 |
"""Retrieves CSV data up to a batch size at a time. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
71 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
72 |
Args: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
73 |
csv_file: A file-like object for reading CSV data. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
74 |
batch_size: Maximum number of CSV rows to yield on each iteration. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
75 |
create_csv_reader, create_csv_writer: Used for dependency injection. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
76 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
77 |
Yields: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
78 |
Tuple (entity_count, csv_content) where: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
79 |
entity_count: Number of entities contained in the csv_content. Will be |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
80 |
less than or equal to the batch_size and greater than 0. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
81 |
csv_content: String containing the CSV content containing the next |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
82 |
entity_count entities. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
83 |
""" |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
84 |
try: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
85 |
csv.field_size_limit(800000) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
86 |
except AttributeError: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
87 |
pass |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
88 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
89 |
reader = create_csv_reader(csv_file, skipinitialspace=True) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
90 |
exhausted = False |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
91 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
92 |
while not exhausted: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
93 |
rows_written = 0 |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
94 |
content = StringIO.StringIO() |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
95 |
writer = create_csv_writer(content) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
96 |
try: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
97 |
for i in xrange(batch_size): |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
98 |
row = reader.next() |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
99 |
writer.writerow(row) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
100 |
rows_written += 1 |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
101 |
except StopIteration: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
102 |
exhausted = True |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
103 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
104 |
if rows_written > 0: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
105 |
yield rows_written, content.getvalue() |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
106 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
107 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
108 |
def PostEntities(host_port, uri, cookie, kind, content): |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
109 |
"""Posts Entity records to a remote endpoint over HTTP. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
110 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
111 |
Args: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
112 |
host_port: String containing the "host:port" pair; the port is optional. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
113 |
uri: Relative URI to access on the remote host (e.g., '/bulkload'). |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
114 |
cookie: String containing the Cookie header to use, if any. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
115 |
kind: Kind of the Entity records being posted. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
116 |
content: String containing the CSV data for the entities. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
117 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
118 |
Raises: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
119 |
BadServerStatusError if the server was contactable but returns an error. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
120 |
PostError If an error occurred while connecting to the server or reading |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
121 |
or writing data. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
122 |
""" |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
123 |
logging.debug('Connecting to %s', host_port) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
124 |
try: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
125 |
body = urllib.urlencode({ |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
126 |
constants.KIND_PARAM: kind, |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
127 |
constants.CSV_PARAM: content, |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
128 |
}) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
129 |
headers = { |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
130 |
'Content-Type': 'application/x-www-form-urlencoded', |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
131 |
'Content-Length': len(body), |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
132 |
'Cookie': cookie, |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
133 |
} |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
134 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
135 |
logging.debug('Posting %d bytes to http://%s%s', len(body), host_port, uri) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
136 |
connection = httplib.HTTPConnection(host_port) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
137 |
try: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
138 |
connection.request('POST', uri, body, headers) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
139 |
response = connection.getresponse() |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
140 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
141 |
status = response.status |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
142 |
reason = response.reason |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
143 |
content = response.read() |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
144 |
logging.debug('Received response code %d: %s', status, reason) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
145 |
if status != httplib.OK: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
146 |
raise BadServerStatusError('Received code %d: %s\n%s' % ( |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
147 |
status, reason, content)) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
148 |
finally: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
149 |
connection.close() |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
150 |
except (IOError, httplib.HTTPException, socket.error), e: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
151 |
logging.debug('Encountered exception accessing HTTP server: %s', e) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
152 |
raise PostError(e) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
153 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
154 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
155 |
def SplitURL(url): |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
156 |
"""Splits an HTTP URL into pieces. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
157 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
158 |
Args: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
159 |
url: String containing a full URL string (e.g., |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
160 |
'http://blah.com:8080/stuff?param=1#foo') |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
161 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
162 |
Returns: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
163 |
Tuple (netloc, uri) where: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
164 |
netloc: String containing the host/port combination from the URL. The |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
165 |
port is optional. (e.g., 'blah.com:8080'). |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
166 |
uri: String containing the relative URI of the URL. (e.g., '/stuff'). |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
167 |
""" |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
168 |
scheme, netloc, path, query, fragment = urlparse.urlsplit(url) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
169 |
return netloc, path |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
170 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
171 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
172 |
def ImportCSV(filename, |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
173 |
post_url, |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
174 |
cookie, |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
175 |
batch_size, |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
176 |
kind, |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
177 |
split_url=SplitURL, |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
178 |
openfile=file, |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
179 |
create_content_generator=ContentGenerator, |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
180 |
post_entities=PostEntities): |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
181 |
"""Imports CSV data using a series of HTTP posts. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
182 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
183 |
Args: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
184 |
filename: File on disk containing CSV data. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
185 |
post_url: URL to post the Entity data to. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
186 |
cookie: Full cookie header to use while connecting. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
187 |
batch_size: Maximum number of Entity objects to post with each request. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
188 |
kind: Entity kind of the objects being posted. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
189 |
split_url, openfile, create_content_generator, post_entities: Used for |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
190 |
dependency injection. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
191 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
192 |
Returns: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
193 |
True if all entities were imported successfully; False otherwise. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
194 |
""" |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
195 |
host_port, uri = split_url(post_url) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
196 |
csv_file = openfile(filename, 'r') |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
197 |
try: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
198 |
content_gen = create_content_generator(csv_file, batch_size) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
199 |
logging.info('Starting import; maximum %d entities per post', batch_size) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
200 |
for num_entities, content in content_gen: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
201 |
logging.info('Importing %d entities in %d bytes', |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
202 |
num_entities, len(content)) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
203 |
try: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
204 |
content = post_entities(host_port, uri, cookie, kind, content) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
205 |
except PostError, e: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
206 |
logging.error('An error occurred while importing: %s', e) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
207 |
return False |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
208 |
finally: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
209 |
csv_file.close() |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
210 |
return True |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
211 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
212 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
213 |
def PrintUsageExit(code): |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
214 |
"""Prints usage information and exits with a status code. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
215 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
216 |
Args: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
217 |
code: Status code to pass to sys.exit() after displaying usage information. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
218 |
""" |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
219 |
print sys.modules['__main__'].__doc__ % sys.argv[0] |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
220 |
sys.stdout.flush() |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
221 |
sys.stderr.flush() |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
222 |
sys.exit(code) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
223 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
224 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
225 |
def ParseArguments(argv): |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
226 |
"""Parses command-line arguments. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
227 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
228 |
Prints out a help message if -h or --help is supplied. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
229 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
230 |
Args: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
231 |
argv: List of command-line arguments. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
232 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
233 |
Returns: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
234 |
Tuple (url, filename, cookie, batch_size, kind) containing the values from |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
235 |
each corresponding command-line flag. |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
236 |
""" |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
237 |
opts, args = getopt.getopt( |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
238 |
argv[1:], |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
239 |
'h', |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
240 |
['debug', |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
241 |
'help', |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
242 |
'url=', |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
243 |
'filename=', |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
244 |
'cookie=', |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
245 |
'batch_size=', |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
246 |
'kind=']) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
247 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
248 |
url = None |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
249 |
filename = None |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
250 |
cookie = '' |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
251 |
batch_size = 10 |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
252 |
kind = None |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
253 |
encoding = None |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
254 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
255 |
for option, value in opts: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
256 |
if option == '--debug': |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
257 |
logging.getLogger().setLevel(logging.DEBUG) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
258 |
if option in ('-h', '--help'): |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
259 |
PrintUsageExit(0) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
260 |
if option == '--url': |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
261 |
url = value |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
262 |
if option == '--filename': |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
263 |
filename = value |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
264 |
if option == '--cookie': |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
265 |
cookie = value |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
266 |
if option == '--batch_size': |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
267 |
batch_size = int(value) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
268 |
if batch_size <= 0: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
269 |
print >>sys.stderr, 'batch_size must be 1 or larger' |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
270 |
PrintUsageExit(1) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
271 |
if option == '--kind': |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
272 |
kind = value |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
273 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
274 |
return (url, filename, cookie, batch_size, kind) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
275 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
276 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
277 |
def main(argv): |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
278 |
"""Runs the importer.""" |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
279 |
logging.basicConfig( |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
280 |
level=logging.INFO, |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
281 |
format='%(levelname)-8s %(asctime)s %(filename)s] %(message)s') |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
282 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
283 |
args = ParseArguments(argv) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
284 |
if [arg for arg in args if arg is None]: |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
285 |
print >>sys.stderr, 'Invalid arguments' |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
286 |
PrintUsageExit(1) |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
287 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
288 |
url, filename, cookie, batch_size, kind = args |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
289 |
if ImportCSV(filename, url, cookie, batch_size, kind): |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
290 |
logging.info('Import succcessful') |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
291 |
return 0 |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
292 |
logging.error('Import failed') |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
293 |
return 1 |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
294 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
295 |
|
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
296 |
if __name__ == '__main__': |
620f9b141567
Load ../../google_appengine into trunk/thirdparty/google_appengine.
Todd Larsen <tlarsen@google.com>
parents:
diff
changeset
|
297 |
sys.exit(main(sys.argv)) |