1 .. _ref-layermapping:

     2 

     3 ====================================

     4 ``LayerMapping`` data import utility

     5 ====================================

     6 

     7 .. module:: django.contrib.gis.utils.layermapping

     8    :synopsis: Spatial data import utility for GeoDjango models.

     9 

    10 .. currentmodule:: django.contrib.gis.utils

    11 

    12 The :class:`LayerMapping` class provides a way to map the contents of

    13 vector spatial data files (e.g. shapefiles) intoto GeoDjango models.

    14 

    15 This utility grew out of the author's personal needs to eliminate

    16 the code repetition that went into pulling geometries and fields out of

    17 a vector layer, converting to another coordinate system (e.g. WGS84), and

    18 then inserting into a GeoDjango model.

    19 

    20 .. note::

    21 

    22     Use of :class:`LayerMapping` requires GDAL.

    23 

    24 .. warning ::

    25 

    26     GIS data sources, like shapefiles, may be very large.  If you find

    27     that :class:`LayerMapping` is using too much memory, set

    28     :setting:`DEBUG` to ``False`` in your settings.  When :setting:`DEBUG`

    29     is set to ``True``, Django :ref:`automatically logs <faq-see-raw-sql-queries>`

    30     *every* SQL query -- thus, when SQL statements contain geometries, it is

    31     easy to consume more memory than is typical.

    32 

    33 Example

    34 =======

    35 

    36 1. You need a GDAL-supported data source, like a shapefile (here we're using

    37    a simple polygon shapefile, ``test_poly.shp``, with three features)::

    38 

    39     >>> from django.contrib.gis.gdal import DataSource

    40     >>> ds = DataSource('test_poly.shp')

    41     >>> layer = ds[0]

    42     >>> print layer.fields # Exploring the fields in the layer, we only want the 'str' field.

    43     ['float', 'int', 'str']

    44     >>> print len(layer) # getting the number of features in the layer (should be 3)

    45     3

    46     >>> print layer.geom_type # Should be 'Polygon'

    47     Polygon

    48     >>> print layer.srs # WGS84 in WKT

    49     GEOGCS["GCS_WGS_1984",

    50         DATUM["WGS_1984",

    51             SPHEROID["WGS_1984",6378137,298.257223563]],

    52         PRIMEM["Greenwich",0],

    53         UNIT["Degree",0.017453292519943295]]

    54 

    55 2. Now we define our corresponding Django model (make sure to use ``syncdb``)::

    56 

    57     from django.contrib.gis.db import models

    58 

    59     class TestGeo(models.Model):

    60         name = models.CharField(max_length=25) # corresponds to the 'str' field

    61         poly = models.PolygonField(srid=4269) # we want our model in a different SRID

    62         objects = models.GeoManager()

    63         def __unicode__(self):

    64             return 'Name: %s' % self.name

    65 

    66 3. Use :class:`LayerMapping` to extract all the features and place them in the

    67    database::

    68 

    69     >>> from django.contrib.gis.utils import LayerMapping

    70     >>> from geoapp.models import TestGeo

    71     >>> mapping = {'name' : 'str', # The 'name' model field maps to the 'str' layer field.

    72                    'poly' : 'POLYGON', # For geometry fields use OGC name.

    73                    } # The mapping is a dictionary

    74     >>> lm = LayerMapping(TestGeo, 'test_poly.shp', mapping)

    75     >>> lm.save(verbose=True) # Save the layermap, imports the data.

    76     Saved: Name: 1

    77     Saved: Name: 2

    78     Saved: Name: 3

    79 

    80 Here, :class:`LayerMapping` just transformed the three geometries from the

    81 shapefile in their original spatial reference system (WGS84) to the spatial

    82 reference system of the GeoDjango model (NAD83).  If no spatial reference

    83 system is defined for the layer, use the ``source_srs`` keyword with a

    84 :class:`~django.contrib.gis.gdal.SpatialReference` object to specify one.

    85 

    86 ``LayerMapping`` API

    87 ====================

    88 

    89 .. class:: LayerMapping(model, data_source, mapping[, layer=0, source_srs=None, encoding=None, transaction_mode='commit_on_success', transform=True, unique=True, using='default'])

    90 

    91 The following are the arguments and keywords that may be used during

    92 instantiation of ``LayerMapping`` objects.

    93 

    94 =================  =========================================================

    95 Argument           Description

    96 =================  =========================================================

    97 ``model``          The geographic model, *not* an instance.

    98 

    99 ``data_source``    The path to the OGR-supported data source file

   100                    (e.g., a shapefile).  Also accepts

   101                    :class:`django.contrib.gis.gdal.DataSource` instances.

   102 

   103 ``mapping``        A dictionary: keys are strings corresponding to

   104                    the model field, and values correspond to

   105                    string field names for the OGR feature, or if the

   106                    model field is a geographic then it should

   107                    correspond to the OGR geometry type,

   108                    e.g., ``'POINT'``, ``'LINESTRING'``, ``'POLYGON'``.

   109 =================  =========================================================

   110 

   111 =====================  =====================================================

   112 Keyword Arguments

   113 =====================  =====================================================

   114 ``layer``              The index of the layer to use from the Data Source

   115                        (defaults to 0)

   116 

   117 ``source_srs``         Use this to specify the source SRS manually (for

   118                        example, some shapefiles don't come with a '.prj'

   119                        file).  An integer SRID, WKT or PROJ.4 strings, and

   120                        :class:`django.contrib.gis.gdal.SpatialReference`

   121                        objects are accepted.

   122 

   123 ``encoding``           Specifies the character set encoding of the strings

   124                        in the OGR data source.  For example, ``'latin-1'``,

   125                        ``'utf-8'``, and ``'cp437'`` are all valid encoding

   126                        parameters.

   127 

   128 ``transaction_mode``   May be ``'commit_on_success'`` (default) or

   129                        ``'autocommit'``.

   130 

   131 ``transform``          Setting this to False will disable coordinate

   132                        transformations.  In other words, geometries will

   133                        be inserted into the database unmodified from their

   134                        original state in the data source.

   135 

   136 ``unique``             Setting this to the name, or a tuple of names,

   137                        from the given  model will create models unique

   138                        only to the given name(s). Geometries will from

   139                        each feature will be added into the collection

   140                        associated with the unique model.  Forces

   141                        the transaction mode to be ``'autocommit'``.

   142 

   143 ``using``              New in version 1.2.  Sets the database to use when

   144                        importing spatial data.  Default is ``'default'``

   145 =====================  =====================================================

   146 

   147 ``save()`` Keyword Arguments

   148 ----------------------------

   149 

   150 .. method:: LayerMapping.save([verbose=False, fid_range=False, step=False, progress=False, silent=False, stream=sys.stdout, strict=False])

   151 

   152 The ``save()`` method also accepts keywords.  These keywords are

   153 used for controlling output logging, error handling, and for importing

   154 specific feature ranges.

   155 

   156 ===========================  =================================================

   157 Save Keyword Arguments       Description

   158 ===========================  =================================================

   159 ``fid_range``                May be set with a slice or tuple of

   160                              (begin, end) feature ID's to map from

   161                              the data source.  In other words, this

   162                              keyword enables the user to selectively

   163                              import a subset range of features in the

   164                              geographic data source.

   165 

   166 ``progress``                 When this keyword is set, status information

   167                              will be printed giving the number of features

   168                              processed and successfully saved.  By default,

   169                              progress information will be printed every 1000

   170                              features processed, however, this default may

   171                              be overridden by setting this keyword with an

   172                              integer for the desired interval.

   173 

   174 ``silent``                   By default, non-fatal error notifications are

   175                              printed to ``sys.stdout``, but this keyword may

   176                              be set to disable these notifications.

   177 

   178 ``step``                     If set with an integer, transactions will

   179                              occur at every step interval. For example, if

   180                              ``step=1000``, a commit would occur after the

   181                              1,000th feature, the 2,000th feature etc.

   182 

   183 

   184 ``stream``                   Status information will be written to this file

   185                              handle.  Defaults to using ``sys.stdout``, but

   186                              any object with a ``write`` method is supported.

   187 

   188 ``strict``                   Execution of the model mapping will cease upon

   189                              the first error encountered.  The default value

   190                              (``False``)

   191                              behavior is to attempt to continue.

   192 

   193 ``verbose``                  If set, information will be printed

   194                              subsequent to each model save

   195                              executed on the database.

   196 ===========================  =================================================

   197 

   198 Troubleshooting

   199 ===============

   200 

   201 Running out of memory

   202 ---------------------

   203 

   204 As noted in the warning at the top of this section, Django stores all SQL

   205 queries when ``DEBUG=True``.  Set ``DEBUG=False`` in your settings, and this

   206 should stop excessive memory use when running ``LayerMapping`` scripts.

   207 

   208 MySQL: ``max_allowed_packet`` error

   209 -----------------------------------

   210 

   211 If you encounter the following error when using ``LayerMapping`` and MySQL::

   212 

   213     OperationalError: (1153, "Got a packet bigger than 'max_allowed_packet' bytes")

   214 

   215 Then the solution is to increase the value of the ``max_allowed_packet``

   216 setting in your MySQL configuration.  For example, the default value may

   217 be something low like one megabyte -- the setting may be modified in MySQL's

   218 configuration file (``my.cnf``) in the ``[mysqld]`` section::

   219 

   220     max_allowed_packet = 10M