=========

Databases

=========

Django attempts to support as many features as possible on all database

backends. However, not all database backends are alike, and we've had to make

design decisions on which features to support and which assumptions we can make

safely.

This file describes some of the features that might be relevant to Django

usage. Of course, it is not intended as a replacement for server-specific

documentation or reference manuals.

.. _postgresql-notes:

PostgreSQL notes

================

PostgreSQL 8.2 to 8.2.4

-----------------------

The implementation of the population statistics aggregates ``STDDEV_POP`` and

``VAR_POP`` that shipped with PostgreSQL 8.2 to 8.2.4 are `known to be

faulty`_. Users of these releases of PostgreSQL are advised to upgrade to

`Release 8.2.5`_ or later. Django will raise a ``NotImplementedError`` if you

attempt to use the ``StdDev(sample=False)`` or ``Variance(sample=False)``

aggregate with a database backend that falls within the affected release range.

.. _known to be faulty: http://archives.postgresql.org/pgsql-bugs/2007-07/msg00046.php

.. _Release 8.2.5: http://developer.postgresql.org/pgdocs/postgres/release-8-2-5.html

Transaction handling

---------------------

:doc:`By default </topics/db/transactions>`, Django starts a transaction when a

database connection is first used and commits the result at the end of the

request/response handling. The PostgreSQL backends normally operate the same

as any other Django backend in this respect.

Autocommit mode

~~~~~~~~~~~~~~~

.. versionadded:: 1.1

If your application is particularly read-heavy and doesn't make many

database writes, the overhead of a constantly open transaction can

sometimes be noticeable. For those situations, if you're using the

``postgresql_psycopg2`` backend, you can configure Django to use

*"autocommit"* behavior for the connection, meaning that each database

operation will normally be in its own transaction, rather than having

the transaction extend over multiple operations. In this case, you can

still manually start a transaction if you're doing something that

requires consistency across multiple database operations. The

autocommit behavior is enabled by setting the ``autocommit`` key in

the :setting:`OPTIONS` part of your database configuration in

:setting:`DATABASES`::

    'OPTIONS': {

        'autocommit': True,

    }

In this configuration, Django still ensures that :ref:`delete()

<topics-db-queries-delete>` and :ref:`update() <topics-db-queries-update>`

queries run inside a single transaction, so that either all the affected

objects are changed or none of them are.

.. admonition:: This is database-level autocommit

    This functionality is not the same as the

    :ref:`topics-db-transactions-autocommit` decorator. That decorator

    is a Django-level implementation that commits automatically after

    data changing operations. The feature enabled using the

    :setting:`OPTIONS` option provides autocommit behavior at the

    database adapter level. It commits after *every* operation.

If you are using this feature and performing an operation akin to delete or

updating that requires multiple operations, you are strongly recommended to

wrap you operations in manual transaction handling to ensure data consistency.

You should also audit your existing code for any instances of this behavior

before enabling this feature. It's faster, but it provides less automatic

protection for multi-call operations.

Indexes for ``varchar`` and ``text`` columns

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. versionadded:: 1.1.2

When specifying ``db_index=True`` on your model fields, Django typically

outputs a single ``CREATE INDEX`` statement.  However, if the database type

for the field is either ``varchar`` or ``text`` (e.g., used by ``CharField``,

``FileField``, and ``TextField``), then Django will create

an additional index that uses an appropriate `PostgreSQL operator class`_

for the column.  The extra index is necessary to correctly perfrom

lookups that use the ``LIKE`` operator in their SQL, as is done with the

``contains`` and ``startswith`` lookup types.

.. _PostgreSQL operator class: http://www.postgresql.org/docs/8.4/static/indexes-opclass.html

.. _mysql-notes:

MySQL notes

===========

Django expects the database to support transactions, referential integrity, and

Unicode (UTF-8 encoding). Fortunately, MySQL_ has all these features as

available as far back as 3.23. While it may be possible to use 3.23 or 4.0,

you'll probably have less trouble if you use 4.1 or 5.0.

MySQL 4.1

---------

`MySQL 4.1`_ has greatly improved support for character sets. It is possible to

set different default character sets on the database, table, and column.

Previous versions have only a server-wide character set setting. It's also the

first version where the character set can be changed on the fly. 4.1 also has

support for views, but Django currently doesn't use views.

MySQL 5.0

---------

`MySQL 5.0`_ adds the ``information_schema`` database, which contains detailed

data on all database schema. Django's ``inspectdb`` feature uses this

``information_schema`` if it's available. 5.0 also has support for stored

procedures, but Django currently doesn't use stored procedures.

.. _MySQL: http://www.mysql.com/

.. _MySQL 4.1: http://dev.mysql.com/doc/refman/4.1/en/index.html

.. _MySQL 5.0: http://dev.mysql.com/doc/refman/5.0/en/index.html

Storage engines

---------------

MySQL has several `storage engines`_ (previously called table types). You can

change the default storage engine in the server configuration.

The default engine is MyISAM_ [#]_. The main drawback of MyISAM is that it

doesn't currently support transactions or foreign keys. On the plus side, it's

currently the only engine that supports full-text indexing and searching.

The InnoDB_ engine is fully transactional and supports foreign key references.

The BDB_ engine, like InnoDB, is also fully transactional and supports foreign

key references. However, its use seems to be deprecated.

`Other storage engines`_, including SolidDB_ and Falcon_, are on the horizon.

For now, InnoDB is probably your best choice.

.. _storage engines: http://dev.mysql.com/doc/refman/5.0/en/storage-engines.html

.. _MyISAM: http://dev.mysql.com/doc/refman/5.0/en/myisam-storage-engine.html

.. _BDB: http://dev.mysql.com/doc/refman/5.0/en/bdb-storage-engine.html

.. _InnoDB: http://dev.mysql.com/doc/refman/5.0/en/innodb.html

.. _Other storage engines: http://dev.mysql.com/doc/refman/5.1/en/storage-engines-other.html

.. _SolidDB: http://forge.mysql.com/projects/project.php?id=139

.. _Falcon: http://dev.mysql.com/doc/falcon/en/index.html

.. [#] Unless this was changed by the packager of your MySQL package. We've

   had reports that the Windows Community Server installer sets up InnoDB as

   the default storage engine, for example.

MySQLdb

-------

`MySQLdb`_ is the Python interface to MySQL. Version 1.2.1p2 or later is

required for full MySQL support in Django.

.. note::

    If you see ``ImportError: cannot import name ImmutableSet`` when trying to

    use Django, your MySQLdb installation may contain an outdated ``sets.py``

    file that conflicts with the built-in module of the same name from Python

    2.4 and later. To fix this, verify that you have installed MySQLdb version

    1.2.1p2 or newer, then delete the ``sets.py`` file in the MySQLdb

    directory that was left by an earlier version.

.. _MySQLdb: http://sourceforge.net/projects/mysql-python

Creating your database

----------------------

You can `create your database`_ using the command-line tools and this SQL::

  CREATE DATABASE <dbname> CHARACTER SET utf8;

This ensures all tables and columns will use UTF-8 by default.

.. _create your database: http://dev.mysql.com/doc/refman/5.0/en/create-database.html

.. _mysql-collation:

Collation settings

~~~~~~~~~~~~~~~~~~

The collation setting for a column controls the order in which data is sorted

as well as what strings compare as equal. It can be set on a database-wide

level and also per-table and per-column. This is `documented thoroughly`_ in

the MySQL documentation. In all cases, you set the collation by directly

manipulating the database tables; Django doesn't provide a way to set this on

the model definition.

.. _documented thoroughly: http://dev.mysql.com/doc/refman/5.0/en/charset.html

By default, with a UTF-8 database, MySQL will use the

``utf8_general_ci_swedish`` collation. This results in all string equality

comparisons being done in a *case-insensitive* manner. That is, ``"Fred"`` and

``"freD"`` are considered equal at the database level. If you have a unique

constraint on a field, it would be illegal to try to insert both ``"aa"`` and

``"AA"`` into the same column, since they compare as equal (and, hence,

non-unique) with the default collation.

In many cases, this default will not be a problem. However, if you really want

case-sensitive comparisons on a particular column or table, you would change

the column or table to use the ``utf8_bin`` collation. The main thing to be

aware of in this case is that if you are using MySQLdb 1.2.2, the database

backend in Django will then return bytestrings (instead of unicode strings) for

any character fields it receive from the database. This is a strong variation

from Django's normal practice of *always* returning unicode strings. It is up

to you, the developer, to handle the fact that you will receive bytestrings if

you configure your table(s) to use ``utf8_bin`` collation. Django itself should

mostly work smoothly with such columns (except for the ``contrib.sessions``

``Session`` and ``contrib.admin`` ``LogEntry`` tables described below), but

your code must be prepared to call ``django.utils.encoding.smart_unicode()`` at

times if it really wants to work with consistent data -- Django will not do

this for you (the database backend layer and the model population layer are

separated internally so the database layer doesn't know it needs to make this

conversion in this one particular case).

If you're using MySQLdb 1.2.1p2, Django's standard

:class:`~django.db.models.CharField` class will return unicode strings even

with ``utf8_bin`` collation. However, :class:`~django.db.models.TextField`

fields will be returned as an ``array.array`` instance (from Python's standard

``array`` module). There isn't a lot Django can do about that, since, again,

the information needed to make the necessary conversions isn't available when

the data is read in from the database. This problem was `fixed in MySQLdb

1.2.2`_, so if you want to use :class:`~django.db.models.TextField` with

``utf8_bin`` collation, upgrading to version 1.2.2 and then dealing with the

bytestrings (which shouldn't be too difficult) as described above is the

recommended solution.

Should you decide to use ``utf8_bin`` collation for some of your tables with

MySQLdb 1.2.1p2 or 1.2.2, you should still use ``utf8_collation_ci_swedish``

(the default) collation for the :class:`django.contrib.sessions.models.Session`

table (usually called ``django_session``) and the

:class:`django.contrib.admin.models.LogEntry` table (usually called

``django_admin_log``). Those are the two standard tables that use

:class:`~django.db.model.TextField` internally.

.. _fixed in MySQLdb 1.2.2: http://sourceforge.net/tracker/index.php?func=detail&aid=1495765&group_id=22307&atid=374932

Connecting to the database

--------------------------

Refer to the :doc:`settings documentation </ref/settings>`.

Connection settings are used in this order:

    1. :setting:`OPTIONS`.

    2. :setting:`NAME`, :setting:`USER`, :setting:`PASSWORD`,

       :setting:`HOST`, :setting:`PORT`

    3. MySQL option files.

In other words, if you set the name of the database in ``OPTIONS``,

this will take precedence over ``NAME``, which would override

anything in a `MySQL option file`_.

Here's a sample configuration which uses a MySQL option file::

    # settings.py

    DATABASES = {

        'default': {

            'ENGINE': 'django.db.backends.mysql',

            'OPTIONS': {

                'read_default_file': '/path/to/my.cnf',

            },

        }

    }

    # my.cnf

    [client]

    database = NAME

    user = USER

    password = PASSWORD

    default-character-set = utf8

Several other MySQLdb connection options may be useful, such as ``ssl``,

``use_unicode``, ``init_command``, and ``sql_mode``. Consult the

`MySQLdb documentation`_ for more details.

.. _MySQL option file: http://dev.mysql.com/doc/refman/5.0/en/option-files.html

.. _MySQLdb documentation: http://mysql-python.sourceforge.net/

Creating your tables

--------------------

When Django generates the schema, it doesn't specify a storage engine, so

tables will be created with whatever default storage engine your database

server is configured for. The easiest solution is to set your database server's

default storage engine to the desired engine.

If you're using a hosting service and can't change your server's default

storage engine, you have a couple of options.

    * After the tables are created, execute an ``ALTER TABLE`` statement to

      convert a table to a new storage engine (such as InnoDB)::

          ALTER TABLE <tablename> ENGINE=INNODB;

      This can be tedious if you have a lot of tables.

    * Another option is to use the ``init_command`` option for MySQLdb prior to

      creating your tables::

          'OPTIONS': {

             'init_command': 'SET storage_engine=INNODB',

          }

      This sets the default storage engine upon connecting to the database.

      After your tables have been created, you should remove this option.

    * Another method for changing the storage engine is described in

      AlterModelOnSyncDB_.

.. _AlterModelOnSyncDB: http://code.djangoproject.com/wiki/AlterModelOnSyncDB

Notes on specific fields

------------------------

Boolean fields

~~~~~~~~~~~~~~

.. versionchanged:: 1.2

In previous versions of Django when running under MySQL ``BooleanFields`` would

return their data as ``ints``, instead of true ``bools``.  See the release

notes for a complete description of the change.

Character fields

~~~~~~~~~~~~~~~~

Any fields that are stored with ``VARCHAR`` column types have their

``max_length`` restricted to 255 characters if you are using ``unique=True``

for the field. This affects :class:`~django.db.models.CharField`,

:class:`~django.db.models.SlugField` and

:class:`~django.db.models.CommaSeparatedIntegerField`.

Furthermore, if you are using a version of MySQL prior to 5.0.3, all of those

column types have a maximum length restriction of 255 characters, regardless

of whether ``unique=True`` is specified or not.

.. _sqlite-notes:

SQLite notes

============

SQLite_ provides an excellent development alternative for applications that

are predominantly read-only or require a smaller installation footprint. As

with all database servers, though, there are some differences that are

specific to SQLite that you should be aware of.

.. _SQLite: http://www.sqlite.org/

.. _sqlite-string-matching:

String matching for non-ASCII strings

--------------------------------------

SQLite doesn't support case-insensitive matching for non-ASCII strings. Some

possible workarounds for this are `documented at sqlite.org`_, but they are

not utilised by the default SQLite backend in Django. Therefore, if you are

using the ``iexact`` lookup type in your queryset filters, be aware that it

will not work as expected for non-ASCII strings.

.. _documented at sqlite.org: http://www.sqlite.org/faq.html#q18

SQLite 3.3.6 or newer strongly recommended

------------------------------------------

Versions of SQLite 3.3.5 and older contains the following bugs:

 * A bug when `handling`_ ``ORDER BY`` parameters. This can cause problems when

   you use the ``select`` parameter for the ``extra()`` QuerySet method. The bug

   can be identified by the error message ``OperationalError: ORDER BY terms

   must not be non-integer constants``.

 * A bug when handling `aggregation`_ together with DateFields and

   DecimalFields.

.. _handling: http://www.sqlite.org/cvstrac/tktview?tn=1768

.. _aggregation: http://code.djangoproject.com/ticket/10031

SQLite 3.3.6 was released in April 2006, so most current binary distributions

for different platforms include newer version of SQLite usable from Python

through either the ``pysqlite2`` or the ``sqlite3`` modules.

However, some platform/Python version combinations include older versions of

SQLite (e.g. the official binary distribution of Python 2.5 for Windows, 2.5.4

as of this writing, includes SQLite 3.3.4). There are (as of Django 1.1) even

some tests in the Django test suite that will fail when run under this setup.

As described :ref:`below<using-newer-versions-of-pysqlite>`, this can be solved

by downloading and installing a newer version of ``pysqlite2``

(``pysqlite-2.x.x.win32-py2.5.exe`` in the described case) that includes and

uses a newer version of SQLite. Python 2.6 for Windows ships with a version of

SQLite that is not affected by these issues.

Version 3.5.9

-------------

The Ubuntu "Intrepid Ibex" (8.10) SQLite 3.5.9-3 package contains a bug that

causes problems with the evaluation of query expressions. If you are using

Ubuntu "Intrepid Ibex", you will need to update the package to version

3.5.9-3ubuntu1 or newer (recommended) or find an alternate source for SQLite

packages, or install SQLite from source.

At one time, Debian Lenny shipped with the same malfunctioning SQLite 3.5.9-3

package. However the Debian project has subsequently issued updated versions

of the SQLite package that correct these bugs. If you find you are getting

unexpected results under Debian, ensure you have updated your SQLite package

to 3.5.9-5 or later.

The problem does not appear to exist with other versions of SQLite packaged

with other operating systems.

Version 3.6.2

--------------

SQLite version 3.6.2 (released August 30, 2008) introduced a bug into ``SELECT

DISTINCT`` handling that is triggered by, amongst other things, Django's

``DateQuerySet`` (returned by the ``dates()`` method on a queryset).

You should avoid using this version of SQLite with Django. Either upgrade to

3.6.3 (released September 22, 2008) or later, or downgrade to an earlier

version of SQLite.

.. _using-newer-versions-of-pysqlite:

Using newer versions of the SQLite DB-API 2.0 driver

----------------------------------------------------

.. versionadded:: 1.1

For versions of Python 2.5 or newer that include ``sqlite3`` in the standard

library Django will now use a ``pysqlite2`` interface in preference to

``sqlite3`` if it finds one is available.

This provides the ability to upgrade both the DB-API 2.0 interface or SQLite 3

itself to versions newer than the ones included with your particular Python

binary distribution, if needed.

"Database is locked" errors

-----------------------------------------------

SQLite is meant to be a lightweight database, and thus can't support a high

level of concurrency. ``OperationalError: database is locked`` errors indicate

that your application is experiencing more concurrency than ``sqlite`` can

handle in default configuration. This error means that one thread or process has

an exclusive lock on the database connection and another thread timed out

waiting for the lock the be released.

Python's SQLite wrapper has

a default timeout value that determines how long the second thread is allowed to

wait on the lock before it times out and raises the ``OperationalError: database

is locked`` error.

If you're getting this error, you can solve it by:

    * Switching to another database backend. At a certain point SQLite becomes

      too "lite" for real-world applications, and these sorts of concurrency

      errors indicate you've reached that point.

    * Rewriting your code to reduce concurrency and ensure that database

      transactions are short-lived.

    * Increase the default timeout value by setting the ``timeout`` database

      option option::

          'OPTIONS': {

              # ...

              'timeout': 20,

              # ...

          }

      This will simply make SQLite wait a bit longer before throwing "database

      is locked" errors; it won't really do anything to solve them.

.. _oracle-notes:

Oracle notes

============

Django supports `Oracle Database Server`_ versions 9i and

higher. Oracle version 10g or later is required to use Django's

``regex`` and ``iregex`` query operators. You will also need at least

version 4.3.1 of the `cx_Oracle`_ Python driver.

Note that due to a Unicode-corruption bug in ``cx_Oracle`` 5.0, that

version of the driver should **not** be used with Django;

``cx_Oracle`` 5.0.1 resolved this issue, so if you'd like to use a

more recent ``cx_Oracle``, use version 5.0.1.

``cx_Oracle`` 5.0.1 or greater can optionally be compiled with the

``WITH_UNICODE`` environment variable.  This is recommended but not

required.

.. _`Oracle Database Server`: http://www.oracle.com/

.. _`cx_Oracle`: http://cx-oracle.sourceforge.net/

In order for the ``python manage.py syncdb`` command to work, your Oracle

database user must have privileges to run the following commands:

    * CREATE TABLE

    * CREATE SEQUENCE

    * CREATE PROCEDURE

    * CREATE TRIGGER

To run Django's test suite, the user needs these *additional* privileges:

    * CREATE USER

    * DROP USER

    * CREATE TABLESPACE

    * DROP TABLESPACE

    * CONNECT WITH ADMIN OPTION

    * RESOURCE WITH ADMIN OPTION

Connecting to the database

--------------------------

Your Django settings.py file should look something like this for Oracle::

    DATABASES = {

        'default': {