PageRenderTime 194ms CodeModel.GetById 17ms RepoModel.GetById 17ms app.codeStats 1ms

Plain Text | 362 lines | 277 code | 85 blank | 0 comment | 0 complexity | a33a23c6ff7ece2dde9521e4e4e297d2 MD5 | raw file
Possible License(s): BSD-3-Clause
  1. ============
  2. Unicode data
  3. ============
  4. Django natively supports Unicode data everywhere. Providing your database can
  5. somehow store the data, you can safely pass around Unicode strings to
  6. templates, models and the database.
  7. This document tells you what you need to know if you're writing applications
  8. that use data or templates that are encoded in something other than ASCII.
  9. Creating the database
  10. =====================
  11. Make sure your database is configured to be able to store arbitrary string
  12. data. Normally, this means giving it an encoding of UTF-8 or UTF-16. If you use
  13. a more restrictive encoding -- for example, latin1 (iso8859-1) -- you won't be
  14. able to store certain characters in the database, and information will be lost.
  15. * MySQL users, refer to the `MySQL manual`_ (section for MySQL 5.1)
  16. for details on how to set or alter the database character set encoding.
  17. * PostgreSQL users, refer to the `PostgreSQL manual`_ (section 21.2.2 in
  18. PostgreSQL 8) for details on creating databases with the correct encoding.
  19. * SQLite users, there is nothing you need to do. SQLite always uses UTF-8
  20. for internal encoding.
  21. .. _MySQL manual:
  22. .. _PostgreSQL manual:
  23. All of Django's database backends automatically convert Unicode strings into
  24. the appropriate encoding for talking to the database. They also automatically
  25. convert strings retrieved from the database into Python Unicode strings. You
  26. don't even need to tell Django what encoding your database uses: that is
  27. handled transparently.
  28. For more, see the section "The database API" below.
  29. General string handling
  30. =======================
  31. Whenever you use strings with Django -- e.g., in database lookups, template
  32. rendering or anywhere else -- you have two choices for encoding those strings.
  33. You can use Unicode strings, or you can use normal strings (sometimes called
  34. "bytestrings") that are encoded using UTF-8.
  35. .. admonition:: Warning
  36. A bytestring does not carry any information with it about its encoding.
  37. For that reason, we have to make an assumption, and Django assumes that all
  38. bytestrings are in UTF-8.
  39. If you pass a string to Django that has been encoded in some other format,
  40. things will go wrong in interesting ways. Usually, Django will raise a
  41. ``UnicodeDecodeError`` at some point.
  42. If your code only uses ASCII data, it's safe to use your normal strings,
  43. passing them around at will, because ASCII is a subset of UTF-8.
  44. Don't be fooled into thinking that if your :setting:`DEFAULT_CHARSET` setting is set
  45. to something other than ``'utf-8'`` you can use that other encoding in your
  46. bytestrings! :setting:`DEFAULT_CHARSET` only applies to the strings generated as
  47. the result of template rendering (and e-mail). Django will always assume UTF-8
  48. encoding for internal bytestrings. The reason for this is that the
  49. :setting:`DEFAULT_CHARSET` setting is not actually under your control (if you are the
  50. application developer). It's under the control of the person installing and
  51. using your application -- and if that person chooses a different setting, your
  52. code must still continue to work. Ergo, it cannot rely on that setting.
  53. In most cases when Django is dealing with strings, it will convert them to
  54. Unicode strings before doing anything else. So, as a general rule, if you pass
  55. in a bytestring, be prepared to receive a Unicode string back in the result.
  56. Translated strings
  57. ------------------
  58. Aside from Unicode strings and bytestrings, there's a third type of string-like
  59. object you may encounter when using Django. The framework's
  60. internationalization features introduce the concept of a "lazy translation" --
  61. a string that has been marked as translated but whose actual translation result
  62. isn't determined until the object is used in a string. This feature is useful
  63. in cases where the translation locale is unknown until the string is used, even
  64. though the string might have originally been created when the code was first
  65. imported.
  66. Normally, you won't have to worry about lazy translations. Just be aware that
  67. if you examine an object and it claims to be a
  68. ``django.utils.functional.__proxy__`` object, it is a lazy translation.
  69. Calling ``unicode()`` with the lazy translation as the argument will generate a
  70. Unicode string in the current locale.
  71. For more details about lazy translation objects, refer to the
  72. :doc:`internationalization </topics/i18n/index>` documentation.
  73. Useful utility functions
  74. ------------------------
  75. Because some string operations come up again and again, Django ships with a few
  76. useful functions that should make working with Unicode and bytestring objects
  77. a bit easier.
  78. Conversion functions
  79. ~~~~~~~~~~~~~~~~~~~~
  80. The ``django.utils.encoding`` module contains a few functions that are handy
  81. for converting back and forth between Unicode and bytestrings.
  82. * ``smart_unicode(s, encoding='utf-8', strings_only=False, errors='strict')``
  83. converts its input to a Unicode string. The ``encoding`` parameter
  84. specifies the input encoding. (For example, Django uses this internally
  85. when processing form input data, which might not be UTF-8 encoded.) The
  86. ``strings_only`` parameter, if set to True, will result in Python
  87. numbers, booleans and ``None`` not being converted to a string (they keep
  88. their original types). The ``errors`` parameter takes any of the values
  89. that are accepted by Python's ``unicode()`` function for its error
  90. handling.
  91. If you pass ``smart_unicode()`` an object that has a ``__unicode__``
  92. method, it will use that method to do the conversion.
  93. * ``force_unicode(s, encoding='utf-8', strings_only=False,
  94. errors='strict')`` is identical to ``smart_unicode()`` in almost all
  95. cases. The difference is when the first argument is a :ref:`lazy
  96. translation <lazy-translations>` instance. While ``smart_unicode()``
  97. preserves lazy translations, ``force_unicode()`` forces those objects to a
  98. Unicode string (causing the translation to occur). Normally, you'll want
  99. to use ``smart_unicode()``. However, ``force_unicode()`` is useful in
  100. template tags and filters that absolutely *must* have a string to work
  101. with, not just something that can be converted to a string.
  102. * ``smart_str(s, encoding='utf-8', strings_only=False, errors='strict')``
  103. is essentially the opposite of ``smart_unicode()``. It forces the first
  104. argument to a bytestring. The ``strings_only`` parameter has the same
  105. behavior as for ``smart_unicode()`` and ``force_unicode()``. This is
  106. slightly different semantics from Python's builtin ``str()`` function,
  107. but the difference is needed in a few places within Django's internals.
  108. Normally, you'll only need to use ``smart_unicode()``. Call it as early as
  109. possible on any input data that might be either Unicode or a bytestring, and
  110. from then on, you can treat the result as always being Unicode.
  111. .. _uri-and-iri-handling:
  112. URI and IRI handling
  113. ~~~~~~~~~~~~~~~~~~~~
  114. Web frameworks have to deal with URLs (which are a type of IRI_). One
  115. requirement of URLs is that they are encoded using only ASCII characters.
  116. However, in an international environment, you might need to construct a
  117. URL from an IRI_ -- very loosely speaking, a URI that can contain Unicode
  118. characters. Quoting and converting an IRI to URI can be a little tricky, so
  119. Django provides some assistance.
  120. * The function ``django.utils.encoding.iri_to_uri()`` implements the
  121. conversion from IRI to URI as required by the specification (`RFC
  122. 3987`_).
  123. * The functions ``django.utils.http.urlquote()`` and
  124. ``django.utils.http.urlquote_plus()`` are versions of Python's standard
  125. ``urllib.quote()`` and ``urllib.quote_plus()`` that work with non-ASCII
  126. characters. (The data is converted to UTF-8 prior to encoding.)
  127. These two groups of functions have slightly different purposes, and it's
  128. important to keep them straight. Normally, you would use ``urlquote()`` on the
  129. individual portions of the IRI or URI path so that any reserved characters
  130. such as '&' or '%' are correctly encoded. Then, you apply ``iri_to_uri()`` to
  131. the full IRI and it converts any non-ASCII characters to the correct encoded
  132. values.
  133. .. note::
  134. Technically, it isn't correct to say that ``iri_to_uri()`` implements the
  135. full algorithm in the IRI specification. It doesn't (yet) perform the
  136. international domain name encoding portion of the algorithm.
  137. The ``iri_to_uri()`` function will not change ASCII characters that are
  138. otherwise permitted in a URL. So, for example, the character '%' is not
  139. further encoded when passed to ``iri_to_uri()``. This means you can pass a
  140. full URL to this function and it will not mess up the query string or anything
  141. like that.
  142. An example might clarify things here::
  143. >>> urlquote(u'Paris & Orlйans')
  144. u'Paris%20%26%20Orl%C3%A9ans'
  145. >>> iri_to_uri(u'/favorites/Franзois/%s' % urlquote(u'Paris & Orlйans'))
  146. '/favorites/Fran%C3%A7ois/Paris%20%26%20Orl%C3%A9ans'
  147. If you look carefully, you can see that the portion that was generated by
  148. ``urlquote()`` in the second example was not double-quoted when passed to
  149. ``iri_to_uri()``. This is a very important and useful feature. It means that
  150. you can construct your IRI without worrying about whether it contains
  151. non-ASCII characters and then, right at the end, call ``iri_to_uri()`` on the
  152. result.
  153. The ``iri_to_uri()`` function is also idempotent, which means the following is
  154. always true::
  155. iri_to_uri(iri_to_uri(some_string)) = iri_to_uri(some_string)
  156. So you can safely call it multiple times on the same IRI without risking
  157. double-quoting problems.
  158. .. _URI:
  159. .. _IRI:
  160. .. _RFC 3987: IRI_
  161. Models
  162. ======
  163. Because all strings are returned from the database as Unicode strings, model
  164. fields that are character based (CharField, TextField, URLField, etc) will
  165. contain Unicode values when Django retrieves data from the database. This
  166. is *always* the case, even if the data could fit into an ASCII bytestring.
  167. You can pass in bytestrings when creating a model or populating a field, and
  168. Django will convert it to Unicode when it needs to.
  169. Choosing between ``__str__()`` and ``__unicode__()``
  170. ----------------------------------------------------
  171. One consequence of using Unicode by default is that you have to take some care
  172. when printing data from the model.
  173. In particular, rather than giving your model a ``__str__()`` method, we
  174. recommended you implement a ``__unicode__()`` method. In the ``__unicode__()``
  175. method, you can quite safely return the values of all your fields without
  176. having to worry about whether they fit into a bytestring or not. (The way
  177. Python works, the result of ``__str__()`` is *always* a bytestring, even if you
  178. accidentally try to return a Unicode object).
  179. You can still create a ``__str__()`` method on your models if you want, of
  180. course, but you shouldn't need to do this unless you have a good reason.
  181. Django's ``Model`` base class automatically provides a ``__str__()``
  182. implementation that calls ``__unicode__()`` and encodes the result into UTF-8.
  183. This means you'll normally only need to implement a ``__unicode__()`` method
  184. and let Django handle the coercion to a bytestring when required.
  185. Taking care in ``get_absolute_url()``
  186. -------------------------------------
  187. URLs can only contain ASCII characters. If you're constructing a URL from
  188. pieces of data that might be non-ASCII, be careful to encode the results in a
  189. way that is suitable for a URL. The ``django.db.models.permalink()`` decorator
  190. handles this for you automatically.
  191. If you're constructing a URL manually (i.e., *not* using the ``permalink()``
  192. decorator), you'll need to take care of the encoding yourself. In this case,
  193. use the ``iri_to_uri()`` and ``urlquote()`` functions that were documented
  194. above_. For example::
  195. from django.utils.encoding import iri_to_uri
  196. from django.utils.http import urlquote
  197. def get_absolute_url(self):
  198. url = u'/person/%s/?x=0&y=0' % urlquote(self.location)
  199. return iri_to_uri(url)
  200. This function returns a correctly encoded URL even if ``self.location`` is
  201. something like "Jack visited Paris & Orlйans". (In fact, the ``iri_to_uri()``
  202. call isn't strictly necessary in the above example, because all the
  203. non-ASCII characters would have been removed in quoting in the first line.)
  204. .. _above: `URI and IRI handling`_
  205. The database API
  206. ================
  207. You can pass either Unicode strings or UTF-8 bytestrings as arguments to
  208. ``filter()`` methods and the like in the database API. The following two
  209. querysets are identical::
  210. qs = People.objects.filter(name__contains=u'Е')
  211. qs = People.objects.filter(name__contains='\xc3\x85') # UTF-8 encoding of Е
  212. Templates
  213. =========
  214. You can use either Unicode or bytestrings when creating templates manually::
  215. from django.template import Template
  216. t1 = Template('This is a bytestring template.')
  217. t2 = Template(u'This is a Unicode template.')
  218. But the common case is to read templates from the filesystem, and this creates
  219. a slight complication: not all filesystems store their data encoded as UTF-8.
  220. If your template files are not stored with a UTF-8 encoding, set the :setting:`FILE_CHARSET`
  221. setting to the encoding of the files on disk. When Django reads in a template
  222. file, it will convert the data from this encoding to Unicode. (:setting:`FILE_CHARSET`
  223. is set to ``'utf-8'`` by default.)
  224. The :setting:`DEFAULT_CHARSET` setting controls the encoding of rendered templates.
  225. This is set to UTF-8 by default.
  226. Template tags and filters
  227. -------------------------
  228. A couple of tips to remember when writing your own template tags and filters:
  229. * Always return Unicode strings from a template tag's ``render()`` method
  230. and from template filters.
  231. * Use ``force_unicode()`` in preference to ``smart_unicode()`` in these
  232. places. Tag rendering and filter calls occur as the template is being
  233. rendered, so there is no advantage to postponing the conversion of lazy
  234. translation objects into strings. It's easier to work solely with Unicode
  235. strings at that point.
  236. E-mail
  237. ======
  238. Django's e-mail framework (in ``django.core.mail``) supports Unicode
  239. transparently. You can use Unicode data in the message bodies and any headers.
  240. However, you're still obligated to respect the requirements of the e-mail
  241. specifications, so, for example, e-mail addresses should use only ASCII
  242. characters.
  243. The following code example demonstrates that everything except e-mail addresses
  244. can be non-ASCII::
  245. from django.core.mail import EmailMessage
  246. subject = u'My visit to Sшr-Trшndelag'
  247. sender = u'Arnbjцrg Rбрormsdуttir <>'
  248. recipients = ['Fred <']
  249. body = u'...'
  250. EmailMessage(subject, body, sender, recipients).send()
  251. Form submission
  252. ===============
  253. HTML form submission is a tricky area. There's no guarantee that the
  254. submission will include encoding information, which means the framework might
  255. have to guess at the encoding of submitted data.
  256. Django adopts a "lazy" approach to decoding form data. The data in an
  257. ``HttpRequest`` object is only decoded when you access it. In fact, most of
  258. the data is not decoded at all. Only the ``HttpRequest.GET`` and
  259. ``HttpRequest.POST`` data structures have any decoding applied to them. Those
  260. two fields will return their members as Unicode data. All other attributes and
  261. methods of ``HttpRequest`` return data exactly as it was submitted by the
  262. client.
  263. By default, the :setting:`DEFAULT_CHARSET` setting is used as the assumed encoding
  264. for form data. If you need to change this for a particular form, you can set
  265. the ``encoding`` attribute on an ``HttpRequest`` instance. For example::
  266. def some_view(request):
  267. # We know that the data must be encoded as KOI8-R (for some reason).
  268. request.encoding = 'koi8-r'
  269. ...
  270. You can even change the encoding after having accessed ``request.GET`` or
  271. ``request.POST``, and all subsequent accesses will use the new encoding.
  272. Most developers won't need to worry about changing form encoding, but this is
  273. a useful feature for applications that talk to legacy systems whose encoding
  274. you cannot control.
  275. Django does not decode the data of file uploads, because that data is normally
  276. treated as collections of bytes, rather than strings. Any automatic decoding
  277. there would alter the meaning of the stream of bytes.