/docs/porting.txt

  1Porting An Application To Silver Lining
  2=======================================
  3
  4This describes some of the steps of taking an existing application and
  5"porting" it to Silver Lining.
  6
  7First steps for a simple Python application
  8-------------------------------------------
  9
 10Imagine you've been developing an application for a while, using your
 11Python framework of choice.  It doesn't matter much *what* framework
 12you used.  You might have to do a little cleanup to start:
 13
 14* Make sure everything runs okay with Python 2.6.
 15* Make sure you've identified all your requirements.
 16
 17It's not required to use a ``setup.py`` file or specify your
 18requirements in ``setup.py`` with ``install_requires``.  These files
 19are good practice, but not necessary.
 20
 21Clean up your ``site-packages``
 22-------------------------------
 23
 24Some people have installed things into their computer as root, e.g.,
 25by using ``sudo easy_install`` or ``sudo pip install``.  This is a bad
 26idea.  This will cause all sorts of weird problems and complications
 27for you, not just with Silver Lining but with everything.
 28
 29I recommend deleting as much from ``/usr/lib/python2.6/site-packages``
 30as you can.  I recommend *keeping* some select stable libraries like
 31PIL, or Database drivers (psycopg, MySQLdb, etc).
 32
 33Create the file layout
 34----------------------
 35
 36Now you are ready to create a location for storing your application.
 37With Silver Lining applications all the code and libraries are stored
 38in a single directory.  I usually name this directory ``mything-app``
 39(where ``mything`` is the name of my application).  So run::
 40
 41    $ silver init mything-app
 42    $ cd mything-app
 43    $ source bin/activate
 44
 45Now you have an activated `virtualenv
 46<http://virtualenv.openplans.org>`_ with some basic Silver Lining
 47layout, though no application.
 48
 49Next, put your application in place::
 50
 51    $ cd src
 52    $ hg/git/svn clone/checkout your-repository mything
 53
 54If you already have your code checked out into, say, ``~/src/mything``
 55then unfortunately you'll either have to move it or deal with two
 56separate checkouts -- the code has to be physically located inside the
 57``mything-app`` directory, because only that directory will be
 58uploaded to the server.  Though you can *probably* do something like
 59this, but I haven't tried it::
 60
 61    $ cd $VIRTUAL_ENV/src
 62    $ ln -s ~/src/mything mything
 63
 64anyway, you probably want to do this work on a branch or separate
 65cloned repository.
 66
 67Next create a new repository for your application's libraries.  You
 68won't install libraries during the regular workflow of developing
 69applications with Silver Lining, even if you are collaborating with
 70other developers or using multiple machines -- you'll only install
 71things when you've actually added, updated, or removed a library that
 72your application uses.  To do that you'll put all your *installed*
 73libraries in another repository.  You can do that like::
 74
 75    $ cd $VIRTUAL_ENV/lib
 76    $ rm -r python
 77    $ hg init python
 78
 79or use ``git init``, or create a repository and clone/check it out.
 80This repository will be separate from your main application.
 81
 82Note that everything gets installed into ``lib/python/``, not into
 83``lib/python2.6/site-packages/`` (which exists, but won't be added
 84to).
 85
 86Installing Stuff
 87----------------
 88
 89Now you have some files in place, but nothing is really "installed".
 90
 91If you have a ``setup.py`` file, then run::
 92
 93    $ cd $VIRTUAL_ENV
 94    $ pip install -e src/mything/
 95
 96This will make the package importable and install any requirements
 97you've specified.
 98
 99If you *have not* written a ``setup.py`` file (common for Django
100applications) then you'll have to make sure the appropriate directory
101is on the path in a more manual fashion.  Let's say you want
102``src/mything/`` directly on the path, then you'll do::
103
104    $ cd $VIRTUAL_ENV/lib/python/
105    $ echo ../../src/mything/ > mything.pth
106
107If you need to install some additional things, go ahead now::
108
109    $ pip install SomeLibrary
110
111If you are developing a library concurrently with your application
112then you should also check that library out into ``src/`` and use
113``pip install -e``.
114
115You should use **pip** specifically and not easy_install, as pip
116cleans up after itself, uninstalling libraries and old versions of
117libraries.
118
119Non-Python Libraries
120--------------------
121
122If you have any libraries that you need that aren't pure-Python, you
123should install them system-wide.  Ideally you should use your
124operating system facilities to install these libraries (e.g.,
125``apt-get install python-lxml``).
126
127Make note of these libraries.  Database connection libraries will be
128installed automatically, but any other libraries will have to be
129explicitly added (more on that later).
130
131Describe your application with ``app.ini``
132------------------------------------------
133
134Now you have to tell Silver Lining how to start up your application,
135and possibly some other details.  This is done through a configuration
136file ``app.ini`` in the root of your app.
137
138There will be an example file in place.  You might just want to edit
139it, making note of these settings::
140
141    [production]
142    app_name = mything
143    runner = src/mything/silver-runner.py
144
145Those are really the most important settings.  Also if you are using a
146database (likely!) then find the appropriate line and uncomment it.
147In our example we'll imagine you are using PostgreSQL::
148
149    service.postgresql =
150
151Note that you don't need any value there.  (Maybe someday there will
152be a service that isn't just "enabled" but is actually configured, at
153which point it will take some value.)
154
155The most important thing you'll note is that now you have a "runner"
156defined in ``src/mything/silver-runner.py``.  You better write that!
157
158The specific code you'll use depends on your web framework.  This file
159will basically look like a mod_wsgi ``.wsgi`` file: it's some Python
160code that creates an object ``application``.
161
162You shouldn't need to set up your path in this file.  In fact, you
163shouldn't need to set up any of your environment, you just find the
164application object.  You can also use a `Paste Deploy
165<http://pythonpaste.org/deploy/>`_ ini file if you want (but even if
166you are using a framework such as Pylons, you can if you want import
167your ``make_app`` function directly and call it).
168
169Here's an example for Django::
170
171    import django.core.handlers.wsgi
172    application = django.core.handlers.wsgi.WSGIHandler()
173
174Setting Up Your Environment
175---------------------------
176
177One nice thing about Silver Lining is that your environment can
178*always* be setup.  That means, everything importable all the time,
179everything configurable all the time, all data available.
180
181To do this there is a specific file for you to customize (well, a file
182for you to create): ``lib/python/silvercustomize.py`` (similar to
183``sitecustomize.py``).
184
185If you are using Django, you should do at least this::
186
187    import os
188    os.environ['DJANGO_SETTINGS_MODULE'] = 'mything.settings'
189
190On other systems you might import or configure other things.  It is
191preferable to put things here rather than in your runner, as this file
192is always loaded, so it will be available in the interactive
193interpreter, in tests, in scripts, etc.
194
195Configuration
196-------------
197
198Silver Lining uses a different model of configuration than is
199typically used.  You can, if you want, ignore configuration entirely
200-- but this is about *porting* an application, so you certainly
201haven't ignored configuration before.  So you need to adapt that
202configuration.
203
204Silver Lining tells the application about the environment through
205`environmental variables <envvariables.html>`_ (``os.environ``).
206These variables are *always* set.  So if you use a PostgreSQL database
207you would look at these:
208
209``os.environ['CONFIG_PG_DBNAME']``: the name of the database you
210should connect to.
211
212``os.environ['CONFIG_PG_HOST']``: the host of the database (empty
213string means localhost).
214
215``os.environ['CONFIG_PG_PORT']``: the port of the database (empty
216string means default port).
217
218``os.environ['CONFIG_PG_USER']``: the user to connect as.
219
220``os.environ['CONFIG_PG_PASSWORD']``: the password to use (empty string
221means no password).
222
223``os.environ['CONFIG_PG_SQLALCHEMY']``: creating the SQLAlchemy
224connection string from all these pieces is somewhat annoying, so this
225does it for you.
226
227These variables will be set both during local development and when
228deployed, so you can always rely on them -- you don't need any special
229conditionals.
230
231``bin/``
232--------
233
234Yet to be written (it's kind of a pain currently).
235
236Setting Up Your Database
237------------------------
238
239Now you have something that will *run*, but with a blank database.
240That's probably not what you want.
241
242You can tell Silver Lining to fetch a URL or call a script every time
243an application is updated.  It is up to this script to do the right
244thing -- to alter tables, to notice tables that already exists, etc.
245All Silver Lining handles is fetching this program.
246
247To do this, add a setting to ``app.ini``::
248
249    [production]
250    ...
251    update_fetch = /update-database
252
253That gets a URL, to call a script::
254
255    update_fetch = script:bin/django-admin.py syncdb
256
257You can have multiple entries, one per line (indent subsequent lines).
258
259If you have to poke at a database remotely there's also a way to do
260that, with ``silver run``.
261
262These URLs or scripts will also be called everytime you start up the
263application locally using ``silver serve``.
264
265Static Files
266------------
267
268If you have static files in your application, they go in ``static/``.
269They can (and should) be symlinks.  Note that ``static`` doesn't
270appear in the URL; so if for example you have a convention that all
271your static files go in ``http://example.com/static/`` then you
272actually have to get the files into ``static/static/``.
273
274So do something like::
275
276    $ cd $VIRTUAL_ENV/static
277    $ ln -s ../src/mything/mything/media media
278
279For Django you might also symlink things from the admin media into a
280location here, like::
281
282    $ ln -s ../lib/python/django/contrib/admin/media admin-media
283
284and in ``settings.py``::
285
286    ADMIN_MEDIA_PREFIX = '/admin-media/'
287
288Non-Python Libraries
289--------------------
290
291Any non-Python libraries have to be installed in production as deb
292packages.  For instance if you need lxml you'll need the
293``python-lxml`` package installed.  You can get this to happen with
294this addition to app.ini::
295
296    packages =
297        python-lxml
298
299You can have multiple packages, one on each line.  This doesn't
300include databases though!  (Well, SQLite is fine, as it runs
301in-process.)
302
303Testing It Out
304--------------
305
306You can run this application locally now, using::
307
308    $ silver serve ~/src/mything-app/
309
310This will run any URLs or scripts in ``update_fetch``, and serve up
311the application at ``http://localhost:8080``.  If you edit any files
312the server will restart to get the refreshed files.
313
314Note that when running locally you have to do things like create
315databases on your own.  You can also `customize the local
316configuration <services.html#local-development-configuration>`_ (this
317is done *outside* the application).
318
319To test an actual deployment you need a server to deploy to.  Silver
320Lining can deploy to any dedicated Ubuntu Karmic machine that has been
321configured by Silver Lining (using ``silver setup-node``), but the
322cloud makes getting a testing server very easy and quite cheap (when
323testing, you can allocate a server just for a few hours for testing
324before getting rid of it), so you might consider opening an account.
325
326
327Problems
328--------
329
330Here are some problems you might encounter, with ideas for solutions:
331
332I can't see errors
333~~~~~~~~~~~~~~~~~~
334
335Silver Lining doesn't do anything very useful with exceptions on its
336own.  You should use whatever facilities your web framework has.
337Projects like `weberror <http://bitbucket.org/bbangert/weberror/>`_
338and `flickzeug <http://dev.pocoo.org/projects/flickzeug/>`_ are a
339generic way of wrapping your application, and could be applied in the
340runner.
341
342To detect if you are in a production environment, you can use::
343
344    from silversupport.env import is_production
345
346    application = normal instantiation
347
348    if not is_production():
349        from flickzeug.debug import DebuggedApplication
350        application = DebuggedApplication(application, evalex=True)
351
352I need custom database setup
353~~~~~~~~~~~~~~~~~~~~~~~~~~~~
354
355If you want to, you can do things more manually than just using
356``update_fetch``.  One way is to use ``silver run``.  For instance,
357instead of using ``script:bin/django-admin.py syncdb`` you can invoke
358this manually on a server like::
359
360    $ silver run mything-host.com django-admin.py syncdb
361
362This command can be interactive.  There isn't good access directly to
363commands like ``psql`` though.
364
365I need a service that isn't supported
366~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
367
368Then write the code!  You can fork `the project
369<http://bitbucket.org/ianb/silverlining>`_ and add code in
370``silversupport/services/``.  It's not very complicated, or at least
371not more complicated than using a database properly anyway.