PageRenderTime 50ms CodeModel.GetById 22ms app.highlight 9ms RepoModel.GetById 1ms app.codeStats 1ms

Plain Text | 619 lines | 480 code | 139 blank | 0 comment | 0 complexity | ba758838b8ba6fbd52820a19e5bd419d MD5 | raw file
  1Things Silver Lining Should Do
  4This document lists out things that I think Silver Lining should do,
  5but does not yet do.  It's also kind of a notepad for design ideas.
  6It should not surprise you if I forget to remove something from this
  7document after implementing it.
  9Far-future expiration of static content
 12It's nice to present static content with far-future expiration dates.
 13Do do this while still allowing this static content to be updated, it
 14is conventional to use a timestamp of some sort in the URL itself.  An
 15example might be::
 17    /static/20100127_1/media/style.css
 19Then if the application is updated this URL will change.  You could
 20use the modification date of the file itself, but a sweeping
 21invalidation of all content is less error-prone and requires very
 22little framework support.
 24I think Silver Lining should add a new location (like in the example)
 25where content in the ``static/`` directory is served with this
 26caching, and an environmental variable ``$STATIC_URL`` which you use
 27to prefix all your media files.  Probably the current pattern will be
 28preserved, as it can be handy when retrofitting an application.  The
 29datestamp_version used in the application directories is apppriately
 32Then in the future there should also be support for uploading these
 33static files to a CDN, at which point the variable will point to the
 36Groups of servers, replication
 39There are notes on this in ``multiserver design notes
 45Backup and restore is implemented in a basic fashion.  It would be
 46very nice if backup files were portable.  Then, for instance, you
 47could download the backups to replicate live data on your site for
 48your development.  (PostGIS is a case where portable backups can be a
 49bit tricky at times.)
 51It would be even more neat if backup files were somewhat version
 52control friendly.  E.g., well-ordered SQL dumps.  This might make the
 53backup process do double duty as a test fixture process, and a process
 54to setup data in new instances.
 56Site-to-site backups should be supported (which would essentially
 57clone the data from one server to another).
 62Log files should also be handled gracefully, tracked, and possibly
 63some frontend (`frontend in progress
 64<>`_).  Though I feel like some
 65kind of management programming interface is better, and then a
 66"deployed" app would use that interface.  Actually, "interface" is
 67probably saying too much; it should be sufficient to simply have a
 68documented and conscious server layout that an application can be
 69written to inspect.
 71There should be a way to get a list of all log files for a host/app.
 72Maybe there should be a standard way to effect the Python standard
 73logging module output, and then put it in a normal location (I think
 74``/var/log/silverlining/apps/NAME/*``).  There are some more notes
 75down in Debugging_.
 80Basically a way is needed to answer the question "do I need to rent
 81bigger (or smaller) nodes?" Basic metrics like load, RAM, swap and
 82requets per second should be monitored and graphed. Munin seems to be a
 83popular choice for that.  Probably a Nagios setup makes sense here.
 84`This post
 86also has some additional ideas.
 88Also similar to ``update_fetch`` etc, there should be an optional
 89defined "ping" URL for an application.  This would be some location
 90that could be safely fetched, wouldn't be time-intensive, and would
 91report general errors.  An application could do self-testing here.
 92Monitoring infrastructure would then be setup to ping this URL.
 93Unlike ``update_fetch`` and other management URLs, this would be
 94fetched in-process in the live server (management URLs get fetched
 95with an analogous but non-mod_wsgi request).  Probably there should be
 96some indication of how often the pinging should happen (i.e., if the
 97ping is cheap do it often, if it's expensive be more reserved).
102CloudKick has started doing a bunch of monitoring things, and has an
103API.  A quick way of doing monitoring might simply be via CloudKick
104(or maybe just an option).
109We should have a good quality rotating random value.  (Like
110``silversupport.secret.get_secret``).  Google has a library for this.
112We should have a way to do security logging.  E.g., log attempted
113attacks.  There should be a whitelist, so that internal people can do
114bad things (e.g., when testing).
119I'd like to integrate Pontoon into applications, so that Silver Lining
120can be a general way to setup applications for localization.  This
121might primarily be a second deployed application that modifies files
122in the other application (somewhere in ``$CONFIG_FiLES``).
124Create Virtual Machine Images
127Local development can for some people be quite hard.  This is
128especially true of Windows users (for which silverlining doesn't have
129any support, and no support is planned).
131It would be nice to be able to run something like:
133    silver setup-develop-image . some-image-filename
135and have it get a bare Ubuntu image (for VMWare, VirtualBox, or
136whatever) and run some adaptation of setup-node, and setup a file
137server to make editing files easy, and then upload the application
138into the machine (version control files and everything).
140To make it more awesome, a built-in web app for managing the image;
141checking in files, restarting, tooling, etc.  Maybe the image could be
142run headless then?  This is more ambitious, but it could in some way
143be an introduction to good open source development practices.
145This is an aside, but probably the generalized "DEST" idea of ``silver
146backup`` would be useful here and elsewhere: anywhere you write
147something (like an image file) allow remote locations.  Of course
148silver backup is done on the remote server and uploaded from there,
149while this is done locally, so the advantage is smaller.
151The core of this is probably two variations:
153* ``setup-node``, except setting up a "development" node.  This would
154  be a node with git/hg/svn installed, somewhat different mod_wsgi
155  configuration, and other such details.
157* ``update``, except again a "development" version, one that copies
158  over VCS metadata (``.hg/`` etc).  Also it wouldn't create
159  versioned/dated directories, but instead support only one version of
160  an application (the version you are developing).
165There should be a kind of "check functions are go" URL or command,
166similar to how ``update_fetch`` works.  This would be fetched on
167update, but before the update was really made live (the internal
168request mechanism can be run on any instance, even one that is not
169live yet).  If this returned a non-2xx result, then the update would
170be aborted (or if it's a test script, exit with an error code).
172Backup/Restore/Revert on update
175Each update is a point when a persistence checkpoint should happen.
176This is most easily done with a local backup.
178There should also be a "run all database updates" URL (also similar to
179``update_fetch``).  This might fail, in which case the backup should
180be restored and the update aborted.
182I think this should be run before test-before-activate, but if
183test-before-activate fails then the databases should also be reverted.
185There are fancy ways of doing really fast checkpoints, like backups
186without actually backing things up.  These should be accessible even
187in a VPS, but they require some fancy sysadmin knowledge that I don't
193I'm fairly conflicted about what Silver Lining should do with respect
194to debugging.  I am inclined to say that applications should be
195instrumented with debugging tools natively, e.g., using Django's
196built-in tools or weberror, etc.
198On the other hand, without this kind of wrapping you get something
199that really sucks -- normal Apache error logs.  Maybe at least
200``environ['wsgi.errors']`` should be fixed (along with stderr) to
201point to something better/reasonable.  (And maybe warnings
202specifically should be handled, so only unique warnings are logged,
203and they are reset between updates.)
205Setting up the ``logging`` module is somewhat similar, it could be
206done manually by applications.  (Sometimes you want to turn up
207logging, *in production* temporarily to see what is going on in a live
208server: this is something not in the current workflow.)
210Also there should be centralized configuration of things like, e.g.,
211where error messages are sent.  Then tools could be configured in a
212more transportable way (e.g., ``DEBUG_EMAIL =
213os.environ['SILVER_DEBUG_EMAIL']``).  This might fit into how
214configuration is generally handled.
216Error reporting for non-requests (like a cron script) is harder.  Most
217of these are done synchronously, so you can just show the user what is
218happening, but at least for cron and perhaps tasks this is not the
219case.  Unifying environment setup might resolve this particular
222Cron and Tasks
225App Engine style cronjobs and tasks are definitely planned.  (I also
226just found some code called PyPeriodic that I think I wrote a long
227time ago and forgot.)
229Cron jobs are based on requests run periodically.  Though I guess they
230could also be commands.  One advantage of keeping to requests is to
231have some kind of monitoring for failed or rejected requests (though
232the internal request mechanism is not as robust currently as the
233external request system, which is managed by mod_wsgi).
235Tasks are things to be done in the future, asynchronously with any
236request.  These are queued, get retried in cases of failure (non-2xx
237response); maybe given multiple servers they could be run on different
240Cron is a higher priority.
242For persistence of the jobs, Redis seems quite possible (small enough
243that I don't feel bad having it always installed, seems relatively
244robust and easy to use).
246API Keys
249This is the obvious kind of non-portable configuration.  There is some
250support for this already, but it is poorly exposed.
252This is something that might be good to manage organization-wide.
253Which just makes the workflow slightly different.
255In some ways this is just global configuration (as opposed to
256application-specific configuration).  This same configuration could
257potentially be used for things like admin email addresses, logging
258information, etc.  Applications would have to pull it in, but it would
259be updated on a machine basis, not a per-application basis (though
260possibly you could have per-application overrides?)
262Inter-application communication
265This will only happen with a strong use case to motivate it; I can
266certainly *imagine* such use cases, but I need something more concrete
267than my imagination.
269In that case, an application could receive configuration information
270for other applications.  Communication would have to be through some
271service (e.g., a database) or HTTP.  Accessing applications on other
272servers would probably not be supported (though potentially it could
273be), but given per-path applications it is reasonable to consider
274cases where this would be useful.  Auth is an obvious case.
276The Silver Lining secret is currently per-server (and will probably be
277per-group in the future), so some things like reading cookies could be
278done across applications already (given the appropriate code).
280Testing and Test Fixture Setup
283It would be nice to have a nice setup for getting test data setup.  I
284think some degree of testing support would be nice, though simply
285installing and using a standard test runner seems mostly fine.  A
286recipe for integration testing would be nice.  A recipe for full-stack
287testing (e.g., with Windmill or Selenium) would be nice.  These are
288probably more recipes than core Silver Lining things.
290It's possible that Silver Lining could setup a second set of "testing"
291services.  So if you ran development off a database named ``myapp``,
292maybe in "testing" mode it would use ``test_myapp`` (and the database
293would be cleared on every run).
295Full-stack development environment
298I can imagine a through-the-browser development environment, based on
299Bespin.  This could be cool.  It could be really VM-friendly as well.
300Circumstances may determine if I really try this out.
302To do this, Bespin would be installed and running, and configured to
303point at the application location.  Version control can be managed
304through Bespin.
306Additionally Hudson could be installed, enabling a kind of built-in
307test suite.  I'm not sure how else Hudson could be enabled (except
308perhaps to list a test script in ``app.ini`` and go from there), but
309specifically for this case it seems like it would be nice.  By
310including a test suite configuration it might generally be possible to
311do a very simple and easy Hudson setup.
313Another interface would control Silver Lining itself, allowing you to
314configure production providers and deploy to them.  It could be very
317Hudson is available as a `third-party Debian package
318<>`_.  Bespin would probably be deployed
319as a Silver Lining application itself.  I imagine you might define a
320base domain (e.g., ``mydevel``) and then you'd have:
322* ``www.mydevel``: the application itself, running
323* ``dev.mydevel``: Bespin
324* ``test.mydevel``: Hudson
325* ``deploy.mydevel``: Silver Lining interface
327One could imagine a load tester somewhere in here too.  And maybe
328a rig for Selenium or other through-the-web tester?
330With all these, heck, maybe I'd develop through a virtual machine even
331though I'm on Linux already.
333Bespin would not preclude file sharing so that you could do editing
334(even version control) as though it was local.  It would simply be
335another alternative.
337Application Packs
340Right now this system emphasizes development.  It would be nice to
341also support ready-to-go applications.  This might be simply a zip
342file created from an application, that can be easily uploaded.  It
343might not be something at all fancy.  But thinking it out would be
346Trac would be an ideal candidate for sometihng like this.  It also has
347enough customization that I can imagine some interesting use cases;
348how do you select the plugins you want and the templates you want to
349override?  If you aren't "developing" the application then this isn't
350as clear.
352Maybe on application upload you could have two directories; one the
353"code" and another the "customizations".  The code would have a script
354to check and merge in the customizations before upload.  One can
355imagine other useful things the code could do with the customizations
356(for instance, support plugin installation through knowledge of the
357app's plugin index and techniques).
359Maybe it would look like::
361    silver update --customize=mycustomizations/
363That's the easy way.  Plugin installation?
367    silver manage-app --customize=mycustomizations/ \
368           --list-commands
369    silver manage-app --customize=mycustomizations/ \
370           install-plugin Akismet
372It feels a bit crude.  Maybe another command besides ``silver`` for
373pre-packaged apps?  Or another set of subcommands.
375Middleware Packs
378Sometimes you want to apply middleware in a deployment-specific
379fashion (not application-specific).  Some examples:
381* Site disabling
382* Error catching
383* Password protecting a site before it is public
384* Ad hoc authorization
386I'd like the ability to apply a piece of middleware to an
387application.  This wouldn't *just* be WSGI middleware, it would
388probably include some extra information (description, etc).  Also a
389PHP equivalent version ("fat" middleware) would be nice to include.
390Using PHP's output buffer operations you can simulate middleware.
392I'm not sure what the command would look like.  Maybe just::
394  silver apply-middleware path/to/middleware LOCATION
396There's also be ``show-middleware`` and ``remove-middleware``
397operations, I suppose.  Like applications, there's a missing notion of
398versions here.  Also middleware would have to be compatible with the
399libraries of the hosted application, as they would run in the same
400process.  Probably middleware libraries would be added after the path,
401and the middleware should be written to be as library-version-agnostic
402as possible.  There should be a fairly finite number of middleware
405Configuration should apply to middleware, though maybe a second set of
406configuration to avoid overlap.
408Configuration Management
411Now that there's a ``--config`` option you can have
412deployment-specific configuration of your application, but it's only
413exposed through ``silver update``.  It should be exposed as a separate
414operation, to view, edit, backup, etc. the configuration, you
415shouldn't have to deploy just to change the configuration.  A complete
416set of commands might be:
418Show the configuration (all past revisions)::
420  silver config-query LOCATION
421  # To show info about files:
422  silver config-query LOCATION --files
423  # To download config:
424  silver config-query LOCATION --dump=path/
426Then operations to actually modify the configuration::
428  # Upload new configuration:
429  silver config LOCATION path/to/config
430  # Revert configuration to some previous version ("PREV" literal, or
431  # some revision number as shown by config-query):
432  silver config LOCATION --revert VERSION
433  # Remove configuration:
434  silver config LOCATION --delete
435  # Copy configuration from another site:
436  silver config LOCATION --copy=SOURCE_LOCATION
438Doing version control on the configuration would be nice.  Also
439supporting an app that did generic through-the-web application
440configuration (needs a DevAuth-style thing again).  Such configuration
441would just do through-the-web-editing of the source files, and
442probably call the validation function if available.  It might also
443allow for operations like reverting, seeing logs, etc.
445Application Versions
448It's hard to figure out exactly what "version" of an application is
449installed somewhere.  There's no particular line, and no one thing is
450the only important version number.
452It'd be nice to look around the files for VCS data and get a version
453snapshot for all pieces.
455Another option is to have a deploy-application-registry, which would
456hold all the files that were deployed.  This would be a parallel
457version control system unrelated to any developer version control, and
458would be used only for auditing.
460Local/CDN Javascript
463I've done some applications that are mostly static pages on the
464frontend with Javascript.  I like to use Google's hosted versions of
465these libraries.  But then offline I lose access to the app :(
467It would be nice to have a local cache of these files in Silver
468Lining.  Except I can't figure out how you'd write a static HTML file
469that could select the local or remote file.  Maybe one option is to
470use Google's loader, but kind of mock it out for local use.  But even
471that is tricky.  It would be possible to do a small amount of
472rewriting of HTML pages, replacing those links (maybe literally) with
473local versions.  During development static files are served fairly
474naively, so it wouldn't be hard to do this substitution.  Also
475minimized files could be replaced with unminimized files.  Because the
476actual HTML would be production-ready there's no compromise, only in
477development are they reverted.
479Silver Lining could ship these libraries, or maybe there could be an
480``app.ini`` setting like::
482    [development]
483    local_html_replacements =
484 /_devel/jquery.js
486Then you'd put that file in ``static/_devel/jquery.js``.  I can't even
487get that on a single line...  maybe these replacements should be put
488in a separate file, and maybe that should be a set of regular
489expressions.  On the other hand, if this is really restricted to URLs
490and local files, you could "sync" these files automatically.  But in
491this case, as an example, you actually want a different file during
492development (the non-minimized ``jquery.js``).
494Also while this is otherwise kind of unresolvable for static files, it
495would be nice if this also worked for dynamic files.  There would then
496have to be some API.  Maybe each of these resources would be "named",
497in a file like::
499    [jquery]
500    production =
501    devel = /_devel/jquery.js
503And then an API like::
505    <script src="${silversupport.href['jquery']}"></script>
507Using ``__getitem__`` would make this more Django-template friendly I
508believe (though a more traditional ``silversupport.href('jquery')`` could
509also be supported).
511This could also be the basis for the CDN setup, and it's possible that
512minimization could happen using this.  Maybe like::
514    [js]
515    jquery.json file = static/js/jquery.json.js
516    app.js file = static/js/app.js
517    all_js combine = jquery.json app.js
519Then in production ``silversupport.href('all_js')`` would return a URL
520pointing to the minimized and combined version of all those files, and
521``silversupport.href('jquery.json')`` and ``silversupport.href('app.js')``
522would both return None.  In development the inverse would be true.
524Using ``silver serve`` there should be an option to do
525minimization, so the minimized versions could be tested.  The same
526ideas apply to CSS.  Something like this might make sense for images,
527but no minimization would occur.  You might not want to enumerate
528images, so you might do:
530    [images]
531    images directory = static/images/
533And then use ``${silversupport.href('images')}/folder.png``
535Start Up Local Service
538There should be a way (probably via ``~/.silverlining.conf``) to
539indicate how to start local services.  Then, for example, you would
540not need to run PostgreSQL or Couch all the time, but could make it
541available as needed.  Maybe it would look like::
543    [service.couchdb]
544    start = sudo /etc/init.d/couchdb start
545    stop = sudo /etc/init.d/couchdb stop
547Of course starting and stopping all the time is annoying, but it
548wouldn't have to happen if the reloader did the restart.  Also perhaps
549Silver Lining could lazily stop the service by daemonizing a
550stop-script on exit.  (Maybe only if there's an additional
551``lazy_stop`` setting.)
553Keep setup-node Version
556It would be good to auto-detect when setup-node has to be rerun.  I
557imagine either simply using a hash of all files that are part of
558setup-node, or using the latest timestamp.  The timestamp would change
559from computer to computer though, giving false positives.
561I imagine if you run ``silver update`` that it would do something
564    $ silver update --node=foo
565    Warning: the node foo is not up to date; setup-node should be run
566    Continue anyway (y)es/(n)o/(u)pdate?
568Yes (which would work with ``--yes``) would just keep going, No
569aborts, Update would run setup-node immediately and then continue.
571Serve general files
574You can use `mod_negotiation
575<>`_ to serve
576static files without extensions.  These are files that include headers
577(that indicate things like Content-Type) as well as the body.
578Probably they'd have a different extension.
580Security Headers
583`Content Security Policy
584<>`_ seems like you
585*might* want to enable it globally (not as part of the application).
586Specifically for static file headers.
588Maybe a middleware pack would handle this case.
590Short-term list
593Here's what I want to do right away:
595* Write a VirtualBox libcloud driver (for testing)
596  - Will use VBoxManage
597* Backup/restore (backup done, restore mostly done)
598  - Consider how this interacts with testing
599  - Specifically the Django ``test_*`` databases
600* Setup up deployment for Weave Sync 2.0 API implementation
601  - Set up multi-server deployments
602* Set up DevAuth for admin access
603* Get https working in some fashion.  (Ad hoc https seems to only get
604  harder over time though)
605* Look into the Varnish logs (Apache logs aren't that complete,
606  really)
607* Make the instantiated app available in the interactive shell
608* Move towards a general schema for commands (mostly these are
609  arguments with known names):
610  * destination location
611  * app dir
612  * provider
613  * node
614  Then some general conventions could be done for default_app, maybe
615  default_provider and some other stuff, as well as info in
616  ``~/.silverlining.conf``
617* There's some nice features in the `Mozilla AMO Hudson instace
618  <>`_ that I would
619  like here (checking coding conventions and other details).