PageRenderTime 321ms CodeModel.GetById 221ms app.highlight 5ms RepoModel.GetById 88ms app.codeStats 0ms

/docs/todo.txt

https://bitbucket.org/ianb/silverlining/
Plain Text | 619 lines | 480 code | 139 blank | 0 comment | 0 complexity | ba758838b8ba6fbd52820a19e5bd419d MD5 | raw file
  1Things Silver Lining Should Do
  2==============================
  3
  4This document lists out things that I think Silver Lining should do,
  5but does not yet do.  It's also kind of a notepad for design ideas.
  6It should not surprise you if I forget to remove something from this
  7document after implementing it.
  8
  9Far-future expiration of static content
 10---------------------------------------
 11
 12It's nice to present static content with far-future expiration dates.
 13Do do this while still allowing this static content to be updated, it
 14is conventional to use a timestamp of some sort in the URL itself.  An
 15example might be::
 16
 17    /static/20100127_1/media/style.css
 18
 19Then if the application is updated this URL will change.  You could
 20use the modification date of the file itself, but a sweeping
 21invalidation of all content is less error-prone and requires very
 22little framework support.
 23
 24I think Silver Lining should add a new location (like in the example)
 25where content in the ``static/`` directory is served with this
 26caching, and an environmental variable ``$STATIC_URL`` which you use
 27to prefix all your media files.  Probably the current pattern will be
 28preserved, as it can be handy when retrofitting an application.  The
 29datestamp_version used in the application directories is apppriately
 30unique.
 31
 32Then in the future there should also be support for uploading these
 33static files to a CDN, at which point the variable will point to the
 34CDN.
 35
 36Groups of servers, replication
 37------------------------------
 38
 39There are notes on this in ``multiserver design notes
 40<multiserver.html>`_.
 41
 42Backups
 43-------
 44
 45Backup and restore is implemented in a basic fashion.  It would be
 46very nice if backup files were portable.  Then, for instance, you
 47could download the backups to replicate live data on your site for
 48your development.  (PostGIS is a case where portable backups can be a
 49bit tricky at times.)
 50
 51It would be even more neat if backup files were somewhat version
 52control friendly.  E.g., well-ordered SQL dumps.  This might make the
 53backup process do double duty as a test fixture process, and a process
 54to setup data in new instances.
 55
 56Site-to-site backups should be supported (which would essentially
 57clone the data from one server to another).
 58
 59Logs
 60----
 61
 62Log files should also be handled gracefully, tracked, and possibly
 63some frontend (`frontend in progress
 64<http://bitbucket.org/ianb/silverlog>`_).  Though I feel like some
 65kind of management programming interface is better, and then a
 66"deployed" app would use that interface.  Actually, "interface" is
 67probably saying too much; it should be sufficient to simply have a
 68documented and conscious server layout that an application can be
 69written to inspect.
 70
 71There should be a way to get a list of all log files for a host/app.
 72Maybe there should be a standard way to effect the Python standard
 73logging module output, and then put it in a normal location (I think
 74``/var/log/silverlining/apps/NAME/*``).  There are some more notes
 75down in Debugging_.
 76
 77Monitoring
 78----------
 79
 80Basically a way is needed to answer the question "do I need to rent
 81bigger (or smaller) nodes?" Basic metrics like load, RAM, swap and
 82requets per second should be monitored and graphed. Munin seems to be a
 83popular choice for that.  Probably a Nagios setup makes sense here.
 84`This post
 85<http://agiletesting.blogspot.com/2010/02/web-site-monitoring-techniques-and.html>`_
 86also has some additional ideas.
 87
 88Also similar to ``update_fetch`` etc, there should be an optional
 89defined "ping" URL for an application.  This would be some location
 90that could be safely fetched, wouldn't be time-intensive, and would
 91report general errors.  An application could do self-testing here.
 92Monitoring infrastructure would then be setup to ping this URL.
 93Unlike ``update_fetch`` and other management URLs, this would be
 94fetched in-process in the live server (management URLs get fetched
 95with an analogous but non-mod_wsgi request).  Probably there should be
 96some indication of how often the pinging should happen (i.e., if the
 97ping is cheap do it often, if it's expensive be more reserved).
 98
 99CloudKick
100~~~~~~~~~
101
102CloudKick has started doing a bunch of monitoring things, and has an
103API.  A quick way of doing monitoring might simply be via CloudKick
104(or maybe just an option).
105
106Security
107--------
108
109We should have a good quality rotating random value.  (Like
110``silversupport.secret.get_secret``).  Google has a library for this.
111
112We should have a way to do security logging.  E.g., log attempted
113attacks.  There should be a whitelist, so that internal people can do
114bad things (e.g., when testing).
115
116Translation
117-----------
118
119I'd like to integrate Pontoon into applications, so that Silver Lining
120can be a general way to setup applications for localization.  This
121might primarily be a second deployed application that modifies files
122in the other application (somewhere in ``$CONFIG_FiLES``).
123
124Create Virtual Machine Images
125-----------------------------
126
127Local development can for some people be quite hard.  This is
128especially true of Windows users (for which silverlining doesn't have
129any support, and no support is planned).
130
131It would be nice to be able to run something like:
132
133    silver setup-develop-image . some-image-filename
134
135and have it get a bare Ubuntu image (for VMWare, VirtualBox, or
136whatever) and run some adaptation of setup-node, and setup a file
137server to make editing files easy, and then upload the application
138into the machine (version control files and everything).
139
140To make it more awesome, a built-in web app for managing the image;
141checking in files, restarting, tooling, etc.  Maybe the image could be
142run headless then?  This is more ambitious, but it could in some way
143be an introduction to good open source development practices.
144
145This is an aside, but probably the generalized "DEST" idea of ``silver
146backup`` would be useful here and elsewhere: anywhere you write
147something (like an image file) allow remote locations.  Of course
148silver backup is done on the remote server and uploaded from there,
149while this is done locally, so the advantage is smaller.
150
151The core of this is probably two variations:
152
153* ``setup-node``, except setting up a "development" node.  This would
154  be a node with git/hg/svn installed, somewhat different mod_wsgi
155  configuration, and other such details.
156
157* ``update``, except again a "development" version, one that copies
158  over VCS metadata (``.hg/`` etc).  Also it wouldn't create
159  versioned/dated directories, but instead support only one version of
160  an application (the version you are developing).
161
162Test-Before-Activate
163--------------------
164
165There should be a kind of "check functions are go" URL or command,
166similar to how ``update_fetch`` works.  This would be fetched on
167update, but before the update was really made live (the internal
168request mechanism can be run on any instance, even one that is not
169live yet).  If this returned a non-2xx result, then the update would
170be aborted (or if it's a test script, exit with an error code).
171
172Backup/Restore/Revert on update
173-------------------------------
174
175Each update is a point when a persistence checkpoint should happen.
176This is most easily done with a local backup.
177
178There should also be a "run all database updates" URL (also similar to
179``update_fetch``).  This might fail, in which case the backup should
180be restored and the update aborted.
181
182I think this should be run before test-before-activate, but if
183test-before-activate fails then the databases should also be reverted.
184
185There are fancy ways of doing really fast checkpoints, like backups
186without actually backing things up.  These should be accessible even
187in a VPS, but they require some fancy sysadmin knowledge that I don't
188have.
189
190Debugging
191---------
192
193I'm fairly conflicted about what Silver Lining should do with respect
194to debugging.  I am inclined to say that applications should be
195instrumented with debugging tools natively, e.g., using Django's
196built-in tools or weberror, etc.
197
198On the other hand, without this kind of wrapping you get something
199that really sucks -- normal Apache error logs.  Maybe at least
200``environ['wsgi.errors']`` should be fixed (along with stderr) to
201point to something better/reasonable.  (And maybe warnings
202specifically should be handled, so only unique warnings are logged,
203and they are reset between updates.)
204
205Setting up the ``logging`` module is somewhat similar, it could be
206done manually by applications.  (Sometimes you want to turn up
207logging, *in production* temporarily to see what is going on in a live
208server: this is something not in the current workflow.)
209
210Also there should be centralized configuration of things like, e.g.,
211where error messages are sent.  Then tools could be configured in a
212more transportable way (e.g., ``DEBUG_EMAIL =
213os.environ['SILVER_DEBUG_EMAIL']``).  This might fit into how
214configuration is generally handled.
215
216Error reporting for non-requests (like a cron script) is harder.  Most
217of these are done synchronously, so you can just show the user what is
218happening, but at least for cron and perhaps tasks this is not the
219case.  Unifying environment setup might resolve this particular
220issue...
221
222Cron and Tasks
223--------------
224
225App Engine style cronjobs and tasks are definitely planned.  (I also
226just found some code called PyPeriodic that I think I wrote a long
227time ago and forgot.)
228
229Cron jobs are based on requests run periodically.  Though I guess they
230could also be commands.  One advantage of keeping to requests is to
231have some kind of monitoring for failed or rejected requests (though
232the internal request mechanism is not as robust currently as the
233external request system, which is managed by mod_wsgi).
234
235Tasks are things to be done in the future, asynchronously with any
236request.  These are queued, get retried in cases of failure (non-2xx
237response); maybe given multiple servers they could be run on different
238servers.
239
240Cron is a higher priority.
241
242For persistence of the jobs, Redis seems quite possible (small enough
243that I don't feel bad having it always installed, seems relatively
244robust and easy to use).
245
246API Keys
247--------
248
249This is the obvious kind of non-portable configuration.  There is some
250support for this already, but it is poorly exposed.
251
252This is something that might be good to manage organization-wide.
253Which just makes the workflow slightly different.
254
255In some ways this is just global configuration (as opposed to
256application-specific configuration).  This same configuration could
257potentially be used for things like admin email addresses, logging
258information, etc.  Applications would have to pull it in, but it would
259be updated on a machine basis, not a per-application basis (though
260possibly you could have per-application overrides?)
261
262Inter-application communication
263-------------------------------
264
265This will only happen with a strong use case to motivate it; I can
266certainly *imagine* such use cases, but I need something more concrete
267than my imagination.
268
269In that case, an application could receive configuration information
270for other applications.  Communication would have to be through some
271service (e.g., a database) or HTTP.  Accessing applications on other
272servers would probably not be supported (though potentially it could
273be), but given per-path applications it is reasonable to consider
274cases where this would be useful.  Auth is an obvious case.
275
276The Silver Lining secret is currently per-server (and will probably be
277per-group in the future), so some things like reading cookies could be
278done across applications already (given the appropriate code).
279
280Testing and Test Fixture Setup
281------------------------------
282
283It would be nice to have a nice setup for getting test data setup.  I
284think some degree of testing support would be nice, though simply
285installing and using a standard test runner seems mostly fine.  A
286recipe for integration testing would be nice.  A recipe for full-stack
287testing (e.g., with Windmill or Selenium) would be nice.  These are
288probably more recipes than core Silver Lining things.
289
290It's possible that Silver Lining could setup a second set of "testing"
291services.  So if you ran development off a database named ``myapp``,
292maybe in "testing" mode it would use ``test_myapp`` (and the database
293would be cleared on every run).
294
295Full-stack development environment
296----------------------------------
297
298I can imagine a through-the-browser development environment, based on
299Bespin.  This could be cool.  It could be really VM-friendly as well.
300Circumstances may determine if I really try this out.
301
302To do this, Bespin would be installed and running, and configured to
303point at the application location.  Version control can be managed
304through Bespin.
305
306Additionally Hudson could be installed, enabling a kind of built-in
307test suite.  I'm not sure how else Hudson could be enabled (except
308perhaps to list a test script in ``app.ini`` and go from there), but
309specifically for this case it seems like it would be nice.  By
310including a test suite configuration it might generally be possible to
311do a very simple and easy Hudson setup.
312
313Another interface would control Silver Lining itself, allowing you to
314configure production providers and deploy to them.  It could be very
315simple.
316
317Hudson is available as a `third-party Debian package
318<http://hudson-ci.org/debian/>`_.  Bespin would probably be deployed
319as a Silver Lining application itself.  I imagine you might define a
320base domain (e.g., ``mydevel``) and then you'd have:
321
322* ``www.mydevel``: the application itself, running
323* ``dev.mydevel``: Bespin
324* ``test.mydevel``: Hudson
325* ``deploy.mydevel``: Silver Lining interface
326
327One could imagine a load tester somewhere in here too.  And maybe
328a rig for Selenium or other through-the-web tester?
329
330With all these, heck, maybe I'd develop through a virtual machine even
331though I'm on Linux already.
332
333Bespin would not preclude file sharing so that you could do editing
334(even version control) as though it was local.  It would simply be
335another alternative.
336
337Application Packs
338-----------------
339
340Right now this system emphasizes development.  It would be nice to
341also support ready-to-go applications.  This might be simply a zip
342file created from an application, that can be easily uploaded.  It
343might not be something at all fancy.  But thinking it out would be
344nice.
345
346Trac would be an ideal candidate for sometihng like this.  It also has
347enough customization that I can imagine some interesting use cases;
348how do you select the plugins you want and the templates you want to
349override?  If you aren't "developing" the application then this isn't
350as clear.
351
352Maybe on application upload you could have two directories; one the
353"code" and another the "customizations".  The code would have a script
354to check and merge in the customizations before upload.  One can
355imagine other useful things the code could do with the customizations
356(for instance, support plugin installation through knowledge of the
357app's plugin index and techniques).
358
359Maybe it would look like::
360
361    silver update someapp.zip --customize=mycustomizations/
362
363That's the easy way.  Plugin installation?
364
365::
366
367    silver manage-app someapp.zip --customize=mycustomizations/ \
368           --list-commands
369    silver manage-app someapp.zip --customize=mycustomizations/ \
370           install-plugin Akismet
371
372It feels a bit crude.  Maybe another command besides ``silver`` for
373pre-packaged apps?  Or another set of subcommands.
374
375Middleware Packs
376----------------
377
378Sometimes you want to apply middleware in a deployment-specific
379fashion (not application-specific).  Some examples:
380
381* Site disabling
382* Error catching
383* Password protecting a site before it is public
384* Ad hoc authorization
385
386I'd like the ability to apply a piece of middleware to an
387application.  This wouldn't *just* be WSGI middleware, it would
388probably include some extra information (description, etc).  Also a
389PHP equivalent version ("fat" middleware) would be nice to include.
390Using PHP's output buffer operations you can simulate middleware.
391
392I'm not sure what the command would look like.  Maybe just::
393
394  silver apply-middleware path/to/middleware LOCATION
395
396There's also be ``show-middleware`` and ``remove-middleware``
397operations, I suppose.  Like applications, there's a missing notion of
398versions here.  Also middleware would have to be compatible with the
399libraries of the hosted application, as they would run in the same
400process.  Probably middleware libraries would be added after the path,
401and the middleware should be written to be as library-version-agnostic
402as possible.  There should be a fairly finite number of middleware
403packs.
404
405Configuration should apply to middleware, though maybe a second set of
406configuration to avoid overlap.
407
408Configuration Management
409------------------------
410
411Now that there's a ``--config`` option you can have
412deployment-specific configuration of your application, but it's only
413exposed through ``silver update``.  It should be exposed as a separate
414operation, to view, edit, backup, etc. the configuration, you
415shouldn't have to deploy just to change the configuration.  A complete
416set of commands might be:
417
418Show the configuration (all past revisions)::
419
420  silver config-query LOCATION
421  # To show info about files:
422  silver config-query LOCATION --files
423  # To download config:
424  silver config-query LOCATION --dump=path/
425
426Then operations to actually modify the configuration::
427
428  # Upload new configuration:
429  silver config LOCATION path/to/config
430  # Revert configuration to some previous version ("PREV" literal, or
431  # some revision number as shown by config-query):
432  silver config LOCATION --revert VERSION
433  # Remove configuration:
434  silver config LOCATION --delete
435  # Copy configuration from another site:
436  silver config LOCATION --copy=SOURCE_LOCATION
437
438Doing version control on the configuration would be nice.  Also
439supporting an app that did generic through-the-web application
440configuration (needs a DevAuth-style thing again).  Such configuration
441would just do through-the-web-editing of the source files, and
442probably call the validation function if available.  It might also
443allow for operations like reverting, seeing logs, etc.
444
445Application Versions
446--------------------
447
448It's hard to figure out exactly what "version" of an application is
449installed somewhere.  There's no particular line, and no one thing is
450the only important version number.
451
452It'd be nice to look around the files for VCS data and get a version
453snapshot for all pieces.
454
455Another option is to have a deploy-application-registry, which would
456hold all the files that were deployed.  This would be a parallel
457version control system unrelated to any developer version control, and
458would be used only for auditing.
459
460Local/CDN Javascript
461--------------------
462
463I've done some applications that are mostly static pages on the
464frontend with Javascript.  I like to use Google's hosted versions of
465these libraries.  But then offline I lose access to the app :(
466
467It would be nice to have a local cache of these files in Silver
468Lining.  Except I can't figure out how you'd write a static HTML file
469that could select the local or remote file.  Maybe one option is to
470use Google's loader, but kind of mock it out for local use.  But even
471that is tricky.  It would be possible to do a small amount of
472rewriting of HTML pages, replacing those links (maybe literally) with
473local versions.  During development static files are served fairly
474naively, so it wouldn't be hard to do this substitution.  Also
475minimized files could be replaced with unminimized files.  Because the
476actual HTML would be production-ready there's no compromise, only in
477development are they reverted.
478
479Silver Lining could ship these libraries, or maybe there could be an
480``app.ini`` setting like::
481
482    [development]
483    local_html_replacements =
484        http://ajax.googleapis.com/ajax/libs/jquery/1.4.0/jquery.min.js /_devel/jquery.js
485
486Then you'd put that file in ``static/_devel/jquery.js``.  I can't even
487get that on a single line...  maybe these replacements should be put
488in a separate file, and maybe that should be a set of regular
489expressions.  On the other hand, if this is really restricted to URLs
490and local files, you could "sync" these files automatically.  But in
491this case, as an example, you actually want a different file during
492development (the non-minimized ``jquery.js``).
493
494Also while this is otherwise kind of unresolvable for static files, it
495would be nice if this also worked for dynamic files.  There would then
496have to be some API.  Maybe each of these resources would be "named",
497in a file like::
498
499    [jquery]
500    production = http://ajax.googleapis.com/ajax/libs/jquery/1.4.0/jquery.min.js
501    devel = /_devel/jquery.js
502
503And then an API like::
504
505    <script src="${silversupport.href['jquery']}"></script>
506
507Using ``__getitem__`` would make this more Django-template friendly I
508believe (though a more traditional ``silversupport.href('jquery')`` could
509also be supported).
510
511This could also be the basis for the CDN setup, and it's possible that
512minimization could happen using this.  Maybe like::
513
514    [js]
515    jquery.json file = static/js/jquery.json.js
516    app.js file = static/js/app.js
517    all_js combine = jquery.json app.js
518
519Then in production ``silversupport.href('all_js')`` would return a URL
520pointing to the minimized and combined version of all those files, and
521``silversupport.href('jquery.json')`` and ``silversupport.href('app.js')``
522would both return None.  In development the inverse would be true.
523
524Using ``silver serve`` there should be an option to do
525minimization, so the minimized versions could be tested.  The same
526ideas apply to CSS.  Something like this might make sense for images,
527but no minimization would occur.  You might not want to enumerate
528images, so you might do:
529
530    [images]
531    images directory = static/images/
532
533And then use ``${silversupport.href('images')}/folder.png``
534
535Start Up Local Service
536----------------------
537
538There should be a way (probably via ``~/.silverlining.conf``) to
539indicate how to start local services.  Then, for example, you would
540not need to run PostgreSQL or Couch all the time, but could make it
541available as needed.  Maybe it would look like::
542
543    [service.couchdb]
544    start = sudo /etc/init.d/couchdb start
545    stop = sudo /etc/init.d/couchdb stop
546
547Of course starting and stopping all the time is annoying, but it
548wouldn't have to happen if the reloader did the restart.  Also perhaps
549Silver Lining could lazily stop the service by daemonizing a
550stop-script on exit.  (Maybe only if there's an additional
551``lazy_stop`` setting.)
552
553Keep setup-node Version
554-----------------------
555
556It would be good to auto-detect when setup-node has to be rerun.  I
557imagine either simply using a hash of all files that are part of
558setup-node, or using the latest timestamp.  The timestamp would change
559from computer to computer though, giving false positives.
560
561I imagine if you run ``silver update`` that it would do something
562like::
563
564    $ silver update --node=foo
565    Warning: the node foo is not up to date; setup-node should be run
566    Continue anyway (y)es/(n)o/(u)pdate?
567
568Yes (which would work with ``--yes``) would just keep going, No
569aborts, Update would run setup-node immediately and then continue.
570
571Serve general files
572-------------------
573
574You can use `mod_negotiation
575<http://httpd.apache.org/docs/2.2/mod/mod_negotiation.html>`_ to serve
576static files without extensions.  These are files that include headers
577(that indicate things like Content-Type) as well as the body.
578Probably they'd have a different extension.
579
580Security Headers
581----------------
582
583`Content Security Policy
584<https://wiki.mozilla.org/Security/CSP/Specification>`_ seems like you
585*might* want to enable it globally (not as part of the application).
586Specifically for static file headers.
587
588Maybe a middleware pack would handle this case.
589
590Short-term list
591---------------
592
593Here's what I want to do right away:
594
595* Write a VirtualBox libcloud driver (for testing)
596  - Will use VBoxManage
597* Backup/restore (backup done, restore mostly done)
598  - Consider how this interacts with testing
599  - Specifically the Django ``test_*`` databases
600* Setup up deployment for Weave Sync 2.0 API implementation
601  - Set up multi-server deployments
602* Set up DevAuth for admin access
603* Get https working in some fashion.  (Ad hoc https seems to only get
604  harder over time though)
605* Look into the Varnish logs (Apache logs aren't that complete,
606  really)
607* Make the instantiated app available in the interactive shell
608* Move towards a general schema for commands (mostly these are
609  arguments with known names):
610  * destination location
611  * app dir
612  * provider
613  * node
614  Then some general conventions could be done for default_app, maybe
615  default_provider and some other stuff, as well as info in
616  ``~/.silverlining.conf``
617* There's some nice features in the `Mozilla AMO Hudson instace
618  <https://hudson.mozilla.org/job/addons.mozilla.org/>`_ that I would
619  like here (checking coding conventions and other details).