/docs/todo.txt

Things Silver Lining Should Do
==============================

This document lists out things that I think Silver Lining should do,
but does not yet do.  It's also kind of a notepad for design ideas.
It should not surprise you if I forget to remove something from this
document after implementing it.

Far-future expiration of static content
---------------------------------------

It's nice to present static content with far-future expiration dates.
Do do this while still allowing this static content to be updated, it
is conventional to use a timestamp of some sort in the URL itself.  An
example might be::

    /static/20100127_1/media/style.css

Then if the application is updated this URL will change.  You could
use the modification date of the file itself, but a sweeping
invalidation of all content is less error-prone and requires very
little framework support.

I think Silver Lining should add a new location (like in the example)
where content in the ``static/`` directory is served with this
caching, and an environmental variable ``$STATIC_URL`` which you use
to prefix all your media files.  Probably the current pattern will be
preserved, as it can be handy when retrofitting an application.  The
datestamp_version used in the application directories is apppriately
unique.

Then in the future there should also be support for uploading these
static files to a CDN, at which point the variable will point to the
CDN.

Groups of servers, replication
------------------------------

There are notes on this in ``multiserver design notes
<multiserver.html>`_.

Backups
-------

Backup and restore is implemented in a basic fashion.  It would be
very nice if backup files were portable.  Then, for instance, you
could download the backups to replicate live data on your site for
your development.  (PostGIS is a case where portable backups can be a
bit tricky at times.)

It would be even more neat if backup files were somewhat version
control friendly.  E.g., well-ordered SQL dumps.  This might make the
backup process do double duty as a test fixture process, and a process
to setup data in new instances.

Site-to-site backups should be supported (which would essentially
clone the data from one server to another).

Logs
----

Log files should also be handled gracefully, tracked, and possibly
some frontend (`frontend in progress
<http://bitbucket.org/ianb/silverlog>`_).  Though I feel like some
kind of management programming interface is better, and then a
"deployed" app would use that interface.  Actually, "interface" is
probably saying too much; it should be sufficient to simply have a
documented and conscious server layout that an application can be
written to inspect.

There should be a way to get a list of all log files for a host/app.
Maybe there should be a standard way to effect the Python standard
logging module output, and then put it in a normal location (I think
``/var/log/silverlining/apps/NAME/*``).  There are some more notes
down in Debugging_.

Monitoring
----------

Basically a way is needed to answer the question "do I need to rent
bigger (or smaller) nodes?" Basic metrics like load, RAM, swap and
requets per second should be monitored and graphed. Munin seems to be a
popular choice for that.  Probably a Nagios setup makes sense here.
`This post
<http://agiletesting.blogspot.com/2010/02/web-site-monitoring-techniques-and.html>`_
also has some additional ideas.

Also similar to ``update_fetch`` etc, there should be an optional
defined "ping" URL for an application.  This would be some location
that could be safely fetched, wouldn't be time-intensive, and would
report general errors.  An application could do self-testing here.
Monitoring infrastructure would then be setup to ping this URL.
Unlike ``update_fetch`` and other management URLs, this would be
fetched in-process in the live server (management URLs get fetched
with an analogous but non-mod_wsgi request).  Probably there should be
some indication of how often the pinging should happen (i.e., if the
ping is cheap do it often, if it's expensive be more reserved).

CloudKick
~~~~~~~~~

CloudKick has started doing a bunch of monitoring things, and has an
API.  A quick way of doing monitoring might simply be via CloudKick
(or maybe just an option).

Security
--------

We should have a good quality rotating random value.  (Like
``silversupport.secret.get_secret``).  Google has a library for this.

We should have a way to do security logging.  E.g., log attempted
attacks.  There should be a whitelist, so that internal people can do
bad things (e.g., when testing).

Translation
-----------

I'd like to integrate Pontoon into applications, so that Silver Lining
can be a general way to setup applications for localization.  This
might primarily be a second deployed application that modifies files
in the other application (somewhere in ``$CONFIG_FiLES``).

Create Virtual Machine Images
-----------------------------

Local development can for some people be quite hard.  This is
especially true of Windows users (for which silverlining doesn't have
any support, and no support is planned).

It would be nice to be able to run something like:

    silver setup-develop-image . some-image-filename

and have it get a bare Ubuntu image (for VMWare, VirtualBox, or
whatever) and run some adaptation of setup-node, and setup a file
server to make editing files easy, and then upload the application
into the machine (version control files and everything).

To make it more awesome, a built-in web app for managing the image;
checking in files, restarting, tooling, etc.  Maybe the image could be
run headless then?  This is more ambitious, but it could in some way
be an introduction to good open source development practices.

This is an aside, but probably the generalized "DEST" idea of ``silver
backup`` would be useful here and elsewhere: anywhere you write
something (like an image file) allow remote locations.  Of course
silver backup is done on the remote server and uploaded from there,
while this is done locally, so the advantage is smaller.

The core of this is probably two variations:

* ``setup-node``, except setting up a "development" node.  This would
  be a node with git/hg/svn installed, somewhat different mod_wsgi
  configuration, and other such details.

* ``update``, except again a "development" version, one that copies
  over VCS metadata (``.hg/`` etc).  Also it wouldn't create
  versioned/dated directories, but instead support only one version of
  an application (the version you are developing).

Test-Before-Activate
--------------------

There should be a kind of "check functions are go" URL or command,
similar to how ``update_fetch`` works.  This would be fetched on
update, but before the update was really made live (the internal
request mechanism can be run on any instance, even one that is not
live yet).  If this returned a non-2xx result, then the update would
be aborted (or if it's a test script, exit with an error code).

Backup/Restore/Revert on update
-------------------------------

Each update is a point when a persistence checkpoint should happen.
This is most easily done with a local backup.

There should also be a "run all database updates" URL (also similar to
``update_fetch``).  This might fail, in which case the backup should
be restored and the update aborted.

I think this should be run before test-before-activate, but if
test-before-activate fails then the databases should also be reverted.

There are fancy ways of doing really fast checkpoints, like backups
without actually backing things up.  These should be accessible even
in a VPS, but they require some fancy sysadmin knowledge that I don't
have.

Debugging
---------

I'm fairly conflicted about what Silver Lining should do with respect
to debugging.  I am inclined to say that applications should be
instrumented with debugging tools natively, e.g., using Django's
built-in tools or weberror, etc.

On the other hand, without this kind of wrapping you get something
that really sucks -- normal Apache error logs.  Maybe at least
``environ['wsgi.errors']`` should be fixed (along with stderr) to
point to something better/reasonable.  (And maybe warnings
specifically should be handled, so only unique warnings are logged,
and they are reset between updates.)

Setting up the ``logging`` module is somewhat similar, it could be
done manually by applications.  (Sometimes you want to turn up
logging, *in production* temporarily to see what is going on in a live
server: this is something not in the current workflow.)

Also there should be centralized configuration of things like, e.g.,
where error messages are sent.  Then tools could be configured in a
more transportable way (e.g., ``DEBUG_EMAIL =
os.environ['SILVER_DEBUG_EMAIL']``).  This might fit into how
configuration is generally handled.

Error reporting for non-requests (like a cron script) is harder.  Most
of these are done synchronously, so you can just show the user what is
happening, but at least for cron and perhaps tasks this is not the
case.  Unifying environment setup might resolve this particular
issue...

Cron and Tasks
--------------

App Engine style cronjobs and tasks are definitely planned.  (I also
just found some code called PyPeriodic that I think I wrote a long
time ago and forgot.)

Cron jobs are based on requests run periodically.  Though I guess they
could also be commands.  One advantage of keeping to requests is to
have some kind of monitoring for failed or rejected requests (though
the internal request mechanism is not as robust currently as the
external request system, which is managed by mod_wsgi).

Tasks are things to be done in the future, asynchronously with any
request.  These are queued, get retried in cases of failure (non-2xx
response); maybe given multiple servers they could be run on different
servers.

Cron is a higher priority.

For persistence of the jobs, Redis seems quite possible (small enough
that I don't feel bad having it always installed, seems relatively
robust and easy to use).

API Keys
--------

This is the obvious kind of non-portable configuration.  There is some
support for this already, but it is poorly exposed.

This is something that might be good to manage organization-wide.
Which just makes the workflow slightly different.

In some ways this is just global configuration (as opposed to
application-specific configuration).  This same configuration could
potentially be used for things like admin email addresses, logging
information, etc.  Applications would have to pull it in, but it would
be updated on a machine basis, not a per-application basis (though
possibly you could have per-application overrides?)

Inter-application communication
-------------------------------

This will only happen with a strong use case to motivate it; I can
certainly *imagine* such use cases, but I need something more concrete
than my imagination.

In that case, an application could receive configuration information
for other applications.  Communication would have to be through some
service (e.g., a database) or HTTP.  Accessing applications on other
servers would probably not be supported (though potentially it could
be), but given per-path applications it is reasonable to consider
cases where this would be useful.  Auth is an obvious case.

The Silver Lining secret is currently per-server (and will probably be
per-group in the future), so some things like reading cookies could be
done across applications already (given the appropriate code).

Testing and Test Fixture Setup
------------------------------

It would be nice to have a nice setup for getting test data setup.  I
think some degree of testing support would be nice, though simply
installing and using a standard test runner seems mostly fine.  A
recipe for integration testing would be nice.  A recipe for full-stack
testing (e.g., with Windmill or Selenium) would be nice.  These are
probably more recipes than core Silver Lining things.

It's possible that Silver Lining could setup a second set of "testing"
services.  So if you ran development off a database named ``myapp``,
maybe in "testing" mode it would use ``test_myapp`` (and the database
would be cleared on every run).

Full-stack development environment
----------------------------------

I can imagine a through-the-browser development environment, based on
Bespin.  This could be cool.  It could be really VM-friendly as well.
Circumstances may determine if I really try this out.

To do this, Bespin would be installed and running, and configured to
point at the application location.  Version control can be managed
through Bespin.

Additionally Hudson could be installed, enabling a kind of built-in
test suite.  I'm not sure how else Hudson could be enabled (except
perhaps to list a test script in ``app.ini`` and go from there), but
specifically for this case it seems like it would be nice.  By
including a test suite configuration it might generally be possible to
do a very simple and easy Hudson setup.

Another interface would control Silver Lining itself, allowing you to
configure production providers and deploy to them.  It could be very
simple.

Hudson is available as a `third-party Debian package
<http://hudson-ci.org/debian/>`_.  Bespin would probably be deployed
as a Silver Lining application itself.  I imagine you might define a
base domain (e.g., ``mydevel``) and then you'd have:

* ``www.mydevel``: the application itself, running
* ``dev.mydevel``: Bespin
* ``test.mydevel``: Hudson
* ``deploy.mydevel``: Silver Lining interface

One could imagine a load tester somewhere in here too.  And maybe
a rig for Selenium or other through-the-web tester?

With all these, heck, maybe I'd develop through a virtual machine even
though I'm on Linux already.

Bespin would not preclude file sharing so that you could do editing
(even version control) as though it was local.  It would simply be
another alternative.

Application Packs
-----------------

Right now this system emphasizes development.  It would be nice to
also support ready-to-go applications.  This might be simply a zip
file created from an application, that can be easily uploaded.  It
might not be something at all fancy.  But thinking it out would be
nice.

Trac would be an ideal candidate for sometihng like this.  It also has
enough customization that I can imagine some interesting use cases;
how do you select the plugins you want and the templates you want to
override?  If you aren't "developing" the application then this isn't
as clear.

Maybe on application upload you could have two directories; one the
"code" and another the "customizations".  The code would have a script
to check and merge in the customizations before upload.  One can
imagine other useful things the code could do with the customizations
(for instance, support plugin installation through knowledge of the
app's plugin index and techniques).

Maybe it would look like::

    silver update someapp.zip --customize=mycustomizations/

That's the easy way.  Plugin installation?

::

    silver manage-app someapp.zip --customize=mycustomizations/ \
           --list-commands
    silver manage-app someapp.zip --customize=mycustomizations/ \
           install-plugin Akismet

It feels a bit crude.  Maybe another command besides ``silver`` for
pre-packaged apps?  Or another set of subcommands.

Middleware Packs
----------------

Sometimes you want to apply middleware in a deployment-specific
fashion (not application-specific).  Some examples:

* Site disabling
* Error catching
* Password protecting a site before it is public
* Ad hoc authorization

I'd like the ability to apply a piece of middleware to an
application.  This wouldn't *just* be WSGI middleware, it would
probably include some extra information (description, etc).  Also a
PHP equivalent version ("fat" middleware) would be nice to include.
Using PHP's output buffer operations you can simulate middleware.

I'm not sure what the command would look like.  Maybe just::

  silver apply-middleware path/to/middleware LOCATION

There's also be ``show-middleware`` and ``remove-middleware``
operations, I suppose.  Like applications, there's a missing notion of
versions here.  Also middleware would have to be compatible with the
libraries of the hosted application, as they would run in the same
process.  Probably middleware libraries would be added after the path,
and the middleware should be written to be as library-version-agnostic
as possible.  There should be a fairly finite number of middleware
packs.

Configuration should apply to middleware, though maybe a second set of
configuration to avoid overlap.

Configuration Management
------------------------

Now that there's a ``--config`` option you can have
deployment-specific configuration of your application, but it's only
exposed through ``silver update``.  It should be exposed as a separate
operation, to view, edit, backup, etc. the configuration, you
shouldn't have to deploy just to change the configuration.  A complete
set of commands might be:

Show the configuration (all past revisions)::

  silver config-query LOCATION
  # To show info about files:
  silver config-query LOCATION --files
  # To download config:
  silver config-query LOCATION --dump=path/

Then operations to actually modify the configuration::

  # Upload new configuration:
  silver config LOCATION path/to/config
  # Revert configuration to some previous version ("PREV" literal, or
  # some revision number as shown by config-query):
  silver config LOCATION --revert VERSION
  # Remove configuration:
  silver config LOCATION --delete
  # Copy configuration from another site:
  silver config LOCATION --copy=SOURCE_LOCATION

Doing version control on the configuration would be nice.  Also
supporting an app that did generic through-the-web application
configuration (needs a DevAuth-style thing again).  Such configuration
would just do through-the-web-editing of the source files, and
probably call the validation function if available.  It might also
allow for operations like reverting, seeing logs, etc.

Application Versions
--------------------

It's hard to figure out exactly what "version" of an application is
installed somewhere.  There's no particular line, and no one thing is
the only important version number.

It'd be nice to look around the files for VCS data and get a version
snapshot for all pieces.

Another option is to have a deploy-application-registry, which would
hold all the files that were deployed.  This would be a parallel
version control system unrelated to any developer version control, and
would be used only for auditing.

Local/CDN Javascript
--------------------

I've done some applications that are mostly static pages on the
frontend with Javascript.  I like to use Google's hosted versions of
these libraries.  But then offline I lose access to the app :(

It would be nice to have a local cache of these files in Silver
Lining.  Except I can't figure out how you'd write a static HTML file
that could select the local or remote file.  Maybe one option is to
use Google's loader, but kind of mock it out for local use.  But even
that is tricky.  It would be possible to do a small amount of
rewriting of HTML pages, replacing those links (maybe literally) with
local versions.  During development static files are served fairly
naively, so it wouldn't be hard to do this substitution.  Also
minimized files could be replaced with unminimized files.  Because the
actual HTML would be production-ready there's no compromise, only in
development are they reverted.

Silver Lining could ship these libraries, or maybe there could be an
``app.ini`` setting like::

    [development]
    local_html_replacements =
        http://ajax.googleapis.com/ajax/libs/jquery/1.4.0/jquery.min.js /_devel/jquery.js

Then you'd put that file in ``static/_devel/jquery.js``.  I can't even
get that on a single line...  maybe these replacements should be put
in a separate file, and maybe that should be a set of regular
expressions.  On the other hand, if this is really restricted to URLs
and local files, you could "sync" these files automatically.  But in
this case, as an example, you actually want a different file during
development (the non-minimized ``jquery.js``).

Also while this is otherwise kind of unresolvable for static files, it
would be nice if this also worked for dynamic files.  There would then
have to be some API.  Maybe each of these resources would be "named",
in a file like::

    [jquery]
    production = http://ajax.googleapis.com/ajax/libs/jquery/1.4.0/jquery.min.js
    devel = /_devel/jquery.js

And then an API like::

    <script src="${silversupport.href['jquery']}"></script>

Using ``__getitem__`` would make this more Django-template friendly I
believe (though a more traditional ``silversupport.href('jquery')`` could
also be supported).

This could also be the basis for the CDN setup, and it's possible that
minimization could happen using this.  Maybe like::

    [js]
    jquery.json file = static/js/jquery.json.js
    app.js file = static/js/app.js
    all_js combine = jquery.json app.js

Then in production ``silversupport.href('all_js')`` would return a URL
pointing to the minimized and combined version of all those files, and
``silversupport.href('jquery.json')`` and ``silversupport.href('app.js')``
would both return None.  In development the inverse would be true.

Using ``silver serve`` there should be an option to do
minimization, so the minimized versions could be tested.  The same
ideas apply to CSS.  Something like this might make sense for images,
but no minimization would occur.  You might not want to enumerate
images, so you might do:

    [images]
    images directory = static/images/

And then use ``${silversupport.href('images')}/folder.png``

Start Up Local Service
----------------------

There should be a way (probably via ``~/.silverlining.conf``) to
indicate how to start local services.  Then, for example, you would
not need to run PostgreSQL or Couch all the time, but could make it
available as needed.  Maybe it would look like::

    [service.couchdb]
    start = sudo /etc/init.d/couchdb start
    stop = sudo /etc/init.d/couchdb stop

Of course starting and stopping all the time is annoying, but it
wouldn't have to happen if the reloader did the restart.  Also perhaps
Silver Lining could lazily stop the service by daemonizing a
stop-script on exit.  (Maybe only if there's an additional
``lazy_stop`` setting.)

Keep setup-node Version
-----------------------

It would be good to auto-detect when setup-node has to be rerun.  I
imagine either simply using a hash of all files that are part of
setup-node, or using the latest timestamp.  The timestamp would change
from computer to computer though, giving false positives.

I imagine if you run ``silver update`` that it would do something
like::

    $ silver update --node=foo
    Warning: the node foo is not up to date; setup-node should be run
    Continue anyway (y)es/(n)o/(u)pdate?

Yes (which would work with ``--yes``) would just keep going, No
aborts, Update would run setup-node immediately and then continue.

Serve general files
-------------------

You can use `mod_negotiation
<http://httpd.apache.org/docs/2.2/mod/mod_negotiation.html>`_ to serve
static files without extensions.  These are files that include headers
(that indicate things like Content-Type) as well as the body.
Probably they'd have a different extension.

Security Headers
----------------

`Content Security Policy
<https://wiki.mozilla.org/Security/CSP/Specification>`_ seems like you
*might* want to enable it globally (not as part of the application).
Specifically for static file headers.

Maybe a middleware pack would handle this case.

Short-term list
---------------

Here's what I want to do right away:

* Write a VirtualBox libcloud driver (for testing)
  - Will use VBoxManage
* Backup/restore (backup done, restore mostly done)
  - Consider how this interacts with testing
  - Specifically the Django ``test_*`` databases
* Setup up deployment for Weave Sync 2.0 API implementation
  - Set up multi-server deployments
* Set up DevAuth for admin access
* Get https working in some fashion.  (Ad hoc https seems to only get
  harder over time though)
* Look into the Varnish logs (Apache logs aren't that complete,
  really)
* Make the instantiated app available in the interactive shell
* Move towards a general schema for commands (mostly these are
  arguments with known names):
  * destination location
  * app dir
  * provider
  * node
  Then some general conventions could be done for default_app, maybe
  default_provider and some other stuff, as well as info in
  ``~/.silverlining.conf``
* There's some nice features in the `Mozilla AMO Hudson instace
  <https://hudson.mozilla.org/job/addons.mozilla.org/>`_ that I would
  like here (checking coding conventions and other details).