/docs/design.txt

Design Decisions
----------------

This document describes some of the design decisions made for Silver
Lining.  It should help you understand some of "why", and if you don't
see something in this document, it may be "unintentional design"; that
is, not something deliberate but just expedient or not a committed
decision.

VCS integration
===============

Several systems, most notably `Heroku <http://heroku.com/>`_,
integrate directly with a VCS system, such as git or Mercurial.  In
these systems you push to a particular destination, and that triggers
a deployment.

Silver Lining doesn't do this, and the more I think about it the more
I think about this it's a bad idea to do VCS integration.  Some
reasons:

1. It's simply not necessary.  What's the real advantage of ``git
   push`` over ``silver update``?  You need an explicit command in
   all cases, because commits don't map one-to-one to deployments.

2. You need a separate tool regardless.  ``silver update`` does
   stuff like updating ``/etc/hosts``, and actually a bunch of other
   things, all of which is easy enough in a separate tool.  It's not
   the kind of thing that is reasonable to do in a post-commit hook; I
   think it is advantageous to handle deployment synchronously.

3. You have to choose a VCS.  I like Mercurial.  Other people like
   git.  Some people even like bzr and svn.  None of this relates to
   what Silver Lining does, it doesn't need to enter that battle.

4. Applications are *assemblies* of code.  This is a big one;
   applications are generally made up of stuff from several
   repositories.  I actually `like putting library files into a
   repository <devpatterns.html>`_, even when that means copying the
   files from an upstream repository.  I like this *sometimes*.  If
   you are writing the library, then this isn't a good idea; you
   should be dealing directly with the appropriate repository, without
   combining code from multiple repositories into one repository.

   A typical application will have at least two repositories: one for
   the "application" (i.e., the code you are developing) and another
   for the "libraries" (everything in ``lib/python``).  You can edit
   code in your application, commit, etc.  Stuff in ``lib/python``
   should not be edited, you should only install or upgrade/downgrade
   those libraries (probably using ``pip``) and commit the results.

5. Try before you commit.  Lots of things are okay to do after only
   changing code and testing in development, but some things are more
   likely to be problematic.  It's nice to test these things with a
   real deployment before actually committing the work.  So long as
   "commit" isn't part of the workflow, this is easy: you call
   ``silver update``, and it doesn't care if you haven't actually
   committed everything yet.

Declarative Application Setup
=============================

The application's deployment needs are generally defined in
``app.ini``.  `It's small <appconfig.html>`_, but it covers all the
details so far.

It's come up that people have wanted hooks to do things like configure
a database.  I don't want that to happen in applications; applications
say what they need, and the Silver Lining services give them what they
need.  Applications in this way are declarative.

One of the primary things I want to do with Silver Lining is to
separate "application" from "environment", and have Silver Lining
manage the environment and applications exist as more of a Platonic
ideal.

If you have needs that aren't met by Silver Lining, the best way is to
modify Silver Lining itself.  It's a good chance you want to put
something in ``silversupport.services.*``.  This isn't perfect, but it has
a side-effect that there's a collective benefit to these new features
(since it's something reusable), and applications stay a bit simpler.
Big features (a bunch are listed in the `todo <todo.html>`_ also
usually belong in Silver Lining -- at least some of them, there are
also of course big features that go right in the application.

No Build Process On The Server
==============================

Tools like `Buildout <http://www.buildout.org/>`_ and `fassembler
<http://blog.ianbicking.org/2008/06/19/my-experience-writing-a-build-system/>`_
target the idea of repeatable and isolated build systems.  With
deployment using these tools, you would run the build process
(probably from scratch) for each deployment.

It's kind of nice for development because it works nicely for
development machines just like production, you just have to run the
build tool and you have a nice local copy.  One of the problems though
is you, as the developer setting up the Buildout, become responsible
for getting *everything* to build.  If you are doing things like
compiling a database this becomes rather burdensome.  If you are using
a database but you aren't compiling it, you then have to figure out
integration with people's systems.  It gives you the power to solve
people's problems -- a genuine benefit -- but it also gives you the
*responsibility* to solve people's problems.  You can't just say
"install X however is appropriate on your system".

Another big problem I have with doing builds on deployment is that you
have to go onto the server, run a bunch of stuff, handle any failures
(which are both possible and fairly common), confirm the result, and
then activate the new build.  This is incredibly un-fun.

There is *absolutely no building* when you deploy an application with
Silver Lining.  Files are copied over.  Maybe your app gets a little
chance to set itself up (``update_fetch``).  In the future Silver Lining
will probably handle backups, simple testing (to see if the app runs
at all) and reverting the application when there's a failure.  But
there are no compiler errors.  New hard-coded paths aren't introduced.
It's simple.

In my experience this *greatly* increases the ease and reliability of
deployments.  If you have a small problem with your application, you
can fix it and deploy it and feel pretty confident your small fix will
have small effects.

Non-.py libraries are handled separately
========================================

You'll notice if you need to use some library that isn't pure-Python,
you need to have the Ubuntu package for that library installed.  This
happens somewhat implicitly for services, and more explicitly with the
``packages`` `configuration setting <appconfig.html>`_.

Generally I've found that Ubuntu packaging of such libraries (a) works
well, (b) is stable and updates are appropriately conservative (c) is
new enough.  Pure-Python libraries generally get updated much more
frequently, but everyone treats these C libraries more
conservatively.  You *should* treat C libraries more conservatively.

This doesn't entirely stop you from handling a volatile C library, or
even developing one in concert with your application.  But you will
have to turn it into a Debian package and probably create a Launchpad
PPA.  It's substantial work, but before you go messing with C
libraries in your application you should be ready to do a lot of work
anyway.

Application API has a small surface area
========================================

Exactly *how* Silver Lining works will change over time.  There are a
lot of places to generalize and expand its operation.  As this happens
it is important that Silver Lining applications *not* change very
much.

Right now the API for Silver Lining is fairly small (from the
perspective of an application).  There are some environmental
variables, there is a small ``app.ini`` configuration file.

While no doubt there are some additions to be made to the application
API, I want to keep those additions as small as possible.  Building up
the infrastructure *around* applications is okay; generally that means
stuff that we can iterate on, figure out, maybe discard.  So long as
applications are kept abstracted from the environment we have a lot of
flexibility.  As soon as we collapse the application with the
environment we're going to have constraints and future problems.  So:
we must resist doing that.

We should also consider Java as a counterexample.  In the JVM
environment they abstracted away anything resembling a specific
operating system.  Python hasn't done that, and I don't want to do
that; if the environment leaks into an application we should still
resist creating an abstraction.  Undocumented application abilities
will get used and may be fragile, but they are better than documented
APIs that get changed.

Fabric
======

This doesn't really qualify as a "design decision" as it's not a
particularly deliberate decision, but because it is asked about a
lot...

`Fabric <http://docs.fabfile.org/>`_ is a library for managing remote
servers.  It has functions to call remote commands, transfer files,
etc., typically over ssh.  As such it seems like a no-brainer to use
for Silver Lining, right?

And maybe it is, but when writing Silver Lining I did not really want
to learn a new tool, as it would only distract from what I was trying
to do (especially at a point in time when I wasn't sure *what* I was
trying to do).  So I kept things simple, calling out to ssh manually
when necessary.  And this actually works reasonably well.

Since then it has gotten a bit more complicated, and there's a bit
more tunneling happening, and the consistency of the codebase has
suffered.  But not *that badly* all considered.  Most of it would be
easily resolvable by simple refactoring (and the same refactoring
would be needed even if switching to Fabric).  So I remain somewhat
reluctant to add Fabric to the mix.

And really the biggest concern I have with Fabric is that underlying
the request is a desire to do ad hoc server manipulations using
Fabric.  A lot of people have deployment systems using Fabric that
basically poke around on the server, installing things, configuring
things, etc, in order to deploy their application.  This is definitely
**not** how Silver Lining should work.  *Internally* Silver Lining
connects to the server and runs commands (though many of the commands
it runs are scripts hosted on the server).  But individual deployments
work at a more abstract level.  They should never be making these
kinds of modifications.  If you are *developing Silver Lining itself*
then sure, you can add new scripts and interactions.  But, as I've
noted, it's not particularly hard to do these remote interactions with
raw ssh callouts.

So while I am not opposed to Fabric, I am not sure it is necessary or
worth the extra layer of abstraction.  Also I am a bit worried about
the added Paramiko dependency (Paramiko implements the ssh
interaction, and doesn't use the normal openssh ssh client, and I
worry is a bit big; but admittedly I haven't looked closely).

Ubuntu
======

This system is based on Ubuntu.

Support for other Linux distributions is not desired.  Few people have
asked about this so far, so I'm hoping this will be a
non-controversial choice.  Many parts of the system expose Ubuntu
(especially the ``packages`` configuration).  The config files are
written based on their locations in Ubuntu and Ubuntu policy.  I don't
want to introduce any indirection to this, or any abstraction layer.
The consistency and reliability of the system is based on the
consistency of its components, and this is an area where I have no
desire to support flexibility at the sacrifice of consistency.

Bare Base Systems
=================

Being based on "cloud" servers, Silver Lining prefers a bare server.
Much like the Ubuntu decision, this is intentional and provides needed
consistency.  It's not a goal to support existing servers that have
been setup in eclectic ways.  You don't have to use a "cloud" server
to get a bare server, of course; but you **do** need a bare server.

Functionality is added from concrete needs
==========================================

I'm trying to avoid implementing things I don't need at the moment.
Of course, patches from people who have needs-at-the-moment are also
cool.  I don't want to predict how things should happen based on
unrealistic ideas of how Silver Lining gets used.  I want to react to
how someone actually wants to use Silver Lining, with a problem
clearly in hand.

We see the same bugs, we fix the same bugs
==========================================

Part of the consistency of the server environment is that we can have
a consistent experience, all of us, one big community of reluctant
sysadmin/programmers (we play dual class characters).  With this
consistency bugs aren't obscure.  We all deal with the same system and
the same bugs.  I want to preserve that.  Forks of the code are fine,
but I really hope they are temporary, because I want us to be fixing
each other's bugs.  There are a lot of moving pieces.  I want this to
be a finely tuned machine; complex, but refined.  That takes a lot of
iterating, and I need a community of people iterating with me to get
there.