PageRenderTime 54ms CodeModel.GetById 30ms app.highlight 2ms RepoModel.GetById 18ms app.codeStats 0ms

/docs/design.txt

https://bitbucket.org/ianb/silverlining/
Plain Text | 266 lines | 220 code | 46 blank | 0 comment | 0 complexity | 0784e78212ec05d834351c9b2d35c64c MD5 | raw file
  1Design Decisions
  2----------------
  3
  4This document describes some of the design decisions made for Silver
  5Lining.  It should help you understand some of "why", and if you don't
  6see something in this document, it may be "unintentional design"; that
  7is, not something deliberate but just expedient or not a committed
  8decision.
  9
 10VCS integration
 11===============
 12
 13Several systems, most notably `Heroku <http://heroku.com/>`_,
 14integrate directly with a VCS system, such as git or Mercurial.  In
 15these systems you push to a particular destination, and that triggers
 16a deployment.
 17
 18Silver Lining doesn't do this, and the more I think about it the more
 19I think about this it's a bad idea to do VCS integration.  Some
 20reasons:
 21
 221. It's simply not necessary.  What's the real advantage of ``git
 23   push`` over ``silver update``?  You need an explicit command in
 24   all cases, because commits don't map one-to-one to deployments.
 25
 262. You need a separate tool regardless.  ``silver update`` does
 27   stuff like updating ``/etc/hosts``, and actually a bunch of other
 28   things, all of which is easy enough in a separate tool.  It's not
 29   the kind of thing that is reasonable to do in a post-commit hook; I
 30   think it is advantageous to handle deployment synchronously.
 31
 323. You have to choose a VCS.  I like Mercurial.  Other people like
 33   git.  Some people even like bzr and svn.  None of this relates to
 34   what Silver Lining does, it doesn't need to enter that battle.
 35
 364. Applications are *assemblies* of code.  This is a big one;
 37   applications are generally made up of stuff from several
 38   repositories.  I actually `like putting library files into a
 39   repository <devpatterns.html>`_, even when that means copying the
 40   files from an upstream repository.  I like this *sometimes*.  If
 41   you are writing the library, then this isn't a good idea; you
 42   should be dealing directly with the appropriate repository, without
 43   combining code from multiple repositories into one repository.
 44
 45   A typical application will have at least two repositories: one for
 46   the "application" (i.e., the code you are developing) and another
 47   for the "libraries" (everything in ``lib/python``).  You can edit
 48   code in your application, commit, etc.  Stuff in ``lib/python``
 49   should not be edited, you should only install or upgrade/downgrade
 50   those libraries (probably using ``pip``) and commit the results.
 51
 525. Try before you commit.  Lots of things are okay to do after only
 53   changing code and testing in development, but some things are more
 54   likely to be problematic.  It's nice to test these things with a
 55   real deployment before actually committing the work.  So long as
 56   "commit" isn't part of the workflow, this is easy: you call
 57   ``silver update``, and it doesn't care if you haven't actually
 58   committed everything yet.
 59
 60Declarative Application Setup
 61=============================
 62
 63The application's deployment needs are generally defined in
 64``app.ini``.  `It's small <appconfig.html>`_, but it covers all the
 65details so far.
 66
 67It's come up that people have wanted hooks to do things like configure
 68a database.  I don't want that to happen in applications; applications
 69say what they need, and the Silver Lining services give them what they
 70need.  Applications in this way are declarative.
 71
 72One of the primary things I want to do with Silver Lining is to
 73separate "application" from "environment", and have Silver Lining
 74manage the environment and applications exist as more of a Platonic
 75ideal.
 76
 77If you have needs that aren't met by Silver Lining, the best way is to
 78modify Silver Lining itself.  It's a good chance you want to put
 79something in ``silversupport.services.*``.  This isn't perfect, but it has
 80a side-effect that there's a collective benefit to these new features
 81(since it's something reusable), and applications stay a bit simpler.
 82Big features (a bunch are listed in the `todo <todo.html>`_ also
 83usually belong in Silver Lining -- at least some of them, there are
 84also of course big features that go right in the application.
 85
 86No Build Process On The Server
 87==============================
 88
 89Tools like `Buildout <http://www.buildout.org/>`_ and `fassembler
 90<http://blog.ianbicking.org/2008/06/19/my-experience-writing-a-build-system/>`_
 91target the idea of repeatable and isolated build systems.  With
 92deployment using these tools, you would run the build process
 93(probably from scratch) for each deployment.
 94
 95It's kind of nice for development because it works nicely for
 96development machines just like production, you just have to run the
 97build tool and you have a nice local copy.  One of the problems though
 98is you, as the developer setting up the Buildout, become responsible
 99for getting *everything* to build.  If you are doing things like
100compiling a database this becomes rather burdensome.  If you are using
101a database but you aren't compiling it, you then have to figure out
102integration with people's systems.  It gives you the power to solve
103people's problems -- a genuine benefit -- but it also gives you the
104*responsibility* to solve people's problems.  You can't just say
105"install X however is appropriate on your system".
106
107Another big problem I have with doing builds on deployment is that you
108have to go onto the server, run a bunch of stuff, handle any failures
109(which are both possible and fairly common), confirm the result, and
110then activate the new build.  This is incredibly un-fun.
111
112There is *absolutely no building* when you deploy an application with
113Silver Lining.  Files are copied over.  Maybe your app gets a little
114chance to set itself up (``update_fetch``).  In the future Silver Lining
115will probably handle backups, simple testing (to see if the app runs
116at all) and reverting the application when there's a failure.  But
117there are no compiler errors.  New hard-coded paths aren't introduced.
118It's simple.
119
120In my experience this *greatly* increases the ease and reliability of
121deployments.  If you have a small problem with your application, you
122can fix it and deploy it and feel pretty confident your small fix will
123have small effects.
124
125Non-.py libraries are handled separately
126========================================
127
128You'll notice if you need to use some library that isn't pure-Python,
129you need to have the Ubuntu package for that library installed.  This
130happens somewhat implicitly for services, and more explicitly with the
131``packages`` `configuration setting <appconfig.html>`_.
132
133Generally I've found that Ubuntu packaging of such libraries (a) works
134well, (b) is stable and updates are appropriately conservative (c) is
135new enough.  Pure-Python libraries generally get updated much more
136frequently, but everyone treats these C libraries more
137conservatively.  You *should* treat C libraries more conservatively.
138
139This doesn't entirely stop you from handling a volatile C library, or
140even developing one in concert with your application.  But you will
141have to turn it into a Debian package and probably create a Launchpad
142PPA.  It's substantial work, but before you go messing with C
143libraries in your application you should be ready to do a lot of work
144anyway.
145
146Application API has a small surface area
147========================================
148
149Exactly *how* Silver Lining works will change over time.  There are a
150lot of places to generalize and expand its operation.  As this happens
151it is important that Silver Lining applications *not* change very
152much.
153
154Right now the API for Silver Lining is fairly small (from the
155perspective of an application).  There are some environmental
156variables, there is a small ``app.ini`` configuration file.
157
158While no doubt there are some additions to be made to the application
159API, I want to keep those additions as small as possible.  Building up
160the infrastructure *around* applications is okay; generally that means
161stuff that we can iterate on, figure out, maybe discard.  So long as
162applications are kept abstracted from the environment we have a lot of
163flexibility.  As soon as we collapse the application with the
164environment we're going to have constraints and future problems.  So:
165we must resist doing that.
166
167We should also consider Java as a counterexample.  In the JVM
168environment they abstracted away anything resembling a specific
169operating system.  Python hasn't done that, and I don't want to do
170that; if the environment leaks into an application we should still
171resist creating an abstraction.  Undocumented application abilities
172will get used and may be fragile, but they are better than documented
173APIs that get changed.
174
175Fabric
176======
177
178This doesn't really qualify as a "design decision" as it's not a
179particularly deliberate decision, but because it is asked about a
180lot...
181
182`Fabric <http://docs.fabfile.org/>`_ is a library for managing remote
183servers.  It has functions to call remote commands, transfer files,
184etc., typically over ssh.  As such it seems like a no-brainer to use
185for Silver Lining, right?
186
187And maybe it is, but when writing Silver Lining I did not really want
188to learn a new tool, as it would only distract from what I was trying
189to do (especially at a point in time when I wasn't sure *what* I was
190trying to do).  So I kept things simple, calling out to ssh manually
191when necessary.  And this actually works reasonably well.
192
193Since then it has gotten a bit more complicated, and there's a bit
194more tunneling happening, and the consistency of the codebase has
195suffered.  But not *that badly* all considered.  Most of it would be
196easily resolvable by simple refactoring (and the same refactoring
197would be needed even if switching to Fabric).  So I remain somewhat
198reluctant to add Fabric to the mix.
199
200And really the biggest concern I have with Fabric is that underlying
201the request is a desire to do ad hoc server manipulations using
202Fabric.  A lot of people have deployment systems using Fabric that
203basically poke around on the server, installing things, configuring
204things, etc, in order to deploy their application.  This is definitely
205**not** how Silver Lining should work.  *Internally* Silver Lining
206connects to the server and runs commands (though many of the commands
207it runs are scripts hosted on the server).  But individual deployments
208work at a more abstract level.  They should never be making these
209kinds of modifications.  If you are *developing Silver Lining itself*
210then sure, you can add new scripts and interactions.  But, as I've
211noted, it's not particularly hard to do these remote interactions with
212raw ssh callouts.
213
214So while I am not opposed to Fabric, I am not sure it is necessary or
215worth the extra layer of abstraction.  Also I am a bit worried about
216the added Paramiko dependency (Paramiko implements the ssh
217interaction, and doesn't use the normal openssh ssh client, and I
218worry is a bit big; but admittedly I haven't looked closely).
219
220Ubuntu
221======
222
223This system is based on Ubuntu.
224
225Support for other Linux distributions is not desired.  Few people have
226asked about this so far, so I'm hoping this will be a
227non-controversial choice.  Many parts of the system expose Ubuntu
228(especially the ``packages`` configuration).  The config files are
229written based on their locations in Ubuntu and Ubuntu policy.  I don't
230want to introduce any indirection to this, or any abstraction layer.
231The consistency and reliability of the system is based on the
232consistency of its components, and this is an area where I have no
233desire to support flexibility at the sacrifice of consistency.
234
235Bare Base Systems
236=================
237
238Being based on "cloud" servers, Silver Lining prefers a bare server.
239Much like the Ubuntu decision, this is intentional and provides needed
240consistency.  It's not a goal to support existing servers that have
241been setup in eclectic ways.  You don't have to use a "cloud" server
242to get a bare server, of course; but you **do** need a bare server.
243
244Functionality is added from concrete needs
245==========================================
246
247I'm trying to avoid implementing things I don't need at the moment.
248Of course, patches from people who have needs-at-the-moment are also
249cool.  I don't want to predict how things should happen based on
250unrealistic ideas of how Silver Lining gets used.  I want to react to
251how someone actually wants to use Silver Lining, with a problem
252clearly in hand.
253
254We see the same bugs, we fix the same bugs
255==========================================
256
257Part of the consistency of the server environment is that we can have
258a consistent experience, all of us, one big community of reluctant
259sysadmin/programmers (we play dual class characters).  With this
260consistency bugs aren't obscure.  We all deal with the same system and
261the same bugs.  I want to preserve that.  Forks of the code are fine,
262but I really hope they are temporary, because I want us to be fixing
263each other's bugs.  There are a lot of moving pieces.  I want this to
264be a finely tuned machine; complex, but refined.  That takes a lot of
265iterating, and I need a community of people iterating with me to get
266there.