Plain Text | 266 lines | 220 code | 46 blank | 0 comment | 0 complexity | 0784e78212ec05d834351c9b2d35c64c MD5 | raw file
1Design Decisions 2---------------- 3 4This document describes some of the design decisions made for Silver 5Lining. It should help you understand some of "why", and if you don't 6see something in this document, it may be "unintentional design"; that 7is, not something deliberate but just expedient or not a committed 8decision. 9 10VCS integration 11=============== 12 13Several systems, most notably `Heroku <http://heroku.com/>`_, 14integrate directly with a VCS system, such as git or Mercurial. In 15these systems you push to a particular destination, and that triggers 16a deployment. 17 18Silver Lining doesn't do this, and the more I think about it the more 19I think about this it's a bad idea to do VCS integration. Some 20reasons: 21 221. It's simply not necessary. What's the real advantage of ``git 23 push`` over ``silver update``? You need an explicit command in 24 all cases, because commits don't map one-to-one to deployments. 25 262. You need a separate tool regardless. ``silver update`` does 27 stuff like updating ``/etc/hosts``, and actually a bunch of other 28 things, all of which is easy enough in a separate tool. It's not 29 the kind of thing that is reasonable to do in a post-commit hook; I 30 think it is advantageous to handle deployment synchronously. 31 323. You have to choose a VCS. I like Mercurial. Other people like 33 git. Some people even like bzr and svn. None of this relates to 34 what Silver Lining does, it doesn't need to enter that battle. 35 364. Applications are *assemblies* of code. This is a big one; 37 applications are generally made up of stuff from several 38 repositories. I actually `like putting library files into a 39 repository <devpatterns.html>`_, even when that means copying the 40 files from an upstream repository. I like this *sometimes*. If 41 you are writing the library, then this isn't a good idea; you 42 should be dealing directly with the appropriate repository, without 43 combining code from multiple repositories into one repository. 44 45 A typical application will have at least two repositories: one for 46 the "application" (i.e., the code you are developing) and another 47 for the "libraries" (everything in ``lib/python``). You can edit 48 code in your application, commit, etc. Stuff in ``lib/python`` 49 should not be edited, you should only install or upgrade/downgrade 50 those libraries (probably using ``pip``) and commit the results. 51 525. Try before you commit. Lots of things are okay to do after only 53 changing code and testing in development, but some things are more 54 likely to be problematic. It's nice to test these things with a 55 real deployment before actually committing the work. So long as 56 "commit" isn't part of the workflow, this is easy: you call 57 ``silver update``, and it doesn't care if you haven't actually 58 committed everything yet. 59 60Declarative Application Setup 61============================= 62 63The application's deployment needs are generally defined in 64``app.ini``. `It's small <appconfig.html>`_, but it covers all the 65details so far. 66 67It's come up that people have wanted hooks to do things like configure 68a database. I don't want that to happen in applications; applications 69say what they need, and the Silver Lining services give them what they 70need. Applications in this way are declarative. 71 72One of the primary things I want to do with Silver Lining is to 73separate "application" from "environment", and have Silver Lining 74manage the environment and applications exist as more of a Platonic 75ideal. 76 77If you have needs that aren't met by Silver Lining, the best way is to 78modify Silver Lining itself. It's a good chance you want to put 79something in ``silversupport.services.*``. This isn't perfect, but it has 80a side-effect that there's a collective benefit to these new features 81(since it's something reusable), and applications stay a bit simpler. 82Big features (a bunch are listed in the `todo <todo.html>`_ also 83usually belong in Silver Lining -- at least some of them, there are 84also of course big features that go right in the application. 85 86No Build Process On The Server 87============================== 88 89Tools like `Buildout <http://www.buildout.org/>`_ and `fassembler 90<http://blog.ianbicking.org/2008/06/19/my-experience-writing-a-build-system/>`_ 91target the idea of repeatable and isolated build systems. With 92deployment using these tools, you would run the build process 93(probably from scratch) for each deployment. 94 95It's kind of nice for development because it works nicely for 96development machines just like production, you just have to run the 97build tool and you have a nice local copy. One of the problems though 98is you, as the developer setting up the Buildout, become responsible 99for getting *everything* to build. If you are doing things like 100compiling a database this becomes rather burdensome. If you are using 101a database but you aren't compiling it, you then have to figure out 102integration with people's systems. It gives you the power to solve 103people's problems -- a genuine benefit -- but it also gives you the 104*responsibility* to solve people's problems. You can't just say 105"install X however is appropriate on your system". 106 107Another big problem I have with doing builds on deployment is that you 108have to go onto the server, run a bunch of stuff, handle any failures 109(which are both possible and fairly common), confirm the result, and 110then activate the new build. This is incredibly un-fun. 111 112There is *absolutely no building* when you deploy an application with 113Silver Lining. Files are copied over. Maybe your app gets a little 114chance to set itself up (``update_fetch``). In the future Silver Lining 115will probably handle backups, simple testing (to see if the app runs 116at all) and reverting the application when there's a failure. But 117there are no compiler errors. New hard-coded paths aren't introduced. 118It's simple. 119 120In my experience this *greatly* increases the ease and reliability of 121deployments. If you have a small problem with your application, you 122can fix it and deploy it and feel pretty confident your small fix will 123have small effects. 124 125Non-.py libraries are handled separately 126======================================== 127 128You'll notice if you need to use some library that isn't pure-Python, 129you need to have the Ubuntu package for that library installed. This 130happens somewhat implicitly for services, and more explicitly with the 131``packages`` `configuration setting <appconfig.html>`_. 132 133Generally I've found that Ubuntu packaging of such libraries (a) works 134well, (b) is stable and updates are appropriately conservative (c) is 135new enough. Pure-Python libraries generally get updated much more 136frequently, but everyone treats these C libraries more 137conservatively. You *should* treat C libraries more conservatively. 138 139This doesn't entirely stop you from handling a volatile C library, or 140even developing one in concert with your application. But you will 141have to turn it into a Debian package and probably create a Launchpad 142PPA. It's substantial work, but before you go messing with C 143libraries in your application you should be ready to do a lot of work 144anyway. 145 146Application API has a small surface area 147======================================== 148 149Exactly *how* Silver Lining works will change over time. There are a 150lot of places to generalize and expand its operation. As this happens 151it is important that Silver Lining applications *not* change very 152much. 153 154Right now the API for Silver Lining is fairly small (from the 155perspective of an application). There are some environmental 156variables, there is a small ``app.ini`` configuration file. 157 158While no doubt there are some additions to be made to the application 159API, I want to keep those additions as small as possible. Building up 160the infrastructure *around* applications is okay; generally that means 161stuff that we can iterate on, figure out, maybe discard. So long as 162applications are kept abstracted from the environment we have a lot of 163flexibility. As soon as we collapse the application with the 164environment we're going to have constraints and future problems. So: 165we must resist doing that. 166 167We should also consider Java as a counterexample. In the JVM 168environment they abstracted away anything resembling a specific 169operating system. Python hasn't done that, and I don't want to do 170that; if the environment leaks into an application we should still 171resist creating an abstraction. Undocumented application abilities 172will get used and may be fragile, but they are better than documented 173APIs that get changed. 174 175Fabric 176====== 177 178This doesn't really qualify as a "design decision" as it's not a 179particularly deliberate decision, but because it is asked about a 180lot... 181 182`Fabric <http://docs.fabfile.org/>`_ is a library for managing remote 183servers. It has functions to call remote commands, transfer files, 184etc., typically over ssh. As such it seems like a no-brainer to use 185for Silver Lining, right? 186 187And maybe it is, but when writing Silver Lining I did not really want 188to learn a new tool, as it would only distract from what I was trying 189to do (especially at a point in time when I wasn't sure *what* I was 190trying to do). So I kept things simple, calling out to ssh manually 191when necessary. And this actually works reasonably well. 192 193Since then it has gotten a bit more complicated, and there's a bit 194more tunneling happening, and the consistency of the codebase has 195suffered. But not *that badly* all considered. Most of it would be 196easily resolvable by simple refactoring (and the same refactoring 197would be needed even if switching to Fabric). So I remain somewhat 198reluctant to add Fabric to the mix. 199 200And really the biggest concern I have with Fabric is that underlying 201the request is a desire to do ad hoc server manipulations using 202Fabric. A lot of people have deployment systems using Fabric that 203basically poke around on the server, installing things, configuring 204things, etc, in order to deploy their application. This is definitely 205**not** how Silver Lining should work. *Internally* Silver Lining 206connects to the server and runs commands (though many of the commands 207it runs are scripts hosted on the server). But individual deployments 208work at a more abstract level. They should never be making these 209kinds of modifications. If you are *developing Silver Lining itself* 210then sure, you can add new scripts and interactions. But, as I've 211noted, it's not particularly hard to do these remote interactions with 212raw ssh callouts. 213 214So while I am not opposed to Fabric, I am not sure it is necessary or 215worth the extra layer of abstraction. Also I am a bit worried about 216the added Paramiko dependency (Paramiko implements the ssh 217interaction, and doesn't use the normal openssh ssh client, and I 218worry is a bit big; but admittedly I haven't looked closely). 219 220Ubuntu 221====== 222 223This system is based on Ubuntu. 224 225Support for other Linux distributions is not desired. Few people have 226asked about this so far, so I'm hoping this will be a 227non-controversial choice. Many parts of the system expose Ubuntu 228(especially the ``packages`` configuration). The config files are 229written based on their locations in Ubuntu and Ubuntu policy. I don't 230want to introduce any indirection to this, or any abstraction layer. 231The consistency and reliability of the system is based on the 232consistency of its components, and this is an area where I have no 233desire to support flexibility at the sacrifice of consistency. 234 235Bare Base Systems 236================= 237 238Being based on "cloud" servers, Silver Lining prefers a bare server. 239Much like the Ubuntu decision, this is intentional and provides needed 240consistency. It's not a goal to support existing servers that have 241been setup in eclectic ways. You don't have to use a "cloud" server 242to get a bare server, of course; but you **do** need a bare server. 243 244Functionality is added from concrete needs 245========================================== 246 247I'm trying to avoid implementing things I don't need at the moment. 248Of course, patches from people who have needs-at-the-moment are also 249cool. I don't want to predict how things should happen based on 250unrealistic ideas of how Silver Lining gets used. I want to react to 251how someone actually wants to use Silver Lining, with a problem 252clearly in hand. 253 254We see the same bugs, we fix the same bugs 255========================================== 256 257Part of the consistency of the server environment is that we can have 258a consistent experience, all of us, one big community of reluctant 259sysadmin/programmers (we play dual class characters). With this 260consistency bugs aren't obscure. We all deal with the same system and 261the same bugs. I want to preserve that. Forks of the code are fine, 262but I really hope they are temporary, because I want us to be fixing 263each other's bugs. There are a lot of moving pieces. I want this to 264be a finely tuned machine; complex, but refined. That takes a lot of 265iterating, and I need a community of people iterating with me to get 266there.