PageRenderTime 112ms CodeModel.GetById 14ms RepoModel.GetById 0ms app.codeStats 1ms

Plain Text | 266 lines | 220 code | 46 blank | 0 comment | 0 complexity | 0784e78212ec05d834351c9b2d35c64c MD5 | raw file
Possible License(s): GPL-2.0
  1. Design Decisions
  2. ----------------
  3. This document describes some of the design decisions made for Silver
  4. Lining. It should help you understand some of "why", and if you don't
  5. see something in this document, it may be "unintentional design"; that
  6. is, not something deliberate but just expedient or not a committed
  7. decision.
  8. VCS integration
  9. ===============
  10. Several systems, most notably `Heroku <>`_,
  11. integrate directly with a VCS system, such as git or Mercurial. In
  12. these systems you push to a particular destination, and that triggers
  13. a deployment.
  14. Silver Lining doesn't do this, and the more I think about it the more
  15. I think about this it's a bad idea to do VCS integration. Some
  16. reasons:
  17. 1. It's simply not necessary. What's the real advantage of ``git
  18. push`` over ``silver update``? You need an explicit command in
  19. all cases, because commits don't map one-to-one to deployments.
  20. 2. You need a separate tool regardless. ``silver update`` does
  21. stuff like updating ``/etc/hosts``, and actually a bunch of other
  22. things, all of which is easy enough in a separate tool. It's not
  23. the kind of thing that is reasonable to do in a post-commit hook; I
  24. think it is advantageous to handle deployment synchronously.
  25. 3. You have to choose a VCS. I like Mercurial. Other people like
  26. git. Some people even like bzr and svn. None of this relates to
  27. what Silver Lining does, it doesn't need to enter that battle.
  28. 4. Applications are *assemblies* of code. This is a big one;
  29. applications are generally made up of stuff from several
  30. repositories. I actually `like putting library files into a
  31. repository <devpatterns.html>`_, even when that means copying the
  32. files from an upstream repository. I like this *sometimes*. If
  33. you are writing the library, then this isn't a good idea; you
  34. should be dealing directly with the appropriate repository, without
  35. combining code from multiple repositories into one repository.
  36. A typical application will have at least two repositories: one for
  37. the "application" (i.e., the code you are developing) and another
  38. for the "libraries" (everything in ``lib/python``). You can edit
  39. code in your application, commit, etc. Stuff in ``lib/python``
  40. should not be edited, you should only install or upgrade/downgrade
  41. those libraries (probably using ``pip``) and commit the results.
  42. 5. Try before you commit. Lots of things are okay to do after only
  43. changing code and testing in development, but some things are more
  44. likely to be problematic. It's nice to test these things with a
  45. real deployment before actually committing the work. So long as
  46. "commit" isn't part of the workflow, this is easy: you call
  47. ``silver update``, and it doesn't care if you haven't actually
  48. committed everything yet.
  49. Declarative Application Setup
  50. =============================
  51. The application's deployment needs are generally defined in
  52. ``app.ini``. `It's small <appconfig.html>`_, but it covers all the
  53. details so far.
  54. It's come up that people have wanted hooks to do things like configure
  55. a database. I don't want that to happen in applications; applications
  56. say what they need, and the Silver Lining services give them what they
  57. need. Applications in this way are declarative.
  58. One of the primary things I want to do with Silver Lining is to
  59. separate "application" from "environment", and have Silver Lining
  60. manage the environment and applications exist as more of a Platonic
  61. ideal.
  62. If you have needs that aren't met by Silver Lining, the best way is to
  63. modify Silver Lining itself. It's a good chance you want to put
  64. something in ``*``. This isn't perfect, but it has
  65. a side-effect that there's a collective benefit to these new features
  66. (since it's something reusable), and applications stay a bit simpler.
  67. Big features (a bunch are listed in the `todo <todo.html>`_ also
  68. usually belong in Silver Lining -- at least some of them, there are
  69. also of course big features that go right in the application.
  70. No Build Process On The Server
  71. ==============================
  72. Tools like `Buildout <>`_ and `fassembler
  73. <>`_
  74. target the idea of repeatable and isolated build systems. With
  75. deployment using these tools, you would run the build process
  76. (probably from scratch) for each deployment.
  77. It's kind of nice for development because it works nicely for
  78. development machines just like production, you just have to run the
  79. build tool and you have a nice local copy. One of the problems though
  80. is you, as the developer setting up the Buildout, become responsible
  81. for getting *everything* to build. If you are doing things like
  82. compiling a database this becomes rather burdensome. If you are using
  83. a database but you aren't compiling it, you then have to figure out
  84. integration with people's systems. It gives you the power to solve
  85. people's problems -- a genuine benefit -- but it also gives you the
  86. *responsibility* to solve people's problems. You can't just say
  87. "install X however is appropriate on your system".
  88. Another big problem I have with doing builds on deployment is that you
  89. have to go onto the server, run a bunch of stuff, handle any failures
  90. (which are both possible and fairly common), confirm the result, and
  91. then activate the new build. This is incredibly un-fun.
  92. There is *absolutely no building* when you deploy an application with
  93. Silver Lining. Files are copied over. Maybe your app gets a little
  94. chance to set itself up (``update_fetch``). In the future Silver Lining
  95. will probably handle backups, simple testing (to see if the app runs
  96. at all) and reverting the application when there's a failure. But
  97. there are no compiler errors. New hard-coded paths aren't introduced.
  98. It's simple.
  99. In my experience this *greatly* increases the ease and reliability of
  100. deployments. If you have a small problem with your application, you
  101. can fix it and deploy it and feel pretty confident your small fix will
  102. have small effects.
  103. libraries are handled separately
  104. ========================================
  105. You'll notice if you need to use some library that isn't pure-Python,
  106. you need to have the Ubuntu package for that library installed. This
  107. happens somewhat implicitly for services, and more explicitly with the
  108. ``packages`` `configuration setting <appconfig.html>`_.
  109. Generally I've found that Ubuntu packaging of such libraries (a) works
  110. well, (b) is stable and updates are appropriately conservative (c) is
  111. new enough. Pure-Python libraries generally get updated much more
  112. frequently, but everyone treats these C libraries more
  113. conservatively. You *should* treat C libraries more conservatively.
  114. This doesn't entirely stop you from handling a volatile C library, or
  115. even developing one in concert with your application. But you will
  116. have to turn it into a Debian package and probably create a Launchpad
  117. PPA. It's substantial work, but before you go messing with C
  118. libraries in your application you should be ready to do a lot of work
  119. anyway.
  120. Application API has a small surface area
  121. ========================================
  122. Exactly *how* Silver Lining works will change over time. There are a
  123. lot of places to generalize and expand its operation. As this happens
  124. it is important that Silver Lining applications *not* change very
  125. much.
  126. Right now the API for Silver Lining is fairly small (from the
  127. perspective of an application). There are some environmental
  128. variables, there is a small ``app.ini`` configuration file.
  129. While no doubt there are some additions to be made to the application
  130. API, I want to keep those additions as small as possible. Building up
  131. the infrastructure *around* applications is okay; generally that means
  132. stuff that we can iterate on, figure out, maybe discard. So long as
  133. applications are kept abstracted from the environment we have a lot of
  134. flexibility. As soon as we collapse the application with the
  135. environment we're going to have constraints and future problems. So:
  136. we must resist doing that.
  137. We should also consider Java as a counterexample. In the JVM
  138. environment they abstracted away anything resembling a specific
  139. operating system. Python hasn't done that, and I don't want to do
  140. that; if the environment leaks into an application we should still
  141. resist creating an abstraction. Undocumented application abilities
  142. will get used and may be fragile, but they are better than documented
  143. APIs that get changed.
  144. Fabric
  145. ======
  146. This doesn't really qualify as a "design decision" as it's not a
  147. particularly deliberate decision, but because it is asked about a
  148. lot...
  149. `Fabric <>`_ is a library for managing remote
  150. servers. It has functions to call remote commands, transfer files,
  151. etc., typically over ssh. As such it seems like a no-brainer to use
  152. for Silver Lining, right?
  153. And maybe it is, but when writing Silver Lining I did not really want
  154. to learn a new tool, as it would only distract from what I was trying
  155. to do (especially at a point in time when I wasn't sure *what* I was
  156. trying to do). So I kept things simple, calling out to ssh manually
  157. when necessary. And this actually works reasonably well.
  158. Since then it has gotten a bit more complicated, and there's a bit
  159. more tunneling happening, and the consistency of the codebase has
  160. suffered. But not *that badly* all considered. Most of it would be
  161. easily resolvable by simple refactoring (and the same refactoring
  162. would be needed even if switching to Fabric). So I remain somewhat
  163. reluctant to add Fabric to the mix.
  164. And really the biggest concern I have with Fabric is that underlying
  165. the request is a desire to do ad hoc server manipulations using
  166. Fabric. A lot of people have deployment systems using Fabric that
  167. basically poke around on the server, installing things, configuring
  168. things, etc, in order to deploy their application. This is definitely
  169. **not** how Silver Lining should work. *Internally* Silver Lining
  170. connects to the server and runs commands (though many of the commands
  171. it runs are scripts hosted on the server). But individual deployments
  172. work at a more abstract level. They should never be making these
  173. kinds of modifications. If you are *developing Silver Lining itself*
  174. then sure, you can add new scripts and interactions. But, as I've
  175. noted, it's not particularly hard to do these remote interactions with
  176. raw ssh callouts.
  177. So while I am not opposed to Fabric, I am not sure it is necessary or
  178. worth the extra layer of abstraction. Also I am a bit worried about
  179. the added Paramiko dependency (Paramiko implements the ssh
  180. interaction, and doesn't use the normal openssh ssh client, and I
  181. worry is a bit big; but admittedly I haven't looked closely).
  182. Ubuntu
  183. ======
  184. This system is based on Ubuntu.
  185. Support for other Linux distributions is not desired. Few people have
  186. asked about this so far, so I'm hoping this will be a
  187. non-controversial choice. Many parts of the system expose Ubuntu
  188. (especially the ``packages`` configuration). The config files are
  189. written based on their locations in Ubuntu and Ubuntu policy. I don't
  190. want to introduce any indirection to this, or any abstraction layer.
  191. The consistency and reliability of the system is based on the
  192. consistency of its components, and this is an area where I have no
  193. desire to support flexibility at the sacrifice of consistency.
  194. Bare Base Systems
  195. =================
  196. Being based on "cloud" servers, Silver Lining prefers a bare server.
  197. Much like the Ubuntu decision, this is intentional and provides needed
  198. consistency. It's not a goal to support existing servers that have
  199. been setup in eclectic ways. You don't have to use a "cloud" server
  200. to get a bare server, of course; but you **do** need a bare server.
  201. Functionality is added from concrete needs
  202. ==========================================
  203. I'm trying to avoid implementing things I don't need at the moment.
  204. Of course, patches from people who have needs-at-the-moment are also
  205. cool. I don't want to predict how things should happen based on
  206. unrealistic ideas of how Silver Lining gets used. I want to react to
  207. how someone actually wants to use Silver Lining, with a problem
  208. clearly in hand.
  209. We see the same bugs, we fix the same bugs
  210. ==========================================
  211. Part of the consistency of the server environment is that we can have
  212. a consistent experience, all of us, one big community of reluctant
  213. sysadmin/programmers (we play dual class characters). With this
  214. consistency bugs aren't obscure. We all deal with the same system and
  215. the same bugs. I want to preserve that. Forks of the code are fine,
  216. but I really hope they are temporary, because I want us to be fixing
  217. each other's bugs. There are a lot of moving pieces. I want this to
  218. be a finely tuned machine; complex, but refined. That takes a lot of
  219. iterating, and I need a community of people iterating with me to get
  220. there.