PageRenderTime 71ms CodeModel.GetById 32ms RepoModel.GetById 0ms app.codeStats 0ms

Plain Text | 619 lines | 480 code | 139 blank | 0 comment | 0 complexity | ba758838b8ba6fbd52820a19e5bd419d MD5 | raw file
Possible License(s): GPL-2.0
  1. Things Silver Lining Should Do
  2. ==============================
  3. This document lists out things that I think Silver Lining should do,
  4. but does not yet do. It's also kind of a notepad for design ideas.
  5. It should not surprise you if I forget to remove something from this
  6. document after implementing it.
  7. Far-future expiration of static content
  8. ---------------------------------------
  9. It's nice to present static content with far-future expiration dates.
  10. Do do this while still allowing this static content to be updated, it
  11. is conventional to use a timestamp of some sort in the URL itself. An
  12. example might be::
  13. /static/20100127_1/media/style.css
  14. Then if the application is updated this URL will change. You could
  15. use the modification date of the file itself, but a sweeping
  16. invalidation of all content is less error-prone and requires very
  17. little framework support.
  18. I think Silver Lining should add a new location (like in the example)
  19. where content in the ``static/`` directory is served with this
  20. caching, and an environmental variable ``$STATIC_URL`` which you use
  21. to prefix all your media files. Probably the current pattern will be
  22. preserved, as it can be handy when retrofitting an application. The
  23. datestamp_version used in the application directories is apppriately
  24. unique.
  25. Then in the future there should also be support for uploading these
  26. static files to a CDN, at which point the variable will point to the
  27. CDN.
  28. Groups of servers, replication
  29. ------------------------------
  30. There are notes on this in ``multiserver design notes
  31. <multiserver.html>`_.
  32. Backups
  33. -------
  34. Backup and restore is implemented in a basic fashion. It would be
  35. very nice if backup files were portable. Then, for instance, you
  36. could download the backups to replicate live data on your site for
  37. your development. (PostGIS is a case where portable backups can be a
  38. bit tricky at times.)
  39. It would be even more neat if backup files were somewhat version
  40. control friendly. E.g., well-ordered SQL dumps. This might make the
  41. backup process do double duty as a test fixture process, and a process
  42. to setup data in new instances.
  43. Site-to-site backups should be supported (which would essentially
  44. clone the data from one server to another).
  45. Logs
  46. ----
  47. Log files should also be handled gracefully, tracked, and possibly
  48. some frontend (`frontend in progress
  49. <>`_). Though I feel like some
  50. kind of management programming interface is better, and then a
  51. "deployed" app would use that interface. Actually, "interface" is
  52. probably saying too much; it should be sufficient to simply have a
  53. documented and conscious server layout that an application can be
  54. written to inspect.
  55. There should be a way to get a list of all log files for a host/app.
  56. Maybe there should be a standard way to effect the Python standard
  57. logging module output, and then put it in a normal location (I think
  58. ``/var/log/silverlining/apps/NAME/*``). There are some more notes
  59. down in Debugging_.
  60. Monitoring
  61. ----------
  62. Basically a way is needed to answer the question "do I need to rent
  63. bigger (or smaller) nodes?" Basic metrics like load, RAM, swap and
  64. requets per second should be monitored and graphed. Munin seems to be a
  65. popular choice for that. Probably a Nagios setup makes sense here.
  66. `This post
  67. <>`_
  68. also has some additional ideas.
  69. Also similar to ``update_fetch`` etc, there should be an optional
  70. defined "ping" URL for an application. This would be some location
  71. that could be safely fetched, wouldn't be time-intensive, and would
  72. report general errors. An application could do self-testing here.
  73. Monitoring infrastructure would then be setup to ping this URL.
  74. Unlike ``update_fetch`` and other management URLs, this would be
  75. fetched in-process in the live server (management URLs get fetched
  76. with an analogous but non-mod_wsgi request). Probably there should be
  77. some indication of how often the pinging should happen (i.e., if the
  78. ping is cheap do it often, if it's expensive be more reserved).
  79. CloudKick
  80. ~~~~~~~~~
  81. CloudKick has started doing a bunch of monitoring things, and has an
  82. API. A quick way of doing monitoring might simply be via CloudKick
  83. (or maybe just an option).
  84. Security
  85. --------
  86. We should have a good quality rotating random value. (Like
  87. ``silversupport.secret.get_secret``). Google has a library for this.
  88. We should have a way to do security logging. E.g., log attempted
  89. attacks. There should be a whitelist, so that internal people can do
  90. bad things (e.g., when testing).
  91. Translation
  92. -----------
  93. I'd like to integrate Pontoon into applications, so that Silver Lining
  94. can be a general way to setup applications for localization. This
  95. might primarily be a second deployed application that modifies files
  96. in the other application (somewhere in ``$CONFIG_FiLES``).
  97. Create Virtual Machine Images
  98. -----------------------------
  99. Local development can for some people be quite hard. This is
  100. especially true of Windows users (for which silverlining doesn't have
  101. any support, and no support is planned).
  102. It would be nice to be able to run something like:
  103. silver setup-develop-image . some-image-filename
  104. and have it get a bare Ubuntu image (for VMWare, VirtualBox, or
  105. whatever) and run some adaptation of setup-node, and setup a file
  106. server to make editing files easy, and then upload the application
  107. into the machine (version control files and everything).
  108. To make it more awesome, a built-in web app for managing the image;
  109. checking in files, restarting, tooling, etc. Maybe the image could be
  110. run headless then? This is more ambitious, but it could in some way
  111. be an introduction to good open source development practices.
  112. This is an aside, but probably the generalized "DEST" idea of ``silver
  113. backup`` would be useful here and elsewhere: anywhere you write
  114. something (like an image file) allow remote locations. Of course
  115. silver backup is done on the remote server and uploaded from there,
  116. while this is done locally, so the advantage is smaller.
  117. The core of this is probably two variations:
  118. * ``setup-node``, except setting up a "development" node. This would
  119. be a node with git/hg/svn installed, somewhat different mod_wsgi
  120. configuration, and other such details.
  121. * ``update``, except again a "development" version, one that copies
  122. over VCS metadata (``.hg/`` etc). Also it wouldn't create
  123. versioned/dated directories, but instead support only one version of
  124. an application (the version you are developing).
  125. Test-Before-Activate
  126. --------------------
  127. There should be a kind of "check functions are go" URL or command,
  128. similar to how ``update_fetch`` works. This would be fetched on
  129. update, but before the update was really made live (the internal
  130. request mechanism can be run on any instance, even one that is not
  131. live yet). If this returned a non-2xx result, then the update would
  132. be aborted (or if it's a test script, exit with an error code).
  133. Backup/Restore/Revert on update
  134. -------------------------------
  135. Each update is a point when a persistence checkpoint should happen.
  136. This is most easily done with a local backup.
  137. There should also be a "run all database updates" URL (also similar to
  138. ``update_fetch``). This might fail, in which case the backup should
  139. be restored and the update aborted.
  140. I think this should be run before test-before-activate, but if
  141. test-before-activate fails then the databases should also be reverted.
  142. There are fancy ways of doing really fast checkpoints, like backups
  143. without actually backing things up. These should be accessible even
  144. in a VPS, but they require some fancy sysadmin knowledge that I don't
  145. have.
  146. Debugging
  147. ---------
  148. I'm fairly conflicted about what Silver Lining should do with respect
  149. to debugging. I am inclined to say that applications should be
  150. instrumented with debugging tools natively, e.g., using Django's
  151. built-in tools or weberror, etc.
  152. On the other hand, without this kind of wrapping you get something
  153. that really sucks -- normal Apache error logs. Maybe at least
  154. ``environ['wsgi.errors']`` should be fixed (along with stderr) to
  155. point to something better/reasonable. (And maybe warnings
  156. specifically should be handled, so only unique warnings are logged,
  157. and they are reset between updates.)
  158. Setting up the ``logging`` module is somewhat similar, it could be
  159. done manually by applications. (Sometimes you want to turn up
  160. logging, *in production* temporarily to see what is going on in a live
  161. server: this is something not in the current workflow.)
  162. Also there should be centralized configuration of things like, e.g.,
  163. where error messages are sent. Then tools could be configured in a
  164. more transportable way (e.g., ``DEBUG_EMAIL =
  165. os.environ['SILVER_DEBUG_EMAIL']``). This might fit into how
  166. configuration is generally handled.
  167. Error reporting for non-requests (like a cron script) is harder. Most
  168. of these are done synchronously, so you can just show the user what is
  169. happening, but at least for cron and perhaps tasks this is not the
  170. case. Unifying environment setup might resolve this particular
  171. issue...
  172. Cron and Tasks
  173. --------------
  174. App Engine style cronjobs and tasks are definitely planned. (I also
  175. just found some code called PyPeriodic that I think I wrote a long
  176. time ago and forgot.)
  177. Cron jobs are based on requests run periodically. Though I guess they
  178. could also be commands. One advantage of keeping to requests is to
  179. have some kind of monitoring for failed or rejected requests (though
  180. the internal request mechanism is not as robust currently as the
  181. external request system, which is managed by mod_wsgi).
  182. Tasks are things to be done in the future, asynchronously with any
  183. request. These are queued, get retried in cases of failure (non-2xx
  184. response); maybe given multiple servers they could be run on different
  185. servers.
  186. Cron is a higher priority.
  187. For persistence of the jobs, Redis seems quite possible (small enough
  188. that I don't feel bad having it always installed, seems relatively
  189. robust and easy to use).
  190. API Keys
  191. --------
  192. This is the obvious kind of non-portable configuration. There is some
  193. support for this already, but it is poorly exposed.
  194. This is something that might be good to manage organization-wide.
  195. Which just makes the workflow slightly different.
  196. In some ways this is just global configuration (as opposed to
  197. application-specific configuration). This same configuration could
  198. potentially be used for things like admin email addresses, logging
  199. information, etc. Applications would have to pull it in, but it would
  200. be updated on a machine basis, not a per-application basis (though
  201. possibly you could have per-application overrides?)
  202. Inter-application communication
  203. -------------------------------
  204. This will only happen with a strong use case to motivate it; I can
  205. certainly *imagine* such use cases, but I need something more concrete
  206. than my imagination.
  207. In that case, an application could receive configuration information
  208. for other applications. Communication would have to be through some
  209. service (e.g., a database) or HTTP. Accessing applications on other
  210. servers would probably not be supported (though potentially it could
  211. be), but given per-path applications it is reasonable to consider
  212. cases where this would be useful. Auth is an obvious case.
  213. The Silver Lining secret is currently per-server (and will probably be
  214. per-group in the future), so some things like reading cookies could be
  215. done across applications already (given the appropriate code).
  216. Testing and Test Fixture Setup
  217. ------------------------------
  218. It would be nice to have a nice setup for getting test data setup. I
  219. think some degree of testing support would be nice, though simply
  220. installing and using a standard test runner seems mostly fine. A
  221. recipe for integration testing would be nice. A recipe for full-stack
  222. testing (e.g., with Windmill or Selenium) would be nice. These are
  223. probably more recipes than core Silver Lining things.
  224. It's possible that Silver Lining could setup a second set of "testing"
  225. services. So if you ran development off a database named ``myapp``,
  226. maybe in "testing" mode it would use ``test_myapp`` (and the database
  227. would be cleared on every run).
  228. Full-stack development environment
  229. ----------------------------------
  230. I can imagine a through-the-browser development environment, based on
  231. Bespin. This could be cool. It could be really VM-friendly as well.
  232. Circumstances may determine if I really try this out.
  233. To do this, Bespin would be installed and running, and configured to
  234. point at the application location. Version control can be managed
  235. through Bespin.
  236. Additionally Hudson could be installed, enabling a kind of built-in
  237. test suite. I'm not sure how else Hudson could be enabled (except
  238. perhaps to list a test script in ``app.ini`` and go from there), but
  239. specifically for this case it seems like it would be nice. By
  240. including a test suite configuration it might generally be possible to
  241. do a very simple and easy Hudson setup.
  242. Another interface would control Silver Lining itself, allowing you to
  243. configure production providers and deploy to them. It could be very
  244. simple.
  245. Hudson is available as a `third-party Debian package
  246. <>`_. Bespin would probably be deployed
  247. as a Silver Lining application itself. I imagine you might define a
  248. base domain (e.g., ``mydevel``) and then you'd have:
  249. * ``www.mydevel``: the application itself, running
  250. * ``dev.mydevel``: Bespin
  251. * ``test.mydevel``: Hudson
  252. * ``deploy.mydevel``: Silver Lining interface
  253. One could imagine a load tester somewhere in here too. And maybe
  254. a rig for Selenium or other through-the-web tester?
  255. With all these, heck, maybe I'd develop through a virtual machine even
  256. though I'm on Linux already.
  257. Bespin would not preclude file sharing so that you could do editing
  258. (even version control) as though it was local. It would simply be
  259. another alternative.
  260. Application Packs
  261. -----------------
  262. Right now this system emphasizes development. It would be nice to
  263. also support ready-to-go applications. This might be simply a zip
  264. file created from an application, that can be easily uploaded. It
  265. might not be something at all fancy. But thinking it out would be
  266. nice.
  267. Trac would be an ideal candidate for sometihng like this. It also has
  268. enough customization that I can imagine some interesting use cases;
  269. how do you select the plugins you want and the templates you want to
  270. override? If you aren't "developing" the application then this isn't
  271. as clear.
  272. Maybe on application upload you could have two directories; one the
  273. "code" and another the "customizations". The code would have a script
  274. to check and merge in the customizations before upload. One can
  275. imagine other useful things the code could do with the customizations
  276. (for instance, support plugin installation through knowledge of the
  277. app's plugin index and techniques).
  278. Maybe it would look like::
  279. silver update --customize=mycustomizations/
  280. That's the easy way. Plugin installation?
  281. ::
  282. silver manage-app --customize=mycustomizations/ \
  283. --list-commands
  284. silver manage-app --customize=mycustomizations/ \
  285. install-plugin Akismet
  286. It feels a bit crude. Maybe another command besides ``silver`` for
  287. pre-packaged apps? Or another set of subcommands.
  288. Middleware Packs
  289. ----------------
  290. Sometimes you want to apply middleware in a deployment-specific
  291. fashion (not application-specific). Some examples:
  292. * Site disabling
  293. * Error catching
  294. * Password protecting a site before it is public
  295. * Ad hoc authorization
  296. I'd like the ability to apply a piece of middleware to an
  297. application. This wouldn't *just* be WSGI middleware, it would
  298. probably include some extra information (description, etc). Also a
  299. PHP equivalent version ("fat" middleware) would be nice to include.
  300. Using PHP's output buffer operations you can simulate middleware.
  301. I'm not sure what the command would look like. Maybe just::
  302. silver apply-middleware path/to/middleware LOCATION
  303. There's also be ``show-middleware`` and ``remove-middleware``
  304. operations, I suppose. Like applications, there's a missing notion of
  305. versions here. Also middleware would have to be compatible with the
  306. libraries of the hosted application, as they would run in the same
  307. process. Probably middleware libraries would be added after the path,
  308. and the middleware should be written to be as library-version-agnostic
  309. as possible. There should be a fairly finite number of middleware
  310. packs.
  311. Configuration should apply to middleware, though maybe a second set of
  312. configuration to avoid overlap.
  313. Configuration Management
  314. ------------------------
  315. Now that there's a ``--config`` option you can have
  316. deployment-specific configuration of your application, but it's only
  317. exposed through ``silver update``. It should be exposed as a separate
  318. operation, to view, edit, backup, etc. the configuration, you
  319. shouldn't have to deploy just to change the configuration. A complete
  320. set of commands might be:
  321. Show the configuration (all past revisions)::
  322. silver config-query LOCATION
  323. # To show info about files:
  324. silver config-query LOCATION --files
  325. # To download config:
  326. silver config-query LOCATION --dump=path/
  327. Then operations to actually modify the configuration::
  328. # Upload new configuration:
  329. silver config LOCATION path/to/config
  330. # Revert configuration to some previous version ("PREV" literal, or
  331. # some revision number as shown by config-query):
  332. silver config LOCATION --revert VERSION
  333. # Remove configuration:
  334. silver config LOCATION --delete
  335. # Copy configuration from another site:
  336. silver config LOCATION --copy=SOURCE_LOCATION
  337. Doing version control on the configuration would be nice. Also
  338. supporting an app that did generic through-the-web application
  339. configuration (needs a DevAuth-style thing again). Such configuration
  340. would just do through-the-web-editing of the source files, and
  341. probably call the validation function if available. It might also
  342. allow for operations like reverting, seeing logs, etc.
  343. Application Versions
  344. --------------------
  345. It's hard to figure out exactly what "version" of an application is
  346. installed somewhere. There's no particular line, and no one thing is
  347. the only important version number.
  348. It'd be nice to look around the files for VCS data and get a version
  349. snapshot for all pieces.
  350. Another option is to have a deploy-application-registry, which would
  351. hold all the files that were deployed. This would be a parallel
  352. version control system unrelated to any developer version control, and
  353. would be used only for auditing.
  354. Local/CDN Javascript
  355. --------------------
  356. I've done some applications that are mostly static pages on the
  357. frontend with Javascript. I like to use Google's hosted versions of
  358. these libraries. But then offline I lose access to the app :(
  359. It would be nice to have a local cache of these files in Silver
  360. Lining. Except I can't figure out how you'd write a static HTML file
  361. that could select the local or remote file. Maybe one option is to
  362. use Google's loader, but kind of mock it out for local use. But even
  363. that is tricky. It would be possible to do a small amount of
  364. rewriting of HTML pages, replacing those links (maybe literally) with
  365. local versions. During development static files are served fairly
  366. naively, so it wouldn't be hard to do this substitution. Also
  367. minimized files could be replaced with unminimized files. Because the
  368. actual HTML would be production-ready there's no compromise, only in
  369. development are they reverted.
  370. Silver Lining could ship these libraries, or maybe there could be an
  371. ``app.ini`` setting like::
  372. [development]
  373. local_html_replacements =
  374. /_devel/jquery.js
  375. Then you'd put that file in ``static/_devel/jquery.js``. I can't even
  376. get that on a single line... maybe these replacements should be put
  377. in a separate file, and maybe that should be a set of regular
  378. expressions. On the other hand, if this is really restricted to URLs
  379. and local files, you could "sync" these files automatically. But in
  380. this case, as an example, you actually want a different file during
  381. development (the non-minimized ``jquery.js``).
  382. Also while this is otherwise kind of unresolvable for static files, it
  383. would be nice if this also worked for dynamic files. There would then
  384. have to be some API. Maybe each of these resources would be "named",
  385. in a file like::
  386. [jquery]
  387. production =
  388. devel = /_devel/jquery.js
  389. And then an API like::
  390. <script src="${silversupport.href['jquery']}"></script>
  391. Using ``__getitem__`` would make this more Django-template friendly I
  392. believe (though a more traditional ``silversupport.href('jquery')`` could
  393. also be supported).
  394. This could also be the basis for the CDN setup, and it's possible that
  395. minimization could happen using this. Maybe like::
  396. [js]
  397. jquery.json file = static/js/jquery.json.js
  398. app.js file = static/js/app.js
  399. all_js combine = jquery.json app.js
  400. Then in production ``silversupport.href('all_js')`` would return a URL
  401. pointing to the minimized and combined version of all those files, and
  402. ``silversupport.href('jquery.json')`` and ``silversupport.href('app.js')``
  403. would both return None. In development the inverse would be true.
  404. Using ``silver serve`` there should be an option to do
  405. minimization, so the minimized versions could be tested. The same
  406. ideas apply to CSS. Something like this might make sense for images,
  407. but no minimization would occur. You might not want to enumerate
  408. images, so you might do:
  409. [images]
  410. images directory = static/images/
  411. And then use ``${silversupport.href('images')}/folder.png``
  412. Start Up Local Service
  413. ----------------------
  414. There should be a way (probably via ``~/.silverlining.conf``) to
  415. indicate how to start local services. Then, for example, you would
  416. not need to run PostgreSQL or Couch all the time, but could make it
  417. available as needed. Maybe it would look like::
  418. [service.couchdb]
  419. start = sudo /etc/init.d/couchdb start
  420. stop = sudo /etc/init.d/couchdb stop
  421. Of course starting and stopping all the time is annoying, but it
  422. wouldn't have to happen if the reloader did the restart. Also perhaps
  423. Silver Lining could lazily stop the service by daemonizing a
  424. stop-script on exit. (Maybe only if there's an additional
  425. ``lazy_stop`` setting.)
  426. Keep setup-node Version
  427. -----------------------
  428. It would be good to auto-detect when setup-node has to be rerun. I
  429. imagine either simply using a hash of all files that are part of
  430. setup-node, or using the latest timestamp. The timestamp would change
  431. from computer to computer though, giving false positives.
  432. I imagine if you run ``silver update`` that it would do something
  433. like::
  434. $ silver update --node=foo
  435. Warning: the node foo is not up to date; setup-node should be run
  436. Continue anyway (y)es/(n)o/(u)pdate?
  437. Yes (which would work with ``--yes``) would just keep going, No
  438. aborts, Update would run setup-node immediately and then continue.
  439. Serve general files
  440. -------------------
  441. You can use `mod_negotiation
  442. <>`_ to serve
  443. static files without extensions. These are files that include headers
  444. (that indicate things like Content-Type) as well as the body.
  445. Probably they'd have a different extension.
  446. Security Headers
  447. ----------------
  448. `Content Security Policy
  449. <>`_ seems like you
  450. *might* want to enable it globally (not as part of the application).
  451. Specifically for static file headers.
  452. Maybe a middleware pack would handle this case.
  453. Short-term list
  454. ---------------
  455. Here's what I want to do right away:
  456. * Write a VirtualBox libcloud driver (for testing)
  457. - Will use VBoxManage
  458. * Backup/restore (backup done, restore mostly done)
  459. - Consider how this interacts with testing
  460. - Specifically the Django ``test_*`` databases
  461. * Setup up deployment for Weave Sync 2.0 API implementation
  462. - Set up multi-server deployments
  463. * Set up DevAuth for admin access
  464. * Get https working in some fashion. (Ad hoc https seems to only get
  465. harder over time though)
  466. * Look into the Varnish logs (Apache logs aren't that complete,
  467. really)
  468. * Make the instantiated app available in the interactive shell
  469. * Move towards a general schema for commands (mostly these are
  470. arguments with known names):
  471. * destination location
  472. * app dir
  473. * provider
  474. * node
  475. Then some general conventions could be done for default_app, maybe
  476. default_provider and some other stuff, as well as info in
  477. ``~/.silverlining.conf``
  478. * There's some nice features in the `Mozilla AMO Hudson instace
  479. <>`_ that I would
  480. like here (checking coding conventions and other details).