PageRenderTime 55ms CodeModel.GetById 24ms app.highlight 25ms RepoModel.GetById 1ms app.codeStats 0ms

/README.markdown

http://github.com/nicolasff/webdis
Markdown | 356 lines | 291 code | 65 blank | 0 comment | 0 complexity | 5350545ce8e658217aba81d286564ca3 MD5 | raw file
  1[![CircleCI](https://circleci.com/gh/nicolasff/webdis.svg?style=shield)](https://circleci.com/gh/nicolasff/webdis)
  2
  3# About
  4
  5A very simple web server providing an HTTP interface to Redis. It uses [hiredis](https://github.com/antirez/hiredis), [jansson](https://github.com/akheron/jansson), [libevent](http://monkey.org/~provos/libevent/), and [http-parser](https://github.com/ry/http-parser/).
  6
  7Webdis depends on libevent-dev. You can install it on Ubuntu by typing `sudo apt-get install libevent-dev` or on OS X by typing `brew install libevent`.
  8<pre>
  9make clean all
 10
 11./webdis &
 12
 13curl http://127.0.0.1:7379/SET/hello/world
 14→ {"SET":[true,"OK"]}
 15
 16curl http://127.0.0.1:7379/GET/hello
 17→ {"GET":"world"}
 18
 19curl -d "GET/hello" http://127.0.0.1:7379/
 20→ {"GET":"world"}
 21
 22</pre>
 23
 24# Try in Docker
 25
 26<pre>
 27$ docker run --rm -d -p 7379:7379 nicolas/webdis
 280d2ce311a4834d403cc3e7cfd571b168ba40cede6a0e155a21507bb0bf7bee81
 29
 30$ curl http://127.0.0.1:7379/PING
 31{"PING":[true,"PONG"]}
 32
 33# To stop it:
 34$ docker stop $(docker ps | grep webdis | cut -c 1-12)
 350d2ce311a483
 36</pre>
 37
 38# Build and run Docker image locally
 39
 40Clone the repository and open a terminal in the webdis directory, then run:
 41<pre>
 42$ docker build -t webdis .
 43[...]
 44
 45$ docker run --rm -d -p 7379:7379 webdis
 46f0a2763fd456ac1f7ebff80eeafd6a5cd0fc7f06c69d0f7717fb2bdcec65926e
 47
 48$ curl http://127.0.0.1:7379/PING
 49{"PING":[true,"PONG"]}
 50
 51# To stop it:
 52$ docker stop $(docker ps | grep webdis | cut -c 1-12)
 53f0a2763fd456
 54</pre>
 55
 56
 57# Features
 58* `GET` and `POST` are supported, as well as `PUT` for file uploads.
 59* JSON output by default, optional JSONP parameter (`?jsonp=myFunction` or `?callback=myFunction`).
 60* Raw Redis 2.0 protocol output with `.raw` suffix
 61* MessagePack output with `.msg` suffix
 62* HTTP 1.1 pipelining (70,000 http requests per second on a desktop Linux machine.)
 63* Multi-threaded server, configurable number of worker threads.
 64* WebSocket support (Currently using the “hixie-76” specification).
 65* Connects to Redis using a TCP or UNIX socket.
 66* Restricted commands by IP range (CIDR subnet + mask) or HTTP Basic Auth, returning 403 errors.
 67* Possible Redis authentication in the config file.
 68* Pub/Sub using `Transfer-Encoding: chunked`, works with JSONP as well. Webdis can be used as a Comet server.
 69* Drop privileges on startup.
 70* Custom Content-Type using a pre-defined file extension, or with `?type=some/thing`.
 71* URL-encoded parameters for binary data or slashes and question marks. For instance, `%2f` is decoded as `/` but not used as a command separator.
 72* Logs, with a configurable verbosity.
 73* Cross-origin requests, usable with XMLHttpRequest2 (Cross-Origin Resource Sharing - CORS).
 74* File upload with PUT.
 75* With the JSON output, the return value of INFO is parsed and transformed into an object.
 76* Optional daemonize: set `"daemonize": true` and `"pidfile": "/var/run/webdis.pid"` in webdis.json.
 77* Default root object: Add `"default_root": "/GET/index.html"` in webdis.json to substitute the request to `/` with a Redis request.
 78* HTTP request limit with `http_max_request_size` (in bytes, set to 128MB by default).
 79* Database selection in the URL, using e.g. `/7/GET/key` to run the command on DB 7.
 80
 81# Ideas, TODO...
 82* Add better support for PUT, DELETE, HEAD, OPTIONS? How? For which commands?
 83	* This could be done using a “strict mode” with a table of commands and the verbs that can/must be used with each command. Strict mode would be optional, configurable. How would webdis know of new commands remains to be determined.
 84* MULTI/EXEC/DISCARD/WATCH are disabled at the moment; find a way to use them.
 85* Support POST of raw Redis protocol data, and execute the whole thing. This could be useful for MULTI/EXEC transactions.
 86* Enrich config file:
 87	* Provide timeout (maybe for some commands only?). What should the response be? 504 Gateway Timeout? 503 Service Unavailable?
 88* Multi-server support, using consistent hashing.
 89* SSL?
 90	* Not sure if this is such a good idea.
 91* SPDY?
 92	* SPDY is mostly useful for parallel fetches. Not sure if it would make sense for Webdis.
 93* Send your ideas using the github tracker, on twitter [@yowgi](http://twitter.com/yowgi) or by mail to n.favrefelix@gmail.com.
 94
 95# HTTP error codes
 96* Unknown HTTP verb: 405 Method Not Allowed.
 97* Redis is unreachable: 503 Service Unavailable.
 98* Matching ETag sent using `If-None-Match`: 304 Not Modified.
 99* Could also be used:
100	* Timeout on the redis side: 503 Service Unavailable.
101	* Missing key: 404 Not Found.
102	* Unauthorized command (disabled in config file): 403 Forbidden.
103
104# Command format
105The URI `/COMMAND/arg0/arg1/.../argN.ext` executes the command on Redis and returns the response to the client. GET, POST, and PUT are supported:
106
107* `GET /COMMAND/arg0/.../argN.ext`
108* `POST /` with `COMMAND/arg0/.../argN` in the HTTP body.
109* `PUT /COMMAND/arg0.../argN-1` with `argN` in the HTTP body (see section on [file uploads](#file-upload).)
110
111`.ext` is an optional extension; it is not read as part of the last argument but only represents the output format. Several formats are available (see below).
112
113Special characters: `/` and `.` have special meanings, `/` separates arguments and `.` changes the Content-Type. They can be replaced by `%2f` and `%2e`, respectively.
114
115# ACL
116Access control is configured in `webdis.json`. Each configuration tries to match a client profile according to two criterias:
117
118* [CIDR](http://en.wikipedia.org/wiki/CIDR) subnet + mask
119* [HTTP Basic Auth](http://en.wikipedia.org/wiki/Basic_access_authentication) in the format of "user:password".
120
121Each ACL contains two lists of commands, `enabled` and `disabled`. All commands being enabled by default, it is up to the administrator to disable or re-enable them on a per-profile basis.
122Examples:
123<pre>
124{
125	"disabled":	["DEBUG", "FLUSHDB", "FLUSHALL"],
126},
127
128{
129	"http_basic_auth": "user:password",
130	"disabled":	["DEBUG", "FLUSHDB", "FLUSHALL"],
131	"enabled":	["SET"]
132},
133
134{
135	"ip": 		"192.168.10.0/24",
136	"enabled":	["SET"]
137},
138
139{
140	"http_basic_auth": "user:password",
141	"ip": 		"192.168.10.0/24",
142	"enabled":	["SET", "DEL"]
143}
144</pre>
145ACLs are interpreted in order, later authorizations superseding earlier ones if a client matches several. The special value "*" matches all commands.
146
147# JSON output
148JSON is the default output format. Each command returns a JSON object with the command as a key and the result as a value.
149
150**Examples:**
151<pre>
152// string
153$ curl http://127.0.0.1:7379/GET/y
154{"GET":"41"}
155
156// number
157$ curl http://127.0.0.1:7379/INCR/y
158{"INCR":42}
159
160// list
161$ curl http://127.0.0.1:7379/LRANGE/x/0/1
162{"LRANGE":["abc","def"]}
163
164// status
165$ curl http://127.0.0.1:7379/TYPE/y
166{"TYPE":[true,"string"]}
167
168// error, which is basically a status
169$ curl http://127.0.0.1:7379/MAKE-ME-COFFEE
170{"MAKE-ME-COFFEE":[false,"ERR unknown command 'MAKE-ME-COFFEE'"]}
171
172// JSONP callback:
173$ curl  "http://127.0.0.1:7379/TYPE/y?jsonp=myCustomFunction"
174myCustomFunction({"TYPE":[true,"string"]})
175</pre>
176
177# RAW output
178This is the raw output of Redis; enable it with the `.raw` suffix.
179<pre>
180
181// string
182$ curl http://127.0.0.1:7379/GET/z.raw
183$5
184hello
185
186// number
187curl http://127.0.0.1:7379/INCR/a.raw
188:2
189
190// list
191$ curl http://127.0.0.1:7379/LRANGE/x/0/-1.raw
192*2
193$3
194abc
195$3
196def
197
198// status
199$ curl http://127.0.0.1:7379/TYPE/y.raw
200+zset
201
202// error, which is basically a status
203$ curl http://127.0.0.1:7379/MAKE-ME-COFFEE.raw
204-ERR unknown command 'MAKE-ME-COFFEE'
205</pre>
206
207# Custom content-type
208Several content-types are available:
209
210* `.json` for `application/json` (this is the default Content-Type).
211* `.msg` for `application/x-msgpack`. See [http://msgpack.org/](http://msgpack.org/) for the specs.
212* `.txt` for `text/plain`
213* `.html` for `text/html`
214* `xhtml` for `application/xhtml+xml`
215* `xml` for `text/xml`
216* `.png` for `image/png`
217* `jpg` or `jpeg` for `image/jpeg`
218* Any other with the `?type=anything/youwant` query string.
219* Add a custom separator for list responses with `?sep=,` query string.
220
221<pre>
222curl -v "http://127.0.0.1:7379/GET/hello.html"
223[...]
224&lt; HTTP/1.1 200 OK
225&lt; Content-Type: text/html
226&lt; Date: Mon, 03 Jan 2011 20:43:36 GMT
227&lt; Content-Length: 137
228&lt;
229&lt;!DOCTYPE html&gt;
230&lt;html&gt;
231[...]
232&lt;/html&gt;
233
234curl -v "http://127.0.0.1:7379/GET/hello.txt"
235[...]
236&lt; HTTP/1.1 200 OK
237&lt; Content-Type: text/plain
238&lt; Date: Mon, 03 Jan 2011 20:43:36 GMT
239&lt; Content-Length: 137
240[...]
241
242curl -v "http://127.0.0.1:7379/GET/big-file?type=application/pdf"
243[...]
244&lt; HTTP/1.1 200 OK
245&lt; Content-Type: application/pdf
246&lt; Date: Mon, 03 Jan 2011 20:45:12 GMT
247[...]
248</pre>
249
250# File upload
251Webdis supports file upload using HTTP PUT. The command URI is slightly different, as the last argument is taken from the HTTP body.
252For example: instead of `/SET/key/value`, the URI becomes `/SET/key` and the value is the entirety of the body. This works for other commands such as LPUSH, etc.
253
254**Uploading a binary file to webdis**:
255<pre>
256$ file redis-logo.png
257redis-logo.png: PNG image, 513 x 197, 8-bit/color RGBA, non-interlaced
258
259$ wc -c redis-logo.png
26016744 redis-logo.png
261
262$ curl -v --upload-file redis-logo.png http://127.0.0.1:7379/SET/logo
263[...]
264&gt; PUT /SET/logo HTTP/1.1
265&gt; User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
266&gt; Host: 127.0.0.1:7379
267&gt; Accept: */*
268&gt; Content-Length: 16744
269&gt; Expect: 100-continue
270&gt;
271&lt; HTTP/1.1 100 Continue
272&lt; HTTP/1.1 200 OK
273&lt; Content-Type: application/json
274&lt; ETag: "0db1124cf79ffeb80aff6d199d5822f8"
275&lt; Date: Sun, 09 Jan 2011 16:48:19 GMT
276&lt; Content-Length: 19
277&lt;
278{"SET":[true,"OK"]}
279
280$ curl -vs http://127.0.0.1:7379/GET/logo.png -o out.png
281&gt; GET /GET/logo.png HTTP/1.1
282&gt; User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
283&gt; Host: 127.0.0.1:7379
284&gt; Accept: */*
285&gt;
286&lt; HTTP/1.1 200 OK
287&lt; Content-Type: image/png
288&lt; ETag: "1991df597267d70bf9066a7d11969da0"
289&lt; Date: Sun, 09 Jan 2011 16:50:51 GMT
290&lt; Content-Length: 16744
291
292$ md5sum redis-logo.png out.png
2931991df597267d70bf9066a7d11969da0  redis-logo.png
2941991df597267d70bf9066a7d11969da0  out.png
295</pre>
296
297The file was uploaded and re-downloaded properly: it has the same hash and the content-type was set properly thanks to the `.png` extension.
298
299# WebSockets
300Webdis supports WebSocket clients implementing [dixie-76](http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-76).  
301Web Sockets are supported with the following formats, selected by the connection URL:
302
303* JSON (on `/` or `/.json`)
304* Raw Redis wire protocol (on `/.raw`)
305
306**Example**:
307<pre>
308function testJSON() {
309	var jsonSocket = new WebSocket("ws://127.0.0.1:7379/.json");
310	jsonSocket.onopen = function() {
311
312		console.log("JSON socket connected!");
313		jsonSocket.send(JSON.stringify(["SET", "hello", "world"]));
314		jsonSocket.send(JSON.stringify(["GET", "hello"]));
315	};
316	jsonSocket.onmessage = function(messageEvent) {
317		console.log("JSON received:", messageEvent.data);
318	};
319}
320testJSON();
321</pre>
322
323This produces the following output:
324<pre>
325JSON socket connected!
326JSON received: {"SET":[true,"OK"]}
327JSON received: {"GET":"world"}
328</pre>
329
330# Pub/Sub with chunked transfer encoding
331Webdis exposes Redis PUB/SUB channels to HTTP clients, forwarding messages in the channel as they are published by Redis. This is done using chunked transfer encoding.
332
333**Example using XMLHttpRequest**:
334<pre>
335var previous_response_length = 0
336xhr = new XMLHttpRequest()
337xhr.open("GET", "http://127.0.0.1:7379/SUBSCRIBE/hello", true);
338xhr.onreadystatechange = checkData;
339xhr.send(null);
340
341function checkData() {
342	if(xhr.readyState == 3)  {
343    	response = xhr.responseText;
344    	chunk = response.slice(previous_response_length);
345    	previous_response_length = response.length;
346    	console.log(chunk);
347    }
348};
349</pre>
350
351Publish messages to redis to see output similar to the following:
352<pre>
353{"SUBSCRIBE":["subscribe","hello",1]}
354{"SUBSCRIBE":["message","hello","some message"]}
355{"SUBSCRIBE":["message","hello","some other message"]} 
356</pre>