PageRenderTime 18ms CodeModel.GetById 3ms app.highlight 3ms RepoModel.GetById 1ms app.codeStats 1ms

/chapter2.txt

https://github.com/tsellon/zguide
Plain Text | 1518 lines | 1101 code | 417 blank | 0 comment | 0 complexity | 6b59a7cb96e60446d7a907be5898c4a3 MD5 | raw file

Large files files are truncated, but you can click here to view the full file

  1.output chapter2.wd
  2++ Chapter Two - Intermediate Stuff
  3
  4In Chapter One we took 0MQ for a drive, with some basic examples of the main 0MQ patterns: request-reply, publish-subscribe, and pipeline. In this chapter we're going to get our hands dirty and start to learn how to use these tools in real programs.
  5
  6We'll cover:
  7
  8* How to create and work with 0MQ sockets.
  9* How to send and receive messages on sockets.
 10* How to build your apps around 0MQ's asynchronous I/O model.
 11* How to handle multiple sockets in one thread.
 12* How to handle fatal and non-fatal errors properly.
 13* How to handle interrupt signals like Ctrl-C.
 14* How to shutdown a 0MQ application cleanly.
 15* How to check a 0MQ application for memory leaks.
 16* How to send and receive multipart messages.
 17* How to forward messages across networks.
 18* How to build a simple message queuing broker.
 19* How to write multithreaded applications with 0MQ.
 20* How to use 0MQ to signal between threads.
 21* How to use 0MQ to coordinate a network of nodes.
 22* How to create durable sockets using socket identities.
 23* How to create and use message envelopes for publish-subscribe.
 24* How to do make durable subscribers that can recover from crashes.
 25* Using the high-water mark (HWM) to protect against memory overflows.
 26
 27+++ The Zen of Zero
 28
 29The Ø in 0MQ is all about tradeoffs. On the one hand this strange name lowers 0MQ's visibility on Google and Twitter. On the other hand it annoys the heck out of some Danish folk who write us things like "ØMG røtfl", and "//Ø is not a funny looking zero!//" and "//Rødgrød med Fløde!//", which is apparently an insult that means "may your neighbours be the direct descendents of Grendel!"  Seems like a fair trade.
 30
 31Originally the zero in 0MQ was meant as "zero broker" and (as close to) "zero latency" (as possible). In the meantime it has come to cover different goals: zero administration, zero cost, zero waste. More generally, "zero" refers to the culture of minimalism that permeates the project. We add power by removing complexity rather than exposing new functionality.
 32
 33+++ The Socket API
 34
 35To be perfectly honest, 0MQ does a kind of switch-and-bait on you. Which we don't apologize for, it's for your own good and hurts us more than it hurts you. It presents a familiar BSD socket API but that hides a bunch of message-processing machines that will slowly fix your world-view about how to design and write distributed software.
 36
 37Sockets are the de-facto standard API for network programming, as well as being useful for stopping your eyes from falling onto your cheeks. One thing that makes 0MQ especially tasty to developers is that it uses a standard socket API. Kudos to Martin Sustrik for pulling this idea off. It turns "Message Oriented Middleware", a phrase guaranteed to send the whole room off to Catatonia, into "Extra Spicy Sockets!" which leaves us with a strange craving for pizza, and a desire to know more.
 38
 39Like a nice pepperoni pizza, 0MQ sockets are easy to digest. Sockets have a life in four parts, just like BSD sockets:
 40
 41* Creating and destroying sockets, which go together to form a karmic circle of socket life (see zmq_socket[3], zmq_close[3]).
 42
 43* Configuring sockets by setting options on them and checking them if necessary (see zmq_setsockopt[3], zmq_getsockopt[3]).
 44
 45* Plugging sockets onto the network topology by creating 0MQ connections to and from them (see zmq_bind[3], zmq_connect[3]).
 46
 47* Using the sockets to carry data by writing and receiving messages on them (see zmq_send[3], zmq_recv[3]).
 48
 49Which looks like this, in C:
 50
 51[[code language="C"]]
 52void *mousetrap;
 53
 54//  Create socket for catching mice
 55mousetrap = zmq_socket (context, ZMQ_PULL);
 56
 57//  Configure the socket
 58int64_t jawsize = 10000;
 59zmq_setsockopt (mousetrap, ZMQ_HWM, &jawsize, sizeof jawsize);
 60
 61//  Plug socket into mouse hole
 62zmq_connect (mousetrap, "tcp://192.168.55.221:5001");
 63
 64//  Wait for juicy mouse to arrive
 65zmq_msg_t mouse;
 66zmq_msg_init (&mouse);
 67zmq_recv (mousetrap, &mouse, 0);
 68//  Destroy the mouse
 69zmq_msg_close (&mouse);
 70
 71//  Destroy the socket
 72zmq_close (mousetrap);
 73[[/code]]
 74
 75Note that sockets are always void pointers, and messages (which we'll come to very soon) are structures. So in C you pass sockets as-such, but you pass addresses of messages in all functions that work with messages, like zmq_send[3] and zmq_recv[3]. As a mnemonic, realize that "in 0MQ all ur sockets are belong to us", but messages are things you actually own in your code.
 76
 77Creating, destroying, and configuring sockets works as you'd expect for any object. But remember that 0MQ is an asynchronous, elastic fabric. This has some impact on how we plug sockets into the network topology, and how we use the sockets after that.
 78
 79+++ Plugging Sockets Into the Topology
 80
 81To create a connection between two nodes you use zmq_bind[3] in one node, and zmq_connect[3] in the other.  As a general rule of thumb, the node which does zmq_bind[3] is a "server", sitting on a well-known network address, and the node which does zmq_connect[3] is a "client", with unknown or arbitrary network addresses. Thus we say that we "bind a socket to an endpoint" and "connect a socket to an endpoint", the endpoint being that well-known network address.
 82
 830MQ connections are somewhat different from old-fashioned TCP connections. The main notable differences are:
 84
 85* They go across an arbitrary transport ({{inproc}}, {{ipc}}, {{tcp}}, {{pgm}} or {{epgm}}). See zmq_inproc[7], zmq_ipc[7], zmq_tcp[7], zmq_pgm[7], and zmq_epgm[7].
 86
 87* They exist when a client does zmq_connect[3] to an endpoint, whether or not a server has already done zmq_bind[3] to that endpoint.
 88
 89* They are asynchronous, and have queues that magically exist where and when needed.
 90
 91* They may express a certain "messaging pattern", according to the type of socket used at each end.
 92
 93* One socket may have many outgoing and many incoming connections.
 94
 95* There is no zmq_accept() method. When a socket is bound to an endpoint it automatically starts accepting connections.
 96
 97* Your application code cannot work with these connections directly; they are encapsulated under the socket.
 98
 99Many architectures follow some kind of client-server model, where the server is the component that is most stable, and the clients are the components that are most dynamic, i.e. they come and go the most. There are sometimes issues of addressing: servers will be visible to clients, but not necessarily vice-versa. So mostly it's obvious which node should be doing zmq_bind[3] (the server) and which should be doing zmq_connect[3] (the client). It also depends on the kind of sockets you're using, with some exceptions for unusual network architectures. We'll look at socket types later.
100
101Now, imagine we start the client //before// we start the server. In traditional networking we get a big red Fail flag. But 0MQ lets us start and stop pieces arbitrarily. As soon as the client node does zmq_connect[3] the connection exists and that node can start to write messages to the socket. At some stage (hopefully before messages queue up so much that they start to get discarded, or the client blocks), the server comes alive, does a zmq_bind[3] and 0MQ starts to deliver messages.
102
103A server node can bind to many endpoints and it can do this using a single socket. This means it will accept connections across different transports:
104
105[[code language="C"]]
106zmq_bind (socket, "tcp://*:5555");
107zmq_bind (socket, "tcp://*:9999");
108zmq_bind (socket, "ipc://myserver.ipc");
109[[/code]]
110
111You cannot bind to the same endpoint twice, that will cause an exception.
112
113Each time a client node does a zmq_connect[3] to any of these endpoints, the server node's socket gets another connection. There is no inherent limit to how many connections a socket can have. A client node can also connect to many endpoints using a single socket.
114
115In most cases, which node acts as client, and which as server, is about network topology rather than message flow. However, there //are// cases (resending when connections are broken) where the same socket type will behave differently if it's a server or if it's a client.
116
117What this means is that you should always think in terms of "servers" as stable parts of your topology, with more-or-less fixed endpoint addresses, and "clients" as dynamic parts that come and go. Then, design your application around this model. The chances that it will "just work" are much better like that.
118
119Sockets have types. The socket type defines the semantics of the socket, its policies for routing messages inwards and outwards, queueing, etc. You can connect certain types of socket together, e.g. a publisher socket and a subscriber socket. Sockets work together in "messaging patterns". We'll look at this in more detail later.
120
121It's the ability to connect sockets in these different ways that gives 0MQ its basic power as a message queuing system. There are layers on top of this, such as devices and topic routing, which we'll get to later. But essentially, with 0MQ you define your network architecture by plugging pieces together like a child's construction toy.
122
123+++ Using Sockets to Carry Data
124
125To send and receive messages you use the zmq_send[3] and zmq_recv[3] methods. The names are conventional but 0MQ's I/O model is different enough from TCP's model that you will need time to get your head around it.
126
127[[code type="textdiagram"]]
128             +------------+
129             |            |
130             |    Node    |
131             |            |
132             +------------+
133             |   Socket   |
134             \------------/
135                   ^
136                   |
137                1 to 1
138                   |
139                   v
140             /------------\
141             |   Socket   |
142             +------------+
143             |            |
144             |    Node    |
145             |            |
146             +------------+
147
148
149   Figure # - TCP sockets are 1 to 1
150[[/code]]
151
152Let's look at the main differences between TCP sockets and 0MQ sockets when it comes to carrying data:
153
154* 0MQ sockets carry messages, rather than bytes (as in TCP) or frames (as in UDP). A message is a length-specified blob of binary data. We'll come to messages shortly, their design is optimized for performance and thus somewhat tricky to understand.
155
156* 0MQ sockets do their I/O in a background thread. This means that messages arrive in a local input queue, and are sent from a local output queue, no matter what your application is busy doing. These are configurable memory queues, by the way.
157
158* 0MQ sockets can, depending on the socket type, be connected to (or from, it's the same) many other sockets. Where TCP emulates a one-to-one phone call, 0MQ implements one-to-many (like a radio broadcast), many-to-many (like a post office), many-to-one (like a mail box), and even one-to-one.
159
160* 0MQ sockets can send to many endpoints (creating a fan-out model), or receive from many endpoints (creating a fan-in model).
161
162[[code type="textdiagram"]]
163        +------------+           +------------+
164        |            |           |            |
165        |    Node    |           |    Node    |
166        |            |           |            |
167        +------------+           +------------+
168        |   Socket   |           |   Socket   |
169        \----+-+-----/           \------+-----/
170             | |                        :
171     1 to N  | +------------------------+
172     Fan out |                          |
173             +------------------------+ | N to 1
174             |                        | | Fan in
175             v                        v v
176        /------------\           /------------\
177        |   Socket   |           |   Socket   |
178        +------------+           +------------+
179        |            |           |            |
180        |    Node    |           |    Node    |
181        |            |           |            |
182        +------------+           +------------+
183
184
185           Figure # - 0MQ sockets are N to N
186[[/code]]
187
188So writing a message to a socket may send the message to one or many other places at once, and conversely, one socket will collect messages from all connections sending messages to it. The zmq_recv[3] method uses a fair-queuing algorithm so each sender gets an even chance.
189
190The zmq_send[3] method does not actually send the message to the socket connection(s). It queues the message so that the I/O thread can send it asynchronously. It does not block except in some exception cases. So the message is not necessarily sent when zmq_send[3] returns to your application. If you created a message using zmq_msg_init_data[3] you cannot reuse the data or free it, otherwise the I/O thread will rapidly find itself writing overwritten or unallocated garbage. This is a common mistake for beginners. We'll see a little later how to properly work with messages.
191
192+++ Unicast Transports
193
1940MQ provides a set of unicast transports ({{inproc}}, {{ipc}}, and {{tcp}}) and multicast transports (epgm, pgm). Multicast is an advanced technique that we'll come to later. Don't even start using it unless you know that your fanout ratios will make 1-to-N unicast impossible.
195
196For most common cases, use **{{tcp}}**, which is a //disconnected TCP// transport. It is elastic, portable, and fast enough for most cases. We call this 'disconnected' because 0MQ's {{tcp}} transport doesn't require that the endpoint exists before you connect to it. Clients and servers can connect and bind at any time, can go and come back, and it remains transparent to applications.
197
198The inter-process transport, **{{ipc}}**, is like {{tcp}} except that it is abstracted from the LAN, so you don't need to specify IP addresses or domain names. This makes it better for some purposes, and we use it quite often in the examples in this book. 0MQ's {{ipc}} transport is disconnected, like {{tcp}}. It has one limitation: it does not work on Windows. This may be fixed in future versions of 0MQ. By convention we use endpoint names with an ".ipc" extension to avoid potential conflict with other file names. On UNIX systems, if you use {{ipc}} endpoints you need to create these with appropriate permissions otherwise they may not be shareable between processes running under different user ids. You must also make sure all processes can access the files, e.g. by running in the same working directory.
199
200The inter-thread transport, **{{inproc}}**, is a connected signaling transport. It is much faster than {{tcp}} or {{ipc}}. This transport has a specific limitation compared to {{ipc}} and {{tcp}}: **you must do bind before connect**. This is something future versions of 0MQ may fix, but at present this defines you use {{inproc}} sockets. We create and bind one socket, start the child threads, which create and connect the other sockets.
201
202+++ 0MQ is Not a Neutral Carrier
203
204A common question that newcomers to 0MQ ask (it's one I asked myself) is something like, "//how do I write a XYZ server in 0MQ?//"  For example, "how do I write an HTTP server in 0MQ?"
205
206The implication is that if we use normal sockets to carry HTTP requests and responses, we should be able to use 0MQ sockets to do the same, only much faster and better.
207
208Sadly the answer is "this is not how it works". 0MQ is not a neutral carrier, it imposes a framing on the transport protocols it uses. This framing is not compatible with existing protocols, which tend to use their own framing. For example, here is an HTTP request, and a 0MQ request, both over TCP/IP:
209
210[[code type="textdiagram"]]
211    +----------------+----+----+----+----+
212    | GET /index.html| 13 | 10 | 13 | 10 |
213    +----------------+----+----+----+----+
214
215           Figure # - HTTP request
216[[/code]]
217
218Where the HTTP request uses CR-LF as its simplest framing delimiter, and 0MQ uses a length-specified frame:
219
220[[code type="textdiagram"]]
221    +---+---+---+---+---+---+
222    | 5 | H | E | L | L | O |
223    +---+---+---+---+---+---+
224
225      Figure # - 0MQ request
226[[/code]]
227
228So you could write a HTTP-like protocol using 0MQ, using for example the request-reply socket pattern. But it would not be HTTP.
229
230There is however a good answer to the question, "how can I make profitable use of 0MQ when making my new XYZ server?"  You need to implement whatever protocol you want to speak in any case, but you can connect that protocol server (which can be extremely thin) to a 0MQ backend that does the real work. The beautiful part here is that you can then extend your backend with code in any language, running locally or remotely, as you wish. Zed Shaw's [http://www.mongrel2.org Mongrel2] web server is a great example of such an architecture.
231
232+++ I/O Threads
233
234We said that 0MQ does I/O in a background thread. One I/O thread (for all sockets) is sufficient for all but the most extreme applications. This is the magic '1' that we use when creating a context, meaning "use one I/O thread":
235
236[[code language="C"]]
237void *context = zmq_init (1);
238[[/code]]
239
240There is a major difference between a 0MQ application and a conventional networked application, which is that you don't create one socket per connection. One socket handles all incoming and outcoming connections for a particular point of work. E.g. when you publish to a thousand subscribers, it's via one socket. When you distribute work among twenty services, it's via one socket. When you collect data from a thousand web applications, it's via one socket.
241
242This has a fundamental impact on how you write applications. A traditional networked application has one process or one thread per remote connection, and that process or thread handles one socket. 0MQ lets you collapse this entire structure into a single thread, and then break it up as necessary for scaling.
243
244+++ Core Messaging Patterns
245
246Underneath the brown paper wrapping of 0MQ's socket API lies the world of messaging patterns. If you have a background in enterprise messaging, these will be vaguely familiar. But to most 0MQ newcomers they are a surprise, we're so used to the TCP paradigm where a socket represents another node.
247
248Let's recap briefly what 0MQ does for you. It delivers blobs of data (messages) to nodes, quickly and efficiently. You can map nodes to threads, processes, or boxes. It gives your applications a single socket API to work with, no matter what the actual transport (like in-process, inter-process, TCP, or multicast). It automatically reconnects to peers as they come and go. It queues messages at both sender and receiver, as needed. It manages these queues carefully to ensure processes don't run out of memory, overflowing to disk when appropriate. It handles socket errors. It does all I/O in background threads. It uses lock-free techniques for talking between nodes, so there are never locks, waits, semaphores, or deadlocks.
249
250But cutting through that, it routes and queues messages according to precise recipes called //patterns//. It is these patterns that provide 0MQ's intelligence. They encapsulate our hard-earned experience of the best ways to distribute data and work. 0MQ's patterns are hard-coded but future versions may allow user-definable patterns.
251
2520MQ patterns are implemented by pairs of sockets with matching types. In other words, to understand 0MQ patterns you need to understand socket types and how they work together. Mostly this just takes learning, there is little that is obvious at this level.
253
254The built-in core 0MQ patterns are:
255
256* **Request-reply**, which connects a set of clients to a set of services. This is a remote procedure call and task distribution pattern.
257
258* **Publish-subscribe**, which connects a set of publishers to a set of subscribers. This is a data distribution pattern.
259
260* **Pipeline**, connects nodes in a fan-out / fan-in pattern that can have multiple steps, and loops. This is a parallel task distribution and collection pattern.
261
262We looked at each of these in the first chapter. There's one more pattern that people tend to try to use when they still think of 0MQ in terms of traditional TCP sockets:
263
264* **Exclusive pair**, which connects two sockets in an exclusive pair. This is a low-level pattern for specific, advanced use-cases. We'll see an example at the end of this chapter.
265
266The zmq_socket[3] man page is fairly clear about the patterns, it's worth reading several times until it starts to make sense. We'll look at each pattern and the use-cases it covers.
267
268These are the socket combinations that are valid for a connect-bind pair (either side can bind):
269
270* PUB and SUB
271* REQ and REP
272* REQ and ROUTER
273* DEALER and REP
274* DEALER and ROUTER
275* DEALER and DEALER
276* ROUTER and ROUTER
277* PUSH and PULL
278* PAIR and PAIR
279
280Any other combination will produce undocumented and unreliable results and future versions of 0MQ will probably return errors if you try them. You can and will of course bridge other socket types //via code//, i.e. read from one socket type and write to another.
281
282+++ High-level Messaging Patterns
283
284These four core patterns are cooked-in to 0MQ. They are part of the 0MQ API, implemented in the core C++ library, and guaranteed to be available in all fine retail stores. If one day the Linux kernel includes 0MQ, for example, these patterns would be there.
285
286On top, we add //high-level patterns//. We build these high-level patterns on top of 0MQ and implement them in whatever language we're using for our application. They are not part of the core library, do not come with the 0MQ package, and exist in their own space, as part of the 0MQ community.
287
288One of the things we aim to provide you with this guide are a set of such high-level patterns, both small (how to handle messages sanely) to large (how to make a reliable publish-subscribe architecture).
289
290+++ Working with Messages
291
292On the wire, 0MQ messages are blobs of any size from zero upwards, fitting in memory. You do your own serialization using Google Protocol Buffers, XDR, JSON, or whatever else your applications need to speak. It's wise to choose a data representation that is portable and fast, but you can make your own decisions about trade-offs.
293
294In memory, 0MQ messages are zmq_msg_t structures (or classes depending on your language). Here are the basic ground rules for using 0MQ messages in C:
295
296* You create and pass around zmq_msg_t objects, not blocks of data.
297
298* To read a message you use zmq_msg_init[3] to create an empty message, and then you pass that to zmq_recv[3].
299
300* To write a message from new data, you use zmq_msg_init_size[3] to create a message and at the same time allocate a block of data of some size. You then fill that data using memcpy, and pass the message to zmq_send[3].
301
302* To release (not destroy) a message you call zmq_msg_close[3]. This drops a reference, and eventually 0MQ will destroy the message.
303
304* To access the message content you use zmq_msg_data[3]. To know how much data the message contains, use zmq_msg_size[3].
305
306* Do not use zmq_msg_move[3], zmq_msg_copy[3], or zmq_msg_init_data[3] unless you read the man pages and know precisely why you need these.
307
308Here is a typical chunk of code working with messages, which should be familiar if you have been paying attention. This is from the zhelpers.h file we use in all the examples:
309
310[[code language="C"]]
311//  Receive 0MQ string from socket and convert into C string
312static char *
313s_recv (void *socket) {
314    zmq_msg_t message;
315    zmq_msg_init (&message);
316    zmq_recv (socket, &message, 0);
317    int size = zmq_msg_size (&message);
318    char *string = malloc (size + 1);
319    memcpy (string, zmq_msg_data (&message), size);
320    zmq_msg_close (&message);
321    string [size] = 0;
322    return (string);
323}
324
325//  Convert C string to 0MQ string and send to socket
326static int
327s_send (void *socket, char *string) {
328    int rc;
329    zmq_msg_t message;
330    zmq_msg_init_size (&message, strlen (string));
331    memcpy (zmq_msg_data (&message), string, strlen (string));
332    rc = zmq_send (socket, &message, 0);
333    assert (!rc);
334    zmq_msg_close (&message);
335    return (rc);
336}
337[[/code]]
338
339You can easily extend this code to send and receive blobs of arbitrary length.
340
341**Note than when you have passed a message to zmq_send(3), ØMQ will clear the message, i.e. set the size to zero. You cannot send the same message twice, and you cannot access the message data after sending it.**
342
343If you want to send the same message more than once, create a second message, initialize it using zmq_msg_init[3] and then use zmq_msg_copy[3] to create a copy of the first message. This does not copy the data but the reference. You can then send the message twice (or more, if you create more copies) and the message will only be finally destroyed when the last copy is sent or closed.
344
3450MQ also supports //multipart// messages, which let you handle a list of blobs as a single message. This is widely used in real applications and we'll look at that later in this chapter and in Chapter Three.
346
347Some other things that are worth knowing about messages:
348
349* 0MQ sends and receives them atomically, i.e. you get a whole message, or you don't get it at all.
350
351* 0MQ does not send a message right away but at some indeterminate later time.
352
353* You can send zero-length messages, e.g. for sending a signal from one thread to another.
354
355* A message must fit in memory. If you want to send files of arbitrary sizes, you should break them into pieces and send each piece as a separate message.
356
357* You must call zmq_msg_close[3] when finished with a message, in languages that don't automatically destroy objects when a scope closes.
358
359And to be necessarily repetitive, do not use zmq_msg_init_data[3], yet. This is a zero-copy method and guaranteed to create trouble for you. There are far more important things to learn about 0MQ before you start to worry about shaving off microseconds.
360
361+++ Handling Multiple Sockets
362
363In all the examples so far, the main loop of most examples has been:
364
365# wait for message on socket
366# process message
367# repeat
368
369What if we want to read from multiple sockets at the same time? The simplest way is to connect one socket to multiple endpoints and get 0MQ to do the fanin for us. This is legal if the remote endpoints are in the same pattern but it would be illegal to e.g. connect a PULL socket to a PUB endpoint. Fun, but illegal. If you start mixing patterns you break future scalability.
370
371The right way is to use zmq_poll[3]. An even better way might be to wrap zmq_poll[3] in a framework that turns it into a nice event-driven //reactor//, but it's significantly more work than we want to cover here.
372
373Let's start with a dirty hack, partly for the fun of not doing it right, but mainly because it lets me show you how to do non-blocking socket reads. Here is a simple example of reading from two sockets using non-blocking reads. This rather confused program acts both as a subscriber to weather updates, and a worker for parallel tasks:
374
375[[code type="example" title="Multiple socket reader" name="msreader"]]
376[[/code]]
377
378The cost of this approach is some additional latency on the first message (the sleep at the end of the loop, when there are no waiting messages to process). This would be a problem in applications where sub-millisecond latency was vital. Also, you need to check the documentation for nanosleep() or whatever function you use to make sure it does not busy-loop.
379
380You can treat the sockets fairly by reading first from one, then the second rather than prioritizing them as we did in this example. This is called "fair-queuing", something that 0MQ does automatically when one socket receives messages from more than one source.
381
382Now let's see the same little senseless application done right, using zmq_poll[3]:
383
384[[code type="example" title="Multiple socket poller" name="mspoller"]]
385[[/code]]
386
387+++ Handling Errors and ETERM
388
3890MQ's error handling philosophy is a mix of fail-fast and resilience. Processes, we believe, should be as vulnerable as possible to internal errors, and as robust as possible against external attacks and errors. To give an analogy, a living cell will self-destruct if it detects a single internal error, yet it will resist attack from the outside by all means possible. Assertions, which pepper the 0MQ code, are absolutely vital to robust code, they just have to be on the right side of the cellular wall. And there should be such a wall. If it is unclear whether a fault is internal or external, that is a design flaw that needs to be fixed.
390
391In C, assertions stop the application immediately with an error. In other languages you may get exceptions or halts.
392
393When 0MQ detects an external fault it returns an error to the calling code. In some rare cases it drops messages silently, if there is no obvious strategy for recovering from the error. In a few places 0MQ still asserts on external faults, but these are considered bugs.
394
395In most of the C examples we've seen so far there's been no error handling. **Real code should do error handling on every single 0MQ call**. If you're using a language binding other than C, the binding may handle errors for you. In C you do need to do this yourself. There are some simple rules, starting with POSIX conventions:
396
397* Methods that create objects will return NULL in case they fail.
398
399* Other methods will return 0 on success and other values (mostly -1) on an exceptional condition (usually failure).
400
401* The error code is provided in {{errno}} or zmq_errno[3].
402
403* A descriptive error text for logging is provided by zmq_strerror[3].
404
405There are two main exceptional conditions that you may want to handle as non-fatal:
406
407* When a thread calls zmq_recv[3] with the NOBLOCK option and there is no waiting data. 0MQ will return -1 and set errno to EAGAIN.
408
409* When a thread calls zmq_term[3] and other threads are doing blocking work. The zmq_term[3] call closes the context and all blocking calls exit with -1, and errno set to ETERM.
410
411What this boils down to is that in most cases you can use assertions on 0MQ calls, like this, in C:
412
413[[code language="C"]]
414void *context = zmq_init (1);
415assert (context);
416void *socket = zmq_socket (context, ZMQ_REP);
417assert (socket);
418int rc;
419rc = zmq_bind (socket, "tcp://*:5555");
420assert (rc == 0);
421[[/code]]
422
423In the first version of this code I put the assert() call around the function. Not a good idea, since an optimized build will turn all assert() macros to null and happily wallop those functions. Use a return code, and assert the return code.
424
425Let's see how to shut down a process cleanly. We'll take the parallel pipeline example from the previous section. If we've started a whole lot of workers in the background, we now want to kill them when the batch is finished. Let's do this by sending a kill message to the workers. The best place to do this is the sink, since it really knows when the batch is done.
426
427How do we connect the sink to the workers? The PUSH/PULL sockets are one-way only. The standard 0MQ answer is: create a new socket flow for each type of problem you need to solve. We'll use a publish-subscribe model to send kill messages to the workers:
428
429* The sink creates a PUB socket on a new endpoint.
430* Workers bind their input socket to this endpoint.
431* When the sink detects the end of the batch it sends a kill to its PUB socket.
432* When a worker detects this kill message, it exits.
433
434It doesn't take much new code in the sink:
435
436[[code language="C"]]
437    void *control = zmq_socket (context, ZMQ_PUB);
438    zmq_bind (control, "tcp://*:5559");
439    ...
440    //  Send kill signal to workers
441    zmq_msg_init_data (&message, "KILL", 5);
442    zmq_send (control, &message, 0);
443    zmq_msg_close (&message);
444[[/code]]
445
446[[code type="textdiagram"]]
447                +-------------+
448                |             |
449                |  Ventilator |
450                |             |
451                +-------------+
452                |    PUSH     |
453                \------+------/
454                       |
455                     tasks
456                       |
457       +---------------+---------------+
458       |               |               |
459       |     /=--------|-----+=--------|-----+------\
460     task    |       task    |        task   |      :
461       |     |         |     |         |     |      |
462       v     v         v     v         v     v      |
463   /------+-----\  /------+-----\  /------+-----\   |
464   | PULL | SUB |  | PULL | SUB |  | PULL | SUB |   |
465   +------+-----+  +------+-----+  +------+-----+   |
466   |            |  |            |  |            |   |
467   |   Worker   |  |   Worker   |  |   Worker   |   |
468   |            |  |            |  |            |   |
469   +------------+  +------------+  +------------+   |
470   |    PUSH    |  |    PUSH    |  |    PUSH    |   |
471   \-----+------/  \-----+------/  \-----+------/   |
472         |               |               |          |
473       result          result          result       |
474         |               |               |          |
475         +---------------+---------------+          |
476                         |                          |
477                      results                       |
478                         |                          |
479                         v                          |
480                  /-------------\                   |
481                  |    PULL     |                   |
482                  +-------------+                   |
483                  |             |                   |
484                  |    Sink     |                   |
485                  |             |                   |
486                  +-------------+                   |
487                  |     PUB     |                   |
488                  \------+------/                   |
489                         |                          |
490                    KILL signal                     |
491                         |                          |
492                         \--------------------------/
493
494
495     Figure # - Parallel Pipeline with Kill signaling
496[[/code]]
497
498Here is the worker process, which manages two sockets (a PULL socket getting tasks, and a SUB socket getting control commands) using the zmq_poll[3] technique we saw earlier:
499
500[[code type="example" title="Parallel task worker with kill signaling" name="taskwork2"]]
501[[/code]]
502
503Here is the modified sink application. When it's finished collecting results it broadcasts a KILL message to all workers:
504
505[[code type="example" title="Parallel task sink with kill signaling" name="tasksink2"]]
506[[/code]]
507
508+++ Handling Interrupt Signals
509
510Realistic applications need to shutdown cleanly when interrupted with Ctrl-C or another signal such as SIGTERM. By default, these simply kill the process, meaning messages won't be flushed, files won't be closed cleanly, etc.
511
512Here is how we handle a signal in various languages:
513
514[[code type="example" title="Handling Ctrl-C cleanly" name="interrupt"]]
515[[/code]]
516
517The program provides s_catch_signals(), which traps Ctrl-C (SIGINT) and SIGTERM. When either of these signals arrive, the s_catch_signals() handler sets the global variable s_interrupted. Your application will not die automatically, you have to now explicitly check for an interrupt, and handle it properly. Here's how:
518
519* Call s_catch_signals() (copy this from interrupt.c) at the start of your main code. This sets-up the signal handling.
520* If your code is blocking in zmq_recv[3], zmq_poll[3], or zmq_send[3], when a signal arrives, the call will return with EINTR.
521* Wrappers like s_recv() return NULL if they are interrupted.
522* So, your application checks for an EINTR return code, a NULL return, and/or s_interrupted.
523
524Here is a typical code fragment:
525
526[[code]]
527s_catch_signals ();
528client = zmq_socket (...);
529while (!s_interrupted) {
530    char *message = s_recv (client);
531    if (!message)
532        break;          //  Ctrl-C used
533}
534zmq_close (client);
535[[/code]]
536
537If you call s_catch_signals() and don't test for interrupts, the your application will become immune to Ctrl-C and SIGTERM, which may be useful, but is usually not.
538
539+++ Detecting Memory Leaks
540
541Any long-running application has to manage memory correctly, or eventually it'll use up all available memory and crash. If you use a language that handles this automatically for you, congratulations. If you program in C or C++ or any other language where you're responsible for memory management, here's a short tutorial on using valgrind, which among other things will report on any leaks your programs have.
542
543* To install valgrind, e.g. on Ubuntu or Debian: {{sudo apt-get install valgrind}}.
544
545* By default, 0MQ will cause valgrind to complain a lot. To remove these warnings, rebuild 0MQ with the ZMQ_MAKE_VALGRIND_HAPPY macro, thus:
546
547[[code]]
548$ cd zeromq2
549$ export CPPFLAGS=-DZMQ_MAKE_VALGRIND_HAPPY
550$ ./configure
551$ make clean; make
552$ sudo make install
553[[/code]]
554
555* Fix your applications to exit cleanly after Ctrl-C. For any application that exits by itself, that's not needed, but for long-running applications (like devices), this is essential, otherwise valgrind will complain about all currently allocated memory.
556
557* Build your application with -DDEBUG, if it's not your default setting. That ensures valgrind can tell you exactly where memory is being leaked.
558
559* Finally, run valgrind thus:
560
561[[code]]
562valgrind --tool=memcheck --leak-check=full someprog
563[[/code]]
564
565And after fixing any errors it reported, you should get the pleasant message:
566
567[[code]]
568==30536== ERROR SUMMARY: 0 errors from 0 contexts...
569[[/code]]
570
571+++ Multipart Messages
572
5730MQ lets us compose a message out of several frames, giving us a 'multipart message'. Realistic applications use multipart messages heavily, especially to make "envelopes". We'll look at them later. What we'll learn now is simply how to safely (but blindly) read and write multipart messages because otherwise the devices we write won't work with applications that use multipart messages.
574
575When you work with multipart messages, each part is a zmq_msg item. E.g. if you are sending a message with five parts, you must construct, send, and destroy five zmq_msg items. You can do this in advance (and store the zmq_msg items in an array or structure), or as you send them, one by one.
576
577Here is how we send the frames in a multipart message (we receive each frame into a message object):
578
579[[code language="C"]]
580zmq_send (socket, &message, ZMQ_SNDMORE);
581...
582zmq_send (socket, &message, ZMQ_SNDMORE);
583...
584zmq_send (socket, &message, 0);
585[[/code]]
586
587Here is how we receive and process all the parts in a message, be it single part or multipart:
588
589[[code language="C"]]
590while (1) {
591    zmq_msg_t message;
592    zmq_msg_init (&message);
593    zmq_recv (socket, &message, 0);
594    //  Process the message part
595    zmq_msg_close (&message);
596    int64_t more;
597    size_t more_size = sizeof (more);
598    zmq_getsockopt (socket, ZMQ_RCVMORE, &more, &more_size);
599    if (!more)
600        break;      //  Last message part
601}
602[[/code]]
603
604Some things to know about multipart messages:
605
606* When you send a multipart message, the first part (and all following parts) are only sent when you send the final part.
607
608* If you are using zmq_poll[3], when you receive the first part of a message, all the rest has also arrived.
609
610* You will receive all parts of a message, or none at all.
611
612* Each part of a message is a separate zmq_msg item.
613
614* You will receive all parts of a message whether or not you check the RCVMORE option.
615
616* On sending, 0MQ queues message parts in memory until the last is received, then sends them all.
617
618* There is no way to cancel a partially sent message, except by closing the socket.
619
620+++ Intermediates and Devices
621
622Any connected set hits a complexity curve as the number of set members increases. A small number of members can all know about each other but as the set gets larger, the cost to each member of knowing all other interesting members grows linearly, and the overall cost of connecting members grows factorially. The solution is to break sets into smaller ones, and use intermediates to connect the sets.
623
624This pattern is extremely common in the real world and is why our societies and economies are filled with intermediaries who have no other real function than to reduce the complexity and scaling costs of larger networks. Intermediaries are typically called wholesalers, distributors, managers, etc.
625
626A 0MQ network like any cannot grow beyond a certain size without needing intermediaries. In 0MQ, we call these "devices". When we use 0MQ we usually start building our applications as a set of nodes on a network with the nodes talking to each other, without intermediaries:
627
628[[code type="textdiagram"]]
629               +---------+
630               |         |
631               |  Node   |
632               |         |
633               +---------+
634               | Socket  |
635               \----+----/
636                    |
637                    |
638             +------+------+
639             |             |
640             |             |
641        /----+----\   /----+----\
642        | Socket  |   | Socket  |
643        +---------+   +---------+
644        |         |   |         |
645        |  Node   |   |  Node   |
646        |         |   |         |
647        +---------+   +---------+
648
649
650Figure # - Small scale 0MQ application
651[[/code]]
652
653And then we extend the application across a wider network, placing devices in specific places and scaling up the number of nodes:
654
655[[code type="textdiagram"]]
656                  +---------+
657                  |         |
658                  |  Node   |
659                  |         |
660                  +---------+
661                  | Socket  |
662                  \----+----/
663                       |
664                       |
665         +-------------+-------------+
666         |             |             |
667         |             |             |
668    /----+----\   /----+----\   /----+----\
669    | Socket  |   | Socket  |   | Socket  |
670    +---------+   +---------+   +---------+
671    |         |   |         |   |         |
672    |  Node   |   |  Node   |   | Device  |
673    |         |   |         |   |         |
674    +---------+   +---------+   +---------+
675                                | Socket  |
676                                \----+----/
677                                     |
678                                     |
679                              +------+------+
680                              |             |
681                              |             |
682                         /----+----\   /----+----\
683                         | Socket  |   | Socket  |
684                         +---------+   +---------+
685                         |         |   |         |
686                         |  Node   |   |  Node   |
687                         |         |   |         |
688                         +---------+   +---------+
689
690
691       Figure # - Larger scale 0MQ application
692[[/code]]
693
6940MQ devices generally connect a set of 'frontend' sockets to a set of 'backend' sockets, though there are no strict design rules. They ideally run with no state, so that it becomes possible to stretch applications over as many intermediates as needed. You can run them as threads within a process, or as stand-alone processes. 0MQ provides some very basic devices but you will in practice develop your own.
695
6960MQ devices can do intermediation of addresses, services, queues, or any other abstraction you care to define above the message and socket layers. Different messaging patterns have different complexity issues and need different kinds of intermediation. For example, request-reply works well with queue and service abstractions, while publish-subscribe works well with streams or topics.
697
698What's interesting about 0MQ as compared to traditional centralized brokers is that you can place devices precisely where you need them, and they can do the optimal intermediation.
699
700++++ A Publish-Subscribe Proxy Server
701
702It is a common requirement to extend a publish-subscribe architecture over more than one network segment or transport. Perhaps there are a group of subscribers sitting at a remote location. Perhaps we want to publish to local subscribers via multicast, and to remote subscribers via TCP.
703
704We're going to write a simple proxy server that sits in between a publisher and a set of subscribers, bridging two networks. This is perhaps the simplest case of a useful device. The device has two sockets, a frontend facing the internal network, where the weather server is sitting, and a backend facing subscribers on the external network. It subscribes to the weather service on the frontend socket, and republishes its data on the backend socket:
705
706[[code type="example" title="Weather update proxy" name="wuproxy"]]
707[[/code]]
708
709We call this a //proxy// because it acts as a subscriber to publishers, and acts as a publisher to subscribers. That means you can slot this device into an existing network without affecting it (of course the new subscribers need to know to speak to the proxy).
710
711[[code type="textdiagram"]]
712
713                   +-----------+
714                   |           |
715                   | Publisher |
716                   |           |
717                   +-----------+
718                   |    PUB    |
719                   \-----------/
720                       bind
721             tcp://192.168.55.210:5556
722                         |
723                         |
724        +----------------+----------------+
725        |                |                |
726        |                |                |
727     connect           connect            |
728  /------------\   /------------\       connect
729  |    SUB     |   |    SUB     |   /------------\
730  +------------+   +------------+   |    SUB     |
731  |            |   |            |   +------------+
732  | Subscriber |   | Subscriber |   |            |
733  |            |   |            |   | Forwarder  |
734  +------------+   +------------+   |            |
735                                    +------------+
736   Internal network                 |    PUB     |
737   ---------------------------------\------------/--------
738   External network                      bind
739                                  tcp://10.1.1.0:8100
740                                          |
741                                          |
742                                 +--------+--------+
743                                 |                 |
744                                 |                 |
745                              connect           connect
746                           /------------\    /------------\
747                           |    SUB     |    |    SUB     |
748                           +------------+    +------------+
749                           |            |    |            |
750                           | Subscriber |    | Subscriber |
751                           |            |    |            |
752                           +------------+    +------------+
753
754
755             Figure # - Forwarder proxy device
756[[/code]]
757
758Note that this application is multipart safe. It correctly detects multipart messages and sends them as it reads them. If we did not set the SNDMORE option on outgoing multipart data, the final recipient would get a corrupted message. You should always make your devices multipart safe so that there is no risk they will corrupt the data they switch.
759
760++++ A Request-Reply Broker
761
762Let's explore how to solve a problem of scale by writing a little message queuing broker in 0MQ. We'll look at the request-reply pattern for this case.
763
764In the Hello World client-server application we have one client that talks to one service. However in real cases we usually need to allow multiple services as well as multiple clients. This lets us scale up the power of the service (many threads or processes or boxes rather than just one). The only constraint is that services must be stateless, all state being in the request or in some shared storage such as a database.
765
766There are two ways to connect multiple clients to multiple servers. The brute-force way is to connect each client socket to multiple service endpoints. One client socket can connect to multiple service sockets, and requests are load-balanced among these services. Let's say you connect a client socket to three service endpoints, A, B, and C. The client makes requests R1, R2, R3, R4. R1 and R4 go to service A, R2 goes to B, and R3 goes to service C.
767
768[[code type="textdiagram"]]
769               +-----------+
770               |           |
771               |   Client  |
772               |           |
773               +-----------+
774               |    REQ    |
775               \-----+-----/
776                     |
777               R1, R2, R3, R4
778                     |
779       +-------------+-------------+
780       |             |             |
781    R1, R4           R2            R3
782       |             |             |
783       v             v             v
784  /---------\   /---------\   /---------\
785  |   REP   |   |   REP   |   |   REP   |
786  +---------+   +---------+   +---------+
787  |         |   |         |   |         |
788  | Service |   | Service |   | Service |
789  |    A    |   |    B    |   |    C    |
790  |         |   |         |   |         |
791  +---------+   +---------+   +---------+
792
793
794   Figure # - Load balancing of requests
795[[/code]]
796
797This design lets you add more clients cheaply. You can also add more services. Each client will load-balance its requests to the services. But each client has to know the service topology. If you have 100 clients and then you decide to add three more services, you need to reconfigure and restart 100 clients in order for the clients to know about the three new services.
798
799That's clearly not the kind of thing we want to be doing at 3am when our supercomputing cluster has run out of resources and we desperately need to add a couple of hundred new service nodes. Too many stable pieces are like liquid concrete: knowledge is distributed and the more stable pieces you have, the more effort it is to change the topology. What we want is something sitting in between clients and services that centralizes all knowledge of the topology. Ideally, we should be able to add and remove services or clients at any time without touching any other part of the topology.
800
801So we'll write a little message queuing broker that gives us t…

Large files files are truncated, but you can click here to view the full file