/README

  1Mordor
  2
  3What is it?
  4
  5Mordor is a high performance I/O library.  It is cross-platform, compiling
  6on Windows, Linux, and Mac (32-bit and 64-bit on all platforms).  It includes
  7several main areas:
  8
  9* Cooperatively scheduled fiber engine, including synchronization primitives
 10* Streams library, for dealing with streams of data and manipulating them.
 11* HTTP library, building on top of Fibers and Streams, to provide a simple to
 12  use, yet extremely powerful HTTP client and server API.
 13* Supporting infrastructure, including logging, configuration, statistics
 14  gathering, and exceptions.
 15* A unit test framework that is lightweight, easy to use, but has several useful
 16  features.
 17
 18One of the main goals of Mordor is to provide very easy to use abstractions and
 19encapsulation of difficult and complex concepts, yet still provide near absolute
 20power in weilding them if necessary.
 21
 22Where should it be used?
 23
 24Any software (server-side or client-side) that need to process a lot of data.
 25It is C++, so is probably overkill for something that could be easily handled
 26with a Python or Ruby script, but can be used for simpler tasks because it does
 27provide some nice abstractions that you won't see elsewhere.  Server
 28applications handling lots of connections will benefit most from the Fiber
 29engine, by transforming an event-based paradigm into a familiar thread-based
 30paradigm, while keeping (and in some cases improving) the performance of an
 31event-based paradigm.
 32
 33How does it change the game?
 34
 35Mordor allows you to focus on performing a logical task, instead of deciding how
 36to make that task conform to a specific threading/event model.  Just because
 37local disk I/O will block, and should be performed in a thread pool, and network
 38I/O should be performed using an event based callback design, doesn't mean you
 39can't do them both _in the same function_.  Mordor allows you to do just that.
 40For example, here's a complete program to read a file from disk, and send it to
 41a socket on the network:
 42
 43#include <iostream>
 44
 45#include <mordor/socket.h>
 46#include <mordor/streams/file.h>
 47#include <mordor/streams/socket.h>
 48#include <mordor/streams/transfer.h>
 49
 50using namespace Mordor;
 51
 52int main(int argc, char **argv)
 53{
 54    if (argc != 3) {
 55        std::cerr << "usage: " << argv[0] << " <file> <destination>" << std::endl;
 56        return 1;
 57    }
 58    try {
 59        std::vector<Address::ptr> addresses = Address::lookup(argv[2], AF_UNSPEC, SOCK_STREAM);
 60        Socket::ptr socket = addresses[0]->createSocket();
 61        socket->connect(addresses[0]);
 62        Stream::ptr fileStream(new FileStream(argv[1], FileStream::OPEN, FileStream::READ));
 63        Stream::ptr socketStream(new SocketStream(socket));
 64        transferStream(fileStream, socketStream);
 65    } catch (...) {
 66        std::cerr << boost::current_exception_diagnostic_information() << std::endl;
 67        return 2;
 68    }
 69    return 0;
 70}
 71
 72This program is quite simple.  It checks for usage, translates the string
 73argument into a network address, creates a socket that is compatible with that
 74address, connects to it, opens a file (as a stream), wraps the socket in a
 75stream, and then sends the file over the socket.  If an error occurs, complete
 76error information is printed on stdout, including the type of error, the OS
 77level error code and description (if applicable), and a complete stacktrace of
 78the error, including debug symbol information, if available.  Looking at it, we
 79can see that there is only a single thread.  Which is all fine and dandy if
 80this is all we're doing.  But what if instead we were sending 1000 files to
 811000 different sockets, but didn't want to create a thread for each one?  Let's
 82say we want one thread for communicating with the network, and four threads for
 83reading the file off the disk.  Let's do it!
 84
 85#include <iostream>
 86
 87#include <mordor/iomanager.h>
 88#include <mordor/scheduler.h>
 89#include <mordor/socket.h>
 90#include <mordor/streams/file.h>
 91#include <mordor/streams/socket.h>
 92#include <mordor/streams/transfer.h>
 93
 94using namespace Mordor;
 95
 96static void doOne(const char *file, const char *destination, IOManager &ioManager, Scheduler &scheduler)
 97{
 98    try {
 99        std::vector<Address::ptr> addresses = Address::lookup(destination, AF_UNSPEC, SOCK_STREAM);
100        Socket::ptr socket = addresses[0]->createSocket(ioManager);
101        socket->connect(addresses[0]);
102        Stream::ptr fileStream(new FileStream(file, FileStream::READ, FileStream::OPEN, &ioManager, &scheduler));
103        Stream::ptr socketStream(new SocketStream(socket));
104        transferStream(fileStream, socketStream);
105    } catch (...) {
106        std::cerr << boost::current_exception_diagnostic_information() << std::endl;
107    }
108}
109
110int main(int argc, char **argv)
111{
112    if (argc % 2 != 1) {
113        std::cerr << "usage: " << argv[0] << " (<file> <destination>)*" << std::endl;
114        return 1;
115    }
116    IOManager ioManager;
117    WorkerPool workerPool(4, false);
118    for (int i = 1; i < argc; i += 2)
119        ioManager.schedule(boost::bind(&doOne, argv[i], argv[i + 1], boost::ref(ioManager), boost::ref(workerPool)));
120    ioManager.dispatch();
121
122    return 0;
123}
124
125So we re-factored most of main into doOne, but other than that it is nearly
126identical.  And it will transfer as many files as you pass on the command line
127in parallel.  Using 5 threads.  The IOManager is the object used for
128non-blocking network I/O, and so is passed to the Socket when it is created.
129WorkerPool is just a generic thread pool, and is passed to the FileStream so
130that it will automatically do its work on those threads, instead of the thread
131it is running on when it is called.  Something to point out here is that when
132the work is scheduled on the IOManager, each bit of work implicitly creates a
133Fiber - a lightweight, cooperatively scheduled, user-mode thread.  The doOne
134function is executed on its own Fiber, and is allowed to switch which thread it
135is running on (inside of FileStream), without having to do any callbacks,
136virtual functions on a defined interface, or anything else.  Internally, when
137FileStream wants to execute on the thread pool, it suspends the current Fiber,
138allowing other Fibers to run on this thread, and is resumed on a thread in the
139WorkerPool.  IOManager and WorkerPool both inherit from Scheduler, which is the
140base functionality for cooperatively scheduling Fibers.  Pretty cool, eh?
141
142
143Dependencies
144
145boost 1.40
146OpenSSL
147Zlib
148Ragel (compile-time only)
149
150
151Compiling for iPhone SDK
152
153The iPhone SDK does not include OpenSSL headers or binaries. Since mordor relies
154on OpenSSL, you must provide these files yourself. The Xcode project file is
155configured to look for files in an iphone directory in the same directory as the
156project file. 
157
158Specifically, the compiler will look for headers in iphone/include/ and 
159libraries in iphone/lib/. We recommend you create a symbolic link called 
160"iphone" which points to the actual directory containing the include/ and lib/ 
161directories.
162
163
164License
165
166Mordor is licensed under the New BSD License, and Copyright (c) 2009, Decho Corp.
167See LICENSE for details.
168
169Authors
170
171Cody Cutrer (cody@mozy.com)
172Patrick Bozeman (peb@mozy.com)
173Jeremy Stanley (jeremy@mozy.com)
174Zach Wily (zach@mozy.com)
175Brian Palmer (brian@mozy.com)