/README
#! | 175 lines | 143 code | 32 blank | 0 comment | 0 complexity | 1be69402f1baa524e9d3511a723165a0 MD5 | raw file
1Mordor 2 3What is it? 4 5Mordor is a high performance I/O library. It is cross-platform, compiling 6on Windows, Linux, and Mac (32-bit and 64-bit on all platforms). It includes 7several main areas: 8 9* Cooperatively scheduled fiber engine, including synchronization primitives 10* Streams library, for dealing with streams of data and manipulating them. 11* HTTP library, building on top of Fibers and Streams, to provide a simple to 12 use, yet extremely powerful HTTP client and server API. 13* Supporting infrastructure, including logging, configuration, statistics 14 gathering, and exceptions. 15* A unit test framework that is lightweight, easy to use, but has several useful 16 features. 17 18One of the main goals of Mordor is to provide very easy to use abstractions and 19encapsulation of difficult and complex concepts, yet still provide near absolute 20power in weilding them if necessary. 21 22Where should it be used? 23 24Any software (server-side or client-side) that need to process a lot of data. 25It is C++, so is probably overkill for something that could be easily handled 26with a Python or Ruby script, but can be used for simpler tasks because it does 27provide some nice abstractions that you won't see elsewhere. Server 28applications handling lots of connections will benefit most from the Fiber 29engine, by transforming an event-based paradigm into a familiar thread-based 30paradigm, while keeping (and in some cases improving) the performance of an 31event-based paradigm. 32 33How does it change the game? 34 35Mordor allows you to focus on performing a logical task, instead of deciding how 36to make that task conform to a specific threading/event model. Just because 37local disk I/O will block, and should be performed in a thread pool, and network 38I/O should be performed using an event based callback design, doesn't mean you 39can't do them both _in the same function_. Mordor allows you to do just that. 40For example, here's a complete program to read a file from disk, and send it to 41a socket on the network: 42 43#include <iostream> 44 45#include <mordor/socket.h> 46#include <mordor/streams/file.h> 47#include <mordor/streams/socket.h> 48#include <mordor/streams/transfer.h> 49 50using namespace Mordor; 51 52int main(int argc, char **argv) 53{ 54 if (argc != 3) { 55 std::cerr << "usage: " << argv[0] << " <file> <destination>" << std::endl; 56 return 1; 57 } 58 try { 59 std::vector<Address::ptr> addresses = Address::lookup(argv[2], AF_UNSPEC, SOCK_STREAM); 60 Socket::ptr socket = addresses[0]->createSocket(); 61 socket->connect(addresses[0]); 62 Stream::ptr fileStream(new FileStream(argv[1], FileStream::OPEN, FileStream::READ)); 63 Stream::ptr socketStream(new SocketStream(socket)); 64 transferStream(fileStream, socketStream); 65 } catch (...) { 66 std::cerr << boost::current_exception_diagnostic_information() << std::endl; 67 return 2; 68 } 69 return 0; 70} 71 72This program is quite simple. It checks for usage, translates the string 73argument into a network address, creates a socket that is compatible with that 74address, connects to it, opens a file (as a stream), wraps the socket in a 75stream, and then sends the file over the socket. If an error occurs, complete 76error information is printed on stdout, including the type of error, the OS 77level error code and description (if applicable), and a complete stacktrace of 78the error, including debug symbol information, if available. Looking at it, we 79can see that there is only a single thread. Which is all fine and dandy if 80this is all we're doing. But what if instead we were sending 1000 files to 811000 different sockets, but didn't want to create a thread for each one? Let's 82say we want one thread for communicating with the network, and four threads for 83reading the file off the disk. Let's do it! 84 85#include <iostream> 86 87#include <mordor/iomanager.h> 88#include <mordor/scheduler.h> 89#include <mordor/socket.h> 90#include <mordor/streams/file.h> 91#include <mordor/streams/socket.h> 92#include <mordor/streams/transfer.h> 93 94using namespace Mordor; 95 96static void doOne(const char *file, const char *destination, IOManager &ioManager, Scheduler &scheduler) 97{ 98 try { 99 std::vector<Address::ptr> addresses = Address::lookup(destination, AF_UNSPEC, SOCK_STREAM); 100 Socket::ptr socket = addresses[0]->createSocket(ioManager); 101 socket->connect(addresses[0]); 102 Stream::ptr fileStream(new FileStream(file, FileStream::READ, FileStream::OPEN, &ioManager, &scheduler)); 103 Stream::ptr socketStream(new SocketStream(socket)); 104 transferStream(fileStream, socketStream); 105 } catch (...) { 106 std::cerr << boost::current_exception_diagnostic_information() << std::endl; 107 } 108} 109 110int main(int argc, char **argv) 111{ 112 if (argc % 2 != 1) { 113 std::cerr << "usage: " << argv[0] << " (<file> <destination>)*" << std::endl; 114 return 1; 115 } 116 IOManager ioManager; 117 WorkerPool workerPool(4, false); 118 for (int i = 1; i < argc; i += 2) 119 ioManager.schedule(boost::bind(&doOne, argv[i], argv[i + 1], boost::ref(ioManager), boost::ref(workerPool))); 120 ioManager.dispatch(); 121 122 return 0; 123} 124 125So we re-factored most of main into doOne, but other than that it is nearly 126identical. And it will transfer as many files as you pass on the command line 127in parallel. Using 5 threads. The IOManager is the object used for 128non-blocking network I/O, and so is passed to the Socket when it is created. 129WorkerPool is just a generic thread pool, and is passed to the FileStream so 130that it will automatically do its work on those threads, instead of the thread 131it is running on when it is called. Something to point out here is that when 132the work is scheduled on the IOManager, each bit of work implicitly creates a 133Fiber - a lightweight, cooperatively scheduled, user-mode thread. The doOne 134function is executed on its own Fiber, and is allowed to switch which thread it 135is running on (inside of FileStream), without having to do any callbacks, 136virtual functions on a defined interface, or anything else. Internally, when 137FileStream wants to execute on the thread pool, it suspends the current Fiber, 138allowing other Fibers to run on this thread, and is resumed on a thread in the 139WorkerPool. IOManager and WorkerPool both inherit from Scheduler, which is the 140base functionality for cooperatively scheduling Fibers. Pretty cool, eh? 141 142 143Dependencies 144 145boost 1.40 146OpenSSL 147Zlib 148Ragel (compile-time only) 149 150 151Compiling for iPhone SDK 152 153The iPhone SDK does not include OpenSSL headers or binaries. Since mordor relies 154on OpenSSL, you must provide these files yourself. The Xcode project file is 155configured to look for files in an iphone directory in the same directory as the 156project file. 157 158Specifically, the compiler will look for headers in iphone/include/ and 159libraries in iphone/lib/. We recommend you create a symbolic link called 160"iphone" which points to the actual directory containing the include/ and lib/ 161directories. 162 163 164License 165 166Mordor is licensed under the New BSD License, and Copyright (c) 2009, Decho Corp. 167See LICENSE for details. 168 169Authors 170 171Cody Cutrer (cody@mozy.com) 172Patrick Bozeman (peb@mozy.com) 173Jeremy Stanley (jeremy@mozy.com) 174Zach Wily (zach@mozy.com) 175Brian Palmer (brian@mozy.com)