PageRenderTime 37ms CodeModel.GetById 18ms RepoModel.GetById 0ms app.codeStats 0ms

/README

http://github.com/mozy/mordor
#! | 175 lines | 143 code | 32 blank | 0 comment | 0 complexity | 1be69402f1baa524e9d3511a723165a0 MD5 | raw file
Possible License(s): BSD-3-Clause
  1. Mordor
  2. What is it?
  3. Mordor is a high performance I/O library. It is cross-platform, compiling
  4. on Windows, Linux, and Mac (32-bit and 64-bit on all platforms). It includes
  5. several main areas:
  6. * Cooperatively scheduled fiber engine, including synchronization primitives
  7. * Streams library, for dealing with streams of data and manipulating them.
  8. * HTTP library, building on top of Fibers and Streams, to provide a simple to
  9. use, yet extremely powerful HTTP client and server API.
  10. * Supporting infrastructure, including logging, configuration, statistics
  11. gathering, and exceptions.
  12. * A unit test framework that is lightweight, easy to use, but has several useful
  13. features.
  14. One of the main goals of Mordor is to provide very easy to use abstractions and
  15. encapsulation of difficult and complex concepts, yet still provide near absolute
  16. power in weilding them if necessary.
  17. Where should it be used?
  18. Any software (server-side or client-side) that need to process a lot of data.
  19. It is C++, so is probably overkill for something that could be easily handled
  20. with a Python or Ruby script, but can be used for simpler tasks because it does
  21. provide some nice abstractions that you won't see elsewhere. Server
  22. applications handling lots of connections will benefit most from the Fiber
  23. engine, by transforming an event-based paradigm into a familiar thread-based
  24. paradigm, while keeping (and in some cases improving) the performance of an
  25. event-based paradigm.
  26. How does it change the game?
  27. Mordor allows you to focus on performing a logical task, instead of deciding how
  28. to make that task conform to a specific threading/event model. Just because
  29. local disk I/O will block, and should be performed in a thread pool, and network
  30. I/O should be performed using an event based callback design, doesn't mean you
  31. can't do them both _in the same function_. Mordor allows you to do just that.
  32. For example, here's a complete program to read a file from disk, and send it to
  33. a socket on the network:
  34. #include <iostream>
  35. #include <mordor/socket.h>
  36. #include <mordor/streams/file.h>
  37. #include <mordor/streams/socket.h>
  38. #include <mordor/streams/transfer.h>
  39. using namespace Mordor;
  40. int main(int argc, char **argv)
  41. {
  42. if (argc != 3) {
  43. std::cerr << "usage: " << argv[0] << " <file> <destination>" << std::endl;
  44. return 1;
  45. }
  46. try {
  47. std::vector<Address::ptr> addresses = Address::lookup(argv[2], AF_UNSPEC, SOCK_STREAM);
  48. Socket::ptr socket = addresses[0]->createSocket();
  49. socket->connect(addresses[0]);
  50. Stream::ptr fileStream(new FileStream(argv[1], FileStream::OPEN, FileStream::READ));
  51. Stream::ptr socketStream(new SocketStream(socket));
  52. transferStream(fileStream, socketStream);
  53. } catch (...) {
  54. std::cerr << boost::current_exception_diagnostic_information() << std::endl;
  55. return 2;
  56. }
  57. return 0;
  58. }
  59. This program is quite simple. It checks for usage, translates the string
  60. argument into a network address, creates a socket that is compatible with that
  61. address, connects to it, opens a file (as a stream), wraps the socket in a
  62. stream, and then sends the file over the socket. If an error occurs, complete
  63. error information is printed on stdout, including the type of error, the OS
  64. level error code and description (if applicable), and a complete stacktrace of
  65. the error, including debug symbol information, if available. Looking at it, we
  66. can see that there is only a single thread. Which is all fine and dandy if
  67. this is all we're doing. But what if instead we were sending 1000 files to
  68. 1000 different sockets, but didn't want to create a thread for each one? Let's
  69. say we want one thread for communicating with the network, and four threads for
  70. reading the file off the disk. Let's do it!
  71. #include <iostream>
  72. #include <mordor/iomanager.h>
  73. #include <mordor/scheduler.h>
  74. #include <mordor/socket.h>
  75. #include <mordor/streams/file.h>
  76. #include <mordor/streams/socket.h>
  77. #include <mordor/streams/transfer.h>
  78. using namespace Mordor;
  79. static void doOne(const char *file, const char *destination, IOManager &ioManager, Scheduler &scheduler)
  80. {
  81. try {
  82. std::vector<Address::ptr> addresses = Address::lookup(destination, AF_UNSPEC, SOCK_STREAM);
  83. Socket::ptr socket = addresses[0]->createSocket(ioManager);
  84. socket->connect(addresses[0]);
  85. Stream::ptr fileStream(new FileStream(file, FileStream::READ, FileStream::OPEN, &ioManager, &scheduler));
  86. Stream::ptr socketStream(new SocketStream(socket));
  87. transferStream(fileStream, socketStream);
  88. } catch (...) {
  89. std::cerr << boost::current_exception_diagnostic_information() << std::endl;
  90. }
  91. }
  92. int main(int argc, char **argv)
  93. {
  94. if (argc % 2 != 1) {
  95. std::cerr << "usage: " << argv[0] << " (<file> <destination>)*" << std::endl;
  96. return 1;
  97. }
  98. IOManager ioManager;
  99. WorkerPool workerPool(4, false);
  100. for (int i = 1; i < argc; i += 2)
  101. ioManager.schedule(boost::bind(&doOne, argv[i], argv[i + 1], boost::ref(ioManager), boost::ref(workerPool)));
  102. ioManager.dispatch();
  103. return 0;
  104. }
  105. So we re-factored most of main into doOne, but other than that it is nearly
  106. identical. And it will transfer as many files as you pass on the command line
  107. in parallel. Using 5 threads. The IOManager is the object used for
  108. non-blocking network I/O, and so is passed to the Socket when it is created.
  109. WorkerPool is just a generic thread pool, and is passed to the FileStream so
  110. that it will automatically do its work on those threads, instead of the thread
  111. it is running on when it is called. Something to point out here is that when
  112. the work is scheduled on the IOManager, each bit of work implicitly creates a
  113. Fiber - a lightweight, cooperatively scheduled, user-mode thread. The doOne
  114. function is executed on its own Fiber, and is allowed to switch which thread it
  115. is running on (inside of FileStream), without having to do any callbacks,
  116. virtual functions on a defined interface, or anything else. Internally, when
  117. FileStream wants to execute on the thread pool, it suspends the current Fiber,
  118. allowing other Fibers to run on this thread, and is resumed on a thread in the
  119. WorkerPool. IOManager and WorkerPool both inherit from Scheduler, which is the
  120. base functionality for cooperatively scheduling Fibers. Pretty cool, eh?
  121. Dependencies
  122. boost 1.40
  123. OpenSSL
  124. Zlib
  125. Ragel (compile-time only)
  126. Compiling for iPhone SDK
  127. The iPhone SDK does not include OpenSSL headers or binaries. Since mordor relies
  128. on OpenSSL, you must provide these files yourself. The Xcode project file is
  129. configured to look for files in an iphone directory in the same directory as the
  130. project file.
  131. Specifically, the compiler will look for headers in iphone/include/ and
  132. libraries in iphone/lib/. We recommend you create a symbolic link called
  133. "iphone" which points to the actual directory containing the include/ and lib/
  134. directories.
  135. License
  136. Mordor is licensed under the New BSD License, and Copyright (c) 2009, Decho Corp.
  137. See LICENSE for details.
  138. Authors
  139. Cody Cutrer (cody@mozy.com)
  140. Patrick Bozeman (peb@mozy.com)
  141. Jeremy Stanley (jeremy@mozy.com)
  142. Zach Wily (zach@mozy.com)
  143. Brian Palmer (brian@mozy.com)