/doc/decorators/jobs_limit.rst

https://code.google.com/p/ruffus/ · ReStructuredText · 71 lines · 48 code · 23 blank · 0 comment · 0 complexity · d688d9a9e2a86c438651534cda643cba MD5 · raw file

  1. .. include:: ../global.inc
  2. .. _decorators.jobs_limit:
  3. .. index::
  4. pair: @jobs_limit; Syntax
  5. See :ref:`Decorators <decorators>` for more decorators
  6. ########################
  7. @jobs_limit
  8. ########################
  9. .. |maximum_num_of_jobs| replace:: `maximum_num_of_jobs`
  10. .. _maximum_num_of_jobs: `decorators.jobs_limit.maximum_num_of_jobs`_
  11. .. |name| replace:: `name`
  12. .. _name: `decorators.jobs_limit.name`_
  13. *****************************************************************************************************************************************
  14. *@jobs_limit* ( |maximum_num_of_jobs|_, [ |name|_ ])
  15. *****************************************************************************************************************************************
  16. **Purpose:**
  17. | Manages the resources available for a task.
  18. | Limits the number of concurrent jobs which can be run in parallel for this task
  19. | Overrides the value for ``multiprocess`` in :ref:`pipeline_run <pipeline_functions.pipeline_run>`
  20. | If an optional ``name`` is given, the same limit is shared across all tasks with the same @job_limit name.
  21. **Parameters:**
  22. .. _decorators.jobs_limit.maximum_num_of_jobs:
  23. * *maximum_num_of_jobs*
  24. The maximum number of concurrent jobs for this task. Must be an integer number
  25. greater than or equal to 1.
  26. .. _decorators.jobs_limit.name:
  27. * *name*
  28. Optional name for the limit. All tasks with the same name share the same limit if they
  29. are running concurrently.
  30. **Example**
  31. ::
  32. from ruffus import *
  33. # make list of 10 files
  34. @split(None, "*.stage1")
  35. def make_files(input_file, output_files):
  36. for i in range(10):
  37. open("%d.stage1" % i, "w")
  38. @jobs_limit(2)
  39. @transform(make_files, suffix(".stage1"), ".stage2")
  40. def stage1(input_file, output_file):
  41. open(output_file, "w")
  42. @transform(stage1, suffix(".stage2"), ".stage3")
  43. def stage2(input_file, output_file):
  44. open(output_file, "w")
  45. pipeline_run([stage2], multiprocess = 5)
  46. will run the 10 jobs of ``stage1`` 2 at a time, while `` stage2`` will
  47. run 5 at a time (from ``multiprocess = 5``):
  48. .. image:: ../images/jobs_limit.png