/doc/decorators/jobs_limit.rst

 1.. include:: ../global.inc
 2.. _decorators.jobs_limit:
 3.. index:: 
 4    pair: @jobs_limit; Syntax
 5
 6See :ref:`Decorators <decorators>` for more decorators
 7
 8
 9########################
10@jobs_limit
11########################
12
13.. |maximum_num_of_jobs| replace:: `maximum_num_of_jobs`
14.. _maximum_num_of_jobs: `decorators.jobs_limit.maximum_num_of_jobs`_
15.. |name| replace:: `name`
16.. _name: `decorators.jobs_limit.name`_
17
18*****************************************************************************************************************************************
19*@jobs_limit* ( |maximum_num_of_jobs|_, [ |name|_ ])
20*****************************************************************************************************************************************
21    **Purpose:**
22        | Manages the resources available for a task.
23        | Limits the number of concurrent jobs which can be run in parallel for this task
24        | Overrides the value for ``multiprocess`` in :ref:`pipeline_run <pipeline_functions.pipeline_run>`
25        | If an optional ``name`` is given, the same limit is shared across all tasks with the same @job_limit name.
26        
27        
28    **Parameters:**
29                
30.. _decorators.jobs_limit.maximum_num_of_jobs:
31
32                
33    * *maximum_num_of_jobs*
34       The maximum number of concurrent jobs for this task. Must be an integer number
35       greater than or equal to 1.
36                
37.. _decorators.jobs_limit.name:
38
39    * *name*
40       Optional name for the limit. All tasks with the same name share the same limit if they
41       are running concurrently.
42
43    **Example**
44        ::
45    
46            from ruffus import *
47            
48            # make list of 10 files
49            @split(None, "*.stage1")
50            def make_files(input_file, output_files):
51                for i in range(10):
52                    open("%d.stage1" % i, "w")
53                
54            @jobs_limit(2)
55            @transform(make_files, suffix(".stage1"), ".stage2")
56            def stage1(input_file, output_file):
57                open(output_file, "w")
58    
59            @transform(stage1, suffix(".stage2"), ".stage3")
60            def stage2(input_file, output_file):
61                open(output_file, "w")
62
63            pipeline_run([stage2], multiprocess = 5)
64
65        will run the 10 jobs of ``stage1`` 2 at a time, while `` stage2`` will
66        run 5 at a time (from ``multiprocess = 5``):
67        
68        .. image:: ../images/jobs_limit.png
69        
70        
71