/doc/decorators/transform.rst

  1.. include:: ../global.inc
  2.. _decorators.transform:
  3.. index:: 
  4    pair: @transform; Syntax
  5
  6See :ref:`Decorators <decorators>` for more decorators
  7
  8########################
  9@transform
 10########################
 11
 12.. |tasks_or_file_names| replace:: `tasks_or_file_names`
 13.. _tasks_or_file_names: `decorators.transform.tasks_or_file_names`_
 14.. |extra_parameters| replace:: `extra_parameters`
 15.. _extra_parameters: `decorators.transform.extra_parameters`_
 16.. |output_pattern| replace:: `output_pattern`
 17.. _output_pattern: `decorators.transform.output_pattern`_
 18.. |matching_regex| replace:: `matching_regex`
 19.. _matching_regex: `decorators.transform.matching_regex`_
 20.. |suffix_string| replace:: `suffix_string`
 21.. _suffix_string: `decorators.transform.suffix_string`_
 22
 23*********************************************************************************************************************************************************************************************************************
 24*@transform* ( |tasks_or_file_names|_, :ref:`suffix<decorators.suffix>`\ *(*\ |suffix_string|_\ *)*\ | :ref:`regex<decorators.regex`\ *(*\ |matching_regex|_\ *)*\ , |output_pattern|_, [|extra_parameters|_,...] )
 25*********************************************************************************************************************************************************************************************************************
 26    **Purpose:**
 27        Applies the task function to transform data from input to output files.
 28
 29        Output file names are determined from |tasks_or_file_names|_, i.e. from the output
 30        of specified tasks, or a list of file names. 
 31
 32        This can be either via matches to the end of the file name (suffix matches) or, more
 33        flexibly, using regular expression pattern substitutions.
 34
 35        Only out of date tasks (comparing input and output files) will be run
 36        
 37    **Simple Example**
 38
 39        Transforms ``*.c`` to ``*.o``::
 40    
 41            @transform(["1.c", "2.c"], suffix(".c"), ".o")
 42            def compile(infile, outfile):
 43                pass
 44    
 45        Same example with a regular expression::
 46            
 47            @transform(["1.c", "2.c"], regex(r".c$"), ".o")
 48            def compile(infile, outfile):
 49                pass
 50
 51        Both result in the following function calls:
 52
 53            ::
 54
 55                # 1.c -> 1.o
 56                # 2.c -> 2.o
 57                compile("1.c", "1.o")
 58                compile("2.c", "2.o")
 59
 60
 61    **Escaping regular expression patterns**
 62
 63        A string like ``universal.h`` in ``add_inputs`` will added *as is*. 
 64        ``r"\1.h"``, however, performs suffix substitution, with the special form ``r"\1"`` matching everything up to the suffix.
 65        Remember to 'escape' ``r"\1"`` otherwise Ruffus will complain and throw an Exception to remind you.
 66        The most convenient way is to use a python "raw" string.
 67
 68    **Parameters:**
 69                
 70.. _decorators.transform.tasks_or_file_names:
 71
 72    * *tasks_or_file_names*
 73       can be a:
 74
 75       #.  Task / list of tasks (as in the example above).
 76            File names are taken from the output of the specified task(s)
 77       #.  (Nested) list of file name strings.
 78            File names containing ``*[]?`` will be expanded as a |glob|_.
 79             E.g.:``"a.*" => "a.1", "a.2"``
 80
 81.. _decorators.transform.suffix_string:
 82
 83    * *suffix_string*
 84       must be wrapped in a :ref:`suffix<decorators.suffix>` indicator object.
 85       The end of each input file name which matches ``suffix_string`` will be replaced by ``output_pattern``.
 86
 87       Input file names which do not match suffix_string will be ignored
 88
 89
 90       The non-suffix part of the match can be referred to using the ``"\1"`` pattern. This
 91       can be useful for putting the output in different directory, for example::
 92    
 93            
 94            @transform(["1.c", "2.c"], suffix(".c"), r"my_path/\1.o")
 95            def compile(infile, outfile):
 96                pass
 97
 98       This results in the following function calls:
 99
100            ::
101
102                # 1.c -> my_path/1.o
103                # 2.c -> my_path/2.o
104                compile("1.c", "my_path/1.o")
105                compile("2.c", "my_path/2.o")
106
107       For convenience and visual clarity, the  ``"\1"`` can be omitted from the output parameter.
108       However, the ``"\1"`` is mandatory for string substitutions in additional parameters, ::
109    
110            
111            @transform(["1.c", "2.c"], suffix(".c"), [r"\1.o", ".o"], "Compiling \1", "verbatim")
112            def compile(infile, outfile):
113                pass
114
115       Results in the following function calls:
116
117            ::
118
119                compile("1.c", ["1.o", "1.o"], "Compiling 1", "verbatim")
120                compile("2.c", ["2.o", "2.o"], "Compiling 2", "verbatim")
121
122       Since r"\1" is optional for the output parameter, ``"\1.o"`` and ``".o"`` are equivalent. 
123       However, strings in other parameters which do not contain r"\1" will be included verbatim, much
124       like the string ``"verbatim"`` in the above example.
125
126
127
128    
129.. _decorators.transform.matching_regex:
130
131    * *matching_regex*
132       is a python regular expression string, which must be wrapped in
133       a :ref:`regex<decorators.regex>`\  indicator object
134       See python `regular expression (re) <http://docs.python.org/library/re.html>`_ 
135       documentation for details of regular expression syntax
136       Each output file name is created using regular expression substitution with ``output_pattern``
137
138.. _decorators.transform.output_pattern:
139
140    * *output_pattern*
141       Specifies the resulting output file name(s).
142                
143.. _decorators.transform.extra_parameters:
144
145    * [*extra_parameters, ...*]
146       Any extra parameters are passed to the task function.
147       
148       If ``regex(matching_regex)`` parameter is used, then regular expression substitution
149       is first applied to (even nested) string parameters. Other data types are passed
150       verbatim.
151       
152       For example::
153       
154             @transform(["a.c", "b.c"], regex(r"(.*).c"), r"\1.o", r"\1")
155             def compile(infile, outfile):
156                 pass
157                 
158       will result in the following function calls::
159       
160            compile("a.c", "a.o", "a")
161            compile("b.c", "b.o", "b")
162                   
163
164
165
166See :ref:`here <decorators.transform_ex>` for more advanced uses of transform.