yaCF source to source tool main task is to make transformations on source codes, from simple high level transformations (like converting matrix to vectors) to complex source translation (like translating OpenMP to CUDA). To achieve this wide variety of transformations, yaCF uses two code patterns: Filter and Mutator. To ease the implementation of transformations and translations, some additional tools are presented. A set of Tree Manipulation Tools are declared on the Tools.Tree. A Source code storage system, capable of managing multiple source files used during the translation, is declared on Tools.SourceStorage.
A Filter is a class which follows the Visitor Pattern. It contains methods to do searches on the Internal Representation, following different orders. All Filters inherits from GenericFilterVisitor.
Creating a filter is as easy as defining a class with the name of the filter, inheriting from the GenericFilterVisitor, and overloading the constructor to define the condition function:
class ExampleFilter(GenericFilterVisitor):
""" Returns the first node matching the example node
"""
def __init__(self):
def condition(node):
if type(node) == whatever_ast.exampleNode:
return True
return False
super(ExampleFilter, self).__init__(condition_func = condition)
The condition function receive always a node parameter, which is the current node being visited, and returns True if a node matches the condition or False otherwise.
Several methods are implemented on GenericFilterVisitor. For example, if you want to get the first node matching a C Declaration, you can use the apply method. However, you can also use the iterator method to get all the matching node, strictly preserving the syntactic order of the code.
Faster methods, which does not guarantee a syntactical-correct tree traversal , are also implemented.
A Mutator is a class capable of making modifications to the Internal Representation. It contains a filter method, which searches for a node in the AST, and a mutatorFunction, which receive a matching node and do some kind of transformation.
All Mutator inherits from Backends.Common.Mutators.AbstractMutator.AbstractMutator, which contains common methods to apply mutations. Mutations can be applied on the first matching node returned by the filter, or can be applied iteratively to all matching nodes:
class ExampleMutator(AbstractMutator):
""" Apply an example mutation"""
def __init__(self, *args, **kwargs):
super(ExampleMutation, self).__init__()
def filter(self, ast):
""" Call a non iterable filter"""
raise NotImplemented
def filter_iterator(self, ast):
""" Call a non interable fast filter """
return NotImplemented
def fast_filter(self, ast):
""" Fast filter , looking for binary expressions """
return LoopInterchangeFilter().dfs_iter(ast)
def mutatorFunction(self, ast):
modify_ast(ast)
return second_loop
Mutators always modify the ast. In addition Mutators are expected to maintain the consistency of the Internal Representation (Symbol table, parent links, etc).
Some kind of transformations require that previously written code patterns are filled and inserted in the AST. Being this a common task, we have implemented extra classes in the yaCF framework to ease the process.
yaCF is currently using Mako Template Engine as template system. This allows developer to express complex modification of destination code using information of the original source. Following is a simplified version of the CUDA kernel template:
__global__ void ${kernelName} (
${', '.join(str(var.type) + " * reduction_cu_" + str(var.name) for var in reduction_vars)}
%if len(reduction_vars) > 0 and len(shared_vars) > 0:
,
%endif
${write_as_parameter_definition(shared_vars)}
)
{
int idx = blockIdx.x * blockDim.x + threadIdx.x;
${loop_vars[0].declaration} ${loop_vars[0]} = idx;
%for var in private_vars:
${var.declaration} ${var};
%endfor
## We need to check if idx is inside limits (for the case we have more threads than iterations)
if ((${loop.init.rvalue} < i) && (${loop.cond}))
${loop.stmt};
}
The example shows different common constructions in templates:
when instantiating the template.
as the result of the code is a string. The environment of this code is the template environment, defined in the template instantiation. Note that declarations from the mutator code are not available here.
writing process, as the %for loop or the conditional. The behaviour of these constructs is similar to their equivalent in python.
template.
The class AbstractMutator contains a parse_snippet method, which receive a template string and a dictionary of template variable substitutions, and returns the Internal Representation of the parsed template.
To parse a template, the user needs to specify all the template variables of the template:
self.parse_snippet(template_code, {'reduction_vars' : reduction_vars,
'shared_vars' : shared_vars}, name = 'Retrieve', show = False)
The name and show variables are only for debugging. In case there is an error in the template, an exception with the template name is shown. Also, if show is True, the template after substitution is printed on the standard output.
Notice that the code inside the snippet must be syntactically valid. Otherwise it won’t parse. This usually makes necessary the creation of fake functions to store the code, which are ignored after the template is parsed.
Most of the time, specially when dealing with pragmas (i.e OpenMP clauses), the templates are filled with information from the clauses. This implies extracting information for declaration lists, and manipulating the declarations in the template. As this is a quite common operation in backends, we have implemented a function called get_template_array.
The list returned by the function contains TemplateVarNode instead of declarations. This nodes can be called directly from templates, to get their string representation, or using the methods of the TemplateVarNode to get more information (like the declarations). In addition, when constructing TemplateVarNode you can specify functions that modify some aspects of the node. This function can make modifications to the names of the functions, to the types, or add debugging information.