Traditionally, compilers have been designed with a layer-oriented approach. Source file is processed in several passes, each one of them is closer to the destination platform. We follow a similar approach within yaCF. Original source file is parsed in the Frontend into the Intermediate Representation (IR), destination-independent transformation are applied in MiddleEnd module and finally, code generation and platform dependant information are grouped in Backends.
To glue these part together, yaCF developers have to write a driver. A driver is a python script who drives the transformation process.
Diagram layered_design showcases the framework layer-oriented architecture.
Uppermost level, Frontend, groups the input-oriented classes. Among them, the Parsing infraestructure , the Symbol Table and the IR are highlighted.
The MiddleEnd module encapsulates local transformations to the IR, for example, loop analysis and transformations, Data Dependency Analysis, Code Outlining, etc.
Finally Backends module contains the classes that allow translations from the IR to different target architectures.
Support classes and methods are available under Tools. Tools to Insert, Remove or Update different stages are stored under Tools.Tree. A Source code storage system, usable in different steps of a source to source transformations, is defined on Tools.SourceStorage.
Documentation have been built using the sphinx system with docstrings along all yaCF sources. Example drivers are available under the bin directory of the main package. C source files for this drivers are available under the examples directory.
When creating source to source transformations, developers usually pretend either to search for a particular pattern in the original source and transform it to another code or to fully translate an input source code to another destination language.
yaCF have been designed to easy implement these two kinds of source to source operations. Starting from the IR, users can write drivers to generate a completely different destination language implementing a Writer. Writers are implementations of the widely known Visitor pattern. A Writer visit all nodes of the IR generating a text output. C99 and OpenMP Writers are available. These writers re-generate the original C99 code from the IR.
For the cases were users intend to search and replace patterns in the original source, two operations are widely used: (1) Retrieving information from the IR and (2) Partial modification of IR. Two patterns have been designed for this purposes:
returning the information matching the specified criteria.
in the context of yaCF the transformation of the IR usually involves heavy changes to the original IR. We rather prefer to name the pattern as Mutator to delve into the abnormal nature of this changes.
Abstract base classes implementing this patterns are available within yaCF. Specific Filters and Mutators might be implemented derived appropriate base classes. Implementing nested Filters and Mutators allows to design source to source transformations following a divide and conquer approach.
When implementing a fully featured compiler Backend, several Filters and Mutators are required to be applied in a particular order, and several files might be involved. To encapsulate this work, a Runner class is instantiated by compiler drivers to start the Backend code generation.
A design pattern is a general reusable solution to a recurring problem within a given context in software design. It is not a finished design ready to be implemented, but a description or template for a particular solution of a problem. A design pattern might be suitable for several different situations, and have to be instantiated into a particular implementation to be useful. In Object-oriented programming, design patterns are commonly used to show relationships and interactions between classes or objects, without defining specific implementations. Design patterns are designed at a high abstraction level, an usually represents relations and interconection between modules or classes.
Internal Representation of yaCF is based on an annotated high-level tree-layered AST (see section XXX for a more specific description). To manipulate this Internal Representation, we heavily use the Visitor pattern. It is important to understand how this pattern work in order to be able to understand most of the concepts behind source transformations in yaCF.
In object-oriented programming and software engineering, the visitor design pattern is a way of separating an algorithm from an object structure on which it operates. A practical result of this separation is the ability to add new operations to existing object structures without modifying those structures.
The Visitor pattern implementation used in this framework have been based on the original pycparser (ref XX) examples, which in turns are based on the ast.NodeVisitor implementation from the python library.
Filter pattern is a particular implementation of the Visitor pattern focused in aspects of IR manipulation. GenericFilterVisitor is the base class which implements the filter in a top-down fashion. An alternative implementation of the filter which transverses the shelves to the root is also available on AbstractReverseVisitor.
Create a Filter is just a matter of creating a class inheriting from the aforementioned GenericFilterVisitor and overload its constructor defining the condition_func parameter:
class ExampleFilter(GenericFilterVisitor):
""" Returns the first node matching the example node
"""
def __init__(self):
def condition(node):
if type(node) == whatever_ast.exampleNode:
return True
return False
super(ExampleFilter, self).__init__(condition_func = condition)
It is also possible to implement a Filter passing directly a function definition, or even a lambda function:
my_filter = GenericFilterVisitor(condition_func = lambda x : x.name == "Name")
However, the previous approach is more powerful, as it allows to implement special visit methods to perform additional actions in specific IR nodes. For example, suppose we want to look for a IR node which is inside a particular construct:
class ExampleFilter(GenericFilterVisitor):
""" Returns the first node matching the example node
"""
def __init__(self):
self._visitedWhatever = False
def condition(node):
if isinstance(node, Something) and self._visitedWhatever:
return True
return False
super(ExampleFilter, self).__init__(condition_func = condition)
def visit_Whatever(self,node):
self._visitedWhatever = True
In this way, if a node of type Something is under a Whatever node , it will be returned by the filter.
Condition function always receive a node parameter, which is the current node being visited, and returns True if it is a valid node or False if it is not. It is important to note that condition function is evaluated before visiting the node.
GenericFilterVisitor implements a wide variety of methods. A basic apply method looks for the first node matching the criteria in strictly grammatical order. An iterator node iterates over all potential matches, also strictly preserving the syntactic order.
Other iteration and search methods, some of them without guarantee to preserve the proper syntactic order, have been implemented too.
A Mutator is a class capable of making modifications to the Intermediate Representation. It contains a filter method, which searches for nodes in the AST, and a mutatorFunction, which receive a matching node and does some kind of transformation.
All Mutators inherit from Backends.Common.Mutators.AbstractMutator.AbstractMutator, which contains common methods to all code mutations. Mutations can be applied on the first matching node returned by the filter, or can be applied iteratively to all matching nodes:
class ExampleMutator(AbstractMutator):
""" Apply an example mutation"""
def __init__(self, *args, **kwargs):
super(ExampleMutation, self).__init__()
def filter(self, ast):
""" Call a non iterable filter"""
raise NotImplemented
def filter_iterator(self, ast):
""" Call an iterable fast filter """
return NotImplemented
def fast_filter(self, ast):
""" Fast filter , looking for binary expressions """
return LoopInterchangeFilter().dfs_iter(ast)
def mutatorFunction(self, ast):
modify_ast(ast)
return second_loop
Mutators always modify the AST. In addition Mutators are expected to keep the consistency of the Intermediate Representation (Symbol table, parent links, etc).
Note
When implementing mutators, take care of the kind of transformation you are applying to the AST. If you modify the IR in such a way that the next iteration might match the condition again, an infinite loop appears. For example: Consider a mutator that looks for nodes with a specific name (let the name be foo), and adds a parent to that node, changing the current visited node. This will produce an infinite loop, because in the next iteration, the actual node will match again foo, thus, repeating the process over and over.
It is common within source to source transformations that some code snippets from the target code be filled with data from the original source code. To avoid the usage of string concatenation, which could potentially lead to unreadable code, yaCF implements a template system that increases the readability of these code snippets and its expressivity.
yaCF is currently using Mako Template Engine as templating language. The template system allows the developer to express complex modifications of destination code using information coming from the original source. Following is an example template:
<%
from Backends.Common.TemplateEngine.Functions import decl_of_id
%>
int ${functionName} ( ${','.join(loop_parameters['inside_vars'])} ) {
## This function is just a test
%for elem in loop_parameters['inside_vars']:
${decl_of_id(elem)}
%endfor
}
In the example, we observe different common constructions in templates:
when instantiating the template.
as the result of the code is a string. The environment of this code is the template environment, defined in the template instantiation.
writing process, as the %for loop or the conditional. The behaviour of these constructs is similar to their equivalent in Python.
template.
The class AbstractMutator contains a parse_snippet method, which receive a template string and a dictionary of template variable substitutions, and returns the Intermediate Representation of the parsed template.
To parse a template, the user has to specify all the template variables of the template:
self.parse_snippet(template_code, {'reduction_vars' : reduction_vars,
'shared_vars' : shared_vars}, name = 'Retrieve', show = False)
The name and show variables are only for debugging purposes. In case of template error, an exception with the template name is shown. Also, if show is set to True, the template with its parameters substituted is printed on the standard output.
Notice that the code inside the snippet must be syntactically valid. Otherwise it won’t parse, and the Frontend will produce a syntactic error.
This usually makes necessary the creation of fake functions to store the code, which are ignored after the template is parsed.
Deprecated since version 1.0.4: This is now task of the analysis module (MiddleEnd.Loop.Analysis).
Most of the time, particularly when dealing with pragmas (i.e OpenMP clauses), the templates are filled with information from the clauses. This implies extracting information from declaration lists, and manipulating the declarations in the template. As this is a quite common operation in backends, we have implemented a function called get_template_array.
Deprecated since version 1.0.4: The new Backend system does not use this ugly class.
The list returned by the function contains TemplateVarNode instead of declarations. These nodes can be called directly from templates, to get their string representation, or using the methods of the TemplateVarNode to get more information (like from declarations). In addition, when constructing TemplateVarNode you can specify functions that modify some aspects of the node. These functions can make modifications to the names of the functions, to the types, or add debugging information.