T4 Text Transformation in MetaSharp

by justin 28. May 2010 17:53

Previously in MetaSharp I had a custom Templating engine for textual transformations. It was a custom grammar and was pretty simple. It worked well and all was good but it was becoming a maintenance burden for me and, I think, turned people off to MetaSharp more than attracted them. So I decided to spend a little time and try to integrate T4 into my templating library as a replacement.

It was actually reasonably simple to accomplish. The hardest part, of course, was working with AppDomains. T4 is a dynamic tool living in a static world and as such it requires an AppDomain to do its magic so it can all be unloaded without a trace. Because of this T4 isn’t very amenable to input. Most examples I found online start with firing up your template then beginning by reading a file or creating a class that connects to a database. In my case I have already have my metadata and would like to pass it into a T4 template but it’s in memory only. The first thing you have to do is make your model Serializable.

Next you need to know about the “inherits” and the “hostspecific” attributes. You add something like this to the top of your template:

<#@ template language="C#" inherits="SongMetaTemplate" hostspecific="true" debug="true" #>

This is for my Song sample application. I then create a SongMetaTemplate class that knows how to cast the model into a Song enumeration.

public abstract class SongMetaTemplate : MetaTemplate
{
    public IEnumerable<Song> Songs
    {
        get { return ((IEnumerable<INode>)this.Model).Cast<Song>(); }
    }
}

Adding a hostspecific attribute to the template causes the template class that inherits from the SongMetaTemplate to also have a Host property which gets loaded up with the Model provided by my pipeline. The MetaTemplate knows how to fetch that host model for me.

public abstract class MetaTemplate : TextTransformation
{
    private object model = null;

    protected object Model
    {
        get
        {
            if (model == null)
            {
                dynamic metame = this;
                var metaHost = metame.Host as MetaSharpTextTemplatingHost;
                if (metaHost != null)
                    this.model = metaHost.Model;
            }

            return model;
        }
    }

    public override string TransformText()
    {
        return this.GenerationEnvironment.ToString();
    }
}

And that’s it! My template can now get a hold of the Song AST parsed from the DSL earlier in the pipeline. Here is an example template in its entirety:

<#@ template language="C#" inherits="SongMetaTemplate" hostspecific="true" debug="true" #>
<#@ assembly name="DynamicSongBuilder" #>
<#@ import namespace="DynamicSongBuilder" #>

<# foreach(var song in this.Songs) { #>
    <# foreach(var b in song.Bars) { #>
        <# foreach(var n in b.Notes) { #>
            play(<#=song.Duration #>, "<#=n.Key #>", <#=n.Octave #>);
        <# }
    }
} #>

The template code is C# but the code generated is actually for my own custom language. In this case it’s parsed into Linq expressions and compiled into a delegate using a context object with a play method. It’s then executed dynamically and also collected by the GC when dereferenced. I plan to replace the custom language also, but its proving difficult without a reasonable parser to replace it.

Tags:

.NET | C# | DSL | dynamic | MetaSharp | T4

First Steps Towards a Parser Generator

by justin 1. February 2010 15:54

I’ve been pretty quiet the last couple of months because I have been very busy at work and at home I have been slowly chipping away at a new Parser for MetaSharp.

It’s very late at night here so I’m not going to go into a lot of detail right now but suffice it to say that currently my Parser is completely hand written and right now I’m working on a Transform step that will let me generate Parsers from a grammar. The goal is to generate the Parser from the Grammar grammar!

I still have a long way to go but here is what I do have. The following Grammar:

grammar Simple < MetaSharp.Transformation.Parsing.Common.BasicParser:
    Alphabet = (A | B | C)+;
    A = "a";
    B = "b";
    C = "c";
end

Produces the following code:

//------------------------------------------------------------------------------
// <auto-generated>
//     This code was generated by a tool.
//     Runtime Version:4.0.21006.1
//
//     Changes to this file may cause incorrect behavior and will be lost if
//     the code is regenerated.
// </auto-generated>
//------------------------------------------------------------------------------

public partial class Simple : MetaSharp.Transformation.Parsing.Common.BasicParser
{

    internal new static System.Collections.Generic.IEnumerable<string> Imports;

    static Simple()
    {
        Simple.Imports = new string[0];
    }

    public Simple(MetaSharp.Transformation.IContext context) :
        base(context)
    {
        MetaSharp.Transformation.IRuleService ruleService =
            this.Context.Locate<MetaSharp.Transformation.IRuleService>();
        this.Add(
            newMetaSharp.Transformation.Patterns.OrPattern(
                ruleService.GetRule("Alphabet", Simple.Imports),
                ruleService.GetRule("A", Simple.Imports),
                ruleService.GetRule("B", Simple.Imports),
                ruleService.GetRule("C", Simple.Imports)));
    }
}

[MetaSharp.Transformation.RuleExportAttribute(typeof(AlphabetRule))]
public class AlphabetRule : MetaSharp.Transformation.Rule
{

    protected override void Initialize()
    {
        this.Body = new MetaSharp.Transformation.Patterns.OneOrMorePattern(
            new MetaSharp.Transformation.Patterns.OrPattern(
                new MetaSharp.Transformation.Patterns.OrPattern(
                    this.Rules.GetRule("A", Simple.Imports),
                    this.Rules.GetRule("B", Simple.Imports)),
                    this.Rules.GetRule("C", Simple.Imports)));
        base.Initialize();
    }
}

[MetaSharp.Transformation.RuleExportAttribute(typeof(ARule))]
public class ARule : MetaSharp.Transformation.Rule
{

    protected override void Initialize()
    {
        this.Body = new MetaSharp.Transformation.Patterns.StringPattern("a");
        base.Initialize();
    }
}

[MetaSharp.Transformation.RuleExportAttribute(typeof(BRule))]
public class BRule : MetaSharp.Transformation.Rule
{

    protected override void Initialize()
    {
        this.Body = new MetaSharp.Transformation.Patterns.StringPattern("b");
        base.Initialize();
    }
}

[MetaSharp.Transformation.RuleExportAttribute(typeof(CRule))]
public class CRule : MetaSharp.Transformation.Rule
{

    protected override voidInitialize()
    {
        this.Body = new MetaSharp.Transformation.Patterns.StringPattern("c");
        base.Initialize();
    }
}

It’s a pretty trivial grammar for now but this should give you an idea of where this is going. The RuleExportAttribute inherits from the MEF ExportAttribute and the concrete IRuleService uses MEF to supply rules using these attributes.

It’s exciting!

Tags: , , , ,

.NET | C# | CodeDom | MetaSharp | DSL

Surprisingly Difficult to Parse a Character

by justin 9. January 2010 00:52

I’ve been working on a custom variant of an OMeta parser for a couple weeks now. It’s coming along pretty well, I think I’ve overcome most of the major hurdles and I’m just trying to go through what I currently have, clean it up and get it to solve some of the edge cases that I need.

Just now I was working on the grammar for parsing a character and realized how hard it really is. It sounds trivial, afterall it’s just two single quotes and a character right? Wrong. Here’s my current grammar:

CharacterLiteralToken
    = '\'' '\\' 'u' Hex#4 '\''
    | '\'' '\\' 'U' Hex#8 '\''
    | '\'' '\\' 'x' Hex Hex? Hex? Hex? '\''
    | '\'' '\\' ('\'' | '\"' | '\\' | '0' | 'a' | 'b' | 'f' | 'n' | 'r' | 't' | 'v') '\''
    | '\'' '\u0000'..'\uffff' '\'';

It turns out that you have to be sure to account for a multitude of escape characters as well as escaped Unicode literals. I didn’t want to have to implement this, but you can see the last rule which just matches every character under the sun needed it.

This will match:

  • '\u0000'
  • '\U00000000'
  • '\x0', '\x00', '\x000', '\x0000'
  • '\'', '\"', '\\', '\0', '\a', '\b', '\f', '\n', '\r', '\t', '\v'
  • 'a' …

Next I get to do the string parser… that should be even more interesting.

Tags: , ,

DSL | MetaSharp | languages

About Me

sweetest hat ever

I'm a software developer from Minnesota and this blog largely focuses on various technical concepts I am thinking about at the moment. I currently work for Microsoft in the St. Paul office of the Expression product group.

RecentPosts