Email templating in Azure Data Factory

Welcome to the first (official) entry of the Function Automate series. In a previous entry we wielded our new found power, a.k.a. Azure Function Apps, to send emails from Data Factory. However, this is only the tip of the iceberg, we can easily execute code within a pipeline and we have the excellent Python SDK libraries from the fine folks at Microsoft to integrate our solutions with the rest of the Azure ecosystem. You can find all the code in GitHub.

In this entry we will simplify the process of building decent looking emails by using Jinja2 templates. Jinja templates are essentially just text files with named placeholders in to which we can substitute values easily. Well, in reality, they can do quite a bit more than that and in this entry we will look at some of the possibilities.

The Code

Or really the lack of it. All that today's function is going to do is retrieve the template file from the storage account associated with the Function App and interpolate in it the values of the incoming request.

Here is main function in all its glory

def main(req):
    template_parameters = get_param(req, "template_parameters")
    template_file = get_param(req, "template_file")
    share_name = get_param(req, "share_name")

    template = get_template(
        conn_str=os.environ["AzureWebJobsStorage"],
        share_name=share_name,
        template_path=template_file,
    )

    completed_template = template.render(template_parameters)

    return func.HttpResponse(json.dumps({"output_text": completed_template}))

All we need to provide to our API through a request are 3 parameters. Where the template_file is within the share, the share_name itself, and last but not least, template_parameters, a dictionary with the parameters that the target template uses.

That is really all there is to the function. The only thing we have left to do is to define the template.

Jinja Templates

At its simplest, Jinja2 templates are just plain text files which have named place holders to subsitute values into them. For example:

Hello {{name}}, can we meet in {{number_of_days}} days.</inc></pre>

The parameters of the template are then passed as a dictionary to the template's render method.

However, Jinja templates are capable of much more, from allowing looping structure to dynamically build lists to filtering and manipulating the input. Here I will barely scratch the surface, just enough to demonstrate the functionality of the new function.

Here is the template I will use for the HTML emails we will be sending out.

This template uses the simple interpolation described above, but also demonstrates how to make a shopping list or filter the most critical issues that your DB is facing.

The body of HTTP request could for example look like this:

{
  "share_name":"blog",
  "template_file":"ADF/email_templates/template.html.jinja2",
  "template_parameters": {
    "name":"John",
    "veggies":["Tomatoes", "Cabbage"],
    "warnings":[
        "Info: A new row was inserted.",
        "Error: The server is literrally on fire!",
    ]
  }
}

Setting Up

Just git clone the repo and publish into your Function app. The repository also includes the function that we developed before to send the emails and which you will need to configure.

Next we need to store the email template in the storage account associated with the Function App so that we can retrieve them easily at run time. For this function I'm keeping the templates in a share file. Once that is out of the way, we only have to setup a test ADF pipeline. The one I've built look like this:

We are generating the email with our compose function and then feeding it into the send email one. The hardest part is to convince ADF to accept our dynamic content. I just follow the recommendation of someone that knows better than myself and build my dynamic content outside ADF and then copy it into the request body.

Here, we are using the output of the Compose Email activity which produced the email contents as the body for the email.

{
  "user":"gb",
  "subject":"Test Subject",
  "recipients":"test_email@gmail.com",
  "body":"@{string(activity('Compose Email').output.output_text)}"
}

Outro

As always you can find the code on GitHub. If you have any ideas for improvements please make a pull request. On the next iteration of Function Automate I'll look into how to trigger ADF pipelines from Python and how this can be leveraged to pause and restart pipelines.

Bye!