let linkIsValid () =
  // Simulate link checking by sleeping for a second
  System.Threading.Thread.Sleep(TimeSpan.FromSeconds(1.))

  true

To measure this function’s execution time, we’ll use the Stopwatch class, which is designed specifically for this purpose:

// This namespace contains the Stopwatch class
open System.Diagnostics

// Create the stopwatch and immediately start measuring time
let stopwatch = Stopwatch.StartNew()

// Execute the function we want to time
let isValid = linkIsValid()

// Stop measuring time
stopwatch.Stop()

// Output the measured (elapsed) time
printfn "%A" stopwatch.Elapsed

If we run this code, the output will look like this:

00:00:01.0155117

To allow re-using the link timing functionality, we’ll encapsulate the above code in a function which returns the link check result and the elapsed time as a tuple:

let linkIsValidWithTiming() =
  let stopwatch = Stopwatch.StartNew()
  let linkStatus = linkIsValid()
  stopwatch.Stop()

  // Return the link status _and_ the elapsed time
  linkStatus, stopwatch.Elapsed

let isValid, elapsed = linkIsValidWithTiming()
printfn "%A" elapsed

Generalizing

What if we want to mesure the execution time of other functions? Well, we can generalize the linkIsValidWithTiming function to work with other functions by taking the function to time as a parameter:

let executeAndTime func =
  let stopwatch = Stopwatch.StartNew()

  // Call the `func` parameter (which is a function)
  let funcResult = func()

  stopwatch.Stop()

  funcResult, stopwatch.Elapsed

We can then pass in the linkIsValid function as an argument to get its result and its timing information:

let isValid, elapsed = executeAndTime linkIsValid
printfn "%A" elapsed

Timing type

To make our code more explicit, let’s define a type to hold timing results:

type Timed<'T> =
  { Result: 'T
    Elapsed: TimeSpan }

We can then return this type from our executeAndTime function:

let executeAndTime func =
  let stopwatch = Stopwatch.StartNew()
  let result = func()
  stopwatch.Stop()

  { Result = result; Elapsed = stopwatch.Elapsed }

let timedResult = executeAndTime linkIsValid
printfn "%A" timedResult.Elapsed

To me, this version has better motivational transparency ¹ as it makes it more explicit what the code is doing.

Timing asynchronous functions

To allow for non-blocking link checking, let’s define an asynchronous version of the linkIsValid function:

let linkIsValidAsync () = async {
  do! Async.Sleep(1000)
  return true
}

If we time this function, the elapsed time seems to be off:

00:00:00.0010091

As the linkIsValidAsync function should run for at least 1000 milliseconds, this can’t be right. The problem is that we don’t wait for the async function to complete. This means that we’re not timing the execution of the function from begin to end, but only the time it takes to start executing. To ensure that the asynchronous function call blocks until it has completed, we can pipe it into Async.RunSynchronously:

let executeAndTime func =
  let stopwatch = Stopwatch.StartNew()
  let funcResult = func() |> Async.RunSynchronously
  stopwatch.Stop()

  { Result = funcResult; Elapsed = stopwatch.Elapsed }

Our asynchronous function is now timed correctly:

00:00:01.0390756

However, if we try to time the old, synchronous linkIsValid function, we get a compile error:

[FS0001] Type mismatch. Expecting a
  'unit -> Async<'a>'
but given a
  'unit -> bool'
The type 'Async<'a>' does not match the type 'bool'

The compiler informs us that Async.RunSynchronously expects an Async<'a> argument, but that it received a bool argument, which is indeed what the (synchronous) linkIsValid function returns.

To fix this, let’s revert back to our previous, synchronous version of the executeAndTime function (by removing Async.RunSynchronously). We can then pass in a lambda as the argument where we pipe the linkIsValidAsync result to Async.RunSynchronously:

let timedResult = executeAndTime (fun() -> linkIsValidAsync() |> Async.RunSynchronously)

Using this approach, we can time both synchronous and asynchronous functions, without the executeAndTime function having to be aware of the difference. The output confirms that we correctly time our asynchronous function:

00:00:01.0390756

Computation Expression

While the function-based approach works well, wouldn’t it be great if we could this instead?

let timedResult = timed {
  return linkIsValid()
}

This code looks very similar to the async syntax we used before in our linkIsValidAsync function:

let linkIsValidAsync () = async {
  do! Async.Sleep(1000)
  return true
}

As it turns out, the async syntax is provided by an F# feature called computation expressions, which provides a convenient syntax for computations that can be sequenced and combined. Besides the built-in computation expressions (like async), one can also define custom computation expressions, so let’s try and build a timed computation expression.

Computation expressions are implemented as classes, which are known as builder types. To get the nice syntax we saw above, one has to bind an instance of the builder type to an identifier. The name of the identifier determines how one can use the computation expression in code.

// Empty build class
type TimedBuilder() = class end

// Builder class instance. The name determines how to use the computation expression
let timed = TimedBuilder()

This will allow us to do:

let timedResult = timed {}

If we try to compile this code, we get can error:

[FS0003] This value is not a function and cannot be applied.

While the message is slightly cryptic, it tries to tell us that we’re not doing anything in our computation expression. This starts to make sense if you realize that computation expressions are translated to method invocations on the builder type instance (in our case: timed). Let’s try to mimic the async computation expression and return the result of calling the linkIsValid function:

let timedResult = timed {
  return linkIsValid()
}

This time, we get the following compile error:

[FS0708] This control construct may only be used if the computation expression builder defines a 'Return' method

This is actually very descriptive! It tells us that to be able to use the return keyword inside our custom computation expression, its builder (type) needs to define a Return method. If we check the documentation, the Return method should be of type 'T -> M<'T>. You should read this as: it takes a “regular” type and returns a computation expression-specific type that “wraps” the “regular” type. For our computation expression, our “wrapped” type is the Timed<'T> type. Let’s add the Return member to the TimedBuilder class:

type TimedBuilder() =
  member x.Return(value) = { Result = value; Elapsed = TimeSpan.Zero }

We can then use this as follows:

let timedResult = timed {
  return linkIsValid()
}
printfn "%A" timedResult.Elapsed

This time, the code runs successfully and outputs:

00:00:00

Let’s try and time the function execution:

type TimedBuilder() =
  member x.Return(value) =
    let stopwatch = Stopwatch.StartNew()
    let result = value()
    stopwatch.Stop()

    { Result = result; Elapsed = stopwatch.Elapsed }

As with our executeAndTime function, the Return method now expects its parameter to be a function, which means that the computation expression should not return the result of invoking the linkIsValid function, but return the function itself:

let timedResult = timed {
  return linkIsValid
}

And this works! Nice and clean, right?

Improved function handling

Our current implementation is not idiomatic though. When executing functions in a computation expression, the Delay member should be implemented. Its signature is (unit -> M<'T>) -> M<'T>, which means that it takes a function without parameters that returns the “wrapped” type and also returns the “wrapped” type. Let’s implement this method:

type TimedBuilder() =
  member x.Delay(func) =
    let stopwatch = Stopwatch.StartNew()
    let timedResult = func()
    stopwatch.Stop()

    { timedResult with Elapsed = timedResult.Elapsed + stopwatch.Elapsed }

The time measuring code is the same, the biggest change is that when we invoke the func parameter, we receive a Timed<'T> value. We then deconstruct this value and return a new Timed<'T> instance, but with the elapsed time as measured by the stopwatch added to the elapsed time returned by the function. If we try to compile this, we get:

[FS0708] This control construct may only be used if the computation expression builder defines a 'Return' method

Whoops. We’re still using the return keyword in our computation expression invocation, which means that we still have to implement the Return method. Note though that any value passed to the Return method is automatically wrapped in a “delay” function and passed to the Delay method to execute. We can use our original Return method implementation:

type TimedBuilder() =
  member x.Return(value) = { Result = value; Elapsed = TimeSpan.Zero }

  member x.Delay(func) =
    let stopwatch = Stopwatch.StartNew()
    let timedResult = func()
    stopwatch.Stop()

    { timedResult with Elapsed = timedResult.Elapsed + stopwatch.Elapsed }

We’ll have to change our computation expression to actually invoke the linkIsValid function again:

let timedResult =
  timed {
    return linkIsValid()
  }

And we’re back to things working. Note that we can now also return a constant value from our computation expression (although that admittedly doesn’t make much sense):

let timedResult =
  timed {
    return 2
  }

And with that, we have created a computation expression that elegantly allows us to measure function execution time using a nice and concise syntax.

Conclusion

Computation expressions are a great way to help make your code more expressive. While writing a computation expression takes a little getting used to, they are not that hard once you’re familiar with their implementation rules. The compiler is also quite helpful, outputting detailed error message that help pinpoint what methods to implement.

When using git, local branches can track remote branches that no longer exist (the remote branch is gone). To identify these branches, we first have to cleanup (prune) the remote’s branches:

$ git fetch -p
From https://test.com
 - [deleted]         (none)     -> origin/disable-feature-x
 - [deleted]         (none)     -> origin/fix-typo
 - [deleted]         (none)     -> origin/grammar-fix

In this case, three remote branches were deleted. Let’s see if we have local branches that are tracking deleted branches:

$ git branch -v
  fix-typo            7b57d4f [gone] Fix typo in README
  grammar-fix         01257bd [gone] Fix some bad grammar
* master              477010d Bump to version 1.1

There are three local branches, of which two (fix-typo and grammar-fix) are marked with [gone]. This indicates that these branches are indeed tracking remote branches that have been deleted.

Unfortunately, git does not have built-in functionality to cleanup these local branches. Our only option is to manually delete them through git branch -d <branch-name>. But what if there are many of these branches? Things would get tedious quickly, so let’s try to automate this!

Identifying the gone branches

The first step in our automation is to identify the branches to delete. A naive approach would be to parse the aforementioned output of git branch -v. However, this is problematic for the following reasons:

1. The output could change in the future. See this blog post by Junio C Hamano (git maintainer).
2. The output can be modified by the user. See this StackOverflow post.

Hmmm, let’s try a different approach. It so happens that the git for-each-ref command lets us specify its output format, neatly circumventing the aforementioned problems.

We can use git for-each-ref to list all branches and their upstream’s branch’s status in our desired format:

$ git for-each-ref --format '%(refname) %(upstream:track)'
refs/heads/bugfix-1
refs/heads/fix-typo [gone]
refs/heads/grammar-fix [gone]
refs/heads/master [behind 15]
refs/heads/new-function [ahead 2, behind 9]
refs/remotes/origin/HEAD
refs/remotes/origin/master
refs/remotes/origin/new-function
refs/stash

Now this is something we can work with! As we’re only interested in our local branches (heads), we’ll filter them by appending refs/heads to our command:

$ git for-each-ref --format '%(refname) %(upstream:track)' refs/heads
refs/heads/bugfix-1
refs/heads/fix-typo [gone]
refs/heads/grammar-fix [gone]
refs/heads/master [behind 15]
refs/heads/new-function [ahead 2, behind 9]

Nice. We can do one last optimization here, and that is to return the branches in their shortened format by using refname:short:

$ git for-each-ref --format '%(refname:short) %(upstream:track)' refs/heads
bugfix-1
fix-typo [gone]
grammar-fix [gone]
master [behind 15]
new-function [ahead 2, behind 9]

Great! We now have a reliable, consistent way to retrieve our local branches and their remote tracking status.

Deleting the gone branches

The next step is to filter the branches which remote branch is gone. We can do this by piping the output to awk, which can filter the branches and print their name (removing the remote tracking status):

$ git for-each-ref --format '%(refname:short) %(upstream:track)' |
  awk '$2 == "[gone]" {print $1}'
fix-typo
grammar-fix

Sweet! The final step is to pipe this output to xargs to delete the branches:

$ git for-each-ref --format '%(refname:short) %(upstream:track)' |
  awk '$2 == "[gone]" {print $1}' |
  xargs -r git branch -D

Deleted branch fix-typo (was 7b57d4f).
Deleted branch grammar-fix (was 01257bd).

And that’s it! We now have a single command to delete all local branches which remote tracking branches have been deleted.

Integrating with git

Our final step is to add our command as a git alias, which allows one to define custom commands that can be called as if they were built into git. Aliases in git are defined in the .gitconfig file found in your home dir. You can edit this file by running: git config --global --edit. Within that file, find the [alias] section (or add it), and add an alias to it named gone:

[alias]
  gone = ! "git fetch -p && git for-each-ref --format '%(refname:short) %(upstream:track)' | awk '$2 == \"[gone]\" {print $1}' | xargs -r git branch -D"

The gone alias does two things:

1. Run git fetch -p (to remove any deleted remote branches).
2. Run our custom command (to remove local branches with a deleted remote branch).

Having added our alias, we can now run git gone as if it was a built-in command:

$ git gone

Deleted branch fix-typo (was 7b57d4f).
Deleted branch grammar-fix (was 01257bd).

The behavior is exactly the same as manually running git fetch -p followed by our custom command. Nice, isn’t it?

You don’t even have to manually edit the .gitconfig file, as the alias can also be added using the following command:

git config --global alias.gone "! git fetch -p && git for-each-ref --format '%(refname:short) %(upstream:track)' | awk '\$2 == \"[gone]\" {print \$1}' | xargs -r git branch -D"

Interestingly, due to the way git is implemented on Windows, the above alias also works on Windows.

Conclusion

In this blog post, we’ve shown how to cleanup local git branches that are tracking remote branches that no longer exist. We did this by combining the git for-each-ref command with the awk and xargs commands. As a bonus, we added a git alias for our cleanup command that allows us to cleanup our local branches using git gone.

Introduction

In this blog post, we’ll build a website that can solve Reverse Polish Notation (RPN) equations. All code (including the website) will be written in F#, but the equations themselves will be solved using WebAssembly.

Reverse Polish Notation

In RPN equations, operators (+, -, etc.) follow their operands (1, 3, etc.). This is known as postfix notation. Its main advantage: parentheses are no longer needed to define precedence. Consider the following equation (in standard mathematical notation):

(1 + 3) * (5 - 2)

This is the same equation, but specified in RPN:

1 3 + 5 2 - *

To solve this RPN equation we can just interpret its parts as a series of stack operations:

• Push 1 on the stack.
• Push 3 on the stack.
• Pop 3 from the stack. Pop 1 from the stack. Apply the + operator to the popped operands. Push 4 on the stack.
• Push 5 on the stack.
• Push 2 on the stack.
• Pop 2 from the stack. Pop 5 from the stack. Apply the - operator to the popped operands. Push 3 on the stack.
• Pop 3 from the stack. Pop 4 from the stack. Apply the * operator to the popped operands. Push 12 on the stack.

As you can imagine, the code to solve RPN equations will be a lot simpler than code to solve equations in standard mathematical notation, as the latter requires one to handle operator precedence rules and nesting (parentheses).

Side note: the famous Edsger Dijkstra invented the Shunting-yard algorithm to convert from infix notation to postfix notation.

In order for our website to be able to solve RPN equations, we’ll start by building a parser. As said, all our code will be written in F#.

Parsing

Each RPN equation is composed of:

1. Operands (the integers).
2. Operators (the functions).

We’ll model this as follows:

type Operator =
    | Add
    | Sub
    | Mul
    | Div

type Operand =
    | Integer of int

type Expression =
    | OperatorExpression of Operator
    | OperandExpression of Operand

type Equation =
    | Equation of Expression list

Our model allows for both empty- and invalid equations. A more advanced model would make illegal states unrepresentable, but for now we’ll keep it simple.

Not all equations will be valid though. There are three types of errors we’ll cover:

• Empty equations: no operands nor operators.
• Invalid equations: one or more unknown tokens.
• Unbalanced equations: equation does not resolve to a single operand.

We’ll model these errors as follows:

type EquationError =
    | Empty
    | Invalid of string
    | Unbalanced

Now on to the actual parsing. First, let’s try to parse an expression (which is either an operand or an operator):

let parseExpression (word: string): Result<Expression, EquationError> =
    match word with
    | "+" -> Ok(OperatorExpression Add)
    | "-" -> Ok(OperatorExpression Sub)
    | "*" -> Ok(OperatorExpression Mul)
    | "/" -> Ok(OperatorExpression Div)
    | Int32 i -> Ok(OperandExpression(Integer i))
    | _ -> Error(Invalid word)

This is fairly straightforward code, where we map strings to operator expressions and integers to operand expressions using a custom active pattern. Note that the return type is not an Expression, but a Result<Expression, EquationError>, as we also want to capture if the word we tried to parse was invalid.

Our next step is to convert a string to its words (represented as a list of strings):

module String =
    let words (str: string): string list =
        str.Trim().Split([| ' ' |], StringSplitOptions.RemoveEmptyEntries)
        |> List.ofArray

Using these two functions, we can start building the parse function:

let parse (str: string): Result<Equation, EquationError> =
    let words = String.words str
    let expressions = List.map parseExpression words
    ...

The expressions value has type Result<Expression, EquationError> list, so it is a list of Result’s. We’d like to convert it to a Result<Expression list, EquationError>, as we can then convert the Expression list to an Equation. We can flatten the Result list into a single Result using this function:

module Result =
    let ofList (results: Result<'a, 'b> list): Result<'a list, 'b> =
        let folder resultState resultElement =
            match resultState, resultElement with
            | Ok results, Ok element -> Ok(element :: results)
            | Error err, _ -> Error err
            | _, Error err -> Error err

        results
        |> List.fold folder (Ok [])
        |> Result.map List.rev

We are now able to create a complete, working version of our parse function:

let parse (str: string): Result<Equation, EquationError> =
    str
    |> String.words
    |> List.map parseExpression
    |> Result.ofList
    |> Result.map Equation

Let’s try calling our parse function to test it:

parse "1 2 +"
val it : Result<Equation,EquationError> =
  Ok
    (Equation
       [OperandExpression (Integer 1); OperandExpression (Integer 2);
        OperatorExpression Add])

Hurray! Parsing works. We’re not completely done with parsing though, as we have two types of errors that we haven’t checked for: empty- and unbalanced equations. To handle these cases, we’ll add the following function:

let validate (expressions: Expression list): Result<Expression list, EquationError> =
    let rec helper remainder stack =
        match remainder, stack with
        | [], [] -> Error Empty
        | [], [ OperandExpression _ ] -> Ok(expressions)
        | OperandExpression i :: xs, _ -> helper xs (OperandExpression i :: stack)
        | OperatorExpression i :: xs, OperandExpression _ :: OperandExpression _ :: ys ->
            helper xs (OperandExpression(Integer 0) :: ys)
        | _ -> Error Unbalanced

    helper expressions []

Using pattern matching and recursion, we’re walking through an Expression list and check to see if it is empty or unbalanced (an operator is found but not enough operands to work with). We can insert this function into our parsing pipeline using Result.bind:

let parse str =
    str
    |> String.words
    |> List.map parseExpression
    |> Result.ofList
    |> Result.bind validate
    |> Result.map Equation

Let’s verify that our pipeline detects these errors:

parse ""
val it : Result<Equation,EquationError> = Error Empty

parse "3 4 + -"
val it : Result<Equation,EquationError> = Error Unbalanced

Looks like it’s working!

REPL

Before we move on to the WebAssembly part, let’s build a tiny REPL in which we can solve RPN equations. First, let’s add code to evaluate (solve) an equation:

type EvaluationResult =
    | EvaluationResult of int

let evaluateExpression (stack: int list) (expression: Expression): int list =
    match expression, stack with
    | OperandExpression(Integer operand), _ -> operand :: stack
    | OperatorExpression Add, x :: y :: xs -> y + x :: xs
    | OperatorExpression Sub, x :: y :: xs -> y - x :: xs
    | OperatorExpression Mul, x :: y :: xs -> y * x :: xs
    | OperatorExpression Div, x :: y :: xs -> y / x :: xs
    | _, _ -> failwith "Invalid expression"

let evaluateEquation (Equation expressions): EvaluationResult =
    expressions
    |> List.fold evaluateExpression []
    |> List.head
    |> EvaluationResult

let evaluate (str: string): Result<EvaluationResult, EquationError> =
    parse str
    |> Result.map evaluateEquation

Here you can clearly see the stack-based algorithm reflected that we mentioned in the RPN introduction.

Now all that’s left is to create a REPL loop:

let printInstructions(): unit =
    printfn "Please enter an equation and evaluate it by pressing enter..."

let read(): unit =
    Console.ReadLine()

let eval (input: string): unit =
    evaluate input

let printOutcome (EvaluationResult outcome): unit =
    printfn "Outcome: %A" outcome

let parserErrorMessage (error: EquationError): string =
    match error with
    | Empty -> "Empty equation"
    | Invalid token -> sprintf "Invalid token: %s" token
    | Unbalanced -> "Equation is unbalanced"

let printError (error: EquationError): unit =
    let errorMessage = parserErrorMessage error
    printfn "Error: %s" errorMessage

let print (result: Result<EvaluationResult, EquationError>): unit =
    match result with
    | Ok outcome -> printOutcome outcome
    | Error error -> printError error

let loop(): unit =
    while true do
        read()
        |> eval
        |> print

[<EntryPoint>]
let main _ =
    printInstructions()
    loop()
    0

Simple and elegant. Let’s try out the REPL:

Please enter an equation and evaluate it by pressing enter...
1 2 +
Outcome: 3
2 4 * + 7
Error: Equation is unbalanced

Nice. We’re ready to move on solving RPN equations using WebAssembly.

WebAssembly

Before we dive into the details, let’s refresh our knowledge on what WebAssembly is. The official website has the following definition:

WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable target for compilation of high-level languages like C/C++/Rust, enabling deployment on the web for client and server applications.

A great way to learn more about WebAssembly is to check out the cartoon intro to WebAssembly by Lin Clark. For now, just keep in mind that WebAssembly can be executed in the browser and that there is both a binary format (WASM) and a text format (WAT).

As said, our goal will be to have a website that can solve RPN equations. The first step is to parse the equation (using the code we just wrote). The parsed equation will then be compiled to (binary) WebAssembly code. The WebAssembly code is then executed in the browser. Finally, we’ll display the result of the solved equation as well as the WebAssembly used to solve it (binary and text format). Sounds fun? Let’s get to it!

Generating WebAssembly

In principle, we could be writing F# to output a generic RPN notation solver in WebAssembly, but to keep things simple, we’ll just compile the minimal WebAssembly needed to solve a RPN equation.

The stack-based algorithm used five types of operations:

• Push a number.
• Add two numbers.
• Subtract two numbers.
• Multiply two numbers.
• Divide two numbers.

If we look at the available WebAssembly instructions, we can find direct counterparts for all these operations, which we’ll model as follows:

type WebAssemblyInstruction =
    | I32Const of int
    | I32Add
    | I32Sub
    | I32Mul
    | I32Div

In our domain, we only support a single numeric type: 32-bit integers. In WebAssembly, these types are referred to as value types:

type WebAssemblyValueType = I32

For our purposes, a WebAssembly function can be modelled as a name, result type and a list of instructions:

type WebAssemblyFunction =
    { Name: string
      Result: WebAssemblyValueType
      Body: WebAssemblyInstruction list }

Functions in WebAssembly are always organized in a module. Our code will only ever have one function so we can make this a 1-1 mapping:

type WebAssemblyModule =
    { Function: WebAssemblyFunction }

We can now convert a parsed equation to its corresponding WebAssembly module:

let expressionToWebAssemblyInstruction (expression: Expression): WebAssemblyInstruction =
    match expression with
    | OperandExpression(Integer i) -> I32Const i
    | OperatorExpression Add -> I32Add
    | OperatorExpression Sub -> I32Sub
    | OperatorExpression Mul -> I32Mul
    | OperatorExpression Div -> I32Div

let equationToWebAssemblyCode (Equation expressions): WebAssemblyInstruction list =
    List.map expressionToWebAssemblyInstruction expressions

let equationToWebAssemblyModule (equation: Equation): WebAssemblyModule =
    { Function =
          { Name = "evaluate"
            Result = I32
            Body = equationToWebAssemblyCode equation } }

In a moment, we’ll write code to convert a WebAssemblyModule to WebAssembly text format (WAT) and binary format (WASM). Let’s start with defining some types for the WebAssembly output:

type WebAssemblyText = WebAssemblyText of string

type WebAssemblyBinary = WebAssemblyBinary of int list

type WebAssembly =
    { Text: WebAssemblyText
      Binary: WebAssemblyBinary }

We can now define the pipeline to compile a RPN equation to WebAssembly:

let equationToWebAssembly (equation: Equation): WebAssembly =
    let webAssemblyModule = equationToWebAssemblyModule equation

    { Text = Text.outputModule webAssemblyModule
      Binary = Binary.outputModule webAssemblyModule }

let convert (str: string): Result<WebAssembly, EquationError> =
    parse str
    |> Result.map equationToWebAssembly

This code will not compile, as we haven’t yet defined the functions to convert the module to text and binary format. Let’s fix this.

Text format

While the WebAssembly text format spec is great, Mozilla’s Understanding WebAssembly text format page was invaluable when trying to understand what the text format looks like in practice. In essence, a WebAssembly module is an s-expression, which is a recursive format designed to represent trees.

Here’s the WAT text we’d like to generate for the RPN equation: 1 2 +:

(module
  (func (export "evaluate") (result i32)
    i32.const 1
    i32.const 2
    i32.add))

There is not much to the WAT output we want to generate. It defines a module with one function. That function takes no parameters, has three instructions and returns an i32 value. Finally, the function is exported with the name “evaluate,” which will allow us to call it from JavaScript later.

The actual code to convert our WebAssembly module to its text format representation is fairly straightforward:

let outputValueType (valueType: WebAssemblyValueType): string =
    match valueType with
    | I32 -> "i32"

let outputResult (resultType: WebAssemblyValueType): string =
    sprintf "(result %s)" (outputValueType resultType)

let outputInstruction (instruction: WebAssemblyInstruction): string =
    match instruction with
    | I32Const i -> sprintf "i32.const %i" i
    | I32Add -> "i32.add"
    | I32Sub -> "i32.sub"
    | I32Mul -> "i32.mul"
    | I32Div -> "i32.div_32"

let outputBody (instructions: WebAssemblyInstruction list): string =
    instructions
    |> List.map outputInstruction
    |> String.concat " "

let outputExport (name: string): string =
    sprintf "(export \"%s\")" name

let outputFunction (function': WebAssemblyFunction): string =
    let export = outputExport function'.Name
    let result = outputResult function'.Result
    let body = outputBody function'.Body
    sprintf "(func %s %s %s)" export result body

let outputModule (module': WebAssemblyModule): WebAssemblyText =
    outputFunction module'.Function
    |> sprintf "(module %s)"
    |> WebAssemblyText

Tiny note: this code doesn’t output the code in an indented format, but it is semantically equivalent.

Binary format

The binary format is a bit more involved than the text format. Once again, the basis is formed by a module, which is represented as a list of bytes.

The first two parts of a module are fixed, and consist of a magic header and a version:

let outputMagicHeader: int list = [ 0x00; 0x61; 0x73; 0x6d ]

let outputVersion: int list = [ 0x01; 0x00; 0x00; 0x00 ]

Note that we defined the values in hexadecimal notation to have them correspond directly to how they are defined in the spec. We could have defined them as bytes, but for the sake of code brevity we’ll use regular ints.

Let’s move on to the contents of a module, which is comprised of zero or more sections. Each section consists of:

• A one-byte section ID.
• The u32 size of the contents, in bytes.
• The actual contents, which depends on the type of section.

While there are (currently) 12 different types of sections, we only need four of them:

type Section =
    | Type
    | Function
    | Export
    | Code

The first part of a section, its ID, can be output as follows:

let outputSectionIndex (section: Section): int =
    match section with
    | Type -> 0x01
    | Function -> 0x03
    | Export -> 0x07
    | Code -> 0x0a

The second and third part deal with the section’s contents. The second part is defined as the u32 size of the contents. If we examine the u32 spec, we can see that u32 values must be encoded using the LEB128 encoding. LEB128 encoding is a variable-length encoding, which means that different numbers can be encoded using a different number of bytes.

The LEB128 format has two different algorithms: one for unsigned integers and one for signed integers. Let’s start with the unsigned integers first:

let outputUnsignedLEB128 (i: int): int list =
    let bitEightMask = 0b1000_0000
    let bitsOneToSevenMask = 0b0111_1111

    let mutable more = true
    let mutable value = i
    let mutable bytes = ResizeArray<int>()

    while more do
        let mutable byte = value &&& bitsOneToSevenMask
        value <- value >>> 7
        more <- value <> 0

        if value <> 0 then byte <- byte ||| bitEightMask

        bytes.Add(byte)

    List.ofSeq bytes

Note that we used mutation here. While immutability is great, it is good to remember that not all F# code has to be immutable. Here, we only use mutation inside our function; it is an implementation detail. The actual returned value is a regular, immutable list.

The signed output is similar, but slightly different:

let outputSignedLEB128 (i: int): int list =
    let bitEightMask = 0b1000_0000
    let bitSevenMask = 0b0100_0000
    let bitsOneToSevenMask = 0b0111_1111

    let mutable more = true
    let mutable value = i
    let mutable bytes = ResizeArray<int>()

    while more do
        let mutable byte = value &&& bitsOneToSevenMask
        value <- value >>> 7

        if value = 0 && (byte &&& bitSevenMask) = 0 then more <- false
        elif value = -1 && (byte &&& bitSevenMask) > 0 then more <- false
        else byte <- byte ||| bitEightMask

        bytes.Add(byte)

    List.ofSeq bytes

Outputting an integer means choosing between the two LEB128 algorithms:

let outputInteger (i: int): int list =
    if i < 0 then outputSignedLEB128 i
    else outputUnsignedLEB128 i

As you might recall, the reason why we needed to output an integer was to output the length of the section’s code. In WebAssembly, bytes prefixed by their length are referred to as a vector:

let outputVector (bytes: int list): int list =
    let length =
        bytes
        |> List.length
        |> outputInteger
    length @ bytes

Outputting a section is then a simple combination of outputting its section index and its vector:

let outputSection (section: Section) (bytes: int list): int list =
    outputSectionIndex section @ outputVector bytes

Now that we have the basics defined, let’s move on to the output of the individual sections.

Type section

The type section encodes the function types of the functions defined in the module (of which we only have one). A function type is defined as the vector of a vector for the parameter types and a vector for the result (return) types:

let outputValueType (valueType: WebAssemblyValueType): int =
    match valueType with
    | I32 -> 0x7f

let outputResult (resultType: WebAssemblyValueType): int =
    outputValueType resultType

let outputFunctionTypes (function': WebAssemblyFunction): int list =
    let parametersCount = outputInteger 0x00
    let resultsCount = outputInteger 0x01
    let resultType = outputResult function'.Result
    let functionType = 0x60

    functionType :: parametersCount @ resultsCount @ [ resultType ]

let outputTypeSection (function': WebAssemblyFunction): int list =
    let functionCount = outputUnsignedLEB128 0x01
    let functionTypes = outputFunctionTypes function'

    functionCount @ functionTypes
    |> outputSection Section.Type

Function section

The function section links a function’s index to the function type. We have only one function, and thus only one function type, so we can hardcode the number of functions and the function index:

let outputFunctionSection: int list =
    let functionsCount = outputInteger 0x01
    let firstFunctionSignatureIndex = outputInteger 0x00

    functionsCount @ firstFunctionSignatureIndex |> outputSection Section.Function

Export section

In the export section, everything is listed that should be exposed to the “outside world.” When running WebAssembly in the browser, this means the exported values can be accessed from JavaScript. As we want to call our RPN solving function from JavaScript, we have to export our function.

As we only have one type we want to export (our function), we can use the following code:

type ExportType = FunctionExport

let outputExportType (exportType: ExportType): int =
    match exportType with
    | FunctionExport -> 0x00

Next, we’ll define a function to output a string, which we’ll use in a moment to define the exported function’s name:

let outputString (str: string): int list =
    str
    |> Seq.map int
    |> Seq.toList
    |> outputVector

Finally, we can glue these things together:

let outputExportSection (function': WebAssemblyFunction): int list =
    let exportsCount = outputInteger 0x01
    let exportName = outputString function'.Name
    let exportType = outputExportType FunctionExport
    let exportFunctionIndex = outputInteger 0x00

    exportsCount @ exportName @ [ exportType ] @ exportFunctionIndex |> outputSection Export

Code section

The final section is the code section, which contains (as you might have guessed) the code that a function will execute.

First, we’ll add a mapping from a WebAssembly instruction to its bytes:

let outputInstruction (instruction: WebAssemblyInstruction): int list =
    match instruction with
    | I32Const i -> 0x41: outputInteger i
    | I32Add -> [ 0x6a ]
    | I32Sub -> [ 0x6b ]
    | I32Mul -> [ 0x6c ]
    | I32Div -> [ 0x6e ]

Next, the body is defined as number of local declarations (of which we have none), the actual instructions, and finally a special marker for the end of the body:

let outputBody (instructions: WebAssemblyInstruction list): int list =
    let localDeclarationsCount = outputInteger 0x00
    let instructions = List.collect outputInstruction instructions
    let bodyEnd = 0x0b

    localDeclarationsCount @ instructions @ [ bodyEnd ] |> outputVector

The final code to output the code section is then simple:

let outputCodeSection (function': WebAssemblyFunction): int list =
    let functionCount = outputInteger 0x01
    let body = outputBody function'.Body

    functionCount @ body |> outputSection Code

Output module

Outputting a module is now a case of appending the outputs of the different sections:

let outputModule (module': WebAssemblyModule): WebAssemblyBinary =
    let typeSection = outputTypeSection module'.Function
    let functionSection = outputFunctionSection
    let exportSection = outputExportSection module'.Function
    let codeSection = outputCodeSection module'.Function

    outputMagicHeader @ outputVersion @ typeSection @ functionSection @ exportSection @ codeSection
    |> WebAssemblyBinary

And we’re done! We can now compiling any RPN equation to its WebAssembly format. Let’s start building the website now.

Building the website

To build the website using only F#, we can use the fabulous (pun intended) Fable compiler. Fable is a compiler that can compile F# code to JavaScript code. Precisely what we need!

Our website will use a particular flavor of Fable called Elmish. With Elmish, one can build applications using the Model-View-Update architecture that was popularized by Elm.

Our first step in building our Elmish application is to define the application’s model. In an Elmish application, the model is used as input to render the website. As such, the model should contain everything that we want to display on our website, which for our application looks like this:

type EvaluationResult =
    { Result: int
      Wasm: int list
      Wat: string }

type EvaluationError =
    | WebAssemblyException of exn
    | EmptyEquation
    | InvalidEquation of string
    | UnbalancedEquation

type Model =
    { Evaluation: Result<EvaluationResult, EvaluationError> option
      Equation: string }

The types are relative simple. There is an evaluation result that contains the result of the evaluation, as well as the generated WASM (binary format) and WAT (text format). The evaluation result itself is an optional part of the model (there is no evaluation initially). The other part of the model is the equation to solve.

For the first render of the website, Elmish requires us to provide a function that returns the initial model:

let init(): Model * Cmd<'a> =
    { Evaluation = None
      Equation = "" }, Cmd.none

Of course, we need to be able to update the model if we are to have any interactivity on our website. In the Elmish model, updating the model means sending a message. These messages are defined as a regular discriminated union:

type Msg =
    | UpdateEquation of string
    | EvaluateEquation
    | EquationEvaluatedSuccessfully of Evaluation
    | EquationEvaluatedWithError of EvaluationError

The next step is to define a function that takes a message and a model, and then returns a model and a command (we’ll get to what a command is):

let update (msg: Msg) (model: Model): Model * Cmd<'a> =
    match msg with
    | UpdateEquation newEquation ->
        { model with Equation = newEquation }, Cmd.none
    | EvaluateEquation ->
        evaluate model
    | EquationEvaluatedSuccessfully evaluation ->
        { model with Evaluation = Some(Ok evaluation) }, Cmd.none
    | EquationEvaluatedWithError error ->
        { model with Evaluation = Some(Error error) }, Cmd.none

The UpdateEquation, EquationEvaluatedSuccessfully and EquationEvaluatedWithError paths are relatively straightforward. They take a message with a payload and then update the model with that payload and return the model. These paths also return Cmd.none to indicate that there is no new message that needs to be processed.

The interesting path is the one where the user has requested that the equation be evaluated. The evaluate function calls the convert method, which takes an equation as input and returns the compiled WebAssembly:

let evaluate (model: Model): Model * Cmd<'a> =
    match convert model.Equation with
    | Ok webAssembly -> equationEvaluatedSuccessfully model webAssembly
    | Error error -> equationEvaluatedWithError model error

In the error case, we map the EquationError to an EvaluationError and then return the (unchanged) model and a new message indicating that the model should be updated with the error:

let equationEvaluatedWithError (model: Model) (error: EvaluationError): Model * Cmd<'a> =
    let evaluationError =
        match error with
        | Empty -> EmptyEquation
        | Invalid token -> InvalidEquation token
        | Unbalanced -> UnbalancedEquation

    model, Cmd.ofMsg (EquationEvaluatedWithError evaluationError)

This code shows that instead of just updating the model and returning Cmd.none, one could also choose to send a new command with a new message, deferring the actual updating of the model to the message handler.

For the success case, where an equation was successfully compiled to WebAssembly, we want to execute the WebAssembly in the browser. This means interacting with browser API’s. Many API’s have type-safe wrappers in Fable, but with the WebAssembly stuff being fairly new, we have to add those wrappers ourselves. Luckily, this is extremely easy:

[<AutoOpen>]
module WebAssembly =
    type Instance =
        { exports: obj }

    type ResultObject =
        { instance: Instance }

    [<Emit("WebAssembly.instantiate($0)")>]
    let instantiate (bufferSource: ArrayBuffer): JS.Promise<ResultObject> = jsNative

[<AutoOpen>]
module Uint8Array =
    [<Emit("Uint8Array.from($0)")>]
    let from (bytes: int list): ArrayBuffer = jsNative

As you can see, we define the browser API’s as regular functions, but with a special Emit attribute that Fable will use when converting the F# to JavaScript.

We now have all the infrastructure we need to execute our compiled WebAssembly:

let equationEvaluatedSuccessfully model webAssembly =
    let (WebAssemblyText(wat)) = webAssembly.Text
    let (WebAssemblyBinary(wasm)) = webAssembly.Binary

    let onSuccess webAssemblyInBrowser =
        EquationEvaluatedSuccessfully
            ({ Result = webAssemblyInBrowser.instance.exports?evaluate ()
               Wasm = wasm
               Wat = wat })

    let onError ex = EquationEvaluatedWithError(WebAssemblyException ex)

    let cmd = Cmd.OfPromise.either WebAssembly.instantiate (Uint8Array.from wasm) onSuccess onError
    model, cmd

Running our WebAssembly function is done through the interop code we just wrote: webAssemblyInBrowser.instance.exports?evaluate(). The evaluate() call is actually our compiled and exported WebAssembly function.

The last part of this function is instructing Fable to execute a function that returns a JavaScript promise (WebAssembly.instantiate), and call one of the two passed callbacks when the promise completed successfully or with an error.

The final step in building our website is to write functions to render our UI. With Elmish, rendering HTML is not done in a regular templating language, but using plain F# functions. Consider the following function to output the evaluation output contents:

let equationOutputContent (resultClass: string) (text: string): ReactElement =
    p []
        [ mark
            [ Name "result"
              Class resultClass ] [ str text ] ]

Calling equationOutputContent "tertiary" "3" will render the following HTML:

<p><mark name="result" class="tertiary">3</mark></p>

Did you note that the return type of the equationOutputContent function was of type ReactElement? This gives away the fact that Fable (by default) compiles to React code, in order to actually render the website.

In Elmish, rendering a UI is not much more than just composing functions. We’ll skip over the other functions, but if you’re interested in this bit you can just check out the source code.

And with that we now have a website where we can solve RPN equations using WebAssembly:

Conclusion

Using F# to parse and solve RPN equations was almost trivial. Modelling the domain using F#’s expressive and low-ceremony type system was an absolute joy. If you are interested in domain modeling using functional techniques, I highly recommend Scott Wlaschin’s Domain Modeling made Functional book.

Generating WebAssembly to solve RPN equations was not very hard. This was helped by the well-written WebAssembly specification. Building the website in F# using Fable was a first for me, but I absolutely loved it. The Elmish part especially was conceptually very appealing, with its simple, functional architecture. Defining HTML output using functions took some time to get used to, but when it did, I really liked it.

The source code of everything described in this blog post can be found at https://github.com/erikschierboom/krakow. You can try out the website at https://krakow-98706.firebaseapp.com/.

In this blog post, we’ll see how we can use the .NET CLI to build a small .NET Core console application. We then show how easy it is to setup a continuous integration pipeline using AppVeyor, Travis and CircleCI.

Note: we will assume that we already have an existing account for AppVeyor, Travis and CircleCI.

Installing the .NET Core SDK

Our first step is to install the .NET Core SDK. This is surprisingly simple. Just go to the download page and follow the on-screen instructions.

Once the installation has completed, open a command prompt and run:

dotnet --version

If the installation was successful, the .NET CLI’s version number will be displayed, which for me was:

1.0.1

Creating the application

The application we’ll build will be a small console application. Although we could use Visual Studio (2017) to build the application, we’ll use the .NET CLI exclusively.

First, we’ll create a directory for our application and navigate to that directory:

mkdir dotnetcore-ci
cd dotnetcore-ci

To quickly generate a basic console application, we can use the .NET CLI:

dotnet new console

This will output:

Content generation time: 47.7522 ms
The template "Console Application" created successfully.

Once finished, the CLI will have created two files in the current directory:

• Program.cs: the C# source file for our application.
• dotnetcore-ci.csproj: a C# project for our application.

These two files is all the .NET CLI needs to build and run our application. Before we can run our application though, we need to restore its dependencies:

dotnet restore

This command will resolve our project’s dependencies:

Restoring packages for /Users/erikschierboom/Programming/dotnetcore-ci/dotnetcore-ci.csproj...
  Generating MSBuild file /Users/erikschierboom/Programming/dotnetcore-ci/obj/dotnetcore-ci.csproj.nuget.g.props.
  Generating MSBuild file /Users/erikschierboom/Programming/dotnetcore-ci/obj/dotnetcore-ci.csproj.nuget.g.targets.
  Writing lock file to disk. Path: /Users/erikschierboom/Programming/dotnetcore-ci/obj/project.assets.json
  Restore completed in 841.1 ms for /Users/erikschierboom/Programming/dotnetcore-ci/dotnetcore-ci.csproj.

  NuGet Config files used:
      /Users/erikschierboom/.nuget/NuGet/NuGet.Config

  Feeds used:
      https://api.nuget.org/v3/index.json

Having restored our project’s dependencies, we can now build our application:

dotnet build

This will output some diagnostic information:

Microsoft (R) Build Engine version 15.1.548.43366
Copyright (C) Microsoft Corporation. All rights reserved.

  dotnetcore-ci -> /Users/erikschierboom/Programming/dotnetcore-ci/bin/Debug/netcoreapp1.1/dotnetcore-ci.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.14

We can see that the build was successful, so let’s run our application using the .NET CLI:

dotnet run

Our application will now be run (using the .NET Core runtime) and will output a familiar message:

Hello World!

The output may not be that exciting, but what is exciting is that we scaffolded, restored, built and ran our application using just the .NET CLI! No IDE was involved! Additionally, we did all this on our MacBook, showing the fully cross-platform nature of the .NET CLI.

Building on a continuous integration server

Now that we have our software building locally, let’s try to setup a continuous integration pipeline. For that, we need to create a public repository on GitHub for our application. Having pushed our local files to the public repository, we are ready to add continuous integration to our application.

AppVeyor

The first CI server we’ll look at is AppVeyor, which is a CI server that runs on Windows.

To configure our application, we add an appveyor.yml file to our root directory. Its contents are very simple:

image: Visual Studio 2017
environment:
  DOTNET_CLI_TELEMETRY_OPTOUT: true
  DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true
before_build:
  - cmd: dotnet restore

In the first line, we specify that we want to use the Visual Studio 2017 image, which is the AppVeyor image that has the .NET Core SDK pre-installed.

Next, we set two environment variables:

1. DOTNET_CLI_TELEMETRY_OPTOUT: true: don’t send any telemetry data.
2. DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true: this will prevent the CLI from pre-populating the packages cache.

We then indicate that dotnet restore should be executed before the application is built. Note that we don’t have to call dotnet build, as AppVeyor will automatically do that for us. You can see this when we examine the build output in AppVeyor:

Build started
git clone -q --branch=master https://github.com/ErikSchierboom/dotnetcore-ci.git C:\projects\dotnetcore-ci
git checkout -qf c6a418e057f44d9604e95429ec273239ef8c01bc

dotnet restore
  Restoring packages for C:\projects\dotnetcore-ci\dotnetcore-ci.csproj...
  Generating MSBuild file
   ...

msbuild "C:\projects\dotnetcore-ci\dotnetcore-ci.csproj" /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll"
Microsoft (R) Build Engine version 15.1.548.43366
Copyright (C) Microsoft Corporation. All rights reserved.
Build started 3/20/2017 2:21:46 PM.
Project "C:\projects\dotnetcore-ci\dotnetcore-ci.csproj" on node 1 (default targets).
PrepareForBuild:
  ...
CoreCompile:
  ...
CopyFilesToOutputDirectory:
  ...
Done Building Project "C:\projects\dotnetcore-ci\dotnetcore-ci.csproj" (default targets).
Build succeeded.
    0 Warning(s)
    0 Error(s)
Time Elapsed 00:00:03.27

Discovering tests...OK
Build success

And with that, we have our .NET Core application building on AppVeyor using the .NET CLI.

Travis

Our second CI server is Travis, which runs on Linux.

Just like AppVeyor, we need to add a configuration file to our root directory, this time named .travis.yml:

language: csharp
dist: trusty
dotnet: 1.0.1
mono: none
env:
  global:
    - DOTNET_SKIP_FIRST_TIME_EXPERIENCE: 1
    - DOTNET_CLI_TELEMETRY_OPTOUT: 1
script:
  - dotnet restore
  - dotnet build

This config file has a bit more going on, but nothing complicated. First, we define the language we’ll be using, C# in our case. Next, we define the Linux distro it runs on, for which we’ll use Ubunty trusty.

We then get to the .NET specific bits, in which we indicate that we want version 1.0.1 of the .NET Core SDK to be installed. We also explicitly specify that we don’t require Mono, which would otherwise be automatically installed due to the language specified being csharp.

Once again, we specify the two .NET CLI specific environment variables.

Finally, we specify the commands to execute, and this time we do need both the dotnet restore and dotnet build commands.

The Travis output log will look like this:

Installing .NET Core
..

git clone --depth=50 --branch=master https://github.com/ErikSchierboom/dotnetcore-ci.git
...

Setting environment variables from .travis.yml
$ export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1
$ export DOTNET_CLI_TELEMETRY_OPTOUT=1

...

dotnet restore
  Restoring packages for C:\projects\dotnetcore-ci\dotnetcore-ci.csproj...
  Generating MSBuild file
   ...

dotnet build
Copyright (C) Microsoft Corporation. All rights reserved.
e/travis/build/ErikSchierboom/dotnetcore-ci/bin/Debug/netcoreapp1.1/dotnetcore-ci.dll
Build succeeded.
    0 Warning(s)
    0 Error(s)
Time Elapsed 00:00:05.60

The command "dotnet build" exited with 0.
Done. Your build exited with 0.

The output is very similar to AppVeyor, with only minor differences. Our continuous integration pipeline now verifies that our application builds both on Windows (AppVeyor) and Linux (Travis).

CircleCI

The last CI server we’ll look at is CircleCI, which is also Linux-based.

Once again, configuration is done by adding a file to the root directory, this time named circle.yml:

version: 2
jobs:
  build:
    working_directory: /temp
    docker:
      - image: microsoft/dotnet:sdk
    environment:
      DOTNET_SKIP_FIRST_TIME_EXPERIENCE: 1
      DOTNET_CLI_TELEMETRY_OPTOUT: 1
    steps:
      - checkout
      - run: dotnet restore
      - run: dotnet build

In the first line, we specify that we want to use version 2 of CircleCI, which is currently in beta. With version 2, builds are based on Docker images. That allows us to use the official .NET Core SDK Docker image to build our application.

Once again, we also set the .NET CLI environment variables.

Finally, we specify the steps to execute. First, we checkout our repository and then we run dotnet restore and dotnet build.

The build output is similar to that of the other two CI servers:

Build-agent version 0.0.2792-4fbb67c (2017-03-20T15:14:26+0000)
Starting container microsoft/dotnet:sdk
  image not cached, downloading microsoft/dotnet:sdk
sdk: Pulling from microsoft/dotnet
...
Status: Downloaded newer image for microsoft/dotnet:sdk
  using image microsoft/dotnet@sha256:3f87a7b7873ced0110a35aee48591f8b4aea3ccb94bf433a80388886bdbd3076

...

git fetch --force origin master:remotes/origin/master
...
Cloning into '.'...
...
HEAD is now at df83619 Add CircleCI support

dotnet restore
Shell: /bin/bash -eo pipefail

  Restoring packages for /tmp/dotnetcore-ci.csproj...
  Generating MSBuild file /tmp/obj/dotnetcore-ci.csproj.nuget.g.props.
  ...

dotnet build

Microsoft (R) Build Engine version 15.1.548.43366
Copyright (C) Microsoft Corporation. All rights reserved.

  dotnetcore-ci -> /tmp/bin/Debug/netcoreapp1.1/dotnetcore-ci.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.94

Conclusion

Creating, building and running a .NET Core application using the .NET CLI is incredibly easy. It it also quite convenient, as it is cross-platform and you can build your application using the same CLI both locally and remotely.

We used the CLI locally to create a simple .NET Core console application. We then used the CLI remotely to setup three continuous integration servers, AppVeyor, Travis and CircleCI, to build our application.

Configuring the CI servers was very simple: they all required only a single, simple YAML configuration file to be added to the root of our project.

The source code of this test application, which includes the three CI server configuration files, can be found here.

To hone my programming skills, I’ve been using exercism.io a lot. With Exercism, you can work on problems in over 30 different languages. In this blog post, we’ll be solving exercism’s wordy problem, which focuses on text-parsing. We’ll be using F# as our language of choice and use various approaches to solve the problem. In the process, we’ll do lots of refactoring.

Setting up Exercism

Setting up Exercism is simple. It involves three steps:

1. Create an account by logging in to Exercism with your GitHub credentials.
2. Install the Exercism command-line client, which is available for Windows, MacOS and Linux.
3. Configure the command-line client with your account’s API key:

exercism configure --key=<your API key>

Once these steps have been completed, you can use the CLI to fetch and submit problems. If you’d like more detailed instructions on how to setup Exercism, please follow their detailed tutorial.

Problem 1: Wordy

To start working on the wordy problem with F#, we use this CLI command:

exercism fetch fsharp wordy

This command will fetch the wordy exercise’s README and test suite, and write them to the file system:

Not Submitted:            1 problem
fsharp (Wordy) /Users/erikschierboom/exercism/fsharp/wordy

New:                      1 problem
fsharp (Wordy) /Users/erikschierboom/exercism/fsharp/wordy

unchanged: 0, updated: 0, new: 1

If we examine the contents of the wordy directory, we’ll find two files:

1. README.md: a textual description of the exercise.
2. WordyTest.fs: the test suite, to verify the correctness of the implementation.

As said, the README file describes the problem we should solve:

Write a program that takes a word problem and returns the answer as an integer.

It also lists some sample word problems, such as:

What is 5 plus 13?

Now that we know the problem to solve, let’s try to solve it. We’ll go at it one test at a time, slowly building up a correct implementation that passes all tests (which are defined in the WordyTest.fs test suite). In the process, we’ll do lots of refactoring, to try to improve the quality of our code. Let’s start!

Test 1: single digit numbers

The first test is defined as followed:

[<Test>]
let ``Can parse and solve addition problems`` () =
    Assert.That(solve "What is 1 plus 1?", Is.EqualTo(Some 2))

From this code we can infer that we need to define a solve function that takes a string parameter (the equation) and returns an int option value (the equation’s result).

Let’s implement a very simple equation solver that removes all non-digit characters and then parses the two remaining digits to calculate the sum:

let solve (question: string) =
    let equation =
        question
            .Replace("What is ", "")
            .Replace(" plus ", "")
            .Replace("?", "")
    let left  = Convert.ToInt32(equation.Substring(0, 1))
    let right = Convert.ToInt32(equation.Substring(1, 1))
    Some (left + right)

If we run the test with this code, it passes.

Note that we aligned the Convert.ToInt calls. Aligning these statements has two advantages:

1. As the statements are so alike, they give users a clue that they probably have similar functionality.
2. It helps us compare the statements. With this alignment, it becomes easy to see which parts of the strings they convert.

Refactoring: simplify integer conversion

Our first refactoring is to use the int function instead of Convert.ToInt32:

let solve (question: string) =
    let equation =
        question
            .Replace("What is ", "")
            .Replace(" plus ", "")
            .Replace("?", "")
    let left  = int <| equation.Substring(0, 1)
    let right = int <| equation.Substring(1, 1)
    Some (left + right)

A quick check verifies that our code still passes the test.

We can also replace the Substring calls with array slice syntax:

let solve (question: string) =
    let equation =
        question
            .Replace("What is ", "")
            .Replace(" plus ", "")
            .Replace("?", "")
    let left  = int equation.[0..0]
    let right = int equation.[1..1]
    Some (left + right)

Refactoring: extract helper functions

Next, we can make the solve function a bit easier to read by extracting some helper functions:

let parseEquation (question: string) =
    question
        .Replace("What is ", "")
        .Replace(" plus ", "")
        .Replace("?", "")

let parseLeft  (equation: string) = int equation.[0..0]
let parseRight (equation: string) = int equation.[1..1]

let solve (question: string) =
    let equation = parseEquation question
    let left  = parseLeft equation
    let right = parseRight equation
    Some (left + right)

The refactored code looks cleaner to me, and it still passes the test! On to the second test.

Test 2: double digit numbers

In the second test, we have to parse double digit numbers:

let ``Can add double digit numbers`` () =
    Assert.That(solve "What is 53 plus 2?", Is.EqualTo(Some 55))

If we run this test, it fails. This is due to us hard-coding the int parsing to single digits. To support double digit numbers, we’ll rewrite our code to use regular expressions:

let solve (question: string) =
    let pattern = "What is (\d+) plus (\d+)\?"
    let matched = Regex.Match(question, pattern)
    let left  = int matched.Groups.[1].Value
    let right = int matched.Groups.[2].Value
    Some (left + right)

This time, the double digit test passes. Although regular expressions can easily become complicated and hard to read, ours is quite simple and understandable. In this case, I feel that using a regular expression does improve the code’s readability.

Note that we assume the regular expression always matches. We’ll defer adding proper error handling until later, when the tests force us to.

Refactoring: using named expressions

Although the regular expression is quite simple, I don’t like how we extract the matched digit values using integer indexing. We can improve this by using named expressions. This regular expression feature lets us attach a name to an expression, which we can then use to retrieve the matched value:

let solve (question: string) =
    let pattern = "What is (?<left>\d+) plus (?<right>\d+)\?"
    let matched = Regex.Match(question, pattern)
    let left  = int matched.Groups.["left"].Value
    let right = int matched.Groups.["right"].Value
    Some (left + right)

You can see that the (\d+) plus (\d+) part of the regular expression has been replaced with (?<left>\d+) plus (?<right>\d+). The ?<left> and ?<right> parts specify the names of the expressions. We then use these names to index into the matched groups, instead of using integer indexers.

Re-running our tests verifies that everything still works.

Test 3: negative numbers

The third test haves us work with negative numbers:

[<Test>]
let ``Can add negative numbers`` () =
    Assert.That(solve "What is -1 plus -10?", Is.EqualTo(Some -11))

Once again, we have to fix our code to make this test pass. Luckily, the fix is as simple as adding an optional minus (-) prefix to the digit matching expressions in our regular expression:

let pattern = "What is (?<left>-?\d+) plus (?<right>-?\d+)\?"

Test 4: large numbers

Our current implementation passes this test, so let’s move on.

Test 5: subtraction

In test number 5, we have to deal with a new operator, the minus operator:

[<Test>]
let ``Can parse and solve subtraction problems`` () =
    Assert.That(solve "What is 4 minus -12?", Is.EqualTo(Some 16))

To support the new operator, we first capture it in our regular expression:

let pattern = "What is (?<left>-?\d+) (?<operator>plus|minus) (?<right>-?\d+)\?"

The final step is to replace the Some (left + right) expression with a match on the operator’s value to determine what value to return:

match matched.Groups.["operator"].Value with
| "plus"  -> Some (left + right)
| "minus" -> Some (left - right)

Running the tests again verifies that they all pass.

Refactoring: using Active Patterns

If we look at our current code, it consists of three simple steps:

1. Match the question with our regular expression.
2. Convert the "left" and "right" groups to int values.
3. Based on the matched "operator", combine the left and right values and return it as an int option value.

This combination of matching and transforming is so commonplace, that F# created a special construct for it: Active Patterns. Let’s create an Active Pattern that will match an equation:

let (|Equation|) question =
    let pattern = "What is (?<left>-?\d+) (?<operator>plus|minus) (?<right>-?\d+)\?"
    let matched = Regex.Match(question, pattern)

    let left  = int matched.Groups.["left"].Value
    let right = int matched.Groups.["right"].Value

    match matched.Groups.["operator"].Value with
    | "plus"  -> Some (left + right)
    | "minus" -> Some (left - right)

The main thing to note is that the function’s body is just literally copy-pasted from the solve method. What’s different though, is the function’s identifier. An Active Pattern’s identifier is defined between pipe (|) characters, placed within parentheses.

We can use this Active Pattern in our solve function as follows:

let solve (question: string) =
    match question with
    | Equation result -> result

You can see that we can match the question string using our Equation Active Pattern, with the value returned from the Active Pattern being bound to the result identifer.

Re-running the tests confirms that everything still works. Looking at our code though, I don’t think the Active Pattern gives us much in terms of readability our functionality, so we’ll revert our code to the previous iteration (which didn’t use an Active Pattern).

Test 6: multiplication

In test 6, multiplication is added to the list of supported operations:

[<Test>]
let ``Can parse and solve multiplication problems`` () =
    Assert.That(solve "What is -3 multiplied by 25?", Is.EqualTo(Some -75))

This is trivial to add. We simply add the "multiplied by" string to the operator part of our regular expression, and then handle this operator in our match clause:

match matched.Groups.["operator"].Value with
| "plus"          -> Some (left + right)
| "minus"         -> Some (left - right)
| "multiplied by" -> Some (left * right)

With that, the test passes.

Test 7: division

Test 7 adds supports for division:

[<Test>]
let ``Can parse and solve division problems`` () =
    Assert.That(solve "What is 33 divided by -3?", Is.EqualTo(Some -11))

Once again, this is trivial to add, so let’s skip the implementation details.

Test 8: add twice

Things start getting more interesting in test 8, where we have to add not once, but twice!

[<Test>]
let ``Can add twice`` () =
    Assert.That(solve "What is 1 plus 1 plus 1?", Is.EqualTo(Some 3))

So far, we have only considered simple expressions with two integers and one operand. However this time, we start with the leftmost number and then apply two of operations. The most hacky solution would be to just match an optional second operation:

let pattern = "What is (?<left>-?\d+) (?<operator1>plus|minus|multiplied by|divided by) (?<right1>-?\d+)( (?<operator2>plus|minus|multiplied by|divided by) (?<right2>-?\d+))\?"

However, I don’t think this is very readable, nor is it very future-proof (what if we want to support five operations?). Let’s take a step back to figure out a better approach. Sometimes it helps to make the state transitions explicit, so let’s try that for the "What is 1 plus 1 plus 1?" question:

1. Set the current state to 1.
2. Add 1 to the current state (1). Returns 2.
3. Add 1 to the current state (2). Returns 3.

I don’t know about you, but this looks an awful lot like a left fold to me. To use a left fold in our implementation, we first have to ensure that we match the added second operation. Our initial attempt at this might look like this:

let pattern = "What is (?<left>-?\d+)(?<operations> (?<operator>plus|minus|multiplied by|divided by) (?<right>-?\d+))+\?"

Here, we define a new capture group named "operations" that matches one or more operator/operand combinations. Unfortunately, this did not work. Let’s try another approach where we split the operations matching part into its own pattern:

let equationPattern = "What is (?<left>-?\d+)(?<operations>.+?)\?"
let operationsPattern = " (?<operator>plus|minus|multiplied by|divided by) (?<right>-?\d+)"

Next, we’ll do two matches:

1. We match the equation (without matching the actual operations).
2. We match the operations using the matched "operations" part of the equation match:

let equationMatch = Regex.Match(question, equationPattern)
let operationMatches = Regex.Matches(equationMatch.Groups.["operations"].Value, operationsPattern)

We can then calculate the equation’s result by folding on the matched operations, with the leftmost integer as the starting state:

let left = int equationMatch.Groups.["left"].Value

let folder result (operation: Match) =
    let right = int operation.Groups.["right"].Value
    match operation.Groups.["operator"].Value with
    | "plus"          -> result + right
    | "minus"         -> result - right
    | "multiplied by" -> result * right
    | "divided by"    -> result / right

let result = Seq.fold folder left (Seq.cast operationMatches)
Some result

Note that we have to add a type annotation on the operation parameter and use Seq.cast on the operationMatches value, due to the MatchCollection type not implementing IEnumerable<Match>. The result is just your typical fold and is quite readable.

Let’s run our tests to see if they now pass, which they do!

Test 9: add then subtract

The next test also involves multiple operations, but this time using two different operations:

[<Test>]
let ``Can add then subtract`` () =
    Assert.That(solve "What is 1 plus 5 minus -2?", Is.EqualTo(Some 8))

Luckily for us, our fold solution already supports multiple different operations, so this test passes without any additional work.

Test 10: subtract twice

A double subtraction is also no problem for our implementation, the test passes.

Test 11: subtract then add

Subtracting and then adding? Already working!

Test 12: multiply twice

Our code also immediately passes the double multiplication test.

Test 13: add then multiply

In test 13, we first add a number and then multiply:

[<Test>]
let ``Can add then multiply`` () =
    Assert.That(solve "What is -3 plus 7 multiplied by -2?", Is.EqualTo(Some -8))

Although our test passes, there is something odd about this test. Shouldn’t the result be -17? Well, yes and no. In mathematics, the multiplication operator has a higher precedence than the addition operator, so the equation should be evaluated as -3 + (7 * -2), which equals -17. However, this does not hold for this exercise, as the README explicitly states:

Remember, that these are verbal word problems, not treated as you normally would treat a written problem. This means that you calculate as you move forward each step. In other words, you should ignore order of operations. 3 + 2 * 3 = 15, not 9.

Ignoring the order of operations is precisely what we do, hence the passing test.

Test 14: divide twice

This test passes.

Test 15: ignore unsupported operation

With test 15, we have our first test that is expected to “fail”, indicated by returning None.

[<Test>]
let ``Cubed is too advanced`` () =
    Assert.That(solve "What is 53 cubed?", Is.EqualTo(None))

At the moment, our code does not pass this test. Why is that? Well, the left part of the equation ("53") matches. The operations regular expression does not match any operation, so it returns an empty sequence. If we fold on an empty sequence, the initial state (53), is returned, hence the value returned by our code equals Some 53, not None.

The fix is simple, we just return None if there are no matched operations:

if operationMatches.Count = 0 then
    None

With this fix, the test passes.

Test 16: irrelevant problems are not valid

The final test also deals with invalid input:

[<Test>]
let ``Irrelevant problems are not valid`` () =
    Assert.That(solve "Who is the president of the United States?", Is.EqualTo(None))

As there are no matching operations, None is returned and this test passes. With that our implementation now passes all tests. Hurray!

Does this mean we’re done? Well, we could use the CLI to submit our solution to Exercism and move on the next exercise. However, let’s try to do some refactoring/cleanup, to see if we can improve the code even more.

Refactoring: fixing compile-time warning

Those of your that know your F#, will have known that when we introduced the match statement, we introduced the following compile-time warning:

What this error message tells us, is that our match statement doesn’t handle all possible cases. This is true of course, as we only handle the four supported operators. We could ignore this warning (the code still compiles after all), but it is always a good idea to heed compiler warnings, so let’s fix it:

match operation.Groups.["operator"].Value with
| "plus"          -> result + right
| "minus"         -> result - right
| "multiplied by" -> result * right
| "divided by"    -> result / right
| _               -> failwith "Unsupported operator"

The added default case simply throws an exception. If we now compile our code, the warning disappears.

Refactoring: regular expression matching, revisited

Previously, we had problems to match both the left and right part of an equation using a single regular expression. We then resorted to matching the equation in two parts: first the equation, then the operations. However, after some more searching, we came across a very interesting regular expressions feature: positive lookahead assertions. With that, we can actually get it all working in a single regex:

let solve (question: string) =
    let equationPattern = "(?<left>-?\d+) (?<operator>-?plus|minus|divided by|multiplied by) (?=(?<right>-?\d+))"
    let equationMatches = Regex.Matches(question, equationPattern)

    if equationMatches.Count = 0 then
        None
    else
        let left = int equationMatches.[0].Groups.["left"].Value

        let folder result (operation: Match) =
            let right = int operation.Groups.["right"].Value
            match operation.Groups.["operator"].Value with
            | "plus"          -> result + right
            | "minus"         -> result - right
            | "multiplied by" -> result * right
            | "divided by"    -> result / right
            | _               -> failwith "Unsupported operator"

        let result = Seq.fold folder left (Seq.cast equationMatches)
        Some result

Refactoring: extracting functions

At the moment, our solve function is quite big and has some repetition. Let’s remove some of the regular expression group matching code duplication by extracting two helper functions:

let regexGroupString (groupName: string) (m: Match) = m.Groups.[groupName].Value
let regexGroupInt    (groupName: string) (m: Match) = regexGroupString groupName m |> int

let solve (question: string) =
    let equationPattern = "(?<left>-?\d+) (?<operator>-?plus|minus|divided by|multiplied by) (?=(?<right>-?\d+))"
    let equationMatches = Regex.Matches(question, equationPattern)

    if equationMatches.Count = 0 then
        None
    else
        let left = regexGroupInt "left" equationMatches.[0]

        let folder result (operation: Match) =
            let right = regexGroupInt "right" operation
            match regexGroupString "operator" operation with
            | "plus"          -> result + right
            | "minus"         -> result - right
            | "multiplied by" -> result * right
            | "divided by"    -> result / right
            | _               -> failwith "Unsupported operator"

        let result = Seq.fold folder left (Seq.cast equationMatches)
        Some result

Better, but we still have to deal with the regular expressions groups in the solve function, which should be an implementation detail the solve function does not know about. Let’s extract the parsing of the various parts into helper functions:

let parseLeftOperand  (matches: MatchCollection) = regexGroupInt "left" matches.[0]
let parseRightOperand (m: Match) = regexGroupInt "right" m
let parseOperator     (m: Match) = regexGroupString "operator" m
let parseOperation    (m: Match) = (parseOperator m, parseRightOperand m)

let parseOperations (matches: MatchCollection) =
    matches
    |> Seq.cast
    |> Seq.map parseOperation

The relevant part of the solve function already looks much cleaner without all the implementation details:

let left = parseLeftOperand equationMatches
let operations = parseOperations equationMatches

let folder result (operator, right) =
    match operator with
    | "plus"          -> result + right
    | "minus"         -> result - right
    | "multiplied by" -> result * right
    | "divided by"    -> result / right
    | _               -> failwith "Unsupported operator"

let result = Seq.fold folder left operations
Some result

Our next step is to extract the equation result calculating code into its own function:

let solveEquation (left, operations) =
    let folder result (operation, right) =
        match operation with
        | "plus"          -> result + right
        | "minus"         -> result - right
        | "multiplied by" -> result * right
        | "divided by"    -> result / right
        | _               -> failwith "Unsupported operator"

    Seq.fold folder left operations

The folder helper function is badly named, so let’s extract it to a better-named function:

let applyOperation acc (operation, right) =
    match operation with
    | "plus"          -> acc + right
    | "minus"         -> acc - right
    | "multiplied by" -> acc * right
    | "divided by"    -> acc / right
    | _               -> failwith "Unsupported operator"

let solveEquation (left, operations) = Seq.fold applyOperation left operations

At the moment, the equation solving has been extracted to a separate function, but the parsing has not. So let’s fix this:

let parseEquation (question: string) =
    let equationPattern = "(?<left>-?\d+) (?<operator>-?plus|minus|divided by|multiplied by) (?=(?<right>-?\d+))"
    let equationMatches = Regex.Matches(question, equationPattern)

    if equationMatches.Count = 0 then
        None
    else
        Some (parseLeftOperand equationMatches, parseOperations equationMatches)

Let’s extract the equation matching into a separate Regex instance:

let equationRegex = new Regex("(?<left>-?\d+) (?<operator>-?plus|minus|divided by|multiplied by) (?=(?<right>-?\d+))")

let parseEquation (question: string) =
    let equationMatches = equationRegex.Matches question

    if equationMatches.Count = 0 then
        None
    else
        Some (parseLeftOperand equationMatches, parseOperations equationMatches)

In general, using an if-statement is discouraged, so let’s use a match expression:

let parseEquation (question: string) =
    match equationRegex.Matches question with
    | matches when matches.Count = 0 -> None
    | matches -> Some (parseLeftOperand matches, parseOperations matches)

The solve function then becomes quite simple:

let solve (question: string) =
    match parseEquation question with
    | Some (left, operations) ->
        let result = solveEquation (left, operations)
        Some result
    | None -> None

Clearly, we don’t need to (de)construct the equation tuple:

let solve (question: string) =
    match parseEquation question with
    | Some equation ->
        let result = solveEquation equation
        Some result
    | None -> None

Better, but this matching of the parseEquation result still looks a bit ugly. However, with the Option.map function and the pipe operator, we can rewrite this to:

let solve (question: string) =
    question
    |> parseEquation
    |> Option.map solveEquation

Not only does this look very pretty, it also perfectly illustrates the pipeline like nature of our implementation:

1. Parse the question into an equation.
2. Solve the equation, if it could be parsed

Through a series of relatively small refactorings, we’ve managed to create a relatively simple implementation that passes all tests. This is the complete, refactored implementation:

let equationRegex = new Regex("(?<left>-?\d+) (?<operator>-?plus|minus|divided by|multiplied by) (?=(?<right>-?\d+))")

let regexGroupString (groupName: string) (m: Match) = m.Groups.[groupName].Value
let regexGroupInt    (groupName: string) (m: Match) = regexGroupString groupName m |> int

let parseLeftOperand  (matches: MatchCollection) = regexGroupInt "left" matches.[0]
let parseRightOperand (m: Match) = regexGroupInt "right" m
let parseOperator     (m: Match) = regexGroupString "operator" m
let parseOperation    (m: Match) = (parseOperator m, parseRightOperand m)

let parseOperations (matches: MatchCollection) =
    matches
    |> Seq.cast
    |> Seq.map parseOperation

let parseEquation (question: string) =
    match equationRegex.Matches question with
    | matches when matches.Count = 0 -> None
    | matches -> Some (parseLeftOperand matches, parseOperations matches)

let applyOperation acc (operation, right) =
    match operation with
    | "plus"          -> acc + right
    | "minus"         -> acc - right
    | "multiplied by" -> acc * right
    | "divided by"    -> acc / right
    | _               -> failwith "Unsupported operator"

let solveEquation (left, operations) = Seq.fold applyOperation left operations

let solve (question: string) =
    question
    |> parseEquation
    |> Option.map solveEquation