Schier Coding for fun

On March 7, 2017, the .NET Core SDK was released. It consists of two parts: the .NET Core runtime and the .NET CLI. The runtime allows you to run .NET Core applications, whereas the CLI is a command-line interface that allows you to develop .NET Core applications.

In this blog post, we’ll see how we can use the .NET CLI to build a small .NET Core console application. We then show how easy it is to setup a continuous integration pipeline using AppVeyor, Travis and CircleCI.

Note: we will assume that we already have an existing account for AppVeyor, Travis and CircleCI.

Installing the .NET Core SDK

Our first step is to install the .NET Core SDK. This is surprisingly simple. Just go to the download page and follow the on-screen instructions.

Once the installation has completed, open a command prompt and run:

 dotnet --version

If the installation was successful, the .NET CLI’s version number will be displayed, which for me was:

 1.0.1

Creating the application

The application we’ll build will be a small console application. Although we could use Visual Studio (2017) to build the application, we’ll use the .NET CLI exclusively.

First, we’ll create a directory for our application and navigate to that directory:

mkdir dotnetcore-ci
cd dotnetcore-ci

To quickly generate a basic console application, we can use the .NET CLI:

dotnet new console

This will output:

Content generation time: 47.7522 ms
The template "Console Application" created successfully.

Once finished, the CLI will have created two files in the current directory:

  • Program.cs: the C# source file for our application.
  • dotnetcore-ci.csproj: a C# project for our application.

These two files is all the .NET CLI needs to build and run our application. Before we can run our application though, we need to restore its dependencies:

dotnet restore

This command will resolve our project’s dependencies:

Restoring packages for /Users/erikschierboom/Programming/dotnetcore-ci/dotnetcore-ci.csproj...
  Generating MSBuild file /Users/erikschierboom/Programming/dotnetcore-ci/obj/dotnetcore-ci.csproj.nuget.g.props.
  Generating MSBuild file /Users/erikschierboom/Programming/dotnetcore-ci/obj/dotnetcore-ci.csproj.nuget.g.targets.
  Writing lock file to disk. Path: /Users/erikschierboom/Programming/dotnetcore-ci/obj/project.assets.json
  Restore completed in 841.1 ms for /Users/erikschierboom/Programming/dotnetcore-ci/dotnetcore-ci.csproj.
  
  NuGet Config files used:
      /Users/erikschierboom/.nuget/NuGet/NuGet.Config
  
  Feeds used:
      https://api.nuget.org/v3/index.json

Having restored our project’s dependencies, we can now build our application:

dotnet build

This will output some diagnostic information:

Microsoft (R) Build Engine version 15.1.548.43366
Copyright (C) Microsoft Corporation. All rights reserved.

  dotnetcore-ci -> /Users/erikschierboom/Programming/dotnetcore-ci/bin/Debug/netcoreapp1.1/dotnetcore-ci.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.14

We can see that the build was successful, so let’s run our application using the .NET CLI:

dotnet run

Our application will now be run (using the .NET Core runtime) and will output a familiar message:

Hello World!

The output may not be that exciting, but what is exciting is that we scaffolded, restored, built and ran our application using just the .NET CLI! No IDE was involved! Additionally, we did all this on our MacBook, showing the fully cross-platform nature of the .NET CLI.

Building on a continuous integration server

Now that we have our software building locally, let’s try to setup a continuous integration pipeline. For that, we need to create a public repository on GitHub for our application. Having pushed our local files to the public repository, we are ready to add continuous integration to our application.

AppVeyor

The first CI server we’ll look at is AppVeyor, which is a CI server that runs on Windows.

To configure our application, we add an appveyor.yml file to our root directory. Its contents are very simple:

image: Visual Studio 2017
environment:
  DOTNET_CLI_TELEMETRY_OPTOUT: true
  DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true
before_build:
  - cmd: dotnet restore

In the first line, we specify that we want to use the Visual Studio 2017 image, which is the AppVeyor image that has the .NET Core SDK pre-installed.

Next, we set two environment variables:

  1. DOTNET_CLI_TELEMETRY_OPTOUT: true: don’t send any telemetry data.
  2. DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true: this will prevent the CLI from pre-populating the packages cache.

We then indicate that dotnet restore should be executed before the application is built. Note that we don’t have to call dotnet build, as AppVeyor will automatically do that for us. You can see this when we examine the build output in AppVeyor:

Build started
git clone -q --branch=master https://github.com/ErikSchierboom/dotnetcore-ci.git C:\projects\dotnetcore-ci
git checkout -qf c6a418e057f44d9604e95429ec273239ef8c01bc

dotnet restore
  Restoring packages for C:\projects\dotnetcore-ci\dotnetcore-ci.csproj...
  Generating MSBuild file
   ...

msbuild "C:\projects\dotnetcore-ci\dotnetcore-ci.csproj" /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll"
Microsoft (R) Build Engine version 15.1.548.43366
Copyright (C) Microsoft Corporation. All rights reserved.
Build started 3/20/2017 2:21:46 PM.
Project "C:\projects\dotnetcore-ci\dotnetcore-ci.csproj" on node 1 (default targets).
PrepareForBuild:
  ...
CoreCompile:
  ...
CopyFilesToOutputDirectory:
  ...
Done Building Project "C:\projects\dotnetcore-ci\dotnetcore-ci.csproj" (default targets).
Build succeeded.
    0 Warning(s)
    0 Error(s)
Time Elapsed 00:00:03.27

Discovering tests...OK
Build success

And with that, we have our .NET Core application building on AppVeyor using the .NET CLI.

Travis

Our second CI server is Travis, which runs on Linux.

Just like AppVeyor, we need to add a configuration file to our root directory, this time named .travis.yml:

language: csharp
dist: trusty
dotnet: 1.0.1
mono: none
env:
  global:
    - DOTNET_SKIP_FIRST_TIME_EXPERIENCE: 1
    - DOTNET_CLI_TELEMETRY_OPTOUT: 1
script:
  - dotnet restore
  - dotnet build

This config file has a bit more going on, but nothing complicated. First, we define the language we’ll be using, C# in our case. Next, we define the Linux distro it runs on, for which we’ll use Ubunty trusty.

We then get to the .NET specific bits, in which we indicate that we want version 1.0.1 of the .NET Core SDK to be installed. We also explicitly specify that we don’t require Mono, which would otherwise be automatically installed due to the language specified being csharp.

Once again, we specify the two .NET CLI specific environment variables.

Finally, we specify the commands to execute, and this time we do need both the dotnet restore and dotnet build commands.

The Travis output log will look like this:

Installing .NET Core
..

git clone --depth=50 --branch=master https://github.com/ErikSchierboom/dotnetcore-ci.git 
...

Setting environment variables from .travis.yml
$ export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1
$ export DOTNET_CLI_TELEMETRY_OPTOUT=1

...

dotnet restore
  Restoring packages for C:\projects\dotnetcore-ci\dotnetcore-ci.csproj...
  Generating MSBuild file
   ...

dotnet build
Copyright (C) Microsoft Corporation. All rights reserved.
e/travis/build/ErikSchierboom/dotnetcore-ci/bin/Debug/netcoreapp1.1/dotnetcore-ci.dll
Build succeeded.
    0 Warning(s)
    0 Error(s)
Time Elapsed 00:00:05.60

The command "dotnet build" exited with 0.
Done. Your build exited with 0.

The output is very similar to AppVeyor, with only minor differences. Our continuous integration pipeline now verifies that our application builds both on Windows (AppVeyor) and Linux (Travis).

CircleCI

The last CI server we’ll look at is CircleCI, which is also Linux-based.

Once again, configuration is done by adding a file to the root directory, this time named circle.yml:

version: 2
jobs:
  build:
    working_directory: /tmp
    docker:
      - image: microsoft/dotnet:sdk
    environment:
      DOTNET_SKIP_FIRST_TIME_EXPERIENCE: 1
      DOTNET_CLI_TELEMETRY_OPTOUT: 1
    steps:
      - checkout
      - run: dotnet restore
      - run: dotnet build

In the first line, we specify that we want to use version 2 of CircleCI, which is currently in beta. With version 2, builds are based on Docker images. That allows us to use the official .NET Core SDK Docker image to build our application.

Once again, we also set the .NET CLI environment variables.

Finally, we specify the steps to execute. First, we checkout our repository and then we run dotnet restore and dotnet build.

The build output is similar to that of the other two CI servers:

Build-agent version 0.0.2792-4fbb67c (2017-03-20T15:14:26+0000)
Starting container microsoft/dotnet:sdk
  image not cached, downloading microsoft/dotnet:sdk
sdk: Pulling from microsoft/dotnet
...
Status: Downloaded newer image for microsoft/dotnet:sdk
  using image microsoft/dotnet@sha256:3f87a7b7873ced0110a35aee48591f8b4aea3ccb94bf433a80388886bdbd3076

...

git fetch --force origin master:remotes/origin/master
...
Cloning into '.'...
...
HEAD is now at df83619 Add CircleCI support

dotnet restore
Shell: /bin/bash -eo pipefail

  Restoring packages for /tmp/dotnetcore-ci.csproj...
  Generating MSBuild file /tmp/obj/dotnetcore-ci.csproj.nuget.g.props.
  ...

dotnet build

Microsoft (R) Build Engine version 15.1.548.43366
Copyright (C) Microsoft Corporation. All rights reserved.

  dotnetcore-ci -> /tmp/bin/Debug/netcoreapp1.1/dotnetcore-ci.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.94

Conclusion

Creating, building and running a .NET Core application using the .NET CLI is incredibly easy. It it also quite convenient, as it is cross-platform and you can build your application using the same CLI both locally and remotely.

We used the CLI locally to create a simple .NET Core console application. We then used the CLI remotely to setup three continuous integration servers, AppVeyor, Travis and CircleCI, to build our application.

Configuring the CI servers was very simple: they all required only a single, simple YAML configuration file to be added to the root of our project.

The source code of this test application, which includes the three CI server configuration files, can be found here.

 

Note: this post is part of the F# Advent Calendar 2016.

To hone my programming skills, I’ve been using exercism.io a lot. With Exercism, you can work on problems in over 30 different languages. In this blog post, we’ll be solving exercism’s wordy problem, which focuses on text-parsing. We’ll be using F# as our language of choice and use various approaches to solve the problem. In the process, we’ll do lots of refactoring.

Setting up Exercism

Setting up Exercism is simple. It involves three steps:

  1. Create an account by logging in to Exercism with your GitHub credentials.
  2. Install the Exercism command-line client, which is available for Windows, MacOS and Linux.
  3. Configure the command-line client with your account’s API key:

exercism configure --key=<your API key>

Once these steps have been completed, you can use the CLI to fetch and submit problems. If you’d like more detailed instructions on how to setup Exercism, please follow their detailed tutorial.

Problem 1: Wordy

To start working on the wordy problem with F#, we use this CLI command:

exercism fetch fsharp wordy

This command will fetch the wordy exercise’s README and test suite, and write them to the file system:

Not Submitted:            1 problem
fsharp (Wordy) /Users/erikschierboom/exercism/fsharp/wordy

New:                      1 problem
fsharp (Wordy) /Users/erikschierboom/exercism/fsharp/wordy

unchanged: 0, updated: 0, new: 1 

If we examine the contents of the wordy directory, we’ll find two files:

  1. README.md: a textual description of the exercise.
  2. WordyTest.fs: the test suite, to verify the correctness of the implementation.

As said, the README file describes the problem we should solve:

Write a program that takes a word problem and returns the answer as an integer.

It also lists some sample word problems, such as:

What is 5 plus 13?

Now that we know the problem to solve, let’s try to solve it. We’ll go at it one test at a time, slowly building up a correct implementation that passes all tests (which are defined in the WordyTest.fs test suite). In the process, we’ll do lots of refactoring, to try to improve the quality of our code. Let’s start!

Test 1: single digit numbers

The first test is defined as followed:

[<Test>]
let ``Can parse and solve addition problems`` () =
    Assert.That(solve "What is 1 plus 1?", Is.EqualTo(Some 2))

From this code we can infer that we need to define a solve function that takes a string parameter (the equation) and returns an int option value (the equation’s result).

Let’s implement a very simple equation solver that removes all non-digit characters and then parses the two remaining digits to calculate the sum:

let solve (question: string) =
    let equation = 
        question
            .Replace("What is ", "")
            .Replace(" plus ", "")
            .Replace("?", "")
    let left  = Convert.ToInt32(equation.Substring(0, 1))
    let right = Convert.ToInt32(equation.Substring(1, 1))
    Some (left + right)

If we run the test with this code, it passes.

Note that we aligned the Convert.ToInt calls. Aligning these statements has two advantages:

  1. As the statements are so alike, they give users a clue that they probably have similar functionality.
  2. It helps us compare the statements. With this alignment, it becomes easy to see which parts of the strings they convert.

Refactoring: simplify integer conversion

Our first refactoring is to use the int function instead of Convert.ToInt32:

let solve (question: string) =
    let equation = 
        question
            .Replace("What is ", "")
            .Replace(" plus ", "")
            .Replace("?", "")
    let left  = int <| equation.Substring(0, 1)
    let right = int <| equation.Substring(1, 1)
    Some (left + right)

A quick check verifies that our code still passes the test.

We can also replace the Substring calls with array slice syntax:

let solve (question: string) =
    let equation = 
        question
            .Replace("What is ", "")
            .Replace(" plus ", "")
            .Replace("?", "")
    let left  = int equation.[0..0]
    let right = int equation.[1..1]
    Some (left + right)

Refactoring: extract helper functions

Next, we can make the solve function a bit easier to read by extracting some helper functions:

let parseEquation (question: string) = 
    question
        .Replace("What is ", "")
        .Replace(" plus ", "")
        .Replace("?", "")

let parseLeft  (equation: string) = int equation.[0..0]
let parseRight (equation: string) = int equation.[1..1]

let solve (question: string) =
    let equation = parseEquation question
    let left  = parseLeft equation
    let right = parseRight equation
    Some (left + right)

The refactored code looks cleaner to me, and it still passes the test! On to the second test.

Test 2: double digit numbers

In the second test, we have to parse double digit numbers:

let ``Can add double digit numbers`` () =
    Assert.That(solve "What is 53 plus 2?", Is.EqualTo(Some 55))

If we run this test, it fails. This is due to us hard-coding the int parsing to single digits. To support double digit numbers, we’ll rewrite our code to use regular expressions:

let solve (question: string) =
    let pattern = "What is (\d+) plus (\d+)\?"
    let matched = Regex.Match(question, pattern)
    let left  = int matched.Groups.[1].Value
    let right = int matched.Groups.[2].Value
    Some (left + right)

This time, the double digit test passes. Although regular expressions can easily become complicated and hard to read, ours is quite simple and understandable. In this case, I feel that using a regular expression does improve the code’s readability.

Note that we assume the regular expression always matches. We’ll defer adding proper error handling until later, when the tests force us to.

Refactoring: using named expressions

Although the regular expression is quite simple, I don’t like how we extract the matched digit values using integer indexing. We can improve this by using named expressions. This regular expression feature lets us attach a name to an expression, which we can then use to retrieve the matched value:

let solve (question: string) =
    let pattern = "What is (?<left>\d+) plus (?<right>\d+)\?"
    let matched = Regex.Match(question, pattern)
    let left  = int matched.Groups.["left"].Value
    let right = int matched.Groups.["right"].Value
    Some (left + right)

You can see that the (\d+) plus (\d+) part of the regular expression has been replaced with (?<left>\d+) plus (?<right>\d+). The ?<left> and ?<right> parts specify the names of the expressions. We then use these names to index into the matched groups, instead of using integer indexers.

Re-running our tests verifies that everything still works.

Test 3: negative numbers

The third test haves us work with negative numbers:

[<Test>]
let ``Can add negative numbers`` () =
    Assert.That(solve "What is -1 plus -10?", Is.EqualTo(Some -11))

Once again, we have to fix our code to make this test pass. Luckily, the fix is as simple as adding an optional minus (-) prefix to the digit matching expressions in our regular expression:

let pattern = "What is (?<left>-?\d+) plus (?<right>-?\d+)\?"

Test 4: large numbers

Our current implementation passes this test, so let’s move on.

Test 5: subtraction

In test number 5, we have to deal with a new operator, the minus operator:

[<Test>]
let ``Can parse and solve subtraction problems`` () =
    Assert.That(solve "What is 4 minus -12?", Is.EqualTo(Some 16))

To support the new operator, we first capture it in our regular expression:

let pattern = "What is (?<left>-?\d+) (?<operator>plus|minus) (?<right>-?\d+)\?"

The final step is to replace the Some (left + right) expression with a match on the operator’s value to determine what value to return:

match matched.Groups.["operator"].Value with 
| "plus"  -> Some (left + right)
| "minus" -> Some (left - right)

Running the tests again verifies that they all pass.

Refactoring: using Active Patterns

If we look at our current code, it consists of three simple steps:

  1. Match the question with our regular expression.
  2. Convert the "left" and "right" groups to int values.
  3. Based on the matched "operator", combine the left and right values and return it as an int option value.

This combination of matching and transforming is so commonplace, that F# created a special construct for it: Active Patterns. Let’s create an Active Pattern that will match an equation:

let (|Equation|) question =
    let pattern = "What is (?<left>-?\d+) (?<operator>plus|minus) (?<right>-?\d+)\?"
    let matched = Regex.Match(question, pattern)

    let left  = int matched.Groups.["left"].Value
    let right = int matched.Groups.["right"].Value

    match matched.Groups.["operator"].Value with 
    | "plus"  -> Some (left + right)
    | "minus" -> Some (left - right)

The main thing to note is that the function’s body is just literally copy-pasted from the solve method. What’s different though, is the function’s identifier. An Active Pattern’s identifier is defined between pipe (|) characters, placed within parentheses.

We can use this Active Pattern in our solve function as follows:

let solve (question: string) =
    match question with
    | Equation result -> result

You can see that we can match the question string using our Equation Active Pattern, with the value returned from the Active Pattern being bound to the result identifer.

Re-running the tests confirms that everything still works. Looking at our code though, I don’t think the Active Pattern gives us much in terms of readability our functionality, so we’ll revert our code to the previous iteration (which didn’t use an Active Pattern).

Test 6: multiplication

In test 6, multiplication is added to the list of supported operations:

[<Test>]
let ``Can parse and solve multiplication problems`` () =
    Assert.That(solve "What is -3 multiplied by 25?", Is.EqualTo(Some -75))

This is trivial to add. We simply add the "multiplied by" string to the operator part of our regular expression, and then handle this operator in our match clause:

match matched.Groups.["operator"].Value with 
| "plus"          -> Some (left + right)
| "minus"         -> Some (left - right)
| "multiplied by" -> Some (left * right)

With that, the test passes.

Test 7: division

Test 7 adds supports for division:

[<Test>]
let ``Can parse and solve division problems`` () =
    Assert.That(solve "What is 33 divided by -3?", Is.EqualTo(Some -11))

Once again, this is trivial to add, so let’s skip the implementation details.

Test 8: add twice

Things start getting more interesting in test 8, where we have to add not once, but twice!

[<Test>]
let ``Can add twice`` () =
    Assert.That(solve "What is 1 plus 1 plus 1?", Is.EqualTo(Some 3))

So far, we have only considered simple expressions with two integers and one operand. However this time, we start with the leftmost number and then apply two of operations. The most hacky solution would be to just match an optional second operation:

let pattern = "What is (?<left>-?\d+) (?<operator1>plus|minus|multiplied by|divided by) (?<right1>-?\d+)( (?<operator2>plus|minus|multiplied by|divided by) (?<right2>-?\d+))\?"

However, I don’t think this is very readable, nor is it very future-proof (what if we want to support five operations?). Let’s take a step back to figure out a better approach. Sometimes it helps to make the state transitions explicit, so let’s try that for the "What is 1 plus 1 plus 1?" question:

  1. Set the current state to 1.
  2. Add 1 to the current state (1). Returns 2.
  3. Add 1 to the current state (2). Returns 3.

I don’t know about you, but this looks an awful lot like a left fold to me. To use a left fold in our implementation, we first have to ensure that we match the added second operation. Our initial attempt at this might look like this:

let pattern = "What is (?<left>-?\d+)(?<operations> (?<operator>plus|minus|multiplied by|divided by) (?<right>-?\d+))+\?"

Here, we define a new capture group named "operations" that matches one or more operator/operand combinations. Unfortunately, this did not work. Let’s try another approach where we split the operations matching part into its own pattern:

let equationPattern = "What is (?<left>-?\d+)(?<operations>.+?)\?"
let operationsPattern = " (?<operator>plus|minus|multiplied by|divided by) (?<right>-?\d+)"

Next, we’ll do two matches:

  1. We match the equation (without matching the actual operations).
  2. We match the operations using the matched "operations" part of the equation match:
let equationMatch = Regex.Match(question, equationPattern)
let operationMatches = Regex.Matches(equationMatch.Groups.["operations"].Value, operationsPattern)

We can then calculate the equation’s result by folding on the matched operations, with the leftmost integer as the starting state:

let left = int equationMatch.Groups.["left"].Value

let folder result (operation: Match) = 
    let right = int operation.Groups.["right"].Value
    match operation.Groups.["operator"].Value with 
    | "plus"          -> result + right
    | "minus"         -> result - right
    | "multiplied by" -> result * right
    | "divided by"    -> result / right

let result = Seq.fold folder left (Seq.cast operationMatches)
Some result

Note that we have to add a type annotation on the operation parameter and use Seq.cast on the operationMatches value, due to the MatchCollection type not implementing IEnumerable<Match>. The result is just your typical fold and is quite readable.

Let’s run our tests to see if they now pass, which they do!

Test 9: add then subtract

The next test also involves multiple operations, but this time using two different operations:

[<Test>]
let ``Can add then subtract`` () =
    Assert.That(solve "What is 1 plus 5 minus -2?", Is.EqualTo(Some 8))

Luckily for us, our fold solution already supports multiple different operations, so this test passes without any additional work.

Test 10: subtract twice

A double subtraction is also no problem for our implementation, the test passes.

Test 11: subtract then add

Subtracting and then adding? Already working!

Test 12: multiply twice

Our code also immediately passes the double multiplication test.

Test 13: add then multiply

In test 13, we first add a number and then multiply:

[<Test>]
let ``Can add then multiply`` () =
    Assert.That(solve "What is -3 plus 7 multiplied by -2?", Is.EqualTo(Some -8))

Although our test passes, there is something odd about this test. Shouldn’t the result be -17? Well, yes and no. In mathematics, the multiplication operator has a higher precedence than the addition operator, so the equation should be evaluated as -3 + (7 * -2), which equals -17. However, this does not hold for this exercise, as the README explicitly states:

Remember, that these are verbal word problems, not treated as you normally would treat a written problem. This means that you calculate as you move forward each step. In other words, you should ignore order of operations. 3 + 2 * 3 = 15, not 9.

Ignoring the order of operations is precisely what we do, hence the passing test.

Test 14: divide twice

This test passes.

Test 15: ignore unsupported operation

With test 15, we have our first test that is expected to “fail”, indicated by returning None.

[<Test>]
let ``Cubed is too advanced`` () =
    Assert.That(solve "What is 53 cubed?", Is.EqualTo(None))

At the moment, our code does not pass this test. Why is that? Well, the left part of the equation ("53") matches. The operations regular expression does not match any operation, so it returns an empty sequence. If we fold on an empty sequence, the initial state (53), is returned, hence the value returned by our code equals Some 53, not None.

The fix is simple, we just return None if there are no matched operations:

if operationMatches.Count = 0 then
    None

With this fix, the test passes.

Test 16: irrelevant problems are not valid

The final test also deals with invalid input:

[<Test>]
let ``Irrelevant problems are not valid`` () =
    Assert.That(solve "Who is the president of the United States?", Is.EqualTo(None))

As there are no matching operations, None is returned and this test passes. With that our implementation now passes all tests. Hurray!

Does this mean we’re done? Well, we could use the CLI to submit our solution to Exercism and move on the next exercise. However, let’s try to do some refactoring/cleanup, to see if we can improve the code even more.

Refactoring: fixing compile-time warning

Those of your that know your F#, will have known that when we introduced the match statement, we introduced the following compile-time warning:

Warning FS0025: Incomplete pattern matches on this expression.

What this error message tells us, is that our match statement doesn’t handle all possible cases. This is true of course, as we only handle the four supported operators. We could ignore this warning (the code still compiles after all), but it is always a good idea to heed compiler warnings, so let’s fix it:

match operation.Groups.["operator"].Value with 
| "plus"          -> result + right
| "minus"         -> result - right
| "multiplied by" -> result * right
| "divided by"    -> result / right
| _               -> failwith "Unsupported operator"

The added default case simply throws an exception. If we now compile our code, the warning disappears.

Refactoring: regular expression matching, revisited

Previously, we had problems to match both the left and right part of an equation using a single regular expression. We then resorted to matching the equation in two parts: first the equation, then the operations. However, after some more searching, we came across a very interesting regular expressions feature: positive lookahead assertions. With that, we can actually get it all working in a single regex:

let solve (question: string) =
    let equationPattern = "(?<left>-?\d+) (?<operator>-?plus|minus|divided by|multiplied by) (?=(?<right>-?\d+))"
    let equationMatches = Regex.Matches(question, equationPattern)

    if equationMatches.Count = 0 then
        None
    else
        let left = int equationMatches.[0].Groups.["left"].Value

        let folder result (operation: Match) = 
            let right = int operation.Groups.["right"].Value
            match operation.Groups.["operator"].Value with 
            | "plus"          -> result + right
            | "minus"         -> result - right
            | "multiplied by" -> result * right
            | "divided by"    -> result / right
            | _               -> failwith "Unsupported operator"

        let result = Seq.fold folder left (Seq.cast equationMatches)
        Some result

Refactoring: extracting functions

At the moment, our solve function is quite big and has some repetition. Let’s remove some of the regular expression group matching code duplication by extracting two helper functions:

let regexGroupString (groupName: string) (m: Match) = m.Groups.[groupName].Value
let regexGroupInt    (groupName: string) (m: Match) = regexGroupString groupName m |> int

let solve (question: string) =
    let equationPattern = "(?<left>-?\d+) (?<operator>-?plus|minus|divided by|multiplied by) (?=(?<right>-?\d+))"
    let equationMatches = Regex.Matches(question, equationPattern)

    if equationMatches.Count = 0 then
        None
    else
        let left = regexGroupInt "left" equationMatches.[0]

        let folder result (operation: Match) = 
            let right = regexGroupInt "right" operation
            match regexGroupString "operator" operation with 
            | "plus"          -> result + right
            | "minus"         -> result - right
            | "multiplied by" -> result * right
            | "divided by"    -> result / right
            | _               -> failwith "Unsupported operator"

        let result = Seq.fold folder left (Seq.cast equationMatches)
        Some result

Better, but we still have to deal with the regular expressions groups in the solve function, which should be an implementation detail the solve function does not know about. Let’s extract the parsing of the various parts into helper functions:

let parseLeftOperand  (matches: MatchCollection) = regexGroupInt "left" matches.[0]
let parseRightOperand (m: Match) = regexGroupInt "right" m
let parseOperator     (m: Match) = regexGroupString "operator" m
let parseOperation    (m: Match) = (parseOperator m, parseRightOperand m)

let parseOperations (matches: MatchCollection) =
    matches 
    |> Seq.cast 
    |> Seq.map parseOperation

The relevant part of the solve function already looks much cleaner without all the implementation details:

let left = parseLeftOperand equationMatches
let operations = parseOperations equationMatches

let folder result (operator, right) = 
    match operator with 
    | "plus"          -> result + right
    | "minus"         -> result - right
    | "multiplied by" -> result * right
    | "divided by"    -> result / right
    | _               -> failwith "Unsupported operator"

let result = Seq.fold folder left operations
Some result

Our next step is to extract the equation result calculating code into its own function:

let solveEquation (left, operations) = 
    let folder result (operation, right) = 
        match operation with 
        | "plus"          -> result + right
        | "minus"         -> result - right
        | "multiplied by" -> result * right
        | "divided by"    -> result / right
        | _               -> failwith "Unsupported operator"

    Seq.fold folder left operations

The folder helper function is badly named, so let’s extract it to a better-named function:

let applyOperation acc (operation, right) = 
    match operation with 
    | "plus"          -> acc + right
    | "minus"         -> acc - right
    | "multiplied by" -> acc * right
    | "divided by"    -> acc / right
    | _               -> failwith "Unsupported operator"

let solveEquation (left, operations) = Seq.fold applyOperation left operations

At the moment, the equation solving has been extracted to a separate function, but the parsing has not. So let’s fix this:

let parseEquation (question: string) =
    let equationPattern = "(?<left>-?\d+) (?<operator>-?plus|minus|divided by|multiplied by) (?=(?<right>-?\d+))"
    let equationMatches = Regex.Matches(question, equationPattern)

    if equationMatches.Count = 0 then
        None
    else
        Some (parseLeftOperand equationMatches, parseOperations equationMatches)

Let’s extract the equation matching into a separate Regex instance:

let equationRegex = new Regex("(?<left>-?\d+) (?<operator>-?plus|minus|divided by|multiplied by) (?=(?<right>-?\d+))")

let parseEquation (question: string) =
    let equationMatches = equationRegex.Matches question

    if equationMatches.Count = 0 then
        None
    else
        Some (parseLeftOperand equationMatches, parseOperations equationMatches)

In general, using an if-statement is discouraged, so let’s use a match expression:

let parseEquation (question: string) =
    match equationRegex.Matches question with
    | matches when matches.Count = 0 -> None
    | matches -> Some (parseLeftOperand matches, parseOperations matches)

The solve function then becomes quite simple:

let solve (question: string) =
    match parseEquation question with
    | Some (left, operations) -> 
        let result = solveEquation (left, operations)
        Some result
    | None -> None

Clearly, we don’t need to (de)construct the equation tuple:

let solve (question: string) =
    match parseEquation question with
    | Some equation ->
        let result = solveEquation equation
        Some result
    | None -> None

Better, but this matching of the parseEquation result still looks a bit ugly. However, with the Option.map function and the pipe operator, we can rewrite this to:

let solve (question: string) =
    question
    |> parseEquation
    |> Option.map solveEquation

Not only does this look very pretty, it also perfectly illustrates the pipeline like nature of our implementation:

  1. Parse the question into an equation.
  2. Solve the equation, if it could be parsed

Through a series of relatively small refactorings, we’ve managed to create a relatively simple implementation that passes all tests. This is the complete, refactored implementation:

let equationRegex = new Regex("(?<left>-?\d+) (?<operator>-?plus|minus|divided by|multiplied by) (?=(?<right>-?\d+))")

let regexGroupString (groupName: string) (m: Match) = m.Groups.[groupName].Value
let regexGroupInt    (groupName: string) (m: Match) = regexGroupString groupName m |> int

let parseLeftOperand  (matches: MatchCollection) = regexGroupInt "left" matches.[0]
let parseRightOperand (m: Match) = regexGroupInt "right" m
let parseOperator     (m: Match) = regexGroupString "operator" m
let parseOperation    (m: Match) = (parseOperator m, parseRightOperand m)

let parseOperations (matches: MatchCollection) =
    matches 
    |> Seq.cast
    |> Seq.map parseOperation

let parseEquation (question: string) =
    match equationRegex.Matches question with
    | matches when matches.Count = 0 -> None
    | matches -> Some (parseLeftOperand matches, parseOperations matches)

let applyOperation acc (operation, right) = 
    match operation with 
    | "plus"          -> acc + right
    | "minus"         -> acc - right
    | "multiplied by" -> acc * right
    | "divided by"    -> acc / right
    | _               -> failwith "Unsupported operator"

let solveEquation (left, operations) = Seq.fold applyOperation left operations

let solve (question: string) =
    question
    |> parseEquation 
    |> Option.map solveEquation

While this implementation is not too bad, there is a fair amount of boilerplate code to handle the regular expression. Furthermore, we use a relatively obscure regular expression feature (positive lookahead assertions), which lessens its readability.

Is there anything we can do to fix these issues? Well, actually there is! Let’s prepare ourselves for another big rewrite, this time using parser combinators.

Refactoring: parser combinators

If we look at our existing implementation, we can see that we parse an equation in several, small steps:

  1. Parse the left operand.
  2. Parse the list of right operator/operand combinations:
    • Parse the right operator.
    • Parse the right operand.

This type of parsing setup is ideal for a specific type of text parsing library: a parser combinator. At its core, a parser combinator is a collection of fairly basic text-parsing functions. The reason it is called a parser combinator, is that you can create complex parsers by combining those basic parsers. Let’s convert our existing, regular expression-based implementation to one that uses a parser combinator library.

Let’s start by installing the FParsec parser combinator library:

Install-Package FParsec

We are now ready to create our first parser.

Parser: left operand

First up: parsing the left operand, which parses a string into an int. As you might have expected, FParsec already has a function to do this for you: pint32 (which you should read as: parse an int32).

To verify the pint32 parser’s behavior, open an F# interactive window and load the FParsec library references. We can then test the parser using the run function:

open FParsec

run pint32 "13";;
 val it : ParserResult<int32,unit> = Success: 13

run pint32 "789";;
 val it : ParserResult<int32,unit> = Success: 789

run pint32 "abc";;
 val it : ParserResult<int32,unit> =
   Failure:
 Error in Ln: 1 Col: 1
 abc
 ^
 Expecting: integer number (32-bit, signed)

The first two inputs were valid int32 values, so the run function returned a Success value containing the corresponding, parsed number. The third input was not a correct int32 value, so a Failure value was returned containing an error message.

Defining our left operand parser is simple:

let parseLeftOperand = pint32

Parser: right operand

Similarly, we can define our right operand parser as:

let parseRightOperand = pint32

Hmmm, the parseLeftOperand and parseRightOperand functions are identical! We can safely merge them into a single function:

let parseOperand = pint32

Previously, this was not possible, as the operand values had to be retrieved from differently named regular expression groups.

Parser: right operator

Our right operator parser is one that matches one of the following string values:

  • “plus”
  • “minus”
  • “divided by”
  • “multiplied by”

Any other string value is considered invalid. We can create a parser to match a specific string using the pstring function:

run (pstring "abc") "abc";;
 val it : ParserResult<string,unit> = Success: "abc"

run (pstring "abc") "abcde";;
 val it : ParserResult<string,unit> = Success: "abc"

run (pstring "abc") "cde";;
 val it : ParserResult<string,unit> =
   Failure:
 Error in Ln: 1 Col: 1
 cde
 ^
 Expecting: 'abc'

Okay, so we can create a parser to match a string, but how to create a parser that matches one of the four operators? Well, simple: we use the <!> (read: or) parser:

let parseOperator = 
    pstring "plus"          <|> 
    pstring "minus"         <|>
    pstring "divided by"    <|>
    pstring "multiplied by"

Here, we use the <|> operator to specify that should its left hand parser fail to parse, it should try to parse its right hand parser. Let’s send this function to the F# interactive to be able to test it:

  Stopped due to error
 System.Exception: Operation could not be completed due to earlier error
 Value restriction. The value 'parseOperator' has been inferred to have generic type
     val parseOperator : Parser<string,obj>    
 Either make the arguments to 'parseOperator' explicit or, if you do not intend for it to be generic, add a type annotation.

Ouch, we get a very weird-looking error! What’s going on? Well, don’t worry, this is a common error when using FParsec. The problem is that the parseOperator function is of type Parser<string,'_a>, but F# doesn’t allow allow a generic type parameter here ('_a must be a concrete type). We can do two things to fix this:

  1. Add a type annotation.
  2. Help the compiler infer the type.

We’ll go for the second option, by also sending a run call to the F# interactive window:

let parseOperator = 
    pstring "plus"          <|> 
    pstring "minus"         <|>
    pstring "divided by"    <|>
    pstring "multiplied by"

run parseOperator "plus"
 val it : ParserResult<string,unit> = Success: "plus"

run parseOperator "divided by";;
 val it : ParserResult<string,unit> = Success: "divided by"

run parseOperator "div";;
 val it : ParserResult<string,unit> =
   Failure:
 Error in Ln: 1 Col: 1
 div
 ^
 Expecting: 'divided by', 'minus', 'multiplied by' or 'plus'

As can be seen, the parser works as expected: it matches if the supplied input equals one of the four possible string values.

Parser: right operator/operand combination

The parser for the right operator/operand combination has to do the following:

  1. Match the operator.
  2. Match a single space.
  3. Match the operand.
  4. Return the operator/operand combination.

Let’s start with step 1, matching the operator:

let parseOperation =
    parseOperator

The next step is to match on a single space, which we can do using pchar ' '. So how do we combine these two parsers? We cannot use the <|> operator, as that is an or match, not an and. As it turns out, we can use several operators. To choose between these operators, we have to decide which parser results we want to retain. Obviously, the matched operator is important, but the single space is not. Therefore, we should use the parser operator that returns the first parser and ignores the second parser. For this purpose, we can use the .>> operator:

let parseOperation =
    parseOperator .>> pchar ' '

The third and fourth step can be combined into a single call to the .>>. operator, which returns the results of both parsers:

let parseOperation =
    parseOperator .>> pchar ' ' .>>. parseOperand

Let’s test this parser:

run parseOperation "plus 7";;
 val it : ParserResult<(string * int32),unit> = Success: ("plus", 7)

run parseOperation "minus 345";;
 val it : ParserResult<(string * int32),unit> = Success: ("minus", 345)

run parseOperation "minus plus";;
 val it : ParserResult<(string * int32),unit> =
   Failure:
 Error in Ln: 1 Col: 7
 minus plus
       ^
 Expecting: integer number (32-bit, signed)

You can see that the result of the parser is a tuple, with the first item being the result of the operator parser, and the second item the result of the operand parser.

Parser: list of right operator/operand combinations

Now that we have a parser for a right operator/operand combination, the next step is to create a parser that can match one or more of those combinations. Once again, there is an existing parser that can do that: the many1 parser. This parser function takes another parser, and matches one of more instances of the supplied parser argument.

Our initial attempt looked like this:

let parseOperations = many1 parseOperation

Let’s test this parser:

run parseOperations "plus 3 minus 4";;
 val it : ParserResult<(string * int32) list,unit> = Success: [("plus", 3)]

run parseOperations "plus 7 multiplied by -2";;
 val it : ParserResult<(string * int32) list,unit> = Success: [("plus", 7)]

Hmmm, the parser does match successfully, but only the first combination. Why is that? Well, there is a very simple explanation for this. When parsing the "plus 3 minus 4" input, it finds a match ("plus 3"). It then tries to parse the remaining input: " minus 4". Did you notice the leading space? That’s what prevents the second combination from being matched, as the parseOperations parser expects the input to start with the operator, not a space.

To fix this, we can do two things:

  • Modify the parseOperation to ignore leading and/or trailing spaces.
  • Use the sepBy1 parser, which matches a parser one or more times using another parser as a separator.

We’ll go with the second option:

let parseOperations = sepBy1 parseOperation (pchar ' ')

run parseOperations "plus 3 minus 4";;
 val it : ParserResult<(string * int32) list,unit> =
   Success: [("plus", 3); ("minus", 4)]

run parseOperations "plus 7 multiplied by -2";;
 val it : ParserResult<(string * int32) list,unit> =
   Success: [("plus", 7); ("multiplied by", -2)]

run parseOperations "multiplied by 6";;
 val it : ParserResult<(string * int32) list,unit> =
   Success: [("multiplied by", 6)]

The modified parseOperations parser now correctly parses a list of operator/operand tuples.

Parser: equation

The final step is to create the equation parser, which combines the left and right parsers and ignores the pre- and postfix strings:

let parseEquation = 
    pstring "What is "
     >>. parseOperand
    .>>  pchar ' '
    .>>. parseOperations
    .>>  pstring "?"

Let’s test this parser:

run parseEquation "What is -12 divided by 2 divided by -3?";;
 val it : ParserResult<(int32 * (string * int32) list),unit> =
   Success: (-12, [("divided by", 2); ("divided by", -3)])

run parseEquation "What is 17 minus 6 plus 3?";;
 val it : ParserResult<(int32 * (string * int32) list),unit> =
   Success: (17, [("minus", 6); ("plus", 3)]) 

Integrating the parser

We can now use our FParsec-based equation parser in the solve function:

let parseToOption parser (input: string) =
    match run parser input with
    | Success(result, _, _)   -> Some result
    | Failure(errorMsg, _, _) -> None

let solve (question: string) =
    parseToOption parseEquation question
    |> Option.map solveEquation

Let’s try to run our tests, and hurray, all tests pass! For reference, this is the full FParsec-based implementation:

open FParsec

let parseOperand = pint32

let parseOperator = 
    pstring "plus"          <|> 
    pstring "minus"         <|>
    pstring "divided by"    <|>
    pstring "multiplied by"

let parseOperation =
    parseOperator .>> pchar ' ' .>>. parseOperand

let parseOperations = sepBy1 parseOperation (pchar ' ')

let parseEquation = 
    pstring "What is "
     >>. parseOperand
    .>>  pchar ' '
    .>>. parseOperations
    .>>  pstring "?"

let parseToOption parser (input: string) =
    match run parser input with
    | Success(result, _, _)   -> Some result
    | Failure(errorMsg, _, _) -> None

let applyOperation acc (operation, right) = 
    match operation with 
    | "plus"          -> acc + right
    | "minus"         -> acc - right
    | "multiplied by" -> acc * right
    | "divided by"    -> acc / right
    | _               -> acc

let solveEquation (left, operations) = Seq.fold applyOperation left operations

let solve (question: string) =
    parseToOption parseEquation question
    |> Option.map solveEquation

The only slightly cryptic part of this implementation are the operators: <|>, >>., .>> and .>>.. Someone new to FParsec will probably not intuitively understand what they do, but to me that’s the only real hard part of this code. The rest is actually quite easy to read (and write).

Refactoring: using an Abstract Syntax Tree

At the moment, we parse an equation into very simple types: strings and integers. Ideally, one should define an Abstract Syntax Tree (AST) to better describe the structure of our equation. As it turns out, this is a piece of cake in F#, so let’s have a go!

First, we define types for each type of syntax node in our equation. This is a perfect use case for discriminated unions. Let’s start with an operand:

type Operand = Int of int

Simple, an operand is a discriminated union with only a single case: Int, which contains an int value. Defining an operand this way allows us to easily add other types of operands later, like floating point numbers, booleans or variables.

Next, we define the type to represent the four supported operators:

type Operator = 
    | Plus
    | Minus
    | DividedBy
    | MultipliedBy 

We then define an operation as an Operator/Operand tuple:

type Operation = Operation of Operator * Operand

Finally, we define the equation:

type Equation = Equation of Operand * Operation list

Note that we haven’t specified types for the leading and trailing strings, nor for the spaces, as they are not relevant to our problem.

The following step is to use these types in our implementation.

Use AST types in parseOperand

Let’s start with the parseOperand function, which should wrap the parsed int value in an Operand type. We can do this using the |>> (read: pipe) operator:

let parseOperand = pint32 |>> Int

run parseOperand "54";;
 val it : ParserResult<Operand,unit> = Success: Operand 54

run parseOperand "123";;
 val it : ParserResult<Operand,unit> = Success: Operand 123

run parseOperand "ab";;
val it : ParserResult<Operand,unit> =
   Failure:
 Error in Ln: 1 Col: 1
 ab
 ^
 Expecting: integer number (32-bit, signed)

Perfect.

Use AST types in parseOperator

Next up: the parseOperator parser:

let parseOperator = 
    (pstring "plus"         |>> (fun _ -> Plus))         <|>
    (pstring "minus"        |>> (fun _ -> Minus))        <|>
    (pstring "divided by"   |>> (fun _ -> DividedBy))    <|>
    (pstring "multiplied by"|>> (fun _ -> MultipliedBy))

run parseOperator "plus"
 val it : ParserResult<Operator,unit> = Success: Plus

run parseOperator "divided by"
 val it : ParserResult<Operator,unit> = Success: DividedBy

run parseOperator "power"
 val it : ParserResult<Operator,unit> =
   Failure:
 Error in Ln: 1 Col: 1
 power
 ^
 Expecting: 'divided by', 'minus', 'multiplied by' or 'plus'

Everything works, but the pipe operator usage looks a bit odd. This is due to the pipe operator expecting a string -> 'a function as its argument. However, we’re not interested in the matched string value, so we ignore it and return the appropriate operator. As it turns out, the >>% operator simplifies this by not expecting a string parameter:

let parseOperator = 
    (pstring "plus"          >>% Plus)         <|>
    (pstring "minus"         >>% Minus)        <|>
    (pstring "divided by"    >>% DividedBy)    <|>
    (pstring "multiplied by" >>% MultipliedBy)

Much better, but there is an even better parser: stringReturn:

let parseOperator = 
    stringReturn "plus"          Plus         <|>
    stringReturn "minus"         Minus        <|>
    stringReturn "divided by"    DividedBy    <|>
    stringReturn "multiplied by" MultipliedBy

Excellent! Clean, simple and fully functional!

Use AST types in parseOperation

With what we know now, modifying the parseOperation to return an Operation type is simple:

let parseOperation =
    parseOperator .>> pchar ' ' .>>. parseOperand |>> Operation

run parseOperation "plus 1"
 val it : ParserResult<Operation,unit> = Success: Operation (Plus,Operand 1)
Use AST types in parseOperations

The parseOperations function already returns the correct type (Operation list), so we don’t need to modify it:

run parseOperations "plus 4 minus 6"
 val it : ParserResult<Operation list,unit> =
   Success: [Operation (Plus,Operand 4); Operation (Minus,Operand 6)]
Use AST types in parseEquation

For the parseEquation, we can just pipe the result into the Equation type:

let parseEquation = 
    pstring "What is "
     >>. parseOperand
    .>>  pchar ' '
    .>>. parseOperations
    .>>  pstring "?"
    |>> Equation

run parseEquation "What is -12 divided by 2 divided by -3?"
 val it : ParserResult<Equation,unit> =
   Success: Equation
   (Operand -12,
    [Operation (DividedBy,Operand 2); Operation (DividedBy,Operand -3)])

As we now use the pchar ' ' parser in several places, we should really define a separate parser for it:

let space = pchar ' '
Use AST types when solving the solution

The final step is to use the AST types when we solve the equation:

let applyOperation acc (Operation (operator, Operand(right))) = 
    match operator with 
    | Plus         -> acc + right
    | Minus        -> acc - right
    | MultipliedBy -> acc * right
    | DividedBy    -> acc / right
    | _            -> acc

let solveEquation (Equation (Operand(left), operations)) = Seq.fold applyOperation left operations

To simplify our code, we pattern-match directly in our parameters. At this point, we run our tests and they all pass! Hurray.

As we now use a discriminated union to represent an operator, the compiler is able to inform us that the catch-all clause will never be reached. We can thus safely remove it. The complete code then becomes:

open FParsec

type Operand = Operand of int

type Operator = 
    | Plus
    | Minus
    | DividedBy
    | MultipliedBy

type Operation = Operation of Operator * Operand

type Equation = Equation of Operand * Operation list

let space = pchar ' '

let parseOperand = pint32 |>> Operand

let parseOperator = 
    stringReturn "plus"          Plus         <|>
    stringReturn "minus"         Minus        <|>
    stringReturn "divided by"    DividedBy    <|>
    stringReturn "multiplied by" MultipliedBy

let parseOperation =
    parseOperator .>> space .>>. parseOperand |>> Operation

let parseOperations = sepBy1 parseOperation space

let parseEquation = 
    pstring "What is "
     >>. parseOperand
    .>>  space
    .>>. parseOperations
    .>>  pstring "?"
    |>> Equation

let parseToOption parser (input: string) =
    match run parser input with
    | Success(result, _, _)   -> Some result
    | Failure(errorMsg, _, _) -> None

let applyOperation acc (Operation (operator, Operand(right))) = 
    match operator with 
    | Plus         -> acc + right
    | Minus        -> acc - right
    | MultipliedBy -> acc * right
    | DividedBy    -> acc / right

let solveEquation (Equation (Operand(left), operations)) = Seq.fold applyOperation left operations

let solve (question: string) =
    parseToOption parseEquation question
    |> Option.map solveEquation

To me, what’s remarkable is that we defined and implemented an AST for our equation with only very code and hardly any effort.

Refactoring: using a different AST

Our previous AST didn’t look too bad, but it might not be the best way to describe an equation. Consider the expression 17 minus 6 plus 3. At the moment, our corresponding AST looks like this:

Equation(
    Operand 17 * 
        [ Operation(Minus, Operand 6); 
          Operation(Plus, Operand 3) ])

What this AST fails to represent, is that our operators always operate on two operands: a left and right part. What we would like is 17 minus 6 to be defined as Minus (Operand 17, Operand 6). The multiple operations in 17 minus 6 plus 3 should be defined as: Plus (Minus (Operand 17, Operand 6), Operand 3). Note that this is an inside-out representation of an equation, the left-most part of the equation is the innermost part of the AST. Solving this AST is thus a recursive, inside-out algorithm:

  1. Plus (Minus (Operand 17, Operand 6), Operand 3),
  2. Plus (Operand 11, Operand 3),
  3. Operand 14

As it turns out, defining this AST in our code is fantastically simple:

type Expression = 
    | Operand of int
    | Plus of Expression * Expression
    | Minus of Expression * Expression
    | DividedBy of Expression * Expression
    | MultipliedBy of Expression * Expression
Use updated AST in parseOperand

The parseOperand function does not need to be changed. The only difference is that its returned value (Operand) is now part of the Expression discriminated union.

Use updated AST in parseOperator

There is also no need to modify the parseOperator function.

Use updated AST in parseOperation and parseOperations

The parseOperation is where things start getting interesting. Previously, an operation was just an operator and an operand, but the operand can now also be another operation. Our initial attempt might look like this:

let parseOperation = 
    parseExpression 
    .>>  space
    .>>. sepBy1 (parseOperator .>> space .>>. parseExpression) space
    |>> TODO

let parseExpression = parseOperand <|> parseOperation

In this modified parseOperation function, we use the new parseExpression parser to parse the left and right hand sides. This new parser then matches on either an operand, or an operation. Unfortunately, we have now introduced a cycle: parseOperation must know of parseExpression, but also the other way around. How can we solve this? Enter the createParserForwardedToRef function.

When you call createParserForwardedToRef, it returns two things:

  1. A parser p which type will be inferred depending on its usage.
  2. A parser reference cell pRef, to which parser p forwards all calls.

With this function, we can start using the parser p, but defer binding its actual implementation until after all required parsers have been defined. Our modified code looks like this:

let expr, exprImpl = createParserForwardedToRef()

let parseOperation = 
    expr 
    .>>  space 
    .>>. sepBy1 (parseOperator .>> space .>>. expr) space
    |>> TODO

let parseExpression = parseOperand <|> parseOperation

exprImpl := parseExpression   

You can see that we use the createParserForwardedToRef to create a parser and its reference cell. We then use this parser in our parseOperation function, and later bind the parseExpression function to the reference cell’s contents. The end result is precisely what we wanted to achieve.

We now have to decide how to process the parsed operation using the |>> operator. If you are starting to become more proficient at reading FParsec’s operators, you will have guessed that we have to process a tuple. The first item of this tuple is the left expression, and the second item is the list of operator/right expression combinations. The first step is thus:

|>> (fun (left, operations) -> TODO)

So how can we merge these two parts into a single expression? Well, we can use a regular fold:

List.fold (fun acc (operator, right) -> operator (acc, right)) left operations

The initial state is the left expression. We then process each operator in sequence, wrapping the previous expression in the new operator. The end result will be an Expression that is structured inside-out, precisely what we want.

It’s probably best to extract the expression building code into its own function. This gives us the following code:

let buildExpression (left, operations) =
    List.fold (fun acc (operator, right) -> operator (acc, right)) left operations

let parseOperation = 
    expr 
    .>>  space 
    .>>. sepBy1 (parseOperator .>> space .>>. expr) space
    |>> buildExpression
Use updated AST in parseEquation

The parseEquation function is very simple, it just uses the parseOperation parser to parse the equation:

let parseEquation = pstring "What is " >>. parseOperation .>> pstring "?"
Use updated AST in solveExpression

The benefit of our modified AST becomes apparent when we work on our equation solving code. As an operator in our updated AST now knows about its operands, solving an expression is a simple, recursive function:

let rec solveExpression =
    function
    | Operand x -> x
    | Plus (left, right)         -> solveExpression left + solveExpression right
    | Minus (left, right)        -> solveExpression left - solveExpression right
    | DividedBy (left, right)    -> solveExpression left / solveExpression right
    | MultipliedBy (left, right) -> solveExpression left * solveExpression right

I think this function is very readable and elegant.

Use updated AST in solve

The solve function does not need any modification, so we can now run all tests, which all pass!

This is the complete, updated implementation:

open FParsec

type Expression = 
    | Operand of int
    | Plus of Expression * Expression
    | Minus of Expression * Expression
    | DividedBy of Expression * Expression
    | MultipliedBy of Expression * Expression

let buildExpression (left, operations) =
    List.fold (fun acc (operator, right) -> operator (acc, right)) left operations

// We create a parser forwarder in order to allow us to 
// define a recursive parser later on
let expr, exprImpl = createParserForwardedToRef()

let space = pchar ' '

let parseOperand = pint32 |>> Operand

let parseOperator = 
    stringReturn "plus"          Plus         <|> 
    stringReturn "minus"         Minus        <|> 
    stringReturn "divided by"    DividedBy    <|> 
    stringReturn "multiplied by" MultipliedBy

let parseOperation = 
    expr 
    .>> space 
    .>>. sepBy1 (parseOperator .>> space .>>. expr) space
    |>> (fun (left, operations) -> 
        List.fold (fun acc (operator, right) -> operator (acc, right)) left operations)

let parseExpression = parseOperand <|> parseOperation

exprImpl := parseExpression

let parseEquation =  pstring "What is " >>. parseOperation .>> pstring "?"

let parseToOption parser (input: string) =
    match run parser input with
    | Success(result, _, _)   -> Some result
    | Failure(errorMsg, _, _) -> None

let rec solveExpression =
    function
    | Operand x -> x
    | Plus (left, right) -> solveExpression left + solveExpression right
    | Minus (left, right) -> solveExpression left - solveExpression right
    | DividedBy (left, right) -> solveExpression left / solveExpression right
    | MultipliedBy (left, right) -> solveExpression left * solveExpression right

let solve (question: string) =
    parseToOption parseEquation question
    |> Option.map solveExpression

I think that this AST implementation is more elegant than our previous one, and also leads to more readable code.

Bonus: implementations in Scala and Haskell

One of the great things about Exercism is that almost all exercises are availabe in more than one language. This also goes for the wordy exercise. We will list our Scala and Haskell implementations of the wordy exercise to allow you to compare them to our F# implementation.

Bonus: Scala implementation

First up: our Scala implementation of the wordy problem:

import scala.util.parsing.combinator._

sealed trait Expression {
  def solve: Int
}
case class Operand(n: Int) extends Expression {
  def solve = n
}
case class Sum(e1: Expression, e2: Expression) extends Expression {
  def solve = e1.solve + e2.solve
}
case class Subtract(e1: Expression, e2: Expression) extends Expression {
  def solve = e1.solve - e2.solve
}
case class Multiply(e1: Expression, e2: Expression) extends Expression {
  def solve = e1.solve * e2.solve
}
case class Divide(e1: Expression, e2: Expression) extends Expression {
  def solve = e1.solve / e2.solve
}

object WordProblem extends RegexParsers {
  def operand: Parser[Operand] = """(0|-?[1-9]\d*)""".r ^^ { str => Operand(str.toInt) }

  def operator: Parser[String] = "plus" | "minus" | "multiplied by" | "divided by"

  def operation: Parser[Expression] =
    expression ~ (operator ~ expression).* ^^ {
      case expr ~ list => list.foldLeft(expr) {
        case (left, "plus" ~ right)          => Sum(left, right)
        case (left, "minus" ~ right)         => Subtract(left, right)
        case (left, "multiplied by" ~ right) => Multiply(left, right)
        case (left, "divided by" ~ right)    => Divide(left, right)
      }
    }

  def expression: Parser[Expression] = operand | operation

  def equation: Parser[Expression] = "What is " ~> operation <~ "?"

  def apply(input: String): Option[Int] = {
    parse(equation, input) match {
      case Success(expression, _) => Some(expression.solve)
      case _ => None
    }
  }
}

This implementation is actually quite similar to our F#, which is in large part due to the fact that we use Scala’s parser combinator library. There are two main differences:

  1. The case classes are Scala’s implementation of discriminated unions. As such, the solution is more object-oriented.
  2. The parser combinator library’s operators are different (^^ instead of |>>).

Bonus: Haskell implementation

For Haskell, we use the Attoparsec parser combinator library:

{-# LANGUAGE OverloadedStrings #-}

module WordProblem (answer) where

import Control.Applicative ((<|>), (<$>))
import Data.Attoparsec.Text (Parser, parse, maybeResult, space, decimal, signed, string, sepBy1)
import Data.Text (pack)

data Expression 
    = Operand Int 
    | Plus Expression Expression 
    | Minus Expression Expression 
    | DividedBy Expression Expression 
    | MultipliedBy Expression Expression
    deriving (Show)

buildExpression :: Expression -> [(Expression -> Expression -> Expression, Expression)] -> Expression 
buildExpression = foldl (\acc (operator, right) -> operator acc right)

operandParser :: Parser Expression 
operandParser = Operand <$> signed decimal

operatorParser :: Parser (Expression -> Expression -> Expression)
operatorParser = 
    (string "plus"          >> return Plus)         <|>
    (string "minus"         >> return Minus)        <|>
    (string "divided by"    >> return DividedBy)    <|>
    (string "multiplied by" >> return MultipliedBy)

operationParser :: Parser Expression
operationParser = do
    left       <- (operandParser <|> operationParser) <* space
    operations <- sepBy1 (do
        operator  <- operatorParser <* space 
        operation <- operandParser <|> operationParser
        return (operator, operation)) space
    return (buildExpression left operations)

equationParser :: Parser Expression
equationParser = string "What is " *> operationParser <* string "?"

solveExpression :: Expression -> Int
solveExpression (Operand x) = x
solveExpression (Plus left right) = solveExpression left + solveExpression right
solveExpression (Minus left right) = solveExpression left - solveExpression right
solveExpression (DividedBy left right) = solveExpression left `div` solveExpression right
solveExpression (MultipliedBy left right) = solveExpression left * solveExpression right

answer :: String -> Maybe Int
answer = fmap solveExpression . maybeResult . parse equationParser . pack

Once again, there are some syntactic differences and some different operators, but the gist of it remains the same. The only significant difference is that we use do notation, a feature similar to Scala’s for comprehensions and F#’s computation expressions.

Bonus: operator precedence

As a final bonus, we’ll look at what we would have to do if the wordy problem did respect the standard operator precedence rules. You might remember that we talked about this when we discussed test 13, which required us to solve "What is -3 plus 7 multiplied by -2?". Due to the exercise ignoring the operator precedence, the result was (-3 + 7) * -2 = -8. However, if we would use the standard convention that multiplication has a higher precedence than addition, the result should be -3 + (7 * -2) = -17. Let’s modify our implementation to follow the standard operator precedence rules.

Luckily for us, we don’t have to write any operator precedence handling code, FParsec has built-in capabilities for operator precedence handling. The basis is formed by the OperatorPrecedenceParser class. The first step is thus to create an OperatorPrecedenceParser instance, supplying it with the type we want it to return (int):

let opp = new OperatorPrecedenceParser<int,unit,unit>()

Note: you can ignore the second and third type parameter, we won’t be needing them.

The next step is to define how terms (which we referred to as operand) should be parsed:

opp.TermParser <- pint32 .>> spaces

For convenience purposes, we’ve added the spaces parser to our parser, which allows us to just ignore any trailing spaces.

We can now specify the operators. There are four types of operators:

Our equation only has infix operators, so we will be using the InfixOperator type. In order to create an InfixOperator, we have to specify five parameters:

  • The operator string.
  • The after-operator parser (which we’ll use to skip trailing spaces).
  • The operator precedence.
  • The associativity: none, left or right.
  • The binary function used to combine terms.

This is how we add the operators to represent our equation:

opp.AddOperator(InfixOperator("plus",          spaces, 1, Associativity.Left, fun x y -> x + y))
opp.AddOperator(InfixOperator("minus",         spaces, 1, Associativity.Left, fun x y -> x - y))
opp.AddOperator(InfixOperator("multiplied by", spaces, 2, Associativity.Left, fun x y -> x * y))
opp.AddOperator(InfixOperator("divided by",    spaces, 2, Associativity.Left, fun x y -> x / y))

Very simple. You can see that the "multiplied by" and "divided by" operators have a higher precedence (2) than the "plus" and "minus" operators (1).

We can of course simplify the term combination function by just passing the operator:

opp.AddOperator(InfixOperator("plus",          spaces, 1, Associativity.Left, (+)))
opp.AddOperator(InfixOperator("minus",         spaces, 1, Associativity.Left, (-)))
opp.AddOperator(InfixOperator("multiplied by", spaces, 2, Associativity.Left, (*)))
opp.AddOperator(InfixOperator("divided by",    spaces, 2, Associativity.Left, (/)))

At this point, the OperatorPrecedenceParser instance has been correctly configured. We can then use its ExpressionParser property in our parseEquation function:

let parseEquation = pstring "What is " >>. opp.ExpressionParser .>>  pstring "?"

If we now try to solve "What is -3 plus 7 multiplied by -2?", the answer will be -17, exactly what we expect.

The updated source that uses the OperatorPrecedenceParser is listed below:

open FParsec

let opp = new OperatorPrecedenceParser<int,unit,unit>()

opp.TermParser <- pint32 .>> spaces

opp.AddOperator(InfixOperator("plus",          spaces, 1, Associativity.Left, (+)))
opp.AddOperator(InfixOperator("minus",         spaces, 1, Associativity.Left, (-)))
opp.AddOperator(InfixOperator("multiplied by", spaces, 2, Associativity.Left, (*)))
opp.AddOperator(InfixOperator("divided by",    spaces, 2, Associativity.Left, (/)))

let parseEquation = pstring "What is " >>. opp.ExpressionParser .>>  pstring "?"

let parseToOption parser (input: string) =
    match run parser input with
    | Success(result, _, _)   -> Some result
    | Failure(errorMsg, _, _) -> None

let solve (question: string) = parseToOption parseEquation question

If we change the expected result for "What is -3 plus 7 multiplied by -2?" to Some -17 and run the tests again, we’ll see that all tests pass! Isn’t that amazing for a grand total of 13 lines of code?

The fun thing is, if we set the operator precedence of all operators to the same value, we have a fully valid implementation of the wordy problem!

Building a parser combinator from scratch

As we are really starting to appreciate parser combinators, let’s build one from scratch!

Got you! You didn’t really think we were going to do that, did you? If you are interested in how one would build a parser combinator library though, I highly recommend the Understanding Parser Combinators series on the fantastic F# for fun and profit website. In that series, Scott Wlaschin gradually builds a fully functional parser combinator library from scratch.

Conclusion

Solving the wordy problem was a very interesting exercise. The first few tests could be solved using just string replacing, splitting and converting. That approach quickly showed its defects, so we tried something else.

The second approach used regular expressions to parse the text. Using more advanced regular expression features (such as positive lookahead assertions), we could solve the problem using relatively little code. The readability was not great though, so it was time for another rewrite.

Our final approach used the FParsec parser combinator library. This resulted in a very elegant and concise solution. For fun, we then defined an Abstract Syntax Tree for the syntax being parsed, which was surprisingly easy and required very little code. We then showed how easy it is to define an expression parser with operator precedence rules in FParsec, further simplifying our implementation.

Of the three approaches we tried, the FParsec one was our favorite by far. It resulted in a simple, elegant implementation that was very readable. This was for a large part due to the many useful parsers FParsec includes out of the box. With these parser building blocks, we quickly and easily built our own, more complex parser. All in all, I can wholeheartedly recommend FParsec, whenever you have a text-parsing problem.

 

In ASP.NET MVC, views are defined by Razor templates. One very useful feature of Razor is that it allows you to generate HTML using simple lambda expressions, strongly-typed to your view model. However, if you have a complex, nested model and want to combine lambda expressions, things stop working. In this post, we’ll see how to solve this problem.

Simple example

Consider the following class that models user-related permissions:

public class UserPermissions
{
    public bool Add { get; set; }
    public bool Edit { get; set; }
    public bool Delete { get; set; }
}

If we set the @model directive of a view to our UserPermissions class, the view’s HTML helper will be bound to this class. We can then specify the property to generate HTML for by passing a lamdba to the HTML helper’s methods:

@model UserPermissions

<dl class="dl-horizontal">
    <dt>@Html.DisplayNameFor(m => m.Add)</dt>
    <dd>@Html.DisplayFor(m => m.Add)</dd>

    <dt>@Html.DisplayNameFor(m => m.Edit)</dt>
    <dd>@Html.DisplayFor(m => m.Edit)</dd>

    <dt>@Html.DisplayNameFor(m => m.Delete)</dt>
    <dd>@Html.DisplayFor(m => m.Delete)</dd>
</dl>

If we assume that Add is true and Edit and Delete are false, the following HTML output is rendered:

<dl class="dl-horizontal">
    <dt>Add</dt>
    <dd>
        <input checked="checked" class="check-box" disabled="disabled" type="checkbox" />
    </dd>

    <dt>Edit</dt>
    <dd>
        <input class="check-box" disabled="disabled" type="checkbox" />
    </dd>

    <dt>Delete</dt>
    <dd>
        <input class="check-box" disabled="disabled" type="checkbox" />
    </dd>

</dl>

Clean and simple. Let’s move to a more complex model.

More complex example

For our more complex example, we’ll create a model that stores the user permissions of several types of users:

public class RolePermissions
{
    public UserPermissions User { get; set; }
    public UserPermissions Moderator { get; set; }
    public UserPermissions Administrator { get; set; }
}

The following view allows us to edit these permissions:

@model RolePermissions

@using (Html.BeginForm())
{
    @Html.AntiForgeryToken()

    <table>
        <thead>
        <tr>
            <th></th>
            <th>@Html.DisplayNameFor(m => m.User)</th>
            <th>@Html.DisplayNameFor(m => m.Moderator)</th>
            <th>@Html.DisplayNameFor(m => m.Administrator)</th>
        </tr>
        </thead>
        <tbody>
        <tr>
            <td>@Html.DisplayNameFor(m => m.User.Add)</td>
            <td>@Html.EditorFor(m => m.User.Add)</td>
            <td>@Html.EditorFor(m => m.Moderator.Add)</td>
            <td>@Html.EditorFor(m => m.Administrator.Add)</td>
        </tr>
        <tr>
            <td>@Html.DisplayNameFor(m => m.User.Edit)</td>
            <td>@Html.EditorFor(m => m.User.Edit)</td>
            <td>@Html.EditorFor(m => m.Moderator.Edit)</td>
            <td>@Html.EditorFor(m => m.Administrator.Edit)</td>
        </tr>
        <tr>
            <td>@Html.DisplayNameFor(m => m.User.Delete)</td>
            <td>@Html.EditorFor(m => m.User.Delete)</td>
            <td>@Html.EditorFor(m => m.Moderator.Delete)</td>
            <td>@Html.EditorFor(m => m.Administrator.Delete)</td>
        </tr>
        </tbody>
    </table>

    <button type="submit">Save</button>
}

Although this view is more complex, it’s still quite readable. The rendered view looks something like this:

Edit model view rendered

It’s not the prettiest of GUI’s, but it is fully functional. To verify that our model is bound properly, let’s first examine the view’s controller:

using System.Web.Mvc;

public class PermissionsController : Controller
{
    [HttpGet]
    public ActionResult Index()
    {
        return View();
    }

    [HttpPost]
    [ValidateAntiForgeryToken]
    public ActionResult Index(RolePermissions model)
    {
        return View(model);
    }
}

Once again, nothing exciting happening here. If we submit our form and put a breakpoint on the return View(model); line, we can verify that model binding still works perfectly.

Removing duplication

Although our view looks nice, it does have a fair amount of duplication. In particular, the rows in the <tbody> section all have the exact same structure:

  • The first column contains the name of the permission.
  • The second column allows editing of that permission for the User role.
  • The third column allows editing of that permission for the Moderator role.
  • The fourth column allows editing of that permission for the Administrator role.

If one or more user types are added, or new types of permissions introduced, we have to edit our template in multiple places. This is tedious and error-prone, so let’s try to remove this repetition.

In Razor, you can extract shared functionality into reusable components using Razor helpers, which are functions that return HTML. What’s great though, is that you use a Razor template to define the HTML it returns. You can think of Razor helpers as a parameterized Razor view.

A first, naive attempt to create such a helper for our <tbody> rows would look something like this:

@helper PermissionRow(Func<UserPermissions, bool> permission)
{
    <tr>
        <td>@Html.DisplayNameFor(m => permission(m.User))</td>
        <td>@Html.EditorFor(m => permission(m.User))</td>
        <td>@Html.EditorFor(m => permission(m.Moderator))</td>
        <td>@Html.EditorFor(m => permission(m.Administrator))</td>
    </tr>
}

We can then eliminate much of the duplication as follows:

<tbody>
@PermissionRow(permissions => permissions.Add)
@PermissionRow(permissions => permissions.Edit)
@PermissionRow(permissions => permissions.Delete)
</tbody>

Unfortunately, if we execute this code, we get the following runtime exception:

Additional information: Templates can be used only with field access, property access, single-dimension array index, or single-parameter custom indexer expressions.

What this error message tells us, is that the HTML helper methods only accept a very restricted set of expressions, namely ones that directly reference a property. In our example, we used a function to reference the property, which is not supported.

In our previous example, we did directly reference the property: @Html.EditorFor(m => m.User.Delete), which worked fine. That lambda’s expression is of type Expression<RolePermissions, bool>, so how can we convert the Func<UserPermissions, bool> parameter of our Razor helper to Expression<RolePermissions, bool>? Let’s find out!

Solution

To convert our Func<UserPermissions, bool> expression to Expression<RolePermissions, bool>, we’ll start by wrapping it in an Expression:

@helper PermissionRow(Expression<Func<UserPermissions, bool>> permission)
{
    <tr>
        <td>@Html.DisplayNameFor(m => permission(m.User))</td>
        <td>@Html.EditorFor(m => permission(m.User))</td>
        <td>@Html.EditorFor(m => permission(m.Moderator))</td>
        <td>@Html.EditorFor(m => permission(m.Administrator))</td>
    </tr>
}

The reason we use an Expression<Func<T>> instead of a Func<T>, is that expressions can be transformed into other expressions. We’ll see how we can use this feature shortly.

The next step is to extract the column rendering from our PermissionRow HTML helper to a separate HTML helper: PermissionColumn. This helper will receive two parameters:

  1. An expression describing a function from UserPermission to bool. This expression points to the permission property: Add, Edit or Delete.
  2. An expression describing a function from RolePermissions to UserPermission. This expression points to the user permission property: User, Moderator or Administrator.

The basic structure of this helper looks like this:

@helper PermissionColumn(
    Expression<Func<UserPermissions, bool>> userPermission, 
    Expression<Func<RolePermissions, UserPermissions>> rolePermission)
{
    <td>@Html.EditorFor("TODO: combine the two expressions")</td>
}

At this point, we are almost there. If we could combine the two expressions into one expression, in which we pipe the output of Func<RolePermissions, UserPermissions> into Func<UserPermissions, bool>, we would end up with an expression of type Expression<Func<RolePermissions, bool>>, precisely what we need!

So how can we combine these two expressions into a single expression? Let’s start with the basic skeleton of the combine method:

public static Expression<Func<RolePermissions, UserPermissions>> 
    Combine<RolePermissions, UserPermissions, bool>
        (Expression<Func<RolePermissions, UserPermissions>> outer,
         Expression<Func<UserPermissions, bool>> inner)
{
    return Expression.Lambda<Func<RolePermissions, bool>>(TODO, TODO);
}

In this method, we use the Expression.Lambda method to create a new expression. It takes two parameters:

  1. The body of the expression.
  2. The parameters passed to the expression in the body.

If you think about it, the second parameter must be equal to the outer expression’s parameters. That’s one step completed:

Expression.Lambda<Func<RolePermissions, bool>>(body, outer.Parameters);

For the first parameter, we need to construct an expression that represents the body of the combined expression. What we want is to pass the output of the Func<RolePermissions, UserPermissions> expression to the input parameter of the Func<UserPermissions, bool> expression. If you think about this backwards, this is the same as replacing the input parameter of the Func<UserPermissions, bool> expression with the output of the body of the Func<RolePermissions, UserPermissions> expression.

To transform/rewrite an expression, the ExpressionVisitor class can be used. Surprisingly, we need very little code to create a visitor that replaces one expression with another:

private class SwapVisitor : ExpressionVisitor
{
    private readonly Expression _from;
    private readonly Expression _to;

    public SwapVisitor(Expression from, Expression to)
    {
        _from = from;
        _to = to;
    }

    public override Expression Visit(Expression node) => node == _from ? _to : base.Visit(node);
}

The constructor takes two Expression parameters:

  1. The from parameter is the expression we want to replace.
  2. The to parameter is the expression we want to replace it with.

The actual rewriting happens in the Visit() method. There, we check if the node parameter equals the from expression. If so, we return the to expression; otherwise, we just continue visiting the node.

We can now use this class to create an expression visitor in which we replace the inner expression’s parameter (which has one parameter) with the outer expression’s body:

var swap = new SwapVisitor(inner.Parameters[0], outer.Body);

The final step is to use this swap visitor to create our target expression, which we do by calling its Visit() method with the inner expression’s body:

using System;
using System.Linq.Expressions;

public static class Expressions
{
    public static Expression<Func<RolePermissions, bool>> 
        Combine<RolePermissions, UserPermissions, bool>
            (Expression<Func<RolePermissions, UserPermissions>> outer,
             Expression<Func<UserPermissions, bool>> inner)
    {
        var swap = new SwapVisitor(inner.Parameters[0], outer.Body);
        return Expression.Lambda<Func<TOuter, TProperty>>(
            swap.Visit(inner.Body), outer.Parameters);
    }    
}

When the Visit method is called, the following happens:

  1. The node parameter of the Visit() method will equal the inner expression’s body. As this expression does not equal the inner.Parameters[0] expression, the expression is passed to base.Visit().
  2. Within base.Visit(), the expression’s children are visited. As the expression is a function, at some point it will visit the expression representing the function’s parameters.
  3. The Visit() method is now called with the inner.Parameters[0] expression, which it recognizes to be equal to the from expression, so the to expression (the outer expression’s body) is returned.

Finally, the Visit method will return the transformed expression, which has the type we were looking for: Expression<Func<RolePermissions, bool>>.

With our PermissionColumn helper is finished, we can use it in our PermissionRow helper:

@helper PermissionRow(Expression<Func<UserPermissions, bool>> permission)
{
    var displayName = Expressions.Combine((RolePermissions model) => model.User, permission);
    <tr>
        <td>@Html.DisplayNameFor(displayName)</td>
        <td>@PermissionColumn(permission, m => m.User)</td>
        <td>@PermissionColumn(permission, m => m.Moderator)</td>
        <td>@PermissionColumn(permission, m => m.Administrator)</td>
    </tr>
}

We can now run our application to verify that everything still works, which it does!

Making the extension generic

The final step is to make the Combine method generic, which is just a matter of replacing the concrete types with type parameters:

public static Expression<Func<TOuter, TProperty>> Combine<TOuter, TInner, TProperty>(
    Expression<Func<TOuter, TInner>> outer,
    Expression<Func<TInner, TProperty>> inner)
{
    var swap = new SwapVisitor(inner.Parameters[0], outer.Body);
    return Expression.Lambda<Func<TOuter, TProperty>>(swap.Visit(inner.Body), outer.Parameters);
}

And there we have it: a fully generic method that can combine Func<TOuter, TInner> and Func<TInner, TProperty> expressions into an Expression<Func<TOuter, TProperty>> expression.

Conclusion

HTML helpers in Razor views are a very convenient feature. In this post we showed how to use the ExpressionVisitor class to combine two expressions into a single expression. We used this feature to allow us to write small Razor helpers with which we were able to greatly reduce duplication in our view.

 

Building mobile apps can be a daunting task. One of the biggest problems is that each platform has its own, specific ecosystem, which prevents you from reusing code between platforms. The Ionic 2 platform lets you build apps for multiple platforms from a single code base, using only HTML, CSS and JavaScript. Ionic 2 is built on the powerful Angular 2 framework.

In this post, we’ll build a native app using Ionic 2. To demonstrate that Ionic 2 apps are real, native mobile apps, we’ll be accessing the device’s hardware to record (and playback) audio.

Installation

The first step to start building Ionic 2 apps is to install the Ionic 2 SDK (which is still in beta):

npm install -g ionic@beta

Once this command has completed, we can use the Ionic CLI to scaffold our app:

ionic start ionic2-recorder blank --v2

With this command, the CLI will scaffold an Ionic 2 app named ionic2-recorder using the blank template:

Creating Ionic app in folder /Users/erikschierboom/Programming/ionic2-recorder based on blank project
Downloading: https://github.com/driftyco/ionic2-app-base/archive/master.zip
[=============================]  100%  0.0s
Downloading: https://github.com/driftyco/ionic2-starter-blank/archive/master.zip
[=============================]  100%  0.0s
Installing npm packages...

Adding initial native plugins
[=============================]  100%  0.0s

Note that scaffolding the app can take quite a while, so this might be a good time to check out the getting started documentation.

After some waiting, the CLI will finish and our app has been scaffolded:

Adding in iOS application by default
add to body class: platform-ios
Saving your Ionic app state of platforms and plugins
Saved platform
Saved plugins
Saved package.json

♬ ♫ ♬ ♫  Your Ionic app is ready to go! ♬ ♫ ♬ ♫

Running in the browser

Before we start building our app, let’s see what the scaffolded app looks like. First, we go to the directory into which our app was scaffolded:

cd ionic2-recorder

Then we ask the CLI to show a preview of our app in the browser:

ionic serve

This will result in some Gulp tasks being run:

Running 'serve:before' gulp task before serve
[12:13:47] Starting 'clean'...
[12:13:47] Finished 'clean' after 6.43 ms
[12:13:47] Starting 'watch'...
[12:13:47] Starting 'sass'...
[12:13:47] Starting 'html'...
[12:13:47] Starting 'fonts'...
[12:13:47] Starting 'scripts'...
[12:13:47] Finished 'html' after 70 ms
[12:13:47] Finished 'scripts' after 72 ms
[12:13:47] Finished 'fonts' after 77 ms
[12:13:48] Finished 'sass' after 782 ms
9.1 MB bytes written (3.96 seconds)
[12:13:54] Finished 'watch' after 7.31 s
[12:13:54] Starting 'serve:before'...
[12:13:54] Finished 'serve:before' after 5.46 μs

Running live reload server: http://localhost:35729
Watching: www/**/*, !www/lib/**/*
√ Running dev server:  http://localhost:8100
Ionic server commands, enter:
  restart or r to restart the client app from the root
  goto or g and a url to have the app navigate to the given url
  consolelogs or c to enable/disable console log output
  serverlogs or s to enable/disable server log output
  quit or q to shutdown the server and exit

ionic $ 

When the Gulp tasks have completed, a browser window will open showing a preview of our app:

Browser preview of first version of the app

It may not look much, but consider this: we are previewing our app in a browser! Not on a device or simulator, but a browser! This means that you can develop apps just like you are developing a website. Let’s see how that works.

Building the app - beginning

To start building our app, open the scaffolded app’s directory in your favorite editor. Don’t be daunted by the number of files and directories, we are only interested in the app directory’s contents, which looks like this:

app
├── pages
|   └── home
|       ├── home.html
|       ├── home.scss
|       └── home.ts
├── theme
|   ├── app.core.scss
|   ├── app.ios.scss
|   ├── app.md.scss
|   ├── app.variables.scss
|   └── app.wp.scss
└── app.ts

Interestingly, this directory contains just three types of files:

  1. TypeScript files: the application’s logic. Compiled to JavaScript.
  2. SCSS files: the application’s styling. Compiled to CSS.
  3. HTML files: the application’s interface. Not compiled.

Clearly, as promised, an Ionic 2 app consists of only HTML, JavaScript and CSS, like any regular website.

At the moment, there is only one page in our app: the home page. Let’s see what the (scaffolded) home.html file looks like:

<ion-header>
  <ion-navbar>
    <ion-title>
      Ionic Blank
    </ion-title>
  </ion-navbar>
</ion-header>

<ion-content padding>
  The world is your oyster.
  <p>
    If you get lost, the <a href="http://ionicframework.com/docs/v2">docs</a> will be your guide.
  </p>
</ion-content>

Although there are some regular HTML tags like the <p> and <a> tags, there are also custom tags like <ion-header> and <ion-navbar>. These tags are specific to the Ionic framework and are known as components (which is an Angular 2 concept). Using these components, you’ll be able to quickly construct an interface for your app, without having to manually create and style them yourselves.

Modifying the app

As our first modification, let’s change the contents of the <ion-title> tag:

<ion-header>
  <ion-navbar>
    <ion-title>
      Ionic 2 Audio Recorder
    </ion-title>
  </ion-navbar>
</ion-header>

<ion-content padding>
  The world is your oyster.
  <p>
    If you get lost, the <a href="http://ionicframework.com/docs/v2">docs</a> will be your guide.
  </p>
</ion-content>

If we save this file, the CLI will rebuild the app and automatically refresh the browser window to show our app with the modified title:

Browser preview of text modification

As a first step towards implementing audio recording functionality, let’s add a “Start recording” button:

<ion-header>
  <ion-navbar>
    <ion-title>
      Ionic 2 Audio Recorder
    </ion-title>
  </ion-navbar>
</ion-header>

<ion-content padding>
  <p>
    <button>Start recording</button>
  </p>
</ion-content>

Note that we just use a regular <button> tag, as there is no <ion-button> component.

The updated app looks like this:

Added button

Improved device preview

Although our browser preview looks fine, it does have fairly odd dimensions for a mobile device. If you are using Google Chrome, you can do one better. First, right-click in our app preview window and click on Inspect. This will open the Developer Tools window. If you look closely, you’ll see an icon that looks like a mobile phone in front of a tablet next to the Elements tab:

Developer tools button

If we click on that icon, Google Chrome will render the page as if it was rendered on a mobile device, correct screen proportions and all:
Device preview

Note that you can also select other devices, change the dimensions, and much more.

Multi-platform preview

One great feature of Ionic is that it will adjust the look and feel of your app depending on the platform it runs on. To easily see the differences between these platforms, you can start the ionic serve command with the --lab parameter:

ionic serve --lab

This results in the preview window showing all three platforms side-by-side:

Compare platforms

Although the difference is not huge in this case, the buttons and headers do have a different look-and-feel for each platform, even though they all run the same code. Pretty neat, right?

Note: you should disable the device preview in the Developer Tools for this to display correctly.

Preparing for recording audio

Now it’s time to start adding audio recording functionality to our app. Recording audio is a native capability: it requires access to a microphone. Therefore, we can’t test our recording functionality in the browser, we’ll have to use an actual device (or simulator).

To run our app on a device or simulator, Ionic uses Cordova. The first step is thus to install Cordova:

npm install -g cordova

Now that Cordova has been installed, the next step is to install the ionic-native plugin:

npm install ionic-native --save

The purpose of Ionic Native is to provide Ionic-friendly wrappers around Cordova plugins, which in turn allow access to the device’s API’s (such as the audio recording API). Schematically, it looks something like this:

App ==> Ionic Native ==> Cordova Plugin ==> Cordova ==> Device

The next step is to find the Ionic Native component that allows us to record audio, which is the MediaPlugin component. According to its documentation, it depends on the cordova-plugin-media plugin, which we can install using the CLI:

ionic plugin add cordova-plugin-media

We now have all the prerequisites to record audio in our app. But what file should contain our audio recording code? Let’s recall the structure of the app directory:

app
├── pages
|   └── home
|       ├── home.html
|       ├── home.scss
|       └── home.ts
├── theme
|   └── ...
└── app.ts

At the moment, we have only one page: the home page, which is defined by the three files in the home directory:

  • home.html: defines the interface of our home page.
  • home.scss: defines any custom styling of our home page.
  • home.ts: defines the logic of the home page.

Clearly, we’ll have to add our audio recording code to home.ts, which looks like this:

import { Component } from '@angular/core';
import { NavController } from 'ionic-angular';

@Component({
  templateUrl: 'build/pages/home/home.html'
})
export class HomePage {
  constructor(public navCtrl: NavController) {

  }
}

It turns out that an Ionic 2 page is just a regular Angular 2 component, nothing fancy.

Note that the template URL must be relative to the directory where the app’s compiled HTML/JavaScript/CSS is located: the build directory.

Recording audio

To import the MediaPlugin component in our home.ts file, we add the following import statement:

import { MediaPlugin } from 'ionic-native';

In the MediaPlugin component’s documentation, we noticed it had a startRecord function, exactly what we need!

Note: if you don’t like to read documentation, most editors also have code-completion (due to Ionic 2 being written in TypeScript):

Visual Studio Code code-completion

Let’s add a startRecording function to our HomePage class, in which we’ll start recording audio:

startRecording() {
  let media = new MediaPlugin('../Library/NoCloud/recording.wav');
  media.startRecord();
}

The final step is to call the startRecording function when the “Start recording” button is clicked:

<button (click)="startRecording()">Start recording</button>

You might be tempted to try this new functionality in the browser, but unfortunately, there will be an error:

Browser recording error

If you look at the error message more closely, it states that:

EXCEPTION: Error in build/pages/home/home.html:10:12
ORIGINAL EXCEPTION: ReferenceError: Media is not defined

Hmmm, it appears it cannot find an instance of Media. Although this error message is a bit cryptic, there is a very simple explanation: we are trying to use the native audio recording capability, which is not supported in the browser. The solution is simple: we should test this code on a physical device or simulator.

Silently failing is not very user-friendly though, so let’s gracefully handle this error. First, we import the AlertController component:

import { NavController, AlertController } from 'ionic-angular';

Next, we require an AlertController instance to be injected into our constructor:

constructor(public navCtrl: NavController, 
            public alertCtrl: AlertController) {
}

We can then use this AlertController instance to create a helper method that will show an alert:

showAlert(message) {
  let alert = this.alertCtrl.create({
    title: 'Error',
    subTitle: message,
    buttons: ['OK']
  });
  alert.present();
}

Finally, we’ll add a try/catch block to our startRecording function and use the showAlert function to show an alert:

startRecording() {
  try {
    let media = new MediaPlugin('../Library/NoCloud/recording.wav');
    media.startRecord();
  }
  catch (e) {
    this.showAlert('Could not start recording.');
  }
}

Now, if we run our application in the browser and click on the “Start recording” button, we’ll see a nice alert:

Browser recording error

Running in a simulator

As said, to record audio we’ll need to use a device our simulator. In both cases, we need to ensure our app supports the iOS platform. We can list the platforms our app currently supports using:

ionic platform

For our app, it outputs the following:

Installed platforms: ios 4.1.1
Available platforms: amazon-fireos, android, blackberry10, browser, firefoxos, osx, webos

Clearly, our app already supports iOS. Note: this was done automatically by the CLI due to us scaffolding the app on a MacBook.

If iOS platform support would not have been installed, we could have installed it manually:

ionic platform add ios

To run our app in the iOS simulator, first make sure you have Xcode and an iOS simulator installed. If you have those prerequisites installed, we can run our app in the simulator using:

ionic emulate ios

This time, there is far more going on in the background:

Running 'emulate:before' gulp task before emulate
...
[11:24:12] Starting 'emulate:before'...
[11:24:12] Finished 'emulate:before' after 6.88 μs

Building project  : /Users/erikschierboom/Programming/ionic2-recorder/platforms/ios/ionic2-recorder.xcodeproj

	Configuration : Debug
	Platform      : emulator

CpResource cordova/build-release.xcconfig build/emulator/ionic2-recorder.app/build-release.xcconfig

CompileXIB ionic2-recorder/Classes/MainViewController.xib
CompileAssetCatalog build/emulator/ionic2-recorder.app ionic2-recorder/Images.xcassets

ProcessInfoPlistFile build/emulator/ionic2-recorder.app/Info.plist ionic2-recorder/ionic2-recorder-Info.plist
ProcessPCH build/sharedpch/ionic2-recorder-Prefix-ecagionshlazuyfofvjakcemdkry/ionic2-recorder-Prefix.pch.pch

CompileC build/ionic2-recorder.build/Debug-iphonesimulator/ionic2-recorder.build/Objects-normal/i386/AppDelegate.o
CompileC build/ionic2-recorder.build/Debug-iphonesimulator/ionic2-recorder.build/Objects-normal/i386/MainViewController.o
CompileC build/ionic2-recorder.build/Debug-iphonesimulator/ionic2-recorder.build/Objects-normal/i386/main.o ionic2-recorder/main.m normal
CompileC build/ionic2-recorder.build/Debug-iphonesimulator/ionic2-recorder.build/Objects-normal/i386/CDVLogger.o 
CompileC build/ionic2-recorder.build/Debug-iphonesimulator/ionic2-recorder.build/Objects-normal/i386/CDVSplashScreen.o
CompileC build/ionic2-recorder.build/Debug-iphonesimulator/ionic2-recorder.build/Objects-normal/i386/CDVDevice.o 
CompileC build/ionic2-recorder.build/Debug-iphonesimulator/ionic2-recorder.build/Objects-normal/i386/CDVViewController+SplashScreen.o
CompileC build/ionic2-recorder.build/Debug-iphonesimulator/ionic2-recorder.build/Objects-normal/i386/CDVStatusBar.o 
CompileC build/ionic2-recorder.build/Debug-iphonesimulator/ionic2-recorder.build/Objects-normal/i386/IonicKeyboard.o.1_0.compiler

Ld build/emulator/ionic2-recorder.app/ionic2-recorder normal i386

GenerateDSYMFile build/emulator/ionic2-recorder.app.dSYM build/emulator/ionic2-recorder.app/ionic2-recorder

Touch build/emulator/ionic2-recorder.app
    cd /Users/erikschierboom/Programming/ionic2-recorder/platforms/ios
    /usr/bin/touch -c /Users/erikschierboom/Programming/ionic2-recorder/platforms/ios/build/emulator/ionic2-recorder.app

** BUILD SUCCEEDED **


No target specified for emulator. Deploying to iPhone-6s-Plus, 9.3 simulator

The build script doesn’t just compile our app to HTML, JavaScript and CSS files, but this time it compiles our app to a real, native iOS app. The reason for this is simple: you can only run native apps in the simulator.

Once the native app has been compiled, the simulator will pop-up and our app is shown:

Simulator preview

Clicking on the “Start recording” button does not result in an error this time. However, we cannot verify if it actually recorded anything, so let’s fix this.

Adding audio playback support

To allow us to playback the recorded audio, we need to add several functions to the HomePage class:

  • A function to stop recording (which will save the recorded audio to file).
  • A function to playback the saved recording.
  • A function to stop the playback of the saved recording.

As all these functions need access to the same MediaPlugin instance, let’s store it in a private field:

export class HomePage {
  media: MediaPlugin = new MediaPlugin('../Library/NoCloud/recording.wav');
}

Next, we modify the existing startRecording function to use this private field and add the other functions:

startRecording() {
  try {
    this.media.startRecord();
  }
  catch (e) {
    this.showAlert('Could not start recording.');
  }
}

stopRecording() {
  try {
    this.media.stopRecord();
  }
  catch (e) {
    this.showAlert('Could not stop recording.');
  }
}

startPlayback() {
  try {
    this.media.play();
  }
  catch (e) {
    this.showAlert('Could not play recording.');
  }
}

stopPlayback() {
  try {
    this.media.stop();
  }
  catch (e) {
    this.showAlert('Could not stop playing recording.');
  }
}

Finally, we’ll update home.html to have a button for each function:

<p>
  <button (click)="startRecording()">Start recording</button>
</p>
<p>
  <button (click)="stopRecording()">Stop recording</button>
</p>
<p>
  <button (click)="startPlayback()">Start playback</button>
</p>
<p>
  <button (click)="stopPlayback()">Stop playback</button>
</p>

Let’s run our app again:

Simulator playback

Unfortunately, the screen is blank, indicating that an error occured. As it turns out, this error is due to the MediaPlugin instance being created when the class is constructed. At that point, the native media plugin will not yet have been loaded, so creating the MediaPlugin instance throws an error.

So what is the right time to create our shared MediaPlugin instance? Well, let’s look at the lifecycle events for an Ionic component (in order of occurance):

  • ionViewLoaded: runs when the page has loaded.
  • ionViewWillEnter: runs when the page is about to enter and become the active page.
  • ionViewDidEnter: runs when the page has fully entered and is now the active page.
  • ionViewWillLeave: runs when the page is about to leave and no longer be the active page.
  • ionViewDidLeave: runs when the page has finished leaving and is no longer the active page.
  • ionViewWillUnload: runs when the page is about to be destroyed and have its elements removed.
  • ionViewDidUnload: runs after the page has been destroyed and its elements have been removed.

Let’s try to initialize the MediaPlugin in the ionViewLoaded function:

export class HomePage {
  media: MediaPlugin;

  ionViewLoaded() {
    this.media = new MediaPlugin('../Library/NoCloud/recording.wav')
  }

  ...
}

If we run our app, the GUI is shown this time, but clicking on the “Start recording” button still issues an error:

Simulator playback

Moving the initialization to ionViewWillEnter also results in the same error, but moving it to ionViewDidEnter solves our problem:

Simulator playback

This time, everything works (provided we click on the buttons in the right order). We can record and playback audio in our Ionic 2 app, using just a couple of lines of JavaScript code!

As a funny side note, the official iOS simulator documentation explicitly states that the microphone can’t be used in the simulator :)

Running in a specific simulator

Each time we ran our app in the simulator, it opened the default simulator. However, it is also possible to specify the simulator that should be used.

The first step is to get a list of all (iOS) simulator device types. We can do this with the following command:

ios-sim showdevicetypes

This will output a list of device types that can be used in the simulator:

iPhone-5, 8.1
iPhone-5, 9.2
iPhone-5, 9.3
iPhone-6-Plus, 8.1
iPhone-6-Plus, 9.2
iPhone-6-Plus, 9.3
iPad-2, 8.1
iPad-2, 9.2
iPad-2, 9.3
iPad-Air, 8.1
iPad-Air, 9.2
iPad-Air, 9.3
Apple-TV-1080p, tvOS 9.2
Apple-Watch-38mm, watchOS 2.2
Apple-Watch-42mm, watchOS 2.2

We then specify the device type we want to use through the --target parameter:

ionic emulate ios --target="iPad-Air, 8.1"

The emulator will now load our app in an iPad Air simulator running iOS 8.1:

Simulator iPad

Running on a device

Although running our app in a simulator is fine, the ultimate test would be to run our app on a physical device. As I want to run our app on my iPhone, I need to make sure our app supports the iOS platform. Luckily for us, we already allowed our app to run on iOS when we ran our app in the simulator.

After having connected our iPhone to our MacBook, we then run:

ionic run ios

This should run our app on our device, but even though my device is recognized, an error occurs:

[....] Waiting up to 1 seconds for iOS device to be connected

[....] Found iPhone 6 Plus 'iPhone van Erik' (b6cb1e8bc407e8d32823fdf11d41d6e6941ae974) connected through USB.

2016-08-31 17:33:16.977 ios-deploy[3114:31946] [ !! ] Can't access app path '/Users/erikschierboom/Programming/ionic2-recorder/platforms/ios/build/device/ionic2-recorder.app' : No such file or directory

Error: Error code 253 for command: ios-deploy with args: --justlaunch,--no-wifi,-d,-b,/Users/erikschierboom/Programming/ionic2-recorder/platforms/ios/build/device/ionic2-recorder.app

After some Googling, I found that this problem can be fixed by appending --device:

ionic run ios --device

Similar to running our app on the simulator, our app is compiled to a native app:

Building project  : /Users/erikschierboom/Programming/ionic2-recorder/platforms/ios/Audio Recorder.xcodeproj

	Configuration : Debug
	Platform      : device

...

ProcessInfoPlistFile build/device/Audio\ Recorder.app/Info.plist Audio\ Recorder/Audio\ Recorder-Info.plist
Packaging application: '/Users/erikschierboom/Programming/ionic2-recorder/platforms/ios/build/device/Audio Recorder.app'

...

Results at '/Users/erikschierboom/Programming/ionic2-recorder/platforms/ios/build/device/Audio Recorder.ipa' 

[....] Waiting up to 1 seconds for iOS device to be connected
[....] Found iPhone 6 Plus 'iPhone van Erik' (b6cb1e8bc407e8d32823fdf11d41d6e6941ae974) connected through USB.
[....] Waiting for iOS device to be connected
[....] Using iPhone 6 Plus 'iPhone van Erik' (b6cb1e8bc407e8d32823fdf11d41d6e6941ae974).

------ Install phase ------
[  0%] Found iPhone 6 Plus 'iPhone van Erik' (b6cb1e8bc407e8d32823fdf11d41d6e6941ae974) connected through USB, beginning install
[  5%] Copying /Users/erikschierboom/Programming/ionic2-recorder/platforms/ios/build/device/Audio Recorder.app/META-INF/ to device

...

[ 49%] Copying /Users/erikschierboom/Programming/ionic2-recorder/platforms/ios/build/device/Audio Recorder.app/www/plugins/ionic-plugin-keyboard/www/ios/keyboard.js to device

[ 52%] CreatingStagingDirectory
[ 57%] ExtractingPackage
[ 60%] InspectingPackage
[ 60%] TakingInstallLock
[ 65%] PreflightingApplication
[ 65%] InstallingEmbeddedProfile
[ 70%] VerifyingApplication
[ 75%] CreatingContainer
[ 80%] InstallingApplication
[ 85%] PostflightingApplication
[ 90%] SandboxingApplication
[ 95%] GeneratingApplicationMap
[100%] Installed package /Users/erikschierboom/Programming/ionic2-recorder/platforms/ios/build/device/Audio Recorder.app

------ Debug phase ------
Starting debug of iPhone 6 Plus 'iPhone van Erik' (b6cb1e8bc407e8d32823fdf11d41d6e6941ae974) connected through USB...

[  0%] Looking up developer disk image
[ 95%] Developer disk image mounted successfully
[100%] Connecting to remote debug server

After a short while, our app will be displayed on our phone (make sure to have your phone unlocked):

App on iPhone

Clicking on the “Start recording” button now issues the following prompt:

Request microphone access on iPhone

If we click “OK”, we can now verify that our app also works on an iPhone.

Running on a device using Ionic View

Besides viewing our app in a browser, simulator or on a device, there is a fourth option: the Ionic View app. With this option, you install the Ionic View app on your iOS or Android device, which you can then use to test your apps with.

To do so, simply run:

ionic upload

Its output will look like this:

WARN: No 'upload:before' gulp task found!
If your app requires a build step, you may want to ensure it runs before upload.

No previous login existed. Attempting to log in now.

To continue, please login to your Ionic account.
Don't have one? Create one at: https://apps.ionic.io/signup

Email: erik_schierboom@hotmail.com
Password: 
Logged in! :)
Uploading app....
Saved app_id, writing to ionic.io.bundle.min.js...
Successfully uploaded (e128c829)

Share your beautiful app with someone:

$ ionic share EMAIL

Saved api_key, writing to ionic.io.bundle.min.js...

To upload our app, we first had to login to our Ionic account. If you don’t have one, you can create one here. After signing in, the app is uploaded and can be accessed in the Ionic View app.

Before we’ll see what that looks like, notice that the build output also issued a warning:

WARN: No 'upload:before' gulp task found!
If your app requires a build step, you may want to ensure it runs before upload.

This warning indicates that the upload Gulp task does not depend on any other task. If you want to be ensure that you’ll always upload the latest version of your app, you need to make sure that the upload task depends on the build task. Fixing this is easy, just add the following line to gulpfile.js:

gulp.task('upload:before', ['build']);

Having uploaded our app, let’s open the Ionic View app to see what it looks like:

Ionic View

We can see that we have access to one app: the one we just uploaded. If we click on it, we get the following menu:

Ionic View app menu

The available options are fairly straightforward. We’ll use the “View app” button to run our app:

App running in Ionic View

As you can see, running the app from the Ionic View app looks identical to running the app directly on our device. Furthermore, as Ionic View is a native app with access to all native capabilities, our app still perfectly records audio.

Sharing with other people

Up until now, we tested our app ourselves. But what if we want other people to test our app? Well, simple! Just use the ionic share command:

ionic share coworker@test.com

So what does this command do? Let’s examine its output:

Sharing app ionic2-recorder (e128c829) with coworker@test.com.
An invite to view your app was sent.

You can see that an email invite was sent to the specified email address. This email contains instructions on how to preview our app. In short, what the user must do is:

  1. Create an Ionic account.
  2. Install the Ionic View app.
  3. Follow the instructions in the email invite.

Having followed these three steps, the user will have access to our app from within their Ionic View app installation.

Note that the version of the app they will receive, will be the latest version that was uploaded through ionic upload.

Extracting the recording functionality

At the moment, the audio recording and playback functionality was added directly to the HomePage class. As a consequence, if we want to use the audio functionality somewhere else in our app, we’d have to either duplicate the code or extract the audio functionality into a separate class. Let’s try the second option.

First, we’ll create a services directory in our app directory. Within the services directory, create a TypeScript file named audiorecorder.ts, in which we’ll define an AudioRecorder class. Extracting the audio recording functionality from our HomePage class, we end up with the following class:

import { MediaPlugin } from 'ionic-native';

export class AudioRecorder {
  media: MediaPlugin;

  constructor() {
    this.media = new MediaPlugin('../Library/NoCloud/recording.wav');
  }

  startRecording() {
    this.media.startRecord();
  }

  stopRecording() {
    this.media.stopRecord();
  }

  startPlayback() {
    this.media.play();
  }

  stopPlayback() {
    this.media.stop();
  }
}

Looks quite nice, agreed? There is another, obvious refactoring. The MediaPlugin instance only really needs to be created when it is first used, so let’s delay creating the MediaPlugin instance:

export class AudioRecorder {
  mediaPlugin: MediaPlugin = null;

  get MediaPlugin(): MediaPlugin {
    if (this.mediaPlugin == null) {
      this.mediaPlugin = new MediaPlugin('../Library/NoCloud/recording.wav');
    }

    return this.mediaPlugin;
  }

  startRecording() {
    this.MediaPlugin.startRecord();
  }

  stopRecording() {
    this.MediaPlugin.stopRecord();
  }

  startPlayback() {
    this.MediaPlugin.play();
  }

  stopPlayback() {
    this.MediaPlugin.stop();
  }
}

Finally, wouldn’t it be nice if we could inject this class, just like we did with the AlertController? Easy. We just import the Injectable decorator and decorate our class with it:

import { Injectable } from '@angular/core';

@Injectable()
export class AudioRecorder {
  ...
}

With all these changes, our HomePage class becomes a lot easier. First, we can replace the import of the MediaPlugin class with an import of our AudioRecorder class:

import { AudioRecorder } from '../../services/audiorecorder';

If you are wondering how we arrived at this relative import path, consider the current layout of the app directory:

app
├── pages
|   └── home
|       ├── home.html
|       ├── home.scss
|       └── home.ts
├── theme
|   └── ...
├── services
|   └── audiorecorder.ts
└── app.ts

You’ll see that after moving up two directories from the home.ts file, we are in the app directory. Then it’s into to the services directory and its audiorecorder.ts file (note that you should omit the file’s extension).

Next, we’ll add the AudioRecorder to the list of providers for our class (to ensure it can be injected) and add it as a parameter to our constructor:

@Component({
  templateUrl: 'build/pages/home/home.html',
  providers: [AudioRecorder]
})
export class HomePage {
  
  constructor(public navCtrl: NavController, 
              public alertCtrl: AlertController,
              public audioRecorder: AudioRecorder) {
  }

  ...
}

Note that due to the lazy initialization of the MediaPlugin instance in the AudioRecorder class, we no longer have to defer creating the instance to the ionViewLoaded function.

Finally, we’ll update the audio handling functions to use our AudioRecorder instance:

startRecording() {
  try {
    this.audioRecorder.startRecording();
  }
  catch (e) {
    this.showAlert('Could not start recording.');
  }
}

stopRecording() {
  try {
    this.audioRecorder.stopRecording();
  }
  catch (e) {
    this.showAlert('Could not stop recording.');
  }
}

startPlayback() {
  try {
    this.audioRecorder.startPlayback();
  }
  catch (e) {
    this.showAlert('Could not play recording.');
  }
}

stopPlayback() {
  try {
    this.audioRecorder.stopPlayback();
  }
  catch (e) {
    this.showAlert('Could not stop playing recording.');
  }
}  

If we test our refactored app, we’ll find it still works as expected, but our code now has much better separation of concerns.

Updating the user-interface

Although we can record audio in our app, the interface could use some refinement. For example, we should only allow people to click on the “Start playback” button once something has actually been recorded. Similarly, we shouldn’t allow people to stop recording when nothing is being recorded. The status of the buttons thus depends on the “state” of the audio recorder. We’ll model this by adding the following enum to the audiorecorder.ts file:

export enum AudioRecorderState {
    Ready,
    Recording,
    Recorded,
    Playing
}

We then add a RecordingState field to the AudioRecorder and update it in each function:

export class AudioRecorder {
  state: AudioRecorderState = AudioRecorderState.Ready;

  startRecording() {
    this.MediaPlugin.startRecord();
    this.state = AudioRecorderState.Recording;
  }

  stopRecording() {
    this.MediaPlugin.stopRecord();
    this.state = AudioRecorderState.Recorded;
  }

  startPlayback() {
    this.MediaPlugin.play();
    this.state = AudioRecorderState.Playing;
  }

  stopPlayback() {
    this.MediaPlugin.stop();
    this.state = AudioRecorderState.Ready;
  }

  ...
}

The last step is to dynamically enable and disable buttons depending on the audio recorder’s state. For that, we’ll modify the home.html file:

<p>
  <button (click)="startRecording()" [disabled]="audioRecorder.state != AudioRecorderState.Ready">Start recording</button>
</p>
<p>
  <button (click)="stopRecording()" [disabled]="audioRecorder.state != AudioRecorderState.Recording">Stop recording</button>
</p>
<p>
  <button (click)="startPlayback()" [disabled]="audioRecorder.state != AudioRecorderState.Recorded">Start playback</button>
</p>
<p>
  <button (click)="stopPlayback()" [disabled]="audioRecorder.state != AudioRecorderState.Playing">Stop playback</button>
</p>

Unfortunately, when we try to run our app, we get an error:

EXCEPTION: Error in build/pages/home/home.html:10:39
ORIGINAL EXCEPTION: TypeError: Cannot read property 'Ready' of undefined

The problem is that our view doesn’t know about the AudioRecorderState enum. To fix this, we’ll also import the AudioRecorderState in our home.ts file:

import { AudioRecorder, AudioRecorderState } from '../../services/audiorecorder';

Then, we add it as a field to our HomePage class, which allows our view to access its values:

export class HomePage {
  AudioRecorderState = AudioRecorderState;

  ...
}

This time, things work as we expected them to:

Dynamically enabled buttons

At first, only the “Start recording” button is enabled. Once we press that, only the “Stop recording” button is enabled, and so on.

Adding color and icons

The finishing touch is to give each button its own color and icon. Let’s start with the colors. First, we’ll add classes to our buttons in home.html:

<p>
  <button (click)="startRecording()" class="buttton-start-recording" [disabled]="audioRecorder.state != AudioRecorderState.Ready">Start recording</button>
</p>
<p>
  <button (click)="stopRecording()" class="buttton-stop-recording" [disabled]="audioRecorder.state != AudioRecorderState.Recording">Stop recording</button>
</p>
<p>
  <button (click)="startPlayback()" class="buttton-start-playback" [disabled]="audioRecorder.state != AudioRecorderState.Recorded">Start playback</button>
</p>
<p>
  <button (click)="stopPlayback()" class="buttton-stop-playback" [disabled]="audioRecorder.state != AudioRecorderState.Playing">Stop playback</button>
</p>

Next, we’ll create the styles for these newly added classes. We’ll do that in home.scss:

.buttton-start-recording,
.buttton-start-recording.activated {
    background-color: red;
}

.buttton-start-recording:hover:not(.disable-hover) {
    background-color: #ff6666;
}

.buttton-stop-recording,
.buttton-stop-recording.activated {
    background-color: orange;
}

.buttton-stop-recording:hover:not(.disable-hover) {
    background-color: #ffc966;
}

.buttton-start-playback,
.buttton-start-playback.activated {
    background-color: green;
}

.buttton-start-playback:hover:not(.disable-hover) {
    background-color: #80ff80;
}

.buttton-stop-playback,
.buttton-stop-playback.activated {
    background-color: blue;
}

.buttton-stop-playback:hover:not(.disable-hover) {
    background-color: #387ef5;
}

If we preview our app using these changes, the buttons will have different background colors:

Button colors

Our final improvement is to add icons to the buttons. Adding an icon to our buttons is simple: just add an <ion-icon> component with the correct name attribute nested in the <button>:

<p>
  <button (click)="startRecording()" class="buttton-start-recording" [disabled]="audioRecorder.state != AudioRecorderState.Ready">
    <ion-icon name="microphone"></ion-icon>
    Start recording
  </button>
</p>
<p>
  <button (click)="stopRecording()" class="buttton-stop-recording" [disabled]="audioRecorder.state != AudioRecorderState.Recording">
    <ion-icon name="mic-off"></ion-icon>
    Stop recording
  </button>
</p>
<p>
  <button (click)="startPlayback()" class="buttton-start-playback" [disabled]="audioRecorder.state != AudioRecorderState.Recorded">
    <ion-icon name="play"></ion-icon>
    Start playback
  </button>
</p>
<p>
  <button (click)="stopPlayback()" class="buttton-stop-playback" [disabled]="audioRecorder.state != AudioRecorderState.Playing">
    <ion-icon name="square"></ion-icon>
    Stop playback
  </button>
</p>

Let’s preview these changes using the multi-platform preview mode:

Button icons

Looks much better, right? Those of you with a keen eye may have noticed that some icons will look different between platforms, which Ionic does automatically to better match each platform’s look-and-feel.

Source code

If you’d like to test the application yourself, the full source code can be found in my ionic2-audio-recorder repository.

Conclusion

Building native apps can be frustrating due to the lack of code sharing between the various platforms. With Ionic 2, you can build native mobile apps for multiple platforms from a single code base, using just HTML, JavaScript and CSS. This makes building an app quite similar to building a regular website.

Building our audio recorder app using Ionic 2 app turned out to be quite easy. Using just HTML and Ionic’s custom components, we created an app that looks like a proper native app. Even building the audio recording code was not very hard, due to Ionic Native giving us an easy to use API to handle audio with.

With helpful tools such as the Ionic CLI, a browser preview mode and the Ionic View app, building and sharing mobile apps has never been as easy or fun!

 

Writing unit tests helps verify the correctness of code. However, most unit tests only test a limited set of pre-defined input values, often just one. Testing with fixed input values is known as example-based tests.

The problem with example-based tests is that they only verify correctness for the pre-defined input values. This can easily lead to an implementation that passes the test for the pre-defined values, but fails for any other value. The implementation could thus be (largely) incorrect, even though the test passes.

Let’s demonstrate this through an example. Note that we’ll use xUnit.net as the test framework in our examples.

Single input value

Suppose we want to write a method that calculates the MD5 hash of a given string. Let’s start by writing a unit test:

[Fact]
public void MD5ReturnsCorrectHash()
{
    var input = "foo";
    var md5 = MD5(input);
    var expected = "acbd18db4cc2f85cedef654fccc4a4d8";
    Assert.Equal(expected, md5);
}

This test method use a single input value ("foo") to verify the correctness of the MD5() method. Therefore, this is an example-based test. To make this test pass, we can implement the MD5() method as follows:

public static string MD5(string input)
{
    return "acbd18db4cc2f85cedef654fccc4a4d8";
}

Clearly, this implementation does not correctly implement the MD5 algorithm, but it passes our test! The danger of using one input value to verify correctness is that you can make the test pass by just hard-coding the expected result.

Note that in TDD, you actually should write the minimal amount of code to make the test pass, so the above implementation is perfectly reasonable when doing TDD.

In the next section, we’ll see how to strengthen our test by using multiple input values.

Multiple input values

The obvious solution to strengten tests that use one input value, is to use several input values. One way to do this is to create copies of the existing test method, but with different input values:

[Fact]
public void MD5ForFooInputReturnsCorrectHash()
{
    var md5 = MD5("foo");
    Assert.Equal("acbd18db4cc2f85cedef654fccc4a4d8", md5);
}

[Fact]
public void MD5ForBarInputReturnsCorrectHash()
{
    var md5 = MD5("bar");
    Assert.Equal("37b51d194a7513e45b56f6524f2d51f2", md5);
}

[Fact]
public void MD5ForBazInputReturnsCorrectHash()
{
    var md5 = MD5("baz");
    Assert.Equal("73feffa4b7f6bb68e44cf984c85f6e88", md5);
}

Although there is nothing wrong with these three tests, we have some code duplication. Luckily, xUnit has the concept of parameterized tests, which allows us to define a single test with more than one input value.

Here is the parameterized test equivalent of our previous three tests:

[Theory]
[InlineData("foo", "acbd18db4cc2f85cedef654fccc4a4d8")]
[InlineData("bar", "37b51d194a7513e45b56f6524f2d51f2")]
[InlineData("baz", "73feffa4b7f6bb68e44cf984c85f6e88")]
public void MD5ReturnsCorrectHash(string input, string expected)
{
    var md5 = MD5(input);
    Assert.Equal(expected, md5);
}

This test differs from our previous tests in several ways:

  • The [Fact] attribute is replaced with the [Theory] attribute. This marks the test as a parameterized test.
  • The test method has two parameters: input and expected, which replace the hard-coded values in our test.
  • Three [InlineData] attributes have been added, one for each input value/expected hash combination.

When xUnit runs this test, it will actually run it three times, with the [InlineData] attributes’ parameters used as the input and expected parameters. Therefore, running our parameterized test results in our test being called three times with the following arguments:

MD5ReturnsCorrectHash("foo", "acbd18db4cc2f85cedef654fccc4a4d8");
MD5ReturnsCorrectHash("bar", "37b51d194a7513e45b56f6524f2d51f2");
MD5ReturnsCorrectHash("baz", "73feffa4b7f6bb68e44cf984c85f6e88");

If we run our parameterized test, it will fail for the "bar" and "baz" input values. To make our test pass, we could again hard-code the expected values:

public static string MD5(string input)
{
    if (input == "foo")
    {
        return "acbd18db4cc2f85cedef654fccc4a4d8";
    }
    if (input == "bar")
    {
        return "37b51d194a7513e45b56f6524f2d51f2";
    }
    if (input == "baz")
    {
        return "73feffa4b7f6bb68e44cf984c85f6e88";
    }
    
    return input;
}

With this modified implementation, the test passes for all three input values. Unfortunately, having multiple tests still allowed us to easily hard-code the implementation; it did not prevent us from incorrectly implementing the MD5 algorithm.

So although having multiple input values leads to stronger tests, it still remains a limited set of input values. Wouldn’t it be great if we could run our tests using all possible input values? Enter property-based testing.

Property-based testing

In property-based testing, we take a different approach to writing our tests. Instead of testing for specific input and output values, we test the properties of the code we are testing. You can think of properties as rules, invariants or requirements. For example, these are some essential properties of the MD5 algorithm:

  1. The hash is 32 characters long.
  2. The hash only contains hexadecimal characters.
  3. Equal inputs have the same hash.
  4. The hash is different from its input.
  5. Similar inputs have significantly different hashes.

So how do we write tests for these properties? First, notice that these five properties are generic: they must be true for all possible input values. Unfortunately, writing tests that actually check all input values is not feasible, as running them would take ages. But if we can’t test all input values, how can we write tests for our properties? Well, we use the next best thing: random input values.

If you test using random input values, hard-coding the expected values is no longer possible as you don’t know beforehand which input values will be tested. This forces you to write a generic implementation that works for any (valid) input value.

Let’s see how that works by writing property-based tests for all five properties we just defined.

Property 1: hash is 32 characters long

Our first property states that an MD5 hash is a string of length 32. If we would write this as a regular, example-based test, it would look like this:

[Fact]
public void MD5ReturnsStringOfCorrectLength()
{
    var input = "foo";
    var hash = MD5(input);
    Assert.Equal(32, hash.Length);
}

We could strengthen our test by converting it to a parameterized test with several input values:

[Theory]
[InlineData("foo")]
[InlineData("bar")]
[InlineData("baz")]
public void MD5ReturnsStringOfCorrectLength(string input)
{
    var hash = MD5(input);
    Assert.Equal(32, hash.Length);
}

However, as said, property-based tests should work with any input value. A property-based test is thus a parameterized test, but with the pre-defined input values replaced by random input values. Our initial attempt at writing a property-based test might look like this:

[Theory]
public void MD5ReturnsStringOfCorrectLength(string input)
{
    var hash = MD5(input);
    Assert.Equal(32, hash.Length);
}

Unfortunately, if we run this test, we’ll find that xUnit reports an error for parameterized tests without input.

To define a test that works with randomly generated input, we’ll use the FsCheck property-based testing framework. We’ll also install the FsCheck.Xunit package, for easy integration of FsCheck with xUnit.

Having installed these libraries, we can convert our test to a property-based test by decorating it with the [Property] attribute:

[Property]
public void MD5ReturnsStringOfCorrectLength(string input)
{
    var hash = MD5(input);
    Assert.Equal(32, hash.Length);
}

Note that we don’t explicitly specify the input value(s) to use, FsCheck will (randomly) generate those. Let’s run this test to see what happens:

MD5ReturnsStringOfCorrectLength [FAIL]

    FsCheck.Xunit.PropertyFailedException : 
    Falsifiable, after 1 test (0 shrinks) (StdGen (766423555,296119444)):
    Original:
    ""

    ---- Assert.Equal() Failure
    Expected: 32
    Actual:   0

The test report indicates that our property-based test failed after one test, for the randomly generated empty string ("") input value. To make our empty string pass the test, we’ll pad unknown inputs to a length of 32:

public static string MD5(string input)
{
    if (input == "foo")
    {
        return "acbd18db4cc2f85cedef654fccc4a4d8";
    }
    if (input == "bar")
    {
        return "37b51d194a7513e45b56f6524f2d51f2";
    }
    if (input == "baz")
    {
        return "73feffa4b7f6bb68e44cf984c85f6e88";
    }
    
    return input.PadRight(32);
}

This should fix the empty string problem, so let’s run the test again:

MD5ReturnsStringOfCorrectLength [FAIL]

    FsCheck.Xunit.PropertyFailedException : 
    Falsifiable, after 12 tests (0 shrinks) (StdGen (1087984417,296119448)):
    Original:
    <null>
        
    ---- System.NullReferenceException : Object reference not set to an instance of an object

Hmmm, our test still fails. This time though, the first 11 tests passed, so we made some progress. The test now failed when FsCheck generated the null input value. Let’s fix this:

public static string MD5(string input)
{
    if (input == "foo")
    {
        return "acbd18db4cc2f85cedef654fccc4a4d8";
    }
    if (input == "bar")
    {
        return "37b51d194a7513e45b56f6524f2d51f2";
    }
    if (input == "baz")
    {
        return "73feffa4b7f6bb68e44cf984c85f6e88";
    }
    
    return (input ?? string.Empty).PadRight(32);
}

And we run our test again:

MD5ReturnsStringOfCorrectLength [FAIL]

    FsCheck.Xunit.PropertyFailedException : 
    Falsifiable, after 43 tests (34 shrinks) (StdGen (964736467,296119643)):
    Original:
    "#e3n+[TC9[Jlbs,x=3U!f\~J'i u+)-y>4VLg]uA("
                
    Shrunk:
    "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"

    ---- Assert.Equal() Failure
    Expected: 32
    Actual:   33

Again, more progress, as it now took 43 tests before the test failed for the input value "#e3n+[TC9[Jlbs,x=3U!f\~J'i u+)-y>4VLg]uA(". There is something different about the test report this time though, as it mentions “34 shrinks” and a “shrunk” value, what is that about?

Shrinking

In property-based testing, shrinking is used to find a minimal counter-example that proves the property does not hold for all possible input values. This minimal counter example is listed as the “shrunk” in our test report. Before we examine how shrinking works, try to figure out for yourself what’s so special about the “shrunk” "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" input value mentioned in the test report.

If you guessed that the specified “shrunk” input was special for its length, you’d be correct! The "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" string contains 33 characters, which significance becomes apparent if we look at the code in our implementation that processes this input value:

return (input ?? string.Empty).PadRight(32);

If we use "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" as our input, the PadRight(32) call doesn’t do anything as our string is already greater than or equal to 32. Therefore, the input value is returned unmodified, which means that the string that is returned also has length 33, which fails our test.

The interesting thing about an input value of length 33 is that 33 is the minimum length for which the test fails; strings of length 32 or less all pass the test. As such, the listed “shrunk” value of length 33 is a minimal counter-example to our property.

The main benefit of having a minimal counter-example is that is helps you locate precisely where your implementation breaks down (in our case for strings of length 33 and greater). You thus spend less time debugging and more writing code.

Finding the “shrunk” value

So how does FsCheck find this shrunk value? To find out, we’ll have FsCheck output the values it generates. Doing that is easy, we just set the Verbose property of our [Property] attribute to true:

[Property(Verbose = true)]
public void MD5ReturnsStringOfCorrectLength(string input)
{
    var hash = MD5(input);
    Assert.Equal(32, hash.Length);
}

Now if we run our test, we’ll see exactly which values FsCheck generated as test input:

MD5ReturnsStringOfCorrectLength [FAIL]

    FsCheck.Xunit.PropertyFailedException : 
    Falsifiable, after 50 tests (38 shrinks) (StdGen (1153044621,296119646)):
    Original:
    "XqyW\O!Lr%ce3]4=H~=6lG, 5lT\aDz%n9"
    Shrunk:
    "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"

    ---- Assert.Equal() Failure
    Expected: 32
    Actual:   33

    Output:
    0: "5z"
    1: "Y"
    2: "r"
    3: ""
    4: "t9[Q"
    ...
    45: <null>
    46: "Qlbz?|perK"
    47: "XP3vO$-`l"
    48: Y{q6oevZA7"0R
    49: "XqyW\O!Lr%ce3]4=H~=6lG, 5lT\aDz%n9"

    shrink: "qyW\O!Lr%ce3]4=H~=6lG, 5lT\aDz%n9"
    shrink: "yW\O!Lr%ce3]4=H~=6lG, 5lT\aDz%n9"
    shrink: "yW\O!Lr%ce3]4=H~=6lG,5lT\aDz%n9"
    shrink: "W\O!Lr%ce3]4=H~=6lG,5lT\aDz%n9"
    shrink: "\O!Lr%ce3]4=H~=6lG,5lT\aDz%n9"
    shrink: "O!Lr%ce3]4=H~=6lG,5lT\aDz%n9"
    shrink: "O!Lr%ce3]4=H~=6lG,5lT\aDz%n9a"
    shrink: "O!Lr%ce3]4=H~=6lG,5lT\aDz%naa"
    shrink: "O!Lr%ce3]4=H~=6lG,5lT\aDz%aaa"
    shrink: "O!Lr%ce3]4=H~=6lG,5lT\aDzaaaa"
    shrink: "O!Lr%ce3]4=H~=6lG,5lT\aDaaaaa"
    shrink: "O!Lr%ce3]4=H~=6lG,5lT\aaaaaaa"
    shrink: "O!Lr%ce3]4=H~=6lG,5lTaaaaaaaa"
    shrink: "O!Lr%ce3]4=H~=6lG,5lTaaaaaaaaa"
    shrink: "O!Lr%ce3]4=H~=6lG,5laaaaaaaaaa"
    shrink: "O!Lr%ce3]4=H~=6lG,5laaaaaaaaaaa"
    shrink: "O!Lr%ce3]4=H~=6lG,5aaaaaaaaaaaa"
    shrink: "O!Lr%ce3]4=H~=6lG,aaaaaaaaaaaaa"
    shrink: "O!Lr%ce3]4=H~=6lG,aaaaaaaaaaaaaa"
    shrink: "O!Lr%ce3]4=H~=6lGaaaaaaaaaaaaaaa"
    shrink: "O!Lr%ce3]4=H~=6laaaaaaaaaaaaaaaa"
    shrink: "O!Lr%ce3]4=H~=6aaaaaaaaaaaaaaaaa"
    shrink: "O!Lr%ce3]4=H~=aaaaaaaaaaaaaaaaaa"
    shrink: "O!Lr%ce3]4=H~aaaaaaaaaaaaaaaaaaa"
    shrink: "O!Lr%ce3]4=Haaaaaaaaaaaaaaaaaaaa"
    shrink: "O!Lr%ce3]4=aaaaaaaaaaaaaaaaaaaaa"
    shrink: "O!Lr%ce3]4aaaaaaaaaaaaaaaaaaaaaa"
    shrink: "O!Lr%ce3]aaaaaaaaaaaaaaaaaaaaaaa"
    shrink: "O!Lr%ce3aaaaaaaaaaaaaaaaaaaaaaaa"
    shrink: "O!Lr%ceaaaaaaaaaaaaaaaaaaaaaaaaa"
    shrink: "O!Lr%caaaaaaaaaaaaaaaaaaaaaaaaaa"
    shrink: "O!Lr%caaaaaaaaaaaaaaaaaaaaaaaaaaa"
    shrink: "O!Lr%aaaaaaaaaaaaaaaaaaaaaaaaaaaa"
    shrink: "O!Lraaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
    shrink: "O!Laaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
    shrink: "O!aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
    shrink: "Oaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
    shrink: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"

The test report starts with the 50 randomly generated input values (note: we omitted most for brevity). The last input value ("XqyW\O!Lr%ce3]4=H~=6lG, 5lT\aDz%n9"), listed as input #49, is the first for which the test failed. Having found an input value that fails the test, FsCheck then starts the shrinking process.

To see how shrinking works, let’s list the failing input value and the first six subsequent shrinks:

# Input Length Passes test
0 “XqyW\O!Lr%ce3]4=H~=6lG, 5lT\aDz%n9” 33 false
1 “qyW\O!Lr%ce3]4=H~=6lG, 5lT\aDz%n9” 32 true
2 “yW\O!Lr%ce3]4=H~=6lG, 5lT\aDz%n9” 31 true
3 “yW\O!Lr%ce3]4=H~=6lG,5lT\aDz%n9” 30 true
4 “W\O!Lr%ce3]4=H~=6lG,5lT\aDz%n9” 29 true
5 “\O!Lr%ce3]4=H~=6lG,5lT\aDz%n9” 28 true
6 “O!Lr%ce3]4=H~=6lG,5lT\aDz%n9” 27 true

The pattern here is quite obvious: with each shrinking step, the previous value is stripped of its first character, decreasing the length by one. Note that all strings of length 32 or less pass the test. FsCheck will use this fact later on.

The next six shrinks follow a different pattern:

# Input Length Passes test
7 “O!Lr%ce3]4=H~=6lG,5lT\aDz%n9a” 28 true
8 “O!Lr%ce3]4=H~=6lG,5lT\aDz%naa” 28 true
9 “O!Lr%ce3]4=H~=6lG,5lT\aDz%aaa” 28 true
10 “O!Lr%ce3]4=H~=6lG,5lT\aDzaaaa” 28 true
11 “O!Lr%ce3]4=H~=6lG,5lT\aDaaaaa” 28 true
12 “O!Lr%ce3]4=H~=6lG,5lT\aaaaaaa” 28 true

This time, each shrink step replaces one character with the 'a' character. FsCheck uses this shrink strategy to check if replacing specific characters in the input string can make the test fail.

As changing characters in the input values did not make the test fail, FsCheck then uses the fact that the only input value that failed the test had a length of 33. It therefore modifies its shrinking strategy and starts generating longer input values, which will lead to generating a 33 length input. Besides generating longer input values, it will additionaly also use the one character modification shrinking strategy as an extra check. This leads to the following sequence of shrinks:

# Input Length Passes test
13 “O!Lr%ce3]4=H~=6lG,5lTaaaaaaaa” 29 true
14 “O!Lr%ce3]4=H~=6lG,5lTaaaaaaaaa” 30 true
15 “O!Lr%ce3]4=H~=6lG,5laaaaaaaaaa” 30 true
16 “O!Lr%ce3]4=H~=6lG,5laaaaaaaaaaa” 31 true
17 “O!Lr%ce3]4=H~=6lG,5aaaaaaaaaaaa” 31 true
18 “O!Lr%ce3]4=H~=6lG,aaaaaaaaaaaaa” 31 true
19 “O!Lr%ce3]4=H~=6lG,aaaaaaaaaaaaaa” 32 true
20 “O!Lr%ce3]4=H~=6lGaaaaaaaaaaaaaaa” 32 true

The 11 shrinks that follow these shrinks are all 32 character strings with one character replaced, which all pass the test. Things start getting interesting from shrink #32, which is a string of length 33 and thus fails the test:

# Input Length Passes test
32 “O!Lr%caaaaaaaaaaaaaaaaaaaaaaaaaaa” 33 false

At this point, FsCheck rightly infers that strings of length 33 form the minimal set of counter-examples to our property. The final shrinking steps again use the single character replacement shrinking strategy, to see if changing a single character in an input value of length 33 can make the test pass:

# Input Length Passes test
33 “O!Lr%aaaaaaaaaaaaaaaaaaaaaaaaaaaa” 33 false
34 “O!Lraaaaaaaaaaaaaaaaaaaaaaaaaaaaa” 33 false
35 “O!Laaaaaaaaaaaaaaaaaaaaaaaaaaaaaa” 33 false
36 “O!aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa” 33 false
37 “Oaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa” 33 false
38 “aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa” 33 false

Even with all characters changed, the input still fails the test. At this point, FsCheck considers the input value "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" to be the minimal counter-example (or “shrunk”) to our property-based test.

Now that we know our test fails for strings of length 33 (and greater), we can fix our implementation as follows:

public static string MD5(string input)
{
    if (input == "foo")
    {
        return "acbd18db4cc2f85cedef654fccc4a4d8";
    }
    if (input == "bar")
    {
        return "37b51d194a7513e45b56f6524f2d51f2";
    }
    if (input == "baz")
    {
        return "73feffa4b7f6bb68e44cf984c85f6e88";
    }
    
    return (input ?? string.Empty).PadRight(32).Substring(0, 32);
}

Now, our test passes all generated inputs, and we have our first, working, property-based test!

Clearly, this single property test did not force us to write a correct implementation, but that is perfectly normal. With property-based testing, you usually need to write several property-based tests before you are finally forced to write a correct implementation, so let’s move on to property two.

Property 2: hash contains only hexadecimal characters

Our second property states that the MD5 hash consists of only hexadecimal characters:

[Property]
public void MD5ReturnsStringWithOnlyAlphaNumericCharacters(string input)
{
    var hash = MD5(input);
    var allowed = "0123456789abcdef".ToCharArray();
    Assert.All(hash, c => Assert.Contains(c, allowed));
}

This test should fail for any input other than the "foo", "bar" or "baz" strings, which indeed it does:

MD5ReturnsStringWithOnlyAlphaNumericCharacters [FAIL]

    FsCheck.Xunit.PropertyFailedException : 
    Falsifiable, after 1 test (0 shrinks) (StdGen (961984695,296121694)):
    Original:
    ""

    ---- Assert.All() Failure: 32 out of 32 items in the collection did not pass.
    [31]: Xunit.Sdk.ContainsException: Assert.Contains() Failure
        Not found: ' '
        In value:  Char[] ['0', '1', '2', '3', '4', ...]

Let’s fix this:

public static string MD5(string input)
{
    if (input == "foo")
    {
        return "acbd18db4cc2f85cedef654fccc4a4d8";
    }
    if (input == "bar")
    {
        return "37b51d194a7513e45b56f6524f2d51f2";
    }
    if (input == "baz")
    {
        return "73feffa4b7f6bb68e44cf984c85f6e88";
    }
    
    return "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
}

This implementation not only passes this test, but also all other tests. On to property three.

Property 3: same hash for same input

This test verifies that when presented with the same input, the same hash is returned:

[Property]
public void MD5ReturnsSameHashForSameInput(string input)
{
    var hash1 = MD5(input);
    var hash2 = MD5(input);
    Assert.Equal(hash1, hash2);
}

Simple enough, and the current implementation passes this test.

Property 4: hash is different from input

Our next property verifies that the hash is different from the input value (which is an essential property of any hashing algorithm):

[Property]
public void MD5ReturnsStringDifferentFromInput(string input)
{
    var hash = MD5(input);
    Assert.NotEqual(input, hash);
}

At the moment, this test will pass for every input string except "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa". That means that unless FsCheck randomly generates the "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" input value, the property-based test will pass and we would think our implementation was correct. Let’s demonstrate this by adding an example-based test:

[Fact]
public void MD5ReturnsStringDifferentFromManyAsInput()
{
    var input = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
    var hash = MD5(input);
    Assert.NotEqual(input, hash);
}

If we run both the property-based and the example-based tests, only the example-based test fails:

MD5ReturnsStringDifferentFromManyAsInput [FAIL]

    Assert.NotEqual() Failure
    Expected: Not "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
    Actual:   "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"

This is one of the drawbacks of testing with random data: you can have tests that sometimes fail. In such cases, augmenting a property-based test with an example-based test can be quite useful.

The fix is simple of course:

public static string MD5(string input)
{
    if (input == "foo")
    {
        return "acbd18db4cc2f85cedef654fccc4a4d8";
    }
    if (input == "bar")
    {
        return "37b51d194a7513e45b56f6524f2d51f2";
    }
    if (input == "baz")
    {
        return "73feffa4b7f6bb68e44cf984c85f6e88";
    }
    if (input == "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")
    {
        return "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb";
    }
    
    return "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
}

Let’s move on to our last property.

Property 5: similar inputs have non-similar hashes

Our last property states a very important property of MD5 hashes: similar inputs should not return similar hashes. For example, the hash for the string "hello" should be completely different from the hash for the string "hello1".

To write a test for this property, we have to define how we calculate the “difference” between two strings. In this case, we’ll just use a simple algorithm that counts the number of positions where the strings have different characters. Obviously, this simple algorithm would normally not suffice, but it works for our purposes. Note that for previty we’ll omit the actual implementation.

The property test looks like this:

[Property]
public void MD5WithSimilarInputsInputsDoesNotReturnSimilarHashes(string input, char addition)
{
    var similar = input + addition;
    var hash1 = MD5(input);
    var hash2 = MD5(similar);
    var difference = Difference(hash1, hash2);
    Assert.InRange(difference, 5, 32);
}

In this test, we let FsCheck generate two input values: the input value and a character to append to the input value. We then calculate the hashes for the original and modified input, calculate the difference and verify that the difference is at least 5 characters (which is was chosen arbitrarily).

In we run our test again, it fails:

MD5WithSimilarInputsInputsDoesNotReturnSimilarHashes [FAIL]

    FsCheck.Xunit.PropertyFailedException : 
    Falsifiable, after 1 test (2 shrinks) (StdGen (1928700473,296121700)):
    Original:
    ("D", 'F')
    Shrunk:
    ("", 'a')

    ---- Assert.InRange() Failure
    Range:  (5 - 32)
    Actual: 0

Modifying our implementation is still possible, but it becomes increasingly harder. At this point, we’ll give in and correctly implement the MD5 algorithm:

public static string MD5(string input)
{
    using (var md5 = System.Security.Cryptography.MD5.Create()) 
    {
        var inputBytes = System.Text.Encoding.ASCII.GetBytes(input);
        var hash = md5.ComputeHash(inputBytes);

        var sb = new StringBuilder();

        for (var i = 0; i < hash.Length; i++)
        {
            sb.Append(hash[i].ToString("x2"));
        }

        return sb.ToString();    
    }
}

Let’s run all our test to verify they all pass:

MD5WithSimilarInputsInputsDoesNotReturnSimilarHashes [FAIL]
      FsCheck.Xunit.PropertyFailedException : 
      Falsifiable, after 13 tests (1 shrink) (StdGen (1520259769,296121702)):
      Original:
      (null, 'm')
      Shrunk:
      (null, 'a')
      
    ---- System.ArgumentNullException : String reference not set to an instance of a String.

MD5ReturnsStringWithOnlyAlphaNumericCharacters [FAIL]
      FsCheck.Xunit.PropertyFailedException : 
      Falsifiable, after 8 tests (0 shrinks) (StdGen (1521993499,296121702)):
      Original:
      <null>
      
    ---- System.ArgumentNullException : String reference not set to an instance of a String.

MD5ReturnsStringDifferentFromInput [FAIL]
      FsCheck.Xunit.PropertyFailedException : 
      Falsifiable, after 4 tests (0 shrinks) (StdGen (1522075879,296121702)):
      Original:
      <null>
      
    ---- System.ArgumentNullException : String reference not set to an instance of a String.

MD5ReturnsSameHashForSameInput [FAIL]
    FsCheck.Xunit.PropertyFailedException : 
    Falsifiable, after 2 tests (0 shrinks) (StdGen (1522087989,296121702)):
    Original:
    <null>
      
    ---- System.ArgumentNullException : String reference not set to an instance of a String.

MD5ReturnsStringOfCorrectLength [FAIL]
    FsCheck.Xunit.PropertyFailedException : 
    Falsifiable, after 28 tests (0 shrinks) (StdGen (1522097819,296121702)):
    Original:
    <null>
      
    ---- System.ArgumentNullException : String reference not set to an instance of a String.

Oh no, by correctly implementing the MD5 method, we made all our property-based tests fail! What happened? Well, previously, we correctly handled null input values in our implementation, but we don’t anymore. If we think about it, null can be consider an invalid input value and throwing an ArgumentNullException is thus perfectly reasonable. We could write an example-based test to verify this behavior:

[Fact]
public void MD5WithNullInputThrowsArgumentNullException()
{
    Assert.Throws<ArgumentNullException>(() => MD5(null));
}

To fix our failing property-based tests, we should rephrase our properties to state that they only hold for valid (non-null) input values. Our next step is to make FsCheck only generate valid, non-null input values.

Customizing input value generation

FsCheck generates values through a concept known as arbitraries. To fix our failing property tests, we’ll define a function that returns a Arbitrary<string>, which FsCheck will use to generate strings with. Our custom Arbitrary<string> instance can generate any string, expect the null string.

To define our custom arbitrary, we create a class with a static method that returns an Arbitrary<string> instance:

public static class NonNullStringArbitrary
{
    public static Arbitrary<string> Strings()
    {
        return Arb.Default.String().Filter(x => x != null);
    }
} 

The Strings() method returns a modified Arbitrary<string> instance that filters out the null string. Note that the name of the method is arbitrary (pun intended), FsCheck will ignore the name and only look at the return type.

We can then use our custom arbitrary in our tests through the Arbitrary property of the [Property] attribute:

[Property(Arbitrary = new[] { typeof(NonNullStringArbitrary) })]
public void MD5ReturnsStringOfCorrectLength(string input)
{
    var hash = MD5(input);
    Assert.Equal(32, hash.Length);
}

If we now run our test, our custom arbitrary is used, which means the null input will not be generated and all our implementation passes all property-based tests!

To make using our custom arbitrary easier, we can create an attribute that derives from PropertyAttribute that automatically sets the Arbitrary property to our custom arbitrary:

public class MD5PropertyAttribute : PropertyAttribute
{
    public MD5PropertyAttribute()
    {
        Arbitrary = new[] { typeof(NonNullStringArbitrary) };
    }
}

We can now use this attribute instead of using a [Property] attribute:

[MD5Property]
public void MD5ReturnsStringOfCorrectLength(string input)
{
    var hash = MD5(input);
    Assert.Equal(32, hash.Length);
}

Much better. The final step is to use this custom attribute on all our property-based tests. To verify we didn’t break anything, we run all tests one more time and thankfully, they all still pass!

Regular tests or property tests?

Now that we know how to write property-based tests, should we stop writing example-based tests? Well, probably not.

One problem with property-based tests is that it can be hard to identify a set of properties that completely cover the desired functionality. In those cases, it can be very useful to also define some example-based tests. This is what we did in our example. The five properties for which we wrote tests did not completely describe the MD5 algorithm though, which is why we also needed an additional parameterized example-based test to ensure the correctness of our implementation.

Furthermore, as example-based tests are less abstract, they are usually easier to understand. Defining some example-based tests in addition to your property-based tests, can thus help explain what you are testing.

Practicing

Propery-based testing is not easy to start with. Especially in the beginning, you’ll probably struggle to identify properties. A good way to practice your property-based testing skills, is to solve the Diamond kata using property-based testing.

If you’d like to learn more about FsCheck and property-based testing, check out the following links:

FsCheck is the most popular property-based framework for the .NET platform, but other platforms have their own implementation. The most well-known are QuickCheck (Haskell), and ScalaCheck (JVM), which both inspired FsCheck.

Conclusion

Although writing tests is important, the way you write your tests is equally important. To strengthen example-based tests, you should use as many inputs as possible. While multiple inputs are better than one input, property-based testing is even better as it works with any input value. It achieves this by randomly generating a large number of input values.

The random nature of property-based testing forces you to approach your tests differently. You don’t focus on specific use cases, but on the general properties of the code you want to test. This helps you better understand the requirements of the code you want to test. Testing with random inputs also makes hard-coding an implementation using pre-defined input values extremely hard. Furthermore, the clever way in which property-based testing frameworks can provide minimal counter-examples, really helps identifying and fixing issues in your implementation.

The main problem with property-based testing is that it can be hard to get started with. In particular, figuring out what properties to test can be quite hard. Furthermore, property-based tests can fail to completely describe all aspects of the implementation. In those cases, you should augment property-based tests with regular example-based tests.