F# Performance Testing: F# Types vs Structs and Classes

F# introduces Record Types and Descriminated Unions to the .NET universe. Although I knew that these types eventually boiled down to reference types, I wanted to make sure that nothing funny was going on behind the scenes when they got instantiated. The results weren’t terribly exciting, but I learned some important things about performance testing along the way.

First Attempt

The first thing I did was create a few basic types:

[<struct>]
type StructWithImplicitConstructor =
    val x : int
    val y : int 

[<struct>]
type StructWithExplicitConstructor =
    val x : int
    val y : int
    new (one, two) = {x = one; y = two}

[<struct>]
type StructWithObjectConstructor(x:int, y:int) =
    member this.X = x
    member this.Y = y

type ObjectWithImplicitConstructor =
    val x : int
    val y : int

type ObjectWithExplicitConstructor =
    val x : int
    val y : int
    new (one, two) = {x = one; y = two}

type ObjectWithObjectConstructor(x:int, y:int) =
    member this.X = x
    member this.Y = y

type RecordType = { x:int; y:int }

type UnionType = 
    | X of int
    | Y of int

Then, I broke out a System.Diagnostics.Stopwatch and started testing how long it took to new up these objects. My first results varied wildly. JITer optimizations and garbage collection behaviors yielded bizarre and inconsistent results. After talking this over with coworkers Bill Wagner and Jay Wren, I made a second pass.

Getting Smarter

This time, I started with a more robust performance testing module:

module PerformanceTesting =
    let Time func =
        let stopwatch = new Stopwatch()
        stopwatch.Start()
        func()
        stopwatch.Stop()
        stopwatch.Elapsed.TotalMilliseconds
        
    let GetAverageTime timesToRun func = 
        Seq.init_infinite (fun _ -> (Time func))
        |> Seq.take timesToRun
        |> Seq.average
        
    let TimeOperation timesToRun =
        GC.Collect()
        GetAverageTime timesToRun
        
    let TimeOperations funcsWithName =
        let randomizer = new Random(int DateTime.Now.Ticks)
        funcsWithName
        |> Seq.sort_by (fun _ -> randomizer.Next())
        |> Seq.map (fun (name, func) -> name, (TimeOperation 100000 func))
    
    let TimeOperationsAFewTimes funcsWithName =
        Seq.init_infinite (fun _ -> (TimeOperations funcsWithName))
        |> Seq.take 50
        |> Seq.concat
        |> Seq.group_by fst
        |> Seq.map (fun (name, individualResults) -> name, (individualResults |> Seq.map snd |> Seq.average))

The above code makes it easy to time operations with greater accuracy. The TimeOperationsAFewTimes function takes a sequence of tuples containing the names of operations to test and the operations themselves. It runs the operations a few times and returns a sequence of tuples that contains the operation names and the average run time for each operation.

There are two important lessons for performance testing to take away from the example. First, the garbage collector is run between tests to make sure that each operation has a clean memory slate. Second, the order that the operations run in is varied between runs in order to help negate the effect of JITer optimizations in our tests.

(As an aside, the TimeOperationsAFewTimes function was really fun code to write. It was one of those moments that made me stop and take notice at the power of the F# language. I realize that the code might be a little difficult to read at first glance, though.)

Results

With the above functions, performance testing became a breeze:

printfn "Starting Tests..."
printfn ""

let tmpRec = {x = 0; y = 50}
 
PerformanceTesting.TimeOperationsAFewTimes 
    [ "RecordType", (fun () -> {x = 1; y = 50}|> ignore)
      "StructWithImplicitConstructor", (fun () -> new StructWithImplicitConstructor() |> ignore) 
      "ObjectWithObjectConstructor", (fun () -> new ObjectWithObjectConstructor(1, 50) |> ignore)
      "ObjectWithExplicitConstructor", (fun () -> new ObjectWithExplicitConstructor(1, 50) |> ignore)
      "StructWithObjectConstructor", (fun () -> new StructWithObjectConstructor(1, 50) |> ignore)
      "StructWithExplicitConstructor", (fun () -> new StructWithExplicitConstructor(1, 50) |> ignore)
      "CreatingUsingWith", (fun () -> { tmpRec with x = 1 } |> ignore)
      "UnionType", (fun () -> X 14 |> ignore)]
|> Seq.sort_by (fun (name, time) -> time)
|> Seq.iter (fun (name, time) -> printfn "%s: %f" name time)

printfn ""
printfn "Press Any Key To Continue..."
      
let tmp = System.Console.ReadKey()

The above code passes a list of tupled names and lamdas to the performance testing functions, sorts them from least to most expensive and prints the results.

Here are the results of a few runs:

Starting Tests…

CreatingUsingWith: 0.001973

StructWithExplicitConstructor: 0.001988

StructWithImplicitConstructor: 0.001992

UnionType: 0.002030

RecordType: 0.002064

ObjectWithExplicitConstructor: 0.002066

ObjectWithObjectConstructor: 0.002081

StructWithObjectConstructor: 0.002092

Press Any Key To Continue…

Starting Tests…

StructWithImplicitConstructor: 0.001407

CreatingUsingWith: 0.001408

RecordType: 0.001423

StructWithExplicitConstructor: 0.001436

ObjectWithObjectConstructor: 0.001438

UnionType: 0.001438

StructWithObjectConstructor: 0.001449

ObjectWithExplicitConstructor: 0.001455

Press Any Key To Continue…

Starting Tests…

RecordType: 0.001365

UnionType: 0.001374

StructWithImplicitConstructor: 0.001380

CreatingUsingWith: 0.001381

StructWithObjectConstructor: 0.001385

StructWithExplicitConstructor: 0.001386

ObjectWithExplicitConstructor: 0.001397

ObjectWithObjectConstructor: 0.001404

Press Any Key To Continue…

The Moral of The Story

The bottom line was that there were not any meaningful differences in the amount of time required to instantiate an object using any of the methods that I tried.

At first, my C and C++ roots were dumbfounded that value types didn’t beat the pants off of everything else. After Thinking critically, I realized that this makes a lot of sense in the managed world because the cost of creating reference types doesn’t include a hefty price for heap allocation.

The other thing that I proved to myself was that Record Types and Discriminated Unions are just as inexpensive to create as any other native .NET type.

This entry was posted in F#. Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.

2 Comments

  1. Ben
    Posted March 10, 2010 at 6:10 am | Permalink

    Hi, you may like to know that I just copied your code to here: http://stackoverflow.com/questions/833180/handy-f-snippets

  2. Posted April 17, 2012 at 3:26 pm | Permalink

    Minor note: call Thread.Sleep(0) before testing. The duration of the test should < 40ms. Test k times, pick the smallest number only. Bigger numbers mean there was a thread context switch.

    Major note:
    You should measure the duration of n * f(). (vs. n times testing duration of f())
    The duration of n * f() = n * actual_duration + n * looping_overhead + general_overhead; So, test for n = 2000, n = 3000, n = 4000 and solve the 3-eq system.

Post a Comment

Your email is never published nor shared. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*
*