This meant that GitFlow and GitHubFlow had completely separate code paths with very little shared, it also meant that things like commit counting varied on different branches. v3 changed all that and now is entirely driven by configuration. This initial pull request weighed in at 78 commits, 196 files, changed 2,500 lines of code and deleted another 700. 106 pull requests later we are shipping.

This configuration based approach has been really important for a number of reasons:

1. Understanding/documenting what GitVersion is doing.
2. When we had bug reports of a version being incorrectly calculated there was never a generic fix.
3. The logic for every branch was different and inconsistent. Then GitFlow and GitHubFlow worked quite differently in some respects
4. It was hard to get GitVersion to work with nightlies and publishing CI builds to NuGet
5. This meant that getting GitVersion working with Octopus deploy was not great

A number of long standing bugs were fixed with the changes made in v3, some of the highlights of the new features

• GitVersion init is a configuration tool which allows you to make GitVersion work the way you want it to
• Decent documentation!
  • Check it out at http://gitversion.readthedocs.org/en/latest
• Configurable version incrementing
  • Continuous Delivery (default) means GitVersion will only increment the version when you release and tag, this works great for some scenarios and not well for otherwise
  • Continuous Deployment, meaning each commit will produce a different SemVer
  • Read more at [http://gitversion.readthedocs.org/en/latest/more-info/version-increments/]
• Better error reporting - when things go wrong (which should be a lot less often now) we try to give you decent information including:
  • Dumping out a cleaned git history graph which you can paste into an issue so we can reproduce the issue
  • Logging out all decisions and sources for versions

Whats next

There are a few features which will be released in 3.1 which will make GitVersion nicer to get started. v4 hopefully will be not too far down the road when we rethink the command line usage and fix a few other things which have got a bit complex over time.

Give it a go

All the links you need are on our GitHub page at https://github.com/GitTools/GitVersion#gitversion

The component which needs to support popping out is the ChatHistory component which takes a set of items which get rendered.

1

<ChatHistory messages={this.props.chat.messages} />

A simplified implementation looks like this

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

class ChatHistory extends React.Component {
  constructor(props) {
    super(props)
    this.popout = this.popout.bind(this)
    this.state = {
      poppedOut: false
    }
  }

  popout() {
    this.setState({ poppedOut: true });
  }

  render() {
    return (
      <div>
        <Graphicon style= icon='popout' onClick={this.popout} />
        {this.props.messages.map(msg =>
          <div>
            {msg.date}
            <pre>{msg.text}</pre>
          </div>
        )}
      </div>
    )
  }
}

Next is how do I pop this content out. First I create a PoppedOutChatHistory component:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

class PoppedOutChatHistory extends React.Component {
  constructor(props) { super(props) }

  render() {
    return (
      <div>
        {this.props.messages.map(msg =>
          <div>
            {item.date}
            <pre>{msg.text}</pre>
          </div>
        )}
      </div>
    )
  }
}

Then I introduced a Popout react component into the ChatHistory render function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

render() {
  if (this.state.isPoppedOut) {
    return (
      <Popout title='Window title' onClosing={this.popupClosed}>
        <PoppedOutChatHistory messages={this.props.messages} />
      </Popout>
    );
  } else {
    return (
      <div>
      <span style= onClick={this.popout} className="buttonGlyphicon glyphicon glyphicon-export"></span>
        {this.props.messages.map(msg =>
          <div>
            {msg.date}
            <pre>{msg.text}</pre>
          </div>
        )}
      </div>
    );
  }
}

popupClosed() {
  this.setState({ poppedOut: false });
}

The Popout component is quite simple, when show is set to true then it will render the content in a new popout window. The nice thing about having this as a React component is that the Popout window’s lifecycle is tied to the lifecycle of the owning component. So when the History component is unmounted the Popout will close the popup it might have had opened.

Summary

I am pretty happy with the result of this Component, I can continue to flow props and handle events. For instance if I wanted to pop up a input form:

1
2
3
4
5
6
7
8

render() {
  return (
    <Popout title='Window title' onClosing={this.popupClosed}>
      <input type='text' onChange={this.textChanged} />
      <input type='button' onClick={this.submit} />
    </Popout>
  );
}

Source and install

npm install react-popup –save

Source on GitHub

Part 1 - Refactoring the InterceptKeys class
Part 2 - Refactoring the MessageProvider class
Part 3 - Introducing the MessageController class
Part 4 - Removing state mutation from the stream

Instead we will dive into the MessageController class which brings the streams from the KeyProvider and MessageProvider together then maintains the list of Messages displayed on the screen. The requirements are:

• Add any new messages into the messages collection
• After 5 seconds set a flag on the message which triggers an animation
• 1 second after that the message is removed from the messages collection
• If a message is updated the 5 second countdown should start again.

Old implementation

The previous implementation was quite simple but it was hard to test. Also Rx can solve this problem very nicely, so why not.

• New messages were added to the messages list
• A timer fired once a second, the callback did:
  • A foreach loop over every item which had a LastUpdated property more than 5 seconds go, then set the IsDeleting flag to true
  • A foreach loop over every item which had a LastUpdated property more than 6 seconds go, then removed it from the list

Lets rewrite this into Rx

Implementing in Rx

Messages stream

1
2

IObservable<KeyPress> keyStream = keyProvider.GetKeyStream();
IObservable<Message> messageStream = messageProvider.GetMessageStream(keyStream);

Step one is to get the messages subscription. Nothing much to see here except the variable names and types.

Adding messages

1

var addMessageSubscription = messageStream.Subscribe(message => messages.Add(message));

Also not much to see here, we subscribe to the messages stream and when a new value is yeilded we add that into the message into the messages collection.

Fading messages out

Now is where things become a little more interesting. Lets start off with simply setting the flag after 5 seconds

1
2
3

var fadeOutMessageSubscription = messageStream
    .Delay(FiveSeconds)
    .Subscribe(m => m.IsDeleting = true);

Great, but this does not take into account the updates. Here is the problem we want to solve visualised.

Each - represents a second, | is a completed stream

1
2
3
4
5

a--------b---------a----*ab----
-----a|
         -----b|
                   ---------ab|
-----a--------b-------------ab

This shows us that we need a nested stream of some sort. Lets rewrite the fade out query into something that will create an inner stream.

1
2
3
4
5
6

var fadeOutMessageStream = messageStream
    .SelectMany(message =>
    {
        // the inner stream goes here
    });
var fadeOutMessageSubscription = fadeOutMessageStream.Subscribe(m => m.IsDeleting = true);

Now we need to write the inner stream. Once again lets visualise the problem (x is an update, @ is the start of an observable.Timer(), o is the stream yeilding a value)

1
2
3
4
5

x---x----x-----
@---|
    @----|
         @-----o|
---------------x|

The thing that is changing and we are reacting to in this query is the updates of the message. A message has a property with signature IObservable<Unit> Updates { get; } which we can subscribe to for updates to that message. We also want to kick off right away when we subscribe (otherwise the timer will not start until an update happens). With this in mind, the start of the inner query is:

1
2

return message.Updated
    .StartWith(Unit.Default);

Now we need to introduce a timer of some sort. Because each update has it’s own timeout we need another nested stream. The marble diagram shows the nested screen pattern off really nicely. To get the nested stream we just do a .Select() which give us:

1
2
3

return message.Updated
    .StartWith(Unit.Default)
    .Select(_ => Observable.Timer(FiveSeconds))

The return type of that select is actually IObservable<IObservable<long>> which is the nested stream. In the marble diagram above you will notice that when we get an update and start a timer we close the previous timer stream. In Rx this is known as the .Switch() statement. When the outer observable yeilds a new inner observable, switch will dispose the last inner observable and subscribe to the new stream instead. Now the code is:

1
2
3
4

return message.Updated
    .StartWith(Unit.Default)
    .Select(_ => Observable.Timer(FiveSeconds))
    .Switch();

We are not finished yet, switch returns us a single stream of type IObservable<long>. We need a stream of messages again:

1
2
3
4
5
6
7
8
9

var fadeOutMessageStream = messageStream
    .SelectMany(message =>
    {
        return message.Updated
            .StartWith(Unit.Default)
            .Select(_ => Observable.Timer(FiveSeconds))
            .Switch()
            .Select(_ => message);
    });

We now are back showing the code which includes the outer stream. Our select we have just added simply ignores what it gets given and returns the message passed by the outer query.

Going back to our marble diagram you will notice there is one thing missing:

1
2
3
4
5

a--------b---------a----*ab----
-----a|
         -----b|
                   ---------ab|
-----a--------b-------------ab

After each inner stream yeilds a value, it immediately terminates. This is an easy fix, we just append .Take(1) to the inner query giving us:

1
2
3
4
5
6
7
8
9
10
11
12

var fadeOutMessageStream = messageStream
    .SelectMany(message =>
    {
        return message.Updated
            .StartWith(Unit.Default)
            .Select(_ => Observable.Timer(FiveSeconds))
            .Switch()
            .Select(_ => message)
            .Take(1);
    });

var fadeOutMessageSubscription = fadeOutMessageStream.Subscribe(m => m.IsDeleting = true);

Onto the Next song

Removing messages

The nice thing about Rx is that you can just build on other streams. So we just take our fadeOutMessageStream, apply a 1 second delay then subscribe.

1
2
3

var removeMessageStream = fadeOutMessageStream
    .Delay(OneSecond)
    .Subscribe(m => messages.Remove(m));

Next steps

One thing all of these streams rely on is that each of them receive the same instance of each message. It is no good to us if each of them are working on a different messages. This is an easy fix using an operator we saw in Part 1 of this series, .Publish().

But this time we will not use .RefCount() as we want to set up our queries then turn them on all at the same time, otherwise we have race conditions that messages might be added but never removed. We want all 3 of our subscriptions to start simultanously.

Lets zoom out to the method level and have a look what the MessageController.Start method looks like after we publish the stream.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

readonly SerialDisposable stopCarnacDisposable = new SerialDisposable();

public void Start()
{
    var keyStream = keyProvider.GetKeyStream();
    IConnectableObservable<Message> messageStream = messageProvider.GetMessageStream(keyStream).Publish();

    IDisposable addMessageSubscription = ...;
    IDisposable fadeOutMessageSubscription = ...;
    IDisposable removeMessageSubscription = ...;

    IDisposable underlyingMessageSubscription = messageStream.Connect();

    stopCarnacDisposable.Disposable = new CompositeDisposable(
        underlyingMessageSubscription,
        addMessageSubscription,
        fadeOutMessageSubscription,
        removeMessageStream);
}

The MessageController will use two different types of Rx Disposables. They are:

• SerialDisposable - If the .Disposable property is set multiple times the current value is Disposed before it is updated to the latest value. I use this so if .Start is called multiple times we will create a new subscription but it makes sure the previous subscription is disposed.
• CompositeDisposable - Takes any number of IDisposable’s as constructor parameters, when it is disposed all disposables passed into the ctor will be disposed

The new CompositeDisposable which takes all our of subscriptions is assigned to the stopCarnacDisposable.Disposable property before exiting the Start method. When the MessageProvider class is disposed stopCarnacDisposable.Dispose() will be called disposing any subscription which has been started by Start.

To walk through what we see above, I publish the message stream, create the three subscriptions and then I call .Connect() on the messageStream (which is now of type IConnectableObservable<Message>).

This allows me to explicitly control the underlying stream, when disposing I simply dispose the underlying stream then all of my other streams and everything is cleaned up nicely.

Specifying Schedulers

One part of the code I have ommited is explicitly specifying on what Scheduler everything will run on. I have introduced an IConcurrencyService which abstracts me from the static Rx schedulers (allowing me to write tests, which is coming up next!). When using Rx you should always specify what schedulers you want your code to run on, in carnac’s case we want everything to run on the UI Thread. Normally this is bad but because we are modifying messages which are bound to by the UI we have to. Maybe a future blog post can be how we made carnacs Rx streams immutable, but not now.

We do that by adding these two lines before every .Subscribe() call in this class.

1
2

.ObserveOn(concurrencyService.MainThreadScheduler)
.SubscribeOn(concurrencyService.MainThreadScheduler)

Error handling

I wanted to call our error handling explicitly, carnac has none. It didn’t before this refactor and it didn’t after. There is a GitHub issue open at the time of writing this blog post to sort this out.

Testing

I mentioned at the start of this post that this would be easier to test, lets put that to the test and write some tests. First off the simplest case.

1
2
3
4
5
6
7
8
9
10
11

[Fact]
public void MessagesAreAddedIntoKeysColletion()
{
    var message = new Message(A);

    sut.Start();
    ProvideMessage(message);

    keysCollection.ShouldContain(message);
    message.IsDeleting.ShouldBe(false);
}

A is a property which returns a KeyPress and ProvideMessage simply yields that message to the messageStream which the KeyProvider subscribes to. Now we have an simple example test, here is a more complex example.

1
2
3
4
5
6
7
8
9
10
11

[Fact]
public void MessagesAreFlaggedAsDeletingAfter5Seconds()
{
    var message = new Message(A);

    sut.Start();
    ProvideMessage(message);
    testScheduler.AdvanceBy(TimeSpan.FromSeconds(5).Ticks);

    message.IsDeleting.ShouldBe(true);
}

In this test we have made use of our ICurrencyService by making it always return a TestScheduler. The TestScheduler is in the Rx.Testing NuGet library and is very useful. Because Rx abstracts time we can write tests which rely on time very quickly and they also execute very fast.

The above test provides a message to the messageStream and then moves time forward by 5 seconds and makes sure that our message has been flagged for deletion, which it has. Lets push this a little harder and make sure that we do not expire when that message has been updated.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

[Fact]
public void MessageTimeoutIsStartedAgainIfMessageIsUpdated()
{
    var message = new Message(A);

    sut.Start();
    ProvideMessage(message);
    testScheduler.AdvanceBy(TimeSpan.FromSeconds(3).Ticks);
    message.Merge(new Message(A)); // Update message
    testScheduler.AdvanceBy(TimeSpan.FromSeconds(3).Ticks);

    message.IsDeleting.ShouldBe(false);
    keysCollection.ShouldContain(message);
}

As you can see the code to test our Rx queries is really simply.

Summary

The MessageController is quite a small class and reasonably simple once you are familiar with Rx concepts but it has some interesting problems which many applications have and demonstrates how they can be easily solved and tested.

Part 4 will remove the mutation of existing messages from the stream. Any sort of mutation of state inside an Rx stream is likely to be a source of bugs and make the application very hard to reason about and debug. Unlike the first 3 posts in this series part 4 will be changing overall architecture and introducing some new concepts into the carnac codebase.

Part 1 - Refactoring the InterceptKeys class
Part 2 - Refactoring the MessageProvider class
Part 3 - Introducing the MessageController class
Part 4 - Removing state mutation from the stream

As a bit of background, in carnac a KeyPress is not directional and it also contains information about if modifiers were pressed at the same time. For Instance ctrl + r would be a KeyPress. A Message is what is shown on the screen.

The message provider as it is does the following:

• Is IObserver<KeyPress>
• It aggregates multiple KeyPresses into logical messages with the following rules:
  • Shortcuts are always shown in their own message
  • If there has been more than a second between the last keypress a new message is created
  • If the key presses were entered into different applications a new messsage is created
• Apply ‘Only show shortcuts’ filter

Here is what the code looked like before the refactor.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123

public class MessageProvider : IMessageProvider, IObserver<KeyPress>
{
    readonly Subject<Message> subject = new Subject<Message>();
    private readonly IShortcutProvider shortcutProvider;
    private IDisposable keyStream;
    private readonly PopupSettings settings;

    public MessageProvider(IKeyProvider keyProvider, IShortcutProvider shortcutProvider, ISettingsProvider settingsProvider)
    {
        this.shortcutProvider = shortcutProvider;
        settings = settingsProvider.GetSettings<PopupSettings>();
        keyStream = keyProvider.Subscribe(this);
    }

    public Message CurrentMessage { get; private set; }

    public IDisposable Subscribe(IObserver<Message> observer)
    {
        return subject.Subscribe(observer);
    }

    public void OnNext(KeyPress value)
    {
        Message message;

        var currentKeyPress = new[] {value};
        var keyPresses = CurrentMessage == null ? currentKeyPress : CurrentMessage.Keys.Concat(currentKeyPress).ToArray();
        var possibleShortcuts = GetPossibleShortcuts(keyPresses).ToList();
        if (possibleShortcuts.Any())
        {
            var shortcut = possibleShortcuts.FirstOrDefault(s => s.IsMatch(keyPresses));
            if (shortcut != null)
            {
                message = CurrentMessage ?? CreateNewMessage(value);
                message.AddKey(value);
                message.ShortcutName = shortcut.Name;
                //Have duplicated as it was easier for now, this should be cleaned up
                return;
            }
        }

        // Haven't matched a Chord, try just the last keypress
        var keyShortcuts = GetPossibleShortcuts(currentKeyPress).ToList();
        if (keyShortcuts.Any())
        {
            var shortcut = keyShortcuts.FirstOrDefault(s => s.IsMatch(currentKeyPress));
            if (shortcut != null)
            {
                //For matching last keypress, we want a new message
                message = CreateNewMessage(value);
                message.AddKey(value);
                message.ShortcutName = shortcut.Name;
                //Have duplicated as it was easier for now, this should be cleaned up
                return;
            }
        }

        if (!value.IsShortcut && settings.DetectShortcutsOnly)
            return;

        if (ShouldCreateNewMessage(value))
        {
            message = CreateNewMessage(value);
        }
        else
            message = CurrentMessage ?? CreateNewMessage(value);

        message.AddKey(value);
    }

    private Message CreateNewMessage(KeyPress value)
    {
        var message = new Message
                              {
                                  StartingTime = DateTime.Now,
                                  ProcessName = value.Process.ProcessName
                              };

        CurrentMessage = message;
        subject.OnNext(message);
        return message;
    }

    private bool ShouldCreateNewMessage(KeyPress value)
    {
        return
            CurrentMessage == null ||
            IsDifferentProcess(value) ||
            IsOlderThanOneSecond() ||
            LastKeyPressWasShortcut() ||
            value.IsShortcut;
    }

    private bool LastKeyPressWasShortcut()
    {
        return CurrentMessage.Keys.Last().IsShortcut;
    }

    private IEnumerable<KeyShortcut> GetPossibleShortcuts(IEnumerable<KeyPress> keyPresses)
    {
        return shortcutProvider.GetShortcutsMatching(keyPresses);
    }

    private bool IsOlderThanOneSecond()
    {
        return CurrentMessage.LastMessage < DateTime.Now.AddSeconds(-1);
    }

    private bool IsDifferentProcess(KeyPress value)
    {
        return CurrentMessage.ProcessName != value.Process.ProcessName;
    }

    public void OnError(Exception error)
    {

    }

    public void OnCompleted()
    {

    }
}

The first thing you will notice is that most of the methods in this class access and manipulate the property CurrentMessage. This makes this class really hard to follow and rationalise what is going on.

Removing IObservable<T>

Like in the previous class we refactored, the first thing we need to do is to stop implementing IObservable.

Our subscribe method will change from IDisposable Subscribe(IObserver<Message> observer) to IObservable<Message> GetMessageStream(IObservable<KeyPress> keyStream)

This means consumers of this class can simply pass an observable in and get a new feed. It also makes this class easy to test.

Visualising the requirements

After looking at the current behaviour I decided that it would make more sense to not show a partial shortcut on the screen until it had either been completed or broken. With that in mind, we will start off with a series of key presses. a, b, ctrl+r, ctrl+r, ctrl+r, a, ↓, ↓ (↓ is the down arrow key). For these examples imagine we have a single shortcut which is ctrl+r, ctrl+r.

If we draw an ascii marble diagram it will look like this

1

a----b----ctrl+r----ctrl+r----ctrl+r----a----↓----↓

The first requirement is that we batch shortcuts into a single message.

1
2

a----b----ctrl+r----ctrl+r----------ctrl+r----a-----------↓---↓
a----b--------------ctrl+r,ctrl+r-------------ctrl+r--a---↓---↓

In the above diagram we see that ctrl+r, ctrl+r is a completed shortcut so our second stream emits the completed shortcut. But when we have ctrl+r, ‘a’ that is not a shortcut. It is instead a broken shortcut so the second stream emits two messages, one directly after the other. a, b and the arrow keys emit a completed message right away because they are not part of any potential shortcuts.

The next step is to merge messages together which we want to display on the screen together. In the above example we want to see this on the screen:

1
2
3
4

ab
ctrl+r, ctrl+r [Rename]
crtl+r
a↓ x 2

Lets add that into our marble diagram. Items prefixed with * are new messages which will replace the messages it has been merged with

1
2
3

a----b----ctrl+r----ctrl+r----------ctrl+r----a-----------↓----↓-------
a----b--------------ctrl+r,ctrl+r-------------ctrl+r--a---↓----↓-------
a----*ab------------ctrl+r,ctrl+r-------------ctrl+r--a---*a↓--*ab↓ x 2

Now we have an idea visually of what our streams will look like we can turn this into Rx.

Writing the query

In Rx when we want to reduce the number of items we have where we need some sort of aggregation function. In this case we want to use the .Scan() operator.

1
2
3
4

return keyStream
    .Scan(new ShortcutAccumulator(), (acc, key) => acc.ProcessKey(shortcutProvider, key))
    .Where(c => c.HasCompletedValue)
    .SelectMany(acc => acc.GetMessages())

Scan calls your accumulation function for each item in the stream, but unlike Aggregate it will emit the new aggregated value. In this case we start off with an empty ShortcutAccumulator, then the keyStream yields a value it is passed to the current ShortcutAccumulator which returns either itself or a new ShortcutAccumulator which .Scan will yield.

The ShortcutAccumulator will check if that key press matches any shortcuts. If it doesn’t, or that key completes the shortcut, it sets the HasCompletedValue property to true and returns itself. When a completed accumulator is asked to process a key it will simply create a new ShortcutAccumulatorand get it to process the key and return that accumulator instead of itself.

Because .Scan will emit the ShortcutAccumulator after each key press, we can simply filter the accumulators which are not completed yet, then select many on each completed ShortcutAccumulator to get the messages it has accumulated. The reason for the select many is when a shortcut is broken we create a message for each accumulated key press. And our stream now matches the second line in our marble diagram.

To do the final line in our marble diagram we need another Scan which merges Messages which need to be merged.

1
2
3
4
5
6

return keyStream
    .Scan(new ShortcutAccumulator(), (acc, key) => acc.ProcessKey(shortcutProvider, key))
    .Where(c => c.HasCompletedValue)
    .SelectMany(c => c.GetMessages())
    .Scan(new Message(), (previousMessage, newMessage) => messageMerger.MergeIfNeeded(previousMessage, newMessage))
    .Where(m => !settings.DetectShortcutsOnly || m.IsShortcut);

Whether a message needs to be merged or not is no longer the responsibility of this class, it has been moved into our MessageMerger class. MergeIfNeeded will check for all the merge conditions like the key presses being over a second apart, from different processes, either message being a shortcut etc and if they can be merged the new message will be merged into the previous. Ideally our entire stream would be immutable but that will have to be a separate task, we are refactoring existing code after all.

Finally we apply the Shortcut Only setting and filter our list if that option is set.

Summary

The end result of the MessageProvider refactoring was a great improvement. The end result is a single method returning an Rx statement which forfills all of our requirements. The shortcut detection logic was moved into another class which has a single responsibility making it much easier to understand what is going on and follow.

Recently Scott Hanselman blogged about a Quake Mode console and mentioned carnac, this triggered Brendan Forster and myself to think about carnac again. When we started writing carnac there was not any Rx experience amongst the team and we did a pretty bad job. I think carnac is a great Rx problem and the code can be improved to take advantage of Rx and be a really good non-trivial Rx sample. This blog series is all about the things we did wrong and the process I went through to refactor carnac into a much simpler code base and really leverage the power of Rx.

Part 1 - Refactoring the InterceptKeys class
Part 2 - Refactoring the MessageProvider class
Part 3 - Introducing the MessageController class
Part 4 - Removing state mutation from the stream

Fixing the InterceptKeys class

Carnac uses windows low level keyboard hooks to listen to key pressess. We have a class called InterceptKeys which is responsible for giving us an Rx stream of KeyEvents. This includes direction so if you press ctrl+r the stream would look like this:

• ctrl (down)
• r (down)
• r (up)
• ctrl (up)

That is all this class has to do. The only other thing to note is InterceptKeys is a singleton and should only ever have a single low level keyboard hook created.

Here is the original code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

[PermissionSet(SecurityAction.LinkDemand, Name = "FullTrust")]
[PermissionSet(SecurityAction.InheritanceDemand, Name="FullTrust")]
public class InterceptKeys : IObservable<InterceptKeyEventArgs>, IDisposable
{
    static InterceptKeys current = new InterceptKeys();
    readonly Win32Methods.LowLevelKeyboardProc callback;
    readonly Subject<InterceptKeyEventArgs> subject;
    bool disposed;
    IntPtr hookId = IntPtr.Zero;
    decimal subscriberCount;

    InterceptKeys()
    {
        subject = new Subject<InterceptKeyEventArgs>();
        callback = HookCallback;
    }

    public static InterceptKeys Current { return current; }

    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this);
    }

    public IDisposable Subscribe(IObserver<InterceptKeyEventArgs> observer)
    {
        IDisposable dispose = subject.Subscribe(observer);
        subscriberCount++;
        if (subscriberCount == 1)
            hookId = SetHook(callback);
        return new DelegateDisposable(() =>
                                          {
                                              subscriberCount--;
                                              if (subscriberCount == 0)
                                                  Win32Methods.UnhookWindowsHookEx(hookId);
                                              dispose.Dispose();
                                          });
    }

    IntPtr HookCallback(int nCode, IntPtr wParam, IntPtr lParam)
    {
        if (nCode >= 0)
        {
            /* snip, not really important, just calculating the variabless used below. */

            var interceptKeyEventArgs = new InterceptKeyEventArgs(key, keyDirection, alt, control, shift);

            subject.OnNext(interceptKeyEventArgs);
            if (interceptKeyEventArgs.Handled)
            {
                return (IntPtr)1; //handled
            }
        }

        return Win32Methods.CallNextHookEx(hookId, nCode, wParam, lParam);
    }

    static IntPtr SetHook(Win32Methods.LowLevelKeyboardProc proc)
    {
        using (Process curProcess = Process.GetCurrentProcess())
        using (ProcessModule curModule = curProcess.MainModule)
        {
            return Win32Methods.SetWindowsHookEx(Win32Methods.WH_KEYBOARD_LL, proc,
                                                  Win32Methods.GetModuleHandle(curModule.ModuleName), 0);
        }
    }

    protected virtual void Dispose(bool disposing)
    {
        if (!disposed)
        {
            if (disposing)
            {
                if (subject != null)
                {
                    subject.Dispose();
                }
            }

            disposed = true;
        }
    }

    class DelegateDisposable : IDisposable
    {
        readonly Action dispose;

        public DelegateDisposable(Action dispose)
        {
            this.dispose = dispose;
        }

        public void Dispose()
        {
            dispose();
        }
    }
}

There are a lot of things we did wrong, lets look at them one at a time.

DelegateDisposable class

This is a simple win. Rx comes with a heap of really useful disposable types. We can delete our nested DelegateDisposable class and replace it’s usage with Disposable.Create(). Quite an easy win.

Implementing IObservable

When using Rx you should never have to implement IObservable<T> or IObserver<T>. If you do then you should be looking for a better way to do it.

Because we are implementing IObservable we have to provide a .Subscribe method. Lets have a look at that.

public IDisposable Subscribe(IObserver<InterceptKeyEventArgs> observer)
{
    IDisposable dispose = subject.Subscribe(observer);
    subscriberCount++;
    if (subscriberCount == 1)
        hookId = SetHook(callback);
    return Disposable.Create(() =>
                      {
                          subscriberCount--;
                          if (subscriberCount == 0)
                              Win32Methods.UnhookWindowsHookEx(hookId);
                          dispose.Dispose();
                      });
}

The first side effect of this is that we have to do subscription management ourselves now! The good news is we can make a very simple change to improve this.

1. No longer inherit from IObservable<InterceptKeyEventArgs>
2. Rename subscribe to GetKeyStream with this signature:
   IObservable<InterceptKeyEventArgs> GetKeyStream()
3. Make it compile again.

To make it compile we need to create the observable that we are going to return. We will do that in the contructor of InterceptKeys so we always return the same observable to all callers (remember InterceptKeys is a singleton).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

InterceptKeys()
{
    keyStream = Observable.Create<InterceptKeyEventArgs>(observer =>
    {
        hookId = SetHook(callback);
        var dispose = subject.Subscribe(observer)

        return Disposable.Create(() =>
        {
            Win32Methods.UnhookWindowsHookEx(hookId);
            dispose.Dispose();
        });
    })
    .Publish().RefCount();
}

public IObservable<InterceptKeyEventArgs> GetKeyStream()
{
    return keyStream;
}

Ok this is looking better! We no longer are doing subscription management ourself. Lets look at the new Rx features we have used.

Observable.Create

If you ever need an observable, it is likely you want to use a static method off the Observable class. Observable.Create passes us an observable which we can call .OnNext(), .OnError() or .OnComplete() on to yield new values to any subscribers.

You then return an IDisposable which is called when the subscription is disposed by the subscriber.

Publish()

The next new thing is the .Publish extension method which is actually just a helper which calls the .Multicast(Subject<T>) extension. Multicast takes a subject (which is both a observer and an observable) and returns an IConnectableObservable<T>. When you subscribe to a connectable observable you are actually subscribing to the subject you passed into .Multicast(). IConnectableObservable<T> has a method called Connect on it, when you call connect it subscribes to the feed which you called Multicast on and passes in your Subject as the observer.

In effect Multicast means all subscribers actually share a single subscription, the subject simply determines the behaviour. Here are three examples:

• .Publish() = .Multicast(new Subject<T>)
• .PublishLast() = .Multicast(new AsyncSubject<T>)
• .Replay() = .Multicast(new ReplaySubject<T>)

As a quick example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

var observable = Observable.Create<InterceptKeyEventArgs>(observer =>
    {
        Debug.WriteLine('Subscribed');
        hookId = SetHook(callback);
        var dispose = subject.Subscribe(observer)

        return Disposable.Create(() =>
        {
            Debug.WriteLine('Unsubscribed');
            Win32Methods.UnhookWindowsHookEx(hookId);
            dispose.Dispose();
        });
    });

// Without Publish
var subscription = observable.Subscribe(_ => { }); // Prints 'Subscribed'
subscription.Dispose(); // Prints 'Unsubscribed'
var subscription2 = observable.Subscribe(_ => { }); // Prints 'Subscribed'
subscription2.Dispose(); // Prints 'Unsubscribed'

// With Publish
var published = observable.Publish();
var subscription = published.Subscribe(_ => { });
var subscription2 = published.Subscribe(_ => { });
var underlyingSubscription = observable.Connect();   // Prints 'Subscribed'
subscription.Dispose();
subscription2.Dispose();
underlyingSubscription.Dispose();  // Prints 'Unsubscribed'

RefCount()

Most of the time you do not want to manage the connection/disconnection of a connected observable yourself. This is where RefCount extension method comes in. It simply counts the number of subscribers and when the observable has subscribers it connects the underlying Published observable. Once all subscribers have Disposed it will disconnect the Published observable.

Now Rx is fully managing our subscriptions and will make sure we only ever have a single low-level keyboard hook!

1
2
3
4
5
6
7
8
9
10
11
12
13

var observable = Observable.Create<InterceptKeyEventArgs>(observer =>
    {
        Debug.WriteLine('Subscribed');
        hookId = SetHook(callback);
        var dispose = subject.Subscribe(observer)

        return Disposable.Create(() =>
        {
            Debug.WriteLine('Unsubscribed');
            Win32Methods.UnhookWindowsHookEx(hookId);
            dispose.Dispose();
        });
    }).Publish().RefCount();

Cleanup

Now that we have made this change, what else can be cleaned up?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

decimal subscriberCount;

public void Dispose()
{
    Dispose(true);
    GC.SuppressFinalize(this);
}

protected virtual void Dispose(bool disposing)
{
    if (!disposed)
    {
        if (disposing)
        {
            if (subject != null)
            {
                subject.Dispose();
            }
        }

        disposed = true;
    }
}

We can delete all of this code now.

Using Subject<InterceptKeyEventArgs>

The next warning sign we should be looking for in our Rx code is the use of Subjects. A subject is both a IObservable<T> AND an IObserver<T>. In this case we have a subject stored in a field which the Low Level Keyboard hook publishes to, inside our keyStream observable we have subscribed to the subject using the observer that Observable.Create gave to us.

To do this we have to move the low level keyboard hook into our Observable.Create so it can publish directly to the observer Rx has created us.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

keyStream = Observable.Create<InterceptKeyEventArgs>(observer =>
{
    Debug.Write("Subscribed to keys");
    IntPtr hookId = IntPtr.Zero;
    // Need to hold onto this callback, otherwise it will get GC'd as it is an unmanged callback
    callback = (nCode, wParam, lParam) =>
    {
        if (nCode >= 0)
        {
            var eventArgs = CreateEventArgs(wParam, lParam);
            observer.OnNext(eventArgs);
            if (eventArgs.Handled)
                return (IntPtr)1;
        }

        return CallNextHookEx(hookId, nCode, wParam, lParam);
    };
    hookId = SetHook(callback);
    return Disposable.Create(() =>
    {
        Debug.Write("Unsubscribed from keys");
        UnhookWindowsHookEx(hookId);
        callback = null;
    });
})
.Publish().RefCount();

This is not the nicest code because of the API of SetWindowsHookEx but we have removed our Subject and the code is far easier to understand now.

Summary

After those refactorings we have a far more maintainable class and we have removed a number of Rx anti-patterns. This is what the class looks like after those refactorings:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

[PermissionSet(SecurityAction.LinkDemand, Name = "FullTrust")]
[PermissionSet(SecurityAction.InheritanceDemand, Name = "FullTrust")]
public class InterceptKeys : IInterceptKeys
{
    public static readonly InterceptKeys Current = new InterceptKeys();
    LowLevelKeyboardProc callback;
    readonly IObservable<InterceptKeyEventArgs> keyStream;

    private InterceptKeys()
    {
        keyStream = Observable.Create<InterceptKeyEventArgs>(observer =>
        {
            Debug.Write("Subscribed to keys");
            IntPtr hookId = IntPtr.Zero;
            // Need to hold onto this callback, otherwise it will get GC'd as it is an unmanged callback
            callback = (nCode, wParam, lParam) =>
            {
                if (nCode >= 0)
                {
                    var eventArgs = CreateEventArgs(wParam, lParam);
                    observer.OnNext(eventArgs);
                    if (eventArgs.Handled)
                        return (IntPtr)1;
                }

                return CallNextHookEx(hookId, nCode, wParam, lParam);
            };
            hookId = SetHook(callback);
            return Disposable.Create(() =>
            {
                Debug.Write("Unsubscribed from keys");
                UnhookWindowsHookEx(hookId);
                callback = null;
            });
        })
        .Publish().RefCount();
    }

    public IObservable<InterceptKeyEventArgs> GetKeyStream()
    {
        return keyStream;
    }

    InterceptKeyEventArgs CreateEventArgs(IntPtr wParam, IntPtr lParam)
    {
        /* snip, not really important, just calculating the variabless used below. */

        return new InterceptKeyEventArgs(key, keyDirection, alt, control, shift);
    }

    static IntPtr SetHook(LowLevelKeyboardProc proc)
    {
        //TODO: This requires FullTrust to use the Process class - is there any options for doing this in MediumTrust?
        //
        using (Process curProcess = Process.GetCurrentProcess())
        using (ProcessModule curModule = curProcess.MainModule)
        {
            return SetWindowsHookEx(WH_KEYBOARD_LL, proc, GetModuleHandle(curModule.ModuleName), 0);
        }
    }
}

Pretty good result in the end. We have reduced this file from 224 lines of code to 100 lines by not implmenting IObservable ourselves, letting Rx manage our subscriptions (with .Publish().RefCount()) and making sure we use types provided by Rx (Disposable.Create instead of our own type).

Next in this series we will be rewriting the class which takes key presses (ctrl + f, a, b, space etc) and turns it into the messages carnac shows on the screen.

Check out part 2

Structure

I normally have two but sometimes three builds for each project. The structure is something like this:

• TeamCity Project
  1. CI
  2. Acceptance/UI Tests (optional)
  3. Release

1. CI builds the solution with correctly versioned assemblies (update assembly info files with version before build), runs all unit tests then creates any packages which are required. This includes NuGet packages, Chocolatey packages, zipped binaries, clickonce installers etc. This build monitors pull requests and is triggered automatically when there are new commits/branches.

2. Acceptance/UI Tests is an extra I use when I have long running or UI tests, for example TestStack.White on TeamCity (sign in as guest) has this build setup and it runs Whites UI tests. The reason I separate it is for speed reasons, I want my CI build to fail fast and only if it is successful do I run this slow build. This build triggers whenever 1. CI succeeds

3. Release (or 2. Release if there is no acceptance/ui test build) is run manually and it releases the artifacts build by 1. CI, if that is a NuGet package it is pushed to NuGet.org, chocolatey packages get pushed to chocolatey.org, zip files get pushed as a GitHub release etc. Once this build succeeds it should tag the VCS root and push that tag.

Build Configuration

VCS Root

First step is to create our VCS root.
Type: Git
VCS Root Name: Project Name (all branches + pull requests)
Default branch: master
Branch specifications:

1
2

+:refs/pull/*/merge
+:refs/heads/*

If you are using stash it would be

1
2

+:refs/pull-requests/*/merge-clean
+:refs/heads/*

You should also put a username/password in to authenticate with the VCS root so we can tag in the release build

1. CI

This has two build steps, for this to work you should install GitVersion on your build server via chocolatey.

Build steps

• GitVersion
  Runner Type: Command Line
  Step Name: GitVersion
  Run: Executable with parameters
  Command executable: GitVersion
  Command parameters: . /updateAssemblyInfo /output buildserver

• Build project
  This should be setup however you used to do it, build your solution, run your build scripts etc.

Triggers

Add trigger VCS Trigger with the default settings which has a branch spec of +:*

Build Features

I have the report status to GitHub plugin installed, this plugin tells github the build status of pull requests so it can display warnings if a pull request will break the build

2. Acceptance test

This build varies greatly, if you need this build it has the same triggers and dependencies setup as 3. Release. I won’t repeat those settings here.

3. Release

VCS Root

The same VCS root which is referenced by 1. CI should also be referenced by this build

Build steps

Normally for me this just a NuGet push step. But whatever you normally do, use the artifact dependencies to get at the artifacts from your CI build to publish

Build Features

Add feature VCS Labelling
Labeling pattern: %system.build.number%
Label builds in branches: +:* (we want to label any release)
Check Label successful builds only

Dependencies

This is the important part, we want to setup a proper TeamCity build chain. 1. Add snapshot dependency for CI build and previous build in chain, so if you have 2. Acceptance tests then add a snapshot dependency for both CI and Acceptance test builds
Tick Do not run new build if there is a suitable one and Only use successful builds from suitable ones
The snapshot dependency is really important, it means that you release the artifacts associated with a specific git commit. If you don’t you can deploy artifacts which came from another branch =/ 2. Add Artifact Dependency to CI Build with these settings:
Get artifacts from: Build from the same chain
Artifacts rules: MyPackage.*.nupkg (or whatever artifacts you need to publish

General Settings

I put this last because you need the dependencies setup before configuring this page. Build Number Format: %dep.MyProject_Ci.build.number% (MyProject_Ci is the project ID of the CI build)

Wrap up

Hopefully this helps you get your TeamCity builds setup, I have found this setup works quite well and is easy to setup and keep running.

I also noticed that many of the RCs and betas were not tagged in Git so you can’t see what was fixed each beta.

I have been working on a little utility to solve exactly this problem called GitReleaseNotes, the idea is that you install it via Chocolatey, then run GitReleaseNotes /outputFile ReleaseNotes.md /allTags and it will connect to your issue tracker (if the issue tracker is a REMOTE in your Git repository) fetch all the closed issues since it was last run and append them into your release notes. For public GitHub repos using GitHub issues these are the same so it just works, for Jira and YouTrack you will need to specify additional command line parameters

You then can manually edit, group and do whatever you want. All your modifications will not be changed when you run GitReleaseNotes again.

The first step was to change the formatting slightly, currently GitVersion assumes all release titles will start with # <Release>, so I changed all of the ##1.2.3 (release date) to # 1.2.3 (release date).

The next step was to tell GitReleaseNotes where to start from, to do this I needed to add the commit range of the last release, the most important one is the last sha. For this I just took when the release notes was updated last, then went back in the history until I found another major release. This gave me:

Commits: [a32f1fc133...f15a8f3b52](https://github.com/chocolatey/chocolatey/compare/a32f1fc133...f15a8f3b52)

But this would work fine too

Commits: a32f1fc133...f15a8f3b52

After I had done that I just ran GitReleaseNotes /o CHANGELOG.md /allTags and this was appended to the top of CHANGELOG.md

1
2
3
4
5
6
7
8
9
10
11
12
13

# vNext

 - [#493](https://github.com/chocolatey/chocolatey/issues/493) - [Enhancement] Chocolatey-Update cleanup
 - [#492](https://github.com/chocolatey/chocolatey/issues/492) - Error messages late in update of chocolatey itself
 - [#487](https://github.com/chocolatey/chocolatey/issues/487) - nuget.exe hangs for packages with many dependencies
 - [#486](https://github.com/chocolatey/chocolatey/pull/486) - Improve Chocolatey setup as administrator and add Test-ProcessAdminRights helper contributed by Jakub Berezanski ([jberezanski](https://github.com/jberezanski))
 - [#424](https://github.com/chocolatey/chocolatey/issues/424) - Update Contributing.md with link to mailing list
 - [#416](https://github.com/chocolatey/chocolatey/pull/416) - [Enhancement] added quiet parameter and forced write-host to honor that param (#411) contributed by Johan Leino ([jole78](https://github.com/jole78))
 - [#411](https://github.com/chocolatey/chocolatey/issues/411) - [Enhancement] absolute "quiet" mode - Allow shutting off "real" Write-Host
 - [#393](https://github.com/chocolatey/chocolatey/pull/393) - Resolve issue with DISM "missing" or with the 32-bit DISM being called on a 64-bit system contributed by Julian Easterling ([dcjulian29](https://github.com/dcjulian29))
 - [#379](https://github.com/chocolatey/chocolatey/issues/379) - [Enhancement] Update NuGet.exe to 2.8+ in Chocolatey install

Commits: [c1ab0a6473...f541d8ca31](https://github.com/chocolatey/chocolatey/compare/c1ab0a6473...f541d8ca31)

That was pretty easy, I submitted this as a pull request and Rob can just edit and next time generate the new release notes with easy. Including all the beta’s. You can view the pull request at https://github.com/chocolatey/chocolatey/pull/496 - it may not be merged, but I figured this was a good guide of how you can start using GitReleaseNotes on your own project. Feel free to post on issue on GitHub if you have issues with your project, it is still a work in progress tool after all!

You can also run GitReleaseNotes /o releasenotes.md /allTags to generate a complete new set of release notes since the start of the project with easy. Give that a go yourself and see what the output is.

Next steps

The next function of GitReleaseNotes will be to help you publish your release. This is not done yet, but you will be able to just go on a publish build:

GitReleaseNotes /publish /version 1.2.3 which will (hopefully):

• Push a GitHub release (which will cause the tag to be created)
• Will automatically read the vNext section of your release notes and use that
• Update the version in the release notes file, and push that

Each of these steps will be able to be done individually. Then with this and GitVersion I will be able to automate MUCH more of my release process.

I am currently using TestStack.BDDfy as my BDD testing framework. v4 has some great changes which make BDDfy really really awesome.

One of the things I am testing is generated confirmation text based on a number of selected form inputs. I want the approved generated text to be put into the BDDfy output. First steps are to read the approved file, here is a quick snipped which does this:

protected string GetApproved()
{
    var namer = new UnitTestFrameworkNamer();
    var basename = string.Format("{0}\\{1}", namer.SourcePath, namer.Name);
    var approvalFilename = new ApprovalTextWriter(string.Empty).GetApprovalFilename(basename);
    var approved = !File.Exists(approvalFilename) ? string.Empty : File.ReadAllText(approvalFilename);
    return approved;
}

Then I can just add my step in BDDfy .Then(_ => ApprovedGeneratedConfirmationShouldMatch(sut.DealSummary.ToString()), string.Format("Approved generated confirmation should be:\r\n{0}", GetApproved()))

Prose is an awesome open source online Jekyll site editor for GitHub. Then if we setup TeamCity to automatically regenerate and deploy our blog when we make any commits we have a really simple way of making quick blog posts online.

Here is a quick rundown of creating a new post with Prose. If you are only interested in the automated TeamCity deployment just skip this section.

1. Go into the _posts folder, then click New File
   
2. Add a title and write your post
   
3. You can define your yaml metadata defaults for prose in your _config.yaml, giving you a nice UI
   

When you hit save Prose will commit that file, pretty sweet. note if you do not create the file in the _posts folder you will not see the meta-data options. note2 Image uploading is broken at the moment, hopefully this gets some love soon

My Prose Config

You can configure Prose by adding some additional metadata in your _config.yaml file. Here is mine:

# ----------------------- #
#   prose.io settings     #
# ----------------------- #
prose:
  rooturl: "source"
  site: "http://jake.ginnivan.net"
  media: "source/assets/posts"
  metadata:
    "source/_posts":
      - name: "layout"
        field:
          element: "hidden"
          value: "post"
      - name: "title"
        field:
          element: "text"
          value: "Title"
      - name: "comments"
        field:
          label: "Allow comments"
          element: "checkbox"
          value: true
      - name: "categories"
        field:
          element: "text"
          value: ""
      - name: "published"
        field:
          label: "Published"
          element: "checkbox"
          value: true

Read more about Prose Configuration.

Setting up continuous deployment from TeamCity

The next step is I want TeamCity to automatically regenerate my Octopress site and push it to GitHub. This also assumes you have your site setup as a user site

1. Install everything you would need to get Octopress building locally on your build server
2. Create new build
3. Add VCS root pointing at source
4. Set TeamCity Checkout mode to checkout on agent
5. Add Command Line Build Step with Exe of bundle and Parameter of install. This will install all the dependencies Octopress needs

6. Next we need TeamCity to clone our repo into the _deploy directory, we can do this with another Command Line build step with the Run parameter set to Custom Script (the reason for git checkout master is that I have the source branch set as default, and the _deploy branch needs to be on master)

    if not exist _deploy (git clone https://%GithubUsername%:%GithubPassword%@github.com/%GithubUsername%/%GithubUsername%.github.io.git _deploy)
    cd _deploy
    git checkout master
    cd ..
   

7. The last build step is to regenerate and deploy our Octopress site. Add a Rake build step with the path set to Rakefile and the task set to gen_deploy. Make sure the bundle exec option is checked

8. Now setup a VCS build trigger so we deploy whenever anything changes
9. Finally go to the Parameters section of your build configuration. There will be two parameters needing values, GitHubUsername and GitHubPassword. GitHubUsername is easy, just put your username in.
10. For GitHubPassword click on Edit next to Spec, then set Display to Hidden and Type to Password. This will make sure your password does not show up in build logs and will be ****’d out instead.

Save your build configuration, and now whenever you push any updates to source your blog will be regenerated and deployed.

Enjoy!

This post is about two new open source tools. GitVersion and GitReleaseNotes which automate both of those things.

Background

About 7 months ago I was talking with Simon about this pain. Simon and the guys at Particular (nServiceBus) had already started on a tool which used the branching conventions of GitFlow to infer the Semantic Version of their software.

I was pretty excited about this, but did not want to convert all my projects to GitFlow just for versioning reasons. We had two choices, I could contribute GitHubFlow support to GitFlowVersion or spin off a new project. After a discussion with Simon and Andreas we made the call that a separate project would be the way to start, then we can combine the best of both worlds later. GitHubFlowVersion was born.

Since then, GitHubFlowVersion and GitFlowVersion have joined forces bringing the best ideas from both projects into GitVersion which uses the conventions in common Git branching strategies to allow you to adopt Semantic Versioning in your project really easily. I will talk more about release notes a bit later.

GitVersion in action

I have created a simple repository on GitHub which I will use for this post at JakeGinnivan/EasyVersioningAndReleases and also setup a CI build (login as guest).

In this example I am going to use GitVersion.exe from my “build script”, check out Command line usage on the GitVersion wiki for more ways you can use GitVersion.exe. Also check out the MSBuild task and the Ruby Gem. When GitVersion.exe is run, it looks at your Git repository and infers the Semantic Version from previous tags, branch names and a few other things. The wiki covers how it does this. The goal though is that it should just work and do sensible things.

Here is my sample build script, all it does is call GitVersion.exe, parse the json that is returned and replace a version token in my powershell module.

$currentDir = Split-Path -Parent -Path $MyInvocation.MyCommand.Definition

$output = . "$currentDir\GitVersion.1.0.0.0\tools\GitVersion.exe"

$joined = $output -join "`n"
$versionInfo = $joined | ConvertFrom-Json
$version = $versionInfo.SemVer

mkdir "$currentDir\Artifacts"

Copy-Item "$currentDir\src\UsefulStuff.psm1" "$currentDir\Artifacts\UsefulStuff.psm1"

(Get-Content "$currentDir\Artifacts\UsefulStuff.psm1") | 
    Foreach-Object {$_ -replace '__version__',"v$version"} |
    Out-File "$currentDir\Artifacts\UsefulStuff.psm1"

Write-Output "##teamcity[buildNumber '$version']"

The json returned from GitVersion is a collection of variables, here is an example of what GitVersion returns

{
  "Major":1,
  "Minor":0,
  "Patch":0,
  "PreReleaseTag":"beta.1",
  "PreReleaseTagWithDash":"-beta.1",
  "BuildMetaData":431,
  "FullBuildMetaData":"431.Branch.release-1.0.0.Sha.c1b1f80cd5e3e8182aa208163e40b4df66461226",
  "MajorMinorPatch":"1.0.0",
  "SemVer":"1.0.0-beta.1",
  "LegacySemVer":"1.0.0-beta1",
  "LegacySemVerPadded":"1.0.0-beta0001",
  "AssemblySemVer":"1.0.0.0",
  "FullSemVer":"1.0.0-beta.1+431",
  "InformationalVersion":"1.0.0-beta.1+431.Branch.release-1.0.0.Sha.c1b1f80cd5e3e8182aa208163e40b4df66461226",
  "ClassicVersion":"1.0.0.431",
  "ClassicVersionWithTag":"1.0.0.431-beta.1",
  "BranchName":"release-1.0.0",
  "Sha":"c1b1f80cd5e3e8182aa208163e40b4df66461226",
  "AssemblyVersion":"1.0.0.0",
  "AssemblyFileVersion":"1.0.0.0",
  "OriginalRelease":"c1b1f80cd5e3e8182aa208163e40b4df66461226.2014-05-11 08:42:24Z"
}

The build metadata in the above example (+431) comes from the number of commits since the last tag/release. This means the SemVer is the same for each build, essentially making each CI build create a release candidate of the release. This works well with continuous delivery.

We then just copy our powershell module into the output folder and replace the version. The last command Write-Output "##teamcity[buildNumber '$version']" updates TeamCity’s build number. Resulting in:

How to SemVer?

Once we have GitVersion running, how do we get it to bump the SemVer?

GitVersion supports two Git branching strategies, GitHubFlow (or the simple pull request model) and GitFlow. I am going to use GitHubFlow as an example.

Ways to bump the version

1. When you tag, the patch will automatically be bumped for the next build
   • So if you Tag 1.2.0, the next build will be 1.2.1
2. Use NextVersion.txt when you want to bump major or minor
   • i.e “2.0.0” > NextVersion.txt – will result in 2.0.0 being built
3. Put the version number into a branch name and merge to master
   • For example if you want to bump major because of a breaking change, you can create a branch called release-2.0.0, which will build as 2.0.0-beta.1 automatically, once it is merged into master 2.0.0 stable will start being built

If you have other idea’s on how we could bump the version, let us know. Some ideas are using Git notes or detecting specific phrases in your commit messages. Like Breaking: would cause major to be increased.

Check out the wiki for some additional info

• https://github.com/Particular/GitVersion/wiki/Advantages-of-GitVersion
• https://github.com/Particular/GitVersion/wiki/GitFlowExamples
• https://github.com/Particular/GitVersion/wiki/GitHubFlowExamples

That is GitVersion in action. You can also just invoke it directly as the first step in you TeamCity build. It also supports a few other build servers.

Generating release notes

The next step in simplifying the maintenance of our open source (or internal) project is generating release notes.

There are two approaches you could take, try and keep your issue tracker and everything up to date so they can be generated from the issue tracker you use when you are release your software.

The second option is to keep your release notes in your repository, and you can edit and update them manually. GitReleaseNotes works with either approach, but the recommended approach is the second one. Let’s go through how you can use GitReleaseNotes.

Note: these examples are with GitHub, but GitReleaseNotes works with Jira and YouTrack currenly. TFS will also be supported. The main requirement is that you use Git as your VCS.

Step 1

Install git release notes from Chocolatey, or download the zip and put into your PATH.

cinst GitReleaseNotes

Step 2

Change directory to any git directory, then run GitReleaseNotes /o ReleaseNotes.md

This will generate some release notes:

• #11 - Added function to load visual studio vars contributed by Jake Ginnivan (JakeGinnivan)
• #10 - Add function to load VS 2013 variables +enhancement
• #9 - Added project info contributed by Jake Ginnivan (JakeGinnivan)
• #8 - Create something to version
• #7 - Fixed header formatting contributed by Jake Ginnivan (JakeGinnivan)
• #1 - Header formatted badly +fix

Commits: d3620015c9…61862ed9e8

Not sure about you, but I think this is pretty handy.. So what has GitReleaseNotes actually done?

• Seen that you have a Git remote pointing at GitHub
• Connects to that remote (for example if upstream is Particular/GitVersion, it will use that repo) and fetches issues and pull requests
  • To connect to a private repo, you will need to login to GitHub and generate an application auth token then pass the auth token to GitReleaseNotes via a command line switch.
• Outputs the closed pull requests and issues since the last tag to the release notes
• If you have specified the /alltags switch then you will get issues grouped by release

You get included:

• Link to the issue/pull request
• Issue title
• Tags (GitReleaseNotes follows http://www.semanticreleasenotes.org/ which uses +tag after the title for tags/categories/labels.
• Shout outs - a really important part of release notes, calling out the awesome people which have contributed to your project! Shout outs are only available for pull requests, and will only be the person who submitted the pull request
• Commit range included in that release (this will become a link to the diff of those commits if supported by the site)

As mentioned above, you could also be using another issue tracker and GitReleaseNotes would connect to it and fetch the issues!

Step 3

[Optional] Edit your release notes, if you only want to list pull requests just delete the issues. If you want to put both issue and pull request links in a single item, just do that. This can be improved as we go based on feedback to reduce the amount of manual work. You can also just leave it as is.

Step 4

If you are using the /allTags switch, then GitReleaseNotes will append new issues to your release notes. Meaning all your modifications will not be changed and new closed issues will simply be appended!

Summary

That’s it, I would love feedback on both of these projects. I think they are massive time savers and I hope they save you some time!

I said I would come back to Particular’s release note generator which can be found at https://github.com/Particular/GitHubReleaseNotes. It is much more strict and uses GitHub milestones and set tags to generate the release notes. Which allows high quality generated release notes. If you use GitHub milestones a lot for release planning, it might work better for you. Check it out.

I just had a very strange issue, I could not type a capital V, lowercase worked fine. Plugging into another machine showed the same issue.

One of the problems with the Sculpt is that the function keys get stuck. Apparently if the F5 key gets stuck, then you can no longer type a capital V. If the F2 key is stuck then you won’t be able to type a capital C.

If you hit this random issue, hopefully you will read this post before returning your keyboard.

Save this as KillThumbsDb.reg and run it as administrator:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Policies\Microsoft\Windows\Explorer]
"DisableThumbsDBOnNetworkFolders"=dword:00000001

I have no idea why thumbs.db is being created in local folders, but this fixes it for me.

git config option merge.tool set to unknown tool: --global
Resetting to default...

This message is displayed because 'merge.tool' is not configured.
See 'git mergetool --tool-help' or 'git help config' for more details.
'git mergetool' will now attempt to use one of the following tools:
tortoisemerge emerge vimdiff
No known merge tool is available.

Somehow (seriously, I have no idea how I did this) I had created a setting in one of my repositories set my mergetool to --global.

If you happen to get yourself into the same issue, run git config --list to see what your config settings are.

For me, I had a rogue merge.tool entry, so I just had to run git config --unset merge.tool which deleted the entry and I was off again.

Please give it a go and report any issues/feedback you have so can fix them before the final v4 release.

New Features

New features include:

• Cucumber Examples support
  • This was one of the biggest missing features of BDDfy. We support text based tables (to support easy migration from SpecFlow) and also examples in code which allows complex types rather than just basic types!
  • If you have not seen cucumber examples before check out https://github.com/cucumber/cucumber/wiki/Scenario-Outlines for more information
  • See the examples blog post for more information about examples and how they work in BDDfy
• Tags
  • .WithTags("Tag1", "Tag2") will be seen in your reports. For example in text reporter you will get Tags: Tag1, Tag2 under the test output. The HTML report will have tags boxes next to the step title
  • Tag functionality is really basic at the moment, and will be improved in v4.1
• Fluent: Step titles are prefixed with the step type
  • For example: this.Given(_ => Something()) will result in a step title of Given something
• Fluent: Removed restriction on order of step types. So you can create more complex tests (for acceptance/UI based tests). i.e .Given().Then().When().Then().And().When().Then() etc.
• Added But step type, in beta 1 this is only available for the fluent syntax. It will be in reflective by the final release
• New Metro HTML report
• Classic HTML Report is now self contained with jQuery inlined in the report. CSS is also minified to make the report source easier to read.
  • You can switch to a CDN version of jQuery via configuration if you do not want it included in the report
• Inline assertions:
  • this.Given(() => { something = 5; }, "Something is 5") so you do not have to create methods for steps if you do not want to.
• Title-only steps: you can have a title only step to explain the context more; e.g. Given(“Given something”)
• StoryMetadata is made more generic to allow different story narratives; e.g. instead of As a …, I want …, So that … you can now have anything including So that …, As a …., I want …
• Story title prefix is now customisable
• Arguments passed to methods are now reported in steps much better, including:
  • Step titles are lazily evaluated, so if previous steps mutate state which is passed into a step then the value at step execution time will be reported
  • Support fields, properties and they will be reported as arguments in step title (i.e this.Given(_ => Given(_three)) will be Given 3
  • And a bunch of other improvements
• BREAKING: A lot less namespaces to make the API easier to use.
  • Many of the namespaces that were necessary before, particularly for using fluent API and executable attributes, are removed and are now accessible via the root TestStack.BDDfy namespace.
• BREAKING: A few types have been renamed to more accurately reflect their role and usage
• BREAKING: Removed story category from .BDDFy and .LazyBDDfy methods
• BREAKING: .net 3.5 support has been dropped, BDDfy now requires > .NET 4.0
• Coming Soon: ReSharper BDDfy plugin, which will stop ReSharper marking steps as unused. Check out it’s progress at https://github.com/TestStack/TestStack.BDDfy.ReSharper - Thanks heaps to Igal Tabachnik (@hmemcpy) for the work on this

Check out the release notes at https://github.com/TestStack/TestStack.BDDfy/blob/release-4.0.0/release-notes.md

Known issues

• Add But to reflective API (https://github.com/TestStack/TestStack.BDDfy/issues/85)
• Update diagnostics report to work with examples (https://github.com/TestStack/TestStack.BDDfy/issues/135)
• Examples do not work with RunStepWithArgsAttribute (https://github.com/TestStack/TestStack.BDDfy/issues/136)

Feedback wanted

This is an exciting release for BDDfy and we want to hear about any issues you hit

From the TestStack team (Jake Ginnivan, Mehdi Khalili, Michael Whelan and Rob Moore)

Scenario: Successful rail card purchases
Given: the buyer is a <buyer category>
And: the buyer selects a <fare>
When: the buyer pays
Then: a sale occurs with an amount of <price>

Examples:

| Buyer Category | Fare          | Price |
| Student        | Monthly Pass  | $76   |
| Senior         | Monthly Pass  | $98   |
| Standard       | Monthly Pass  | $146  |
| Student        | Weekly Pass   | $23   |
| Senior         | Weekly Pass   | $30   |
| Standard       | Weekly Pass   | $44   |
| Student        | Day Pass      | $4    |
| Senior         | Day Pass      | $5    |
| Standard       | Day Pass      | $7    |
| Student        | Single Ticket | $1.5  |
| Senior         | Single Ticket | $2    |
| Standard       | Single Ticket | $3    |

This is an example of a Cucumber test with examples, for each row in the examples table the test will run effectively giving us a data driven BDD style test.

I have picked this test because it is a bit more complex than the standard Given I have 3 beers in my fridge, When I drink 2, Then I have 1 beer left.. Yay, how interesting..

In this example, I will use BDDfy’s fluent API to write this test. First without examples:

this.Given(_ => TheBuyerIsA(BuyerCategory.Student))
    .And(_ => TheBuyerSelectsA(new MonthlyPass()))
    .When(_ => TheBuyerPays())
    .Then(_ => ASaleOccursWithAnAmountOf(new Money(76)))
    .BDDfy("Successful rail card purchases");

Which gives us a nicely formatted output of

Scenario: Successful rail card purchases
  The buyer is a Student
    The buyer selects a Monthly Pass
  The buyer pays
  A sale occurs with an amount of $76.00

The next step is to turn this into a test with examples.

Step 1

Create somewhere to put the examples, BDDfy supports private/public fields and properties as well as local variables in your test method. The following all work:

private Fare fare;
private BuyerCategory _buyerCategory;
public Money Price { get; set; }

public void MyTest()
{
      var Fare = default(Fare);
}

Local variables are the cleanest and recommended approach, but feel free to use fields or properties when examples are shared between tests.

Step 2

Then we should replace the arguments we are passing to our steps to be the fields/properties

var buyerCategory = default(BuyerCategory);
var fare = default(Fare);
var price = default(Money);

this.Given(_ => TheBuyerIsA(buyerCategory))
    .And(_ => TheBuyerSelectsA(fare))
    .When(_ => TheBuyerPays())
    .Then(_ => ASaleOccursWithAnAmountOf(Price))
    .BDDfy("Successful rail card purchases");

Step 3

Use the .WithExamples() extension method to pass your examples to BDDfy

this.Given(_ => TheBuyerIsA(buyerCategory))
    .And(_ => TheBuyerSelectsA(fare))
    .When(_ => TheBuyerPays())
    .Then(_ => ASaleOccursWithAnAmountOf(price))
    .WithExamples(new ExampleTable(
        "Buyer Category", "Fare", "Price")
    {
        { BuyerCategory.Student, new MonthlyPass(), new Money(76) },
        { BuyerCategory.Senior, new MonthlyPass(), new Money(98) },
        { BuyerCategory.Standard, new MonthlyPass(), new Money(146) },
        { BuyerCategory.Student, new WeeklyPass(), new Money(23) },
        { BuyerCategory.Senior, new WeeklyPass(), new Money(30) },
        { BuyerCategory.Standard, new WeeklyPass(), new Money(44) },
        { BuyerCategory.Student, new DayPass(), new Money(4) },
        { BuyerCategory.Senior, new DayPass(), new Money(5) },
        { BuyerCategory.Standard, new DayPass(), new Money(7) },
        { BuyerCategory.Student, new SingleTicket(), new Money(1.5m) },
        { BuyerCategory.Senior, new SingleTicket(), new Money(2m) },
        { BuyerCategory.Standard, new SingleTicket(), new Money(3m) }
    })
    .BDDfy("Successful rail card purchases");

BDDfy supports two ways of providing examples, if you are migrating from SpecFlow and you are using simple types (unlike the example above where my examples are actually classes) then we support the normal text table:

.WithExamples(@"
    | Header 1 | Header 2     | Header3    |
    | Value 1  | 2            | 3          |
    |          | 14 Mar 2010  | Transition |");

Step 4

Thats it, simply run your tests and the output will be something like this:

Scenario: Successful rail card purchases
  The buyer is a <Buyer Category>        
    The buyer selects a <Fare>           
  The buyer pays                         
  A sale occurs with an amount of <Price>

Examples: 
| Buyer Category | Fare         | Price   |
| Student        | Monthly Pass | $76.00  |
| Senior         | Monthly Pass | $98.00  |
| Standard       | Monthly Pass | $146.00 |
| Student        | Weekly Pass  | $23.00  |
| Senior         | Weekly Pass  | $30.00  |
| Standard       | Weekly Pass  | $44.00  |
| Student        | Day Pass     | $4.00   |
| Senior         | Day Pass     | $5.00   |
| Standard       | Day Pass     | $7.00   |
| Student        | Day Pass     | $1.50   |
| Senior         | Day Pass     | $2.00   |
| Standard       | Day Pass     | $3.00   |

Errors will be reported against examples rather than steps when examples are used. For example:

Examples: 
| Buyer Category | Fare         | Price   | Error                                                  |
| Student        | Monthly Pass | $76.00  |                                                        |
| Senior         | Monthly Pass | $98.00  |                                                        |
| Standard       | Monthly Pass | $146.00 |                                                        |
| Student        | Weekly Pass  | $23.00  | ChuckedAWobblyException: $24 should be $23 [Details 1] |
| Senior         | Weekly Pass  | $30.00  |                                                        |
| Standard       | Weekly Pass  | $44.00  |                                                        |
| Student        | Day Pass     | $4.00   |                                                        |
| Senior         | Day Pass     | $5.00   |                                                        |
| Standard       | Day Pass     | $7.00   |                                                        |
| Student        | Day Pass     | $1.50   |                                                        |
| Senior         | Day Pass     | $2.00   |                                                        |
| Standard       | Day Pass     | $3.00   |                                                        |

Exceptions:
1. ChuckedAWobblyException: $24 should be $23
<stacktrace>

This works with all of our built in reports. If you have custom reports then you will get a test for each example. Have a look at the built in reporters for examples on how to build custom reporters with examples support.

Other things to note

Step names

There are multiple ways to put placeholders in step names, the one shown above allows you to simply use the parameter on the step method. If it matches a column in your example table it will be substituted in the step name. If it doesn’t match you will just get the value like you currently do.

We also support the following:

GivenIHave__initialCount__Beers() # Then access the value from a field/property directly in the step
.Given(_ => InitialCount(), "Given I have <initial count> beers")

If you have more ideas on how we could make this even better, let us know!

Errors

BDDfy will try it’s best to tell you when something is wrong, for example your test will fail if examples in your example table are not put anywhere. This is handy when you make a spelling mistake or refactor things and they no longer match. We will also give you row and column information of values if they cannot be assigned, for example trying to assign ‘null’ to a value type.

I really hope you enjoy using this feature, we have wanted to add this to BDDfy for a long time but it is a pretty massive feature so it has taken us a while. We also not only wanted to make it as good as the Gherkin first languages, but take advantage of the fact that BDDfy tests are written in code to allow complex types to be used in examples.

Recently I have upgraded to nCrunch 2.2 beta and ReSharper 8.1 both which ship with the upgraded nUnit 2.6.3 runner. After I upgraded these tools I noticed tests which should have been failing were passing and passing tests were being reported as failing but showing stack traces from a different test..

Also I was getting different results in nCrunch, R# and nUnits console runner. Something was broken.

NOTE: Async and TPL support is not supported in nUnit 2.x, but will be officially supported in v3.x and that it was a coincidence that it worked in 2.6.2. My discussions about the issues are at here on the nunit discussion board.
As far as I can tell, there is no reason that 4.0 cannot be supported because to offer framework support does not need any new features OR the classes in the Async Targeting Pack or .NET 4.5. At a minimum tests returning Task should be supported as TPL was introduced into the CLR for net40.
Recently I added async void and Task support to BDDfy which targets .NET 4.0, also xUnit 1.9.x supports Tasks in the current released version and has backported async void support to the 1.9.x codebase from the 2.0 and will be released if there is a need to release another patch release before 2.0 is released.

.NET 4.5 is fully supported and works as expected with nUnit, it is only .net 4.0 which is affected

When I first saw this it really really confused me, so I have been diving into async, nUnit and trying all different things which is where my blog post on async and synchronisation contexts came from.

So what’s broken?

But in 2.6.3 tests returning Task and async void tests will not wait for completion. In addition to that, nUnit 2.6.3 will flat out refuse to run tests which return Task and the console runner will return an error code. Other runners just silently skip the tests…

This means that if you use async/await or TPL and have upgraded to R# 8.1, nCrunch 2.2 or 2.3 beta, the nUnit.Runners NuGet project or any other tools which have upgraded to use the nUnit 2.6.3 runner internally your async tests will be completing unobserved and may be failing without your knowledge.

Repro Solution

I put together a sample solution showing the issues (and the screenshot above is from) the GitHub repo is available at https://github.com/JakeGinnivan/nUnit_net4.0AsyncIssues

Here are the test results from 2.6.2:

Tests run: 3, Errors: 1, Failures: 0, Inconclusive: 0, Time: 1.4863562 seconds
  Not run: 0, Invalid: 0, Ignored: 0, Skipped: 0

Errors and Failures:
1) Test Error : ClassLibrary1.Class1Tests.Test1
   System.InvalidOperationException : Operation is not valid due to the current state of the object.

Server stack trace:
   at ClassLibrary1.Class1Tests.<Test1>d__0.MoveNext() in c:\Users\Jake\_Code\WpfApplication4\ClassLibrary1\Class1Tests.cs:line 19

Exception rethrown at [0]:
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<ThrowAsync>b__0(Object state)
   at NUnit.Core.AsyncSynchronizationContext.AsyncOperationQueue.InvokePendingOperations()
   at NUnit.Core.AsyncSynchronizationContext.AsyncOperationQueue.InvokeAll()
   at NUnit.Core.NUnitAsyncTestMethod.RunVoidAsyncMethod(TestResult testResult)

And 2.6.3:

Tests run: 2, Errors: 0, Failures: 0, Inconclusive: 0, Time: 0.676135209417321 seconds
  Not run: 1, Invalid: 1, Ignored: 0, Skipped: 0

Errors and Failures:

Tests Not Run:
1) NotRunnable : ClassLibrary1.Class1Tests.Test3
   Test method has non-void return type, but no result is expected

What to do about it

Luckily we can fix this and go back to a working version of nUnit (which as far as I can tell works perfectly for both async void and Task tests).

First you will need to download nUnit 2.6.2 and extract it to a known location

Fixing ReSharper

ReSharper is pretty easy as the options dialog allows you to override the runner it uses.

Go into the Resharper menu, then Options. Scroll down to Unit Testing under tools and select NUnit.

Then you can point ReSharper at the nUnit lib directory like so:

Now ReSharper will be using a non-broken version of nUnit.

Fixing nCrunch

nCrunch is a little harder, but this fix works fine.

First open up C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2013, then overwrite nunit.core.dll and nunit.core.interfaces.dll with the ones from the 2.6.2 zip you downloaded earlier.

Now nCrunch will be using the non-broken version. Yay

Summary

If you are using .NET 4.0, async/await or TPL and nUnit, do not upgrade your runners to 2.6.3 and if any tools/build servers you are using upgrade, make sure you set them back to using 2.6.2 manually

In this post I will cover:

• What .ConfigureAwait(false) does in .net 4.0 and in .net 4.5, and why they act differently
• Some guidance around using await in a rich client app (i.e await for offloading, not scalability) .net 4.0 in particular, .net 4.5 behaves better and the guidance is not as relevant
• How you can use custom awaiters to switch contexts

Quick SynchronizationContext Intro

If you do not know what synchronization context’s are, then I will quickly cover them here. If you know what they do, skip to the next section.

A SynchronizationContext allows you to queue a unit of work on another context, a context can be a specific thread or it may be shared between multiple threads. For example you can get the SynchronizationContext of the current thread by going var uiSyncContext = SynchronizationContext.Current, you can use it to run a delegate on the UI thread (or whatever context the SynchronizationContext represents) with uiSyncContext.Send(state => { MyProperty = state; }, state: "bar"). This will pass the string “bar” through as state (without implicitly capturing the outside closure) then assign to MyProperty on the UI thread.

SynchronizationContext’s also get notified when asynchronous work is started, which is how frameworks like (nUnit, xUnit 2.0, BDDfy and others supports tests which have a method signature of async void

How await uses SynchronisationContext’s

Async/await is just compiler magic to make it easier to compose asynchronous stuff, it does not make your code run asynchronously all it does is when it sees an await, it splits your method up into a state machine then when the thing you are awaiting finishes executing, the state machine is resumed and your code continues running. Resuming execution is called the continuation.
A great feature of the await keyword is that it captures the current SynchronizationContext before it runs the asynchronous operation, then it will post the continuation to that SynchronizationContext, meaning if you are on the UI Thread when you await Foo() once Foo() finishes running your code will continue execution on the UI Thread.

This scenario here is called offloading, because the UI thread is an important thread you want to run as little code as possible on the UI thread.

Controlling await with .ConfigureAwait()

To change the default behaviour TPL gives us the .ConfigureAwait(bool continueOnCapturedContext), this allows you to write code like this:

public async Task<ObservableCollection<DataViewModel>> DoSomeStuff() {
    var results = await _service.GetSomeData().ConfigureAwait(false);
    var mappedData = MapDataToViewModels(results); // Just pretend this is CPU bound work
    return mappedData;
}

Because of the .ConfigureAwait(false) our continuation and the MapDataToViewModels() method call will be run on a background thread (unless the awaited task is already complete, then the continuation will be executed in-line). Lets take this one step further

public async Task<ObservableCollection<DataViewModel>> DoSomeStuff() {
    var results = await _service.GetSomeData().ConfigureAwait(false);
    var mappedData = MapDataToViewModels(results);
    var moreData = await _service.GetMoreData();
    UpdateViewModelsWithAdditionalData(moreData);
    return mappedData;
}

In which context does UpdateViewModelsWithAdditionalData(moreData); run?

The answer is it depends, on .net 4.0 UpdateViewModelsWithAdditionalData(moreData); runs on the UI thread, on .net 4.5 it will run on the threadpool. Why is this? To answer that we need to dive into something called the execution context.

Execution Context?

When our .NET code is executing, these is a bunch of additional metadata which floats around with our thread, these are things like security information and the synchronisation context. For more in depth information on the .net execution context have a read of http://blogs.msdn.com/b/pfxteam/archive/2012/06/15/executioncontext-vs-synchronizationcontext.aspx.

The important thing to know about the execution context is that the .NET framework flows it around so things work as we would expect, for instance if we start a new thread, out execution context will go with us. Task.Run, ThreadPool.QueueUserWorkItem() etc all will flow the execution context (but will supress the flow of the SynchronizationContext).

Differences between .NET 4.0 and 4.5

So we have explained some of the moving parts when we are dealing with async/await. (If you would like me to do a more in depth post on async/await in general and how it works starting from the basics, leave a comment below).

Above I showed this code

public async Task<ObservableCollection<DataViewModel>> DoSomeStuff() {
    var results = await _service.GetSomeData().ConfigureAwait(false);
    var mappedData = MapDataToViewModels(results);
    var moreData = await _service.GetMoreData();
    UpdateViewModelsWithAdditionalData(moreData);
    return mappedData;
}

And said that in .NET 4.0 UpdateViewModelsWithAdditionalData(moreData); would run on the UI thread and in .NET 4.5 it would run on the ThreadPool. This is because of how the different implementations capture the ExecutionContext.

Breakdown of .NET 4.0

If we break this method down into what will be executed. This is a mixture of the generated state machine, library code and code to just demonstrate what is happening (because there are heaps of moving parts).

int state;
Results results;

void IAsyncStateMachine.MoveNext()
{
    switch 1:
        var mappedData = MapDataToViewModels(results);
        // etc...
        break;
    // etc...
    default:
        var ex = ExecutionContext.Capture();
        var task = _service.GetSomeData();

        TaskAwaiter.OnCompletedInternal(task, ()=> {
            ex.Run(()=>
            {
                results = task.Result;
                state = 1;
                MoveNext();
            });
        }, continueOnCapturedContext: false);
}

Now there are a few things to look at in this small bit of code, first is var ex = ExecutionContext.Capture();, this captures the current execution context and the ExecutionContext contains the SynchronizationContext so when we execute MoveNext() our SynchronizationContext has been restored BUT because the TaskAwaiter.OnCompletedInternal has been told to not continue on the captured context it will be run on the default task scheduler.

The end result of all this is when var mappedData = MapDataToViewModels(results); runs we will:

• Be running on the ThreadPool
• SynchronizationContext.Current will still be the DispatcherSynchronizationContext

So on the next line when we await and do not use .ConfigureAwait(false) the continuation will run on the captured SynchronizationContext

.NET 4.5 Behaviour

Lets have a look at what the code looks like in .NET 4.5 land.

int state;
Results results;

void IAsyncStateMachine.MoveNext()
{
    switch 1:
        var mappedData = MapDataToViewModels(results);
        // etc...
        break;
    // etc...
    default:
        var ex = ExecutionContext.CaptureInternal(ExecutionContext.CaptureOptions.IgnoreSyncCtx); 
        var task = _service.GetSomeData();

        TaskAwaiter.OnCompletedInternal(task, ()=> {
            ex.Run(()=>
            {
                results = task.Result;
                state = 1;
                MoveNext();
            });
        }, continueOnCapturedContext: false);
}

Notice the line where the execution context is captured, in .NET 4.5 the AsyncMethodBuilderCore calls an internal method on the ExecutionContext which allows the caller to specify that they do not want the SynchronizationContext to be captured as part of the ExecutionContext.

The reason .NET 4.0 and .NET 4.5 are different is that this internal cannot be called from the Async Targeting Pack because it’s a library, in .NET 4.5 the AsyncMethodBuilderCore is part of the framework so it can call this internal methods.

How does TaskEx.Run not capture SynchronizationContext?

Because TaskEx.Run delegates to Task.Factory.StartNew() which is part of the framework, it passes the option to not capture the SynchronizationContext

Some guidance for .NET 4.0

The general guidance is any library code should use .ConfigureAwait(false) so our continuations do not constantly post back to the UI thread, because .NET 4.0 always flows the SynchronizationContext it means that all of our await calls should have .ConfigureAwait(false) which is pretty ugly. Also because .ConfigureAwait(false) still checks if the task is complete, using .ConfigureAwait(false) does not guarantee that we will no longer be executing on the UI thread.

Rather than using .ConfigureAwait() we can explicitly get our code running on the ThreadPool as soon as possible. There are two options for this (that I can see currently):

In ViewModels when you invoke any sort of async service, wrap it in a await TaskEx.Run(()=>_myService.DoStuffAsync());, this is quite good because if your service calls other services you do not get the problem where all your services have TaskEx.Run scattered everywhere and you are scheduling a heap more things into the ThreadPool.

Create a app services layer which your viewmodels interact with, in this layer you delegate to your domain or existing app services, but wrap it in a TaskEx.Run(()=>..);

| UI/ViewModels |   ==>   | Service Layer (TaskEx.Run) |   ==>   | Rest of app (never has sync context) |

Both of these options basically try and make sure that only code executing in the ViewModels have a SynchronizationContext, and the rest of your application had no threading concerns.

Additional Option for .NET 4.5

If you are using .NET 4.5, then another option is to create a custom awaiter (note this feature was removed from the async CTP, possibly because it doesn’t work properly in .NET 4.0 and also because it can confuse people)

await TaskHelper.SwitchToThreadPool();

Just awaiting this will cause our async method to switch onto the ThreadPool, personally I think this is quite nice because you do not have to pay the Lambda Tax of Task.Run(..)

The code is pretty simple

public static class TaskHelper
{
    public static SwitchContextToThreadPoolAwaiter SwitchToThreadPool()
    {
        return new SwitchContextToThreadPoolAwaiter();
    }
}

public struct SwitchContextToThreadPoolAwaiter : INotifyCompletion
{
    public SwitchContextToThreadPoolAwaiter GetAwaiter() { return this; }

    public bool IsCompleted { get { return false; } }

    public void OnCompleted(Action continuation)
    {
        if (!Thread.CurrentThread.IsThreadPoolThread)
            ThreadPool.QueueUserWorkItem(state => ((Action)state)(), continuation);
        else
            continuation();
    }

    public void GetResult() { }
}

Summary

I dived into this because of randomly failing tests when I upgraded to nCrunch 2.2 beta, this was due to some false assumptions about how async/await worked in .net 4.0 and also because of some helper classes which scheduled work on the threadpool or on the main UI thread. Because of the way the SyncContext flowed, the nUnit synchronisation context was present where we thought it should not be. Because of this our code and helpers were not working correctly as the WPF UI DispatcherSynchronizationContext can be captured as a TaskScheduler and has a few other differences.

In short

• .NET 4.0 flows the SynchronizationContext always, even when you use .ConfigureAwait(false)
• .NET 4.5 does not flow the SynchronizationContext when you use .ConfigureAwait(false)
• Don’t try and use the nUnit SynchronizationContext as a task scheduler
• If you are using .NET 4.0, use TaskEx.Run to get off the UI Thread as fast as possible, then use an await in your viewmodel to capture the UI thread and bring you back to it
• Make sure your application code does not need to run on the UI thread, if you have services (like say an IDialogService) then it should post to the UI dispatcher in that service so it is effectively mocked in your unit tests. This was the main source of our issues with tests failing..
• Don’t mix threading concerns and application code

I Hope that helps a few people understand SynchronizationContexts, ExecutionContexts and how async/await works with either of them.

Resources

Stephen Toub on ExecutionContext vs SynchronizationContext (talks about .net 4.5 behaviour)

Those of you subscribed to my RSS feeds will likely get duplications, sorry about that.

LinqPad Migration script

Migrates posts, and images (putting them in the correct spot for Jekyll sites and rewriting links in your blog posts as it goes)

Available at http://share.linqpad.net/76oqrd.linq

Other Links

https://github.com/tokkonopapa/octopress-tagcloud

I am excited about this project because it will save me a heap of time and effort with managing the open source projects I do releases for.

GitReleaseNotes was another project I kicked off about the same time that Simon Cropp and Andreas Öhlund were kicking off similar projects, see https://github.com/Particular/ReleaseNotesCompiler. We decided to start the projects off down different roads to start with, then maybe merge later once we both could experiment with ideas.

What does it do

The concept is quite simple, you can optionally specify a Git tag (it will select the newest tag if you do not specify one). It will then scan all newer commits for references to issues.

Once it has all the referenced commits, it will connect the issue tracker you specify, currently GitHub or Jira (and soon YouTrack, TFS and BitBucket) and fetch all the closed issues which have been referenced. It will then output your release notes in Semantic Release Notes format (which is also markdown) to a file you specify.

It can also publish a release on GitHub, including the generated release notes.

Here are some examples of the types of release notes it can generate:

Simple single issue release
Multiple releases

How to use it

GitReleaseNotes is a .NET exe, but can be used on ALL project types (JavaScript, Ruby, Java, whatever) because it simply works with Git and whatever issue tracker you use.

GitReleaseNotes.exe /IssueTracker Github /Repo JakeGinnivan/GitReleaseNotes /Token ######################### /OutputFile ReleaseNotes.md

To Publish, just add the /Publish switch and specify the version you want to publish with the /Version switch. Like so

GitReleaseNotes.exe /IssueTracker Github /Repo JakeGinnivan/GitReleaseNotes /Token ######################### /Publish /Version 0.2.0

How to setup on TeamCity

TeamCity is not required, but I like being able to press a button and my latest CI build gets published.

Well, I am using GitHubFlowVersion which helps me do Semantic Versioning with ease, but if you have another versioning strategy most of the instructions are the same.

Assuming you have a CI build which creates the artifacts, or you have an existing build you will be adding to.

1. Head to https://github.com/settings/applications and create an access token
2. Create a new configuration parameter in teamcity called GitHubToken, set the value to the access token you have just created and edit the spec field and paste in password display='hidden', which will mean your GitHub access token will not be published into any build logs and will just be ####ed out.
3. Your VCS root must be set to checkout Automatically on Agent, otherwise GitReleaseNotes will not run
4. Make sure you check GitReleaseNotes.exe into source control so you can access it
5. Create a new build step which is running a command line application, and make it look something like this
   GitReleaseNotes\GitReleaseNotes.exe /IssueTracker GitHub /Publish /Token %GitHubToken% /Repo JakeGinnivan/GitReleaseNotes /Version %dep.OpenSourceProjects_GitReleaseNotes_CI.system.GitHubFlowVersion.SemVer%

My version number is the SemVer of my CI build, if you are using GitHubFlowVersion you will have to create a dummy system.GitHubFlowVersion.SemVer variable, otherwise you cannot reference it across builds (doesn’t exist at configure time, it is created when you run GitHubFlowVersion).

And thats it, you can have a build publishing your release notes to GitHub and this will also tag master with the version you have just published.

There are still plenty of issues with this project and heaps of work to do, but I like the way it is shaping up. Feedback/contributions are welcome.

Next stop on my trip was looking into Git-Flow and how it manages releases, initially it seemed very waterfally and too heavy to use on an open source project, but I thought it may fit at different clients. I was instantly drawn to the fact that releases have an explicit step to version at the start of the release process, this is great, I can manage my project, merge pull requests, add features then when I am ready, I can decide to release, put together release notes and figure out if this is a major, minor or patch release.

Much to my surpise, it is actually very light-weight and you can drop much of it if your process is simpler (say for open source).

https://github.com/TestStack/ConventionTests is my guinea pig!

To get started, I decided to try and implement git-flow manually to really understand what is going on. There are plenty of explainations out there, so I will just be running through what I did, and how it works from my point of view. I always have two remotes setup for my projects, ‘upstream’ is the main repository, ‘origin’ is my fork.

1. Convert the repo over to git-flow

From your git command line

git checkout master
git fetch upstream
git merge upstream/master
git checkout develop
git push upstream develop

This pushes the develop branch into your repo, then head to the project settings in github and change the default branch to develop.

2. Install GitFlowVersion

I am a big fan on Simon Cropp’s (https://twitter.com/SimonCropp) work. He has been working on https://github.com/Particular/GitFlowVersion with Andreas Öhlund.

Basically GitFlowVersion uses the conventions in place in git-flow to make it really easy to version your software.

Once up and running, tags off master get a normal version, say v2.1.0 (assume this is the LAST tagged release), develop CI builds get the version v2.2.0-unstable20 where the minor is LASTMINOR + 1 and there has been 20 commits since the last tag on master. Or v<major>.<minor+1>.0-unstable<#commitssincelastrelease> Release branches have a version of v2.2.0-beta where 2.2.0 is the version you have put in the release branch name (in this case release-2.2.0) You can also tag a release branch as rc1 or rc2 and the version will become v2.2.0-<tag>.

If this sounds confusing, its not, and it is all pretty automatic, read on to see what it actually means for you maintaining a project.

3. Contributing/Pull Requests

I always use feature branches, but not for an entire feature. I use very short lived branches, which I put up as a pull request as soon as I am done. I am known to submit 3+ pull requests all within the space of an hour when I work on a project because I just fix a bunch of small things.

Nothing much changes here, except you take you branch from develop. So

git checkout develop
git fetch upstream
git merge upstream/develop
git checkout -b FixingSomething

Pull requests now target develop, so I do my commits, push to origin like normal and submit my pull request targeting develop

Your CI build (assuming you are building pull requests) will trigger with build number v2.2.0-PullRequest50, where the version is the last release, with the minor bumped, just like develop version numbers, except the semver tag is PullRequest<PR#>. I think that is pretty neat.

4. Releasing a new version

When you decide you want to release, or start preparing a release you can either take a release branch, or merge develop straight into master and tag master with the release (afaik this doesn’t cause any issues :P)

In my mind there are two styles to run a project,

1. every checkin builds, then if tests pass autodeploys. These projects cannot adhere to semantic versioning, but the changes are often so tiny that upgrades pose little risk.
2. At some point in time, the decision to release is made, this could be as soon as a pull request is merged, or after a particular milestone is reached. All the projects I am currently contributing to work in this way.

4.1 Using a release branch

Using a release branch allows you to make the decision I am going to release, there may be a few things I know I have to do before releasing (DbUp is a great example of this). But I don’t want to stop being able to merge pull requests. To release using a branch (assume develop is up to date), and I am only fixing a bug in the current v2.1.0 release

git checkout -b release-2.1.1

or

git flow release start 2.1.1

I can now leave this branch open for a bit, do the work I need to do to release 2.1.1 or send it to someone else on the team to OK. I could even release the 2.1.1 build as a pre-release package on NuGet. It would have version 2.1.1-beta1 remember.

Once I am happy, either the pre-release package has got the feedback I wanted, or I have made the additional changes I need I simply merge the branch to master (with the –no-ff option), tag, then delete the remote branchs. Remember this is if you are doing it manually, you can also just go git flow release finish if you have the git-flow extensions installed, or use source tree and click a button :P The first two commands are not needed if you haven’t committed anything to the release branch

git checkout develop
git merge release-2.1.1 --no-ff
git checkout master
git merge release-2.1.1 --no-ff
git tag 2.1.1

or

git flow release finish 2.1.1

Then publish

git push upstream master
git push --tags
git branch -d release-2.1.1
git push upstream :release-2.1.1

or

git flow release publish 2.1.1

4.2 Skip release branch

If you don’t want to bother with the release branch, you could also just go

git checkout master
git merge develop --no-ff
git tag 2.1.1
git push upstream master
git push --tags

5. Publish the build

Now you have the release build from your CI you can progress through your build pipeline and release to NuGet or publish to a test environment or whatever the first step is in your Continuous Delivery pipeline.

That’s it

I think Git-Flow is actually not too bad for releasing projects using semantic versioning, most of the time in this is writing the release notes, seeing what has changed since the last release (which is really easy due to the conventions in place!) and checking to see if you have any breaking changes which would mean a major version bump.

As a summary. Here is me fixing and releasing a feature as a pre-release package. The current version on NuGet is currently v2.2.0

git fetch upstream
git checkout develop
git merge upstream/develop
git checkout -b SomeFeature

# Work..
git commit -am "Some experimental feature"
git push origin SomeFeature

# Submit pull request, so NOTHING has changed yet, this PR is merged
git fetch upstream
git checkout develop
git merge upstream/develop # to get my changes into my local develop branch

# So now I am starting the release, nothing before here is new if you are using GitHub flow
git checkout -b release-2.3.0
git push upstream release-2.3.0

Now my CI builds, and I click the button to promote the last CI build as a pre-release NuGet package. Once I am happy, I have got my feedback

git checkout master
git merge release-2.3.0 --no-ff
git tag 2.3.0
git push upstream master
git push upstream --tags

My CI will build, and I can click the publish button on teamcity to release to NuGet.

Thoughts? Do you know of an easier way. I am REALLY open to suggestions and willing to try stuff out at the moment :)