The default options work fine if all you want to access is the Synology system itself, but I noticed that after connecting my OS X client, I couldn’t reach any other computers on my home network.

After much googling and head scratching, I figured out that L2TP does not have a built in mechanism for configuring classless static routes and that VPN Server does not go out of its way to make this work out of the box.

After establishing a connection over L2TP, a client will typically issue a DHCP Inform broadcast message over the connection requesting such routes. I let my WiFi router do DHCP so my Synology DHCP service is disabled. I don’t know if it would reply to these messages over VPN or not.

Anyway, since the DHCP Inform message is sent on the L2TP virtual network, it isn’t rebroadcast on the local LAN, so my WiFi router DHCP server never sees it and can’t respond.

I was able to solve this problem by installing the 3rd party (unsigned) SynDnsmasq package on my Synology box.

Once I had the package installed, I set the contents of /volume1/@appstore/dnsmasq/etc/dnsmasq.conf to:

except-interface=eth0
except-interface=eth1
except-interface=eth2
except-interface=eth3

domain=localnet

dhcp-range=192.168.2.0,static
dhcp-option=option:router
dhcp-option=121,10.0.0.0/24,192.168.2.0
dhcp-option=249,10.0.0.0/24,192.168.2.0
dhcp-option=vendor:MSFT,2,1i

You’ll need to adjust the subnets/masks and router IPs to match your home network and L2TP address range.

What this does is respond to DHCP Inform requests with a classless static route for the home network (10.0.0.0/24) via the L2TP server IP address (192.168.2.0).

I restarted SynDnsmasq, connected my OS X VPN client, and was off to the races.

One issue I ran into, of course, is dealing with cross-origin requests in the browser. For my tests, my Ember app is running on localhost:4200 and my Solr server is running on localhost:8983. The browser, rightly, will refuse to let requests from my Ember app execute on Solr, unless the Solr server enables Cross-Origin Resource Sharing headers.

Solr 5 ships with Jetty, and it turns out there’s already a servlet filter that can do this for us. All we have to do is download it and configure it.

First, figure out which version of Jetty your Solr shipped with. Solr 5.0 comes with Jetty 8.1.10. Now head over to the jetty-servlets folder on the maven.org repository, drill down into the corresponding version, and download jetty-servlets-version.jar. Save the jar into $SOLR_HOME/server/lib.

Next, open $SOLR_HOME/server/etc/webdefault.xml in a text editor. Insert the following at bottom of the file, right before the closing </web-app> tag:

1
2
3
4
5
6
7
8
9
10
11
12
13

  <filter>
    <filter-name>cross-origin</filter-name>
    <filter-class>org.eclipse.jetty.servlets.CrossOriginFilter</filter-class>
    <init-param>
      <param-name>chainPreflight</param-name>
      <param-value>false</param-value>
    </init-param>
  </filter>

  <filter-mapping>
    <filter-name>cross-origin</filter-name>
    <url-pattern>/*</url-pattern>
  </filter-mapping>

The third and final step is to start (or restart) your Solr server.

This example enables CORS requests for all origins. If you are going to do something like this on your production server, you probably want to lock this down using the allowedOrigins parameter.

The chainPreflight parameter is set to false so the filter will short-circuit when a browser sends a pre-flight OPTIONS request. This is necessary for Solr because the Solr update handler would respond with a 400 Bad Request since it doesn’t know what to do with an OPTIONS request.

You can find the other configuration parameters for the filter at wiki.eclipse.org.

Alternatives

If you can’t or don’t want to use CORS, there are a couple other options available to you. You could have your Ember app proxy requests to Solr, you could deploy your Ember app on the same origin as your Solr server, or you could try to use JSON-P.

There’s been quite a bit of work off and on over the months, but I’ve kept postponing a new version to add Just One More Feature. Scope creep, anyone?

One remaining issue I’m now working through is dealing with normal NuGet packages, NuGet symbol packages, and looking ahead at Microsoft.NET 5 (aka .NET vNext).

Ignoring vNext, the nuget pack command takes an optional -symbols argument which asks it to build two separate zip files: one that contains the DLLs, documentation, content, scripts and other artifacts that a project would consume from your package, and another one that contains the DLLs, PDBs (debugging symbols), and source code for your package.

The symbol packages are intended to be uploaded somewhere like SymbolSource.org where they will be processed for consumption by debuggers like Visual Studio and WinDbg.

Klondike allows both normal packages and symbol packages to be uploaded to a single end point. This enables Klondike users to use NuGet for private libraries without losing useful source level integration with Visual Studio.

On the surface, there is little to tell a symbol package from a normal one. Both packages will have virtually identical manifests and metadata. The metadata will have a package ID and version. Apart from the name of the package file (MyThing.1.0.nupkg vs MyThing.1.0.symbols.nupkg), we have to fall back to heuristics to suss out one from the other.

This is what Klondike does: look at the files in the package and if there’s at least one file in the src directory, and at least one pdb file in the lib directory, then the package is deemed to be a symbol package.

With vNext, the waters are muddied. Commands like kpm pack produce a single package that contains consumable contents but also includes PDBs and source. This sounds great. Two use cases in a single payload.

The trouble is in how we distinguish between a normal package and a symbol package. Klondike needs to treat them differently so when a client asks to download a package, it gets the full normal package and not just the symbols and source.

At present, Klondike wrongly concludes that a vNext package is a symbol package. My heuristic is wrong when dealing with such packages.

It’s the file name stupid

My first thought was, it is obvious from a file name like MyThing.1.0.symbols.nupkg that a package should be treated as symbols-only. When files are uploaded using multipart/mime, a helpful Content-Disposition header should tell me the name of the file being uploaded.

Well, as it turns out, regardless of what the file is named on disk, the nuget push command always lies to the server and says it’s uploading a file named package.

Push MyThing.1.0.nupkg and the server sees package. Push MyThing.1.0.symbols.nupkg and the server sees package. Push a file that isn’t a package and the server sees… you get it.

It’s the endpoint stupid

The client knows if a package is symbols or not. Maybe the client should use a different endpoint to upload symbol packages from normal ones. Perhaps trying to handle both package types from the same URL was a bad decision in the first place.

Changing it now would break any automated scripts that currently push packages to Klondike. Maybe there’s not a ton of clients out there today, but it still rubs me the wrong way.

It’s something else?

Apart from requiring the client to use a different URL to push symbols, I’m at a loss of how else to reliably tell the difference between symbols and normal packages. Since the problem only affects vNext, I could add more complicated heuristics to see if aspnet50 or aspnetcore50 are supported by the package. This does not sounds like an approach that won’t need constant attention as new frameworks and versions are introduced.

What do

It’s a bummer that symbol packages don’t have a simple flag in their manifests that proclaim what they are. It’s a bummer that nuget push uses a hard-coded file name no matter what type of package is being uploaded. It’s a bummer that vNext appears to be changing the rules of what a normal package may contain.

Unless I’ve overlooked something, the right choice seems to be to require clients to push symbol packages to an alternate URL.

I’ve been using Grunt (and therefore Node) to develop Klondike for about 6 months and it never occurred to me that hosting managed .net code from Node was an option.

After reading Scott’s post, my immediate question was if I could host an OWIN application from Node. Some quick googling revealed connect-owin is well on its way.

It took a little elbow grease to get from the toy app samples to being able to initialize Klondike and serve requests.

Proof of Concept

On my Windows VM, I ran:

1

npm install connect-owin express

Then created this barebones Node program:

1
2
3
4
5
6
7

var owin = require('connect-owin'),
    express = require('express');

var app = express();
app.all('/api/*', owin('bin/debug/NuGet.Lucene.Web.OwinHost.Sample.exe'));

app.listen(3000);

AppDomains and Configuration

Klondike uses appSettings to configure where files are kept and which options are enabled. It also has a fair amount of binding redirects that make sure all the various assemblies pulled in from NuGet packages agree to use the same version of DLL dependencies.

I realized quickly that when my assembly is loaded from Node, my config file was being ignored.

I asked a question on the Edge Github project to confirm, but inspecting source code it appears to be the case that when Edge loads a managed assembly, it isn’t creating a full AppDomain and specifying a config file to go with it.

To “trick” my configuration into being loaded, I copied node.exe from it’s home in Program Files to my bin/Debug directory, and then copied my config file to node.exe.config in the same local directory.

Sure enough, running .\bin\Debug\node.exe server.js happily pulled in my config settings and binding redirects.

It would be nice if Issue 131 gets first-class support, but in the mean time at least I know there’s a way to load complex applications into Edge.

Whither Mono?

Edge (and therefore connect-owin) have first-class support for the Mono Framework, although unfortunately I was unable to get beyond an issue with Ninject. As it turns out, Ninject has a separate build configuration to support Mono and the packages on nuget.org won’t work when you try to use them with Mono.

This is pretty unfortunate and I’ll be looking for a new IoC container to replace Ninject with. One that has built in integration with WebApi and SignalR would be nice.

The Future

The Future is now. Being able to use Bower, Grunt and Node with integrated .NET request processing means teams can use the same great front-end tools that are sweeping the community by storm while still building rich server-side components powered by .NET.

I’m a huge fan of c# and .NET, but love that this enables front-end developers who aren’t to be able to contribute to these projects without forcing them to stop using Grunt, Bower and whatever other favorite tools they bring from their toolboxes.

Are Edge and connect-owin production ready? I don’t know, but I also don’t really care. To me, these are ideal tools for development and testing, but once grunt build gives me an app tree in ./dist I’d rather just push my app to the cloud and let someone else worry about hosting.

Source

My efforts to convert Klondike to support OWIN self hosting are a work-in-progress, but you can see the source code on the webapi-oata branch of NuGet.Lucene on Github.

The NuGet OData implementation provides links to download a package by using the default stream concept.

In WCF Data Services, an entity indicates that it has a stream using the aptly named HasStreamAttribute.

1
2
3
4
5
6

[DataServiceKey("Id", "Version")]
[HasStream]
public class DataServicePackage : IEquatable<DataServicePackage>
{
    //snip
}

To build a URI for each instance, a custom implementation of IDataServiceStreamProvider is wired into our DataService.

This is done differently in WebApi which does not include the HasStreamAttribute at all.

Over a year ago, I asked on twitter:

At the time, the answer was no. However, I’m happy to report that recent versions of WebApi do support this feature.

First, we need to tell our EDM model builder that the entity has a default stream:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

public void MapODataRoutes(HttpConfiguration config)
{
    var builder = new ODataConventionModelBuilder();

    var entity = builder.EntitySet<ODataPackage>("Packages");
    entity.EntityType.HasKey(pkg => pkg.Id);
    entity.EntityType.HasKey(pkg => pkg.Version);

    var entityType = model
      .FindDeclaredType(typeof(ODataPackage).FullName);

    model = builder.GetEdmModel();
    model.SetHasDefaultStream(
       entityType as IEdmEntityType, hasStream: true);

    // snip
}

Next we’ll need to implement a custom ODataSerializerProvider and ODataEdmTypeSerializer to provide a default stream URL for entity instances:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87

public class ODataPackageDefaultStreamAwareSerializerProvider
    : DefaultODataSerializerProvider
{
    private readonly ODataEdmTypeSerializer entitySerializer;

    public ODataPackageDefaultStreamAwareSerializerProvider()
    {
        this.entitySerializer =
          new ODataPackageDefaultStreamAwareEntityTypeSerializer(this);
    }

    public override ODataEdmTypeSerializer GetEdmTypeSerializer(
      IEdmTypeReference edmType)
    {
        if (edmType.IsEntity())
        {
            return entitySerializer;
        }

        return base.GetEdmTypeSerializer(edmType);
    }
}

public abstract class DefaultStreamAwareEntityTypeSerializer<T>
    : ODataEntityTypeSerializer where T : class
{
    protected DefaultStreamAwareEntityTypeSerializer(
      ODataSerializerProvider serializerProvider)
        : base(serializerProvider)
    {
    }

    public override ODataEntry CreateEntry(
      SelectExpandNode selectExpandNode,
      EntityInstanceContext entityInstanceContext)
    {
        var entry = base.CreateEntry(selectExpandNode,
          entityInstanceContext);

        var instance = entityInstanceContext.EntityInstance as T;

        if (instance != null)
        {
            var link = BuildLinkForStreamProperty(
              instance, entityInstanceContext);

            entry.MediaResource = new ODataStreamReferenceValue
            {
                ContentType = ContentType,
                ReadLink = new Uri(link)
            };
        }
        return entry;
    }

    protected virtual string ContentType
    {
        get { return "application/octet-stream"; }
    }

    protected abstract string BuildLinkForStreamProperty(
      T entity,
      EntityInstanceContext entityInstanceContext);
}

public class ODataPackageDefaultStreamAwareEntityTypeSerializer : DefaultStreamAwareEntityTypeSerializer<ODataPackage>
{
    public ODataPackageDefaultStreamAwareEntityTypeSerializer(
      ODataSerializerProvider serializerProvider)
      : base(serializerProvider)
    {
    }

    protected override string BuildLinkForStreamProperty(
      ODataPackage package,
      EntityInstanceContext context)
    {
        var url = new UrlHelper(context.Request);
        var routeParams = new { package.Id, package.Version };
        return url.Link(RouteNames.Packages.Download, routeParams);
    }

    protected override string ContentType
    {
        get { return "application/zip"; }
    }
}

The most relevant part above is the override of CreateEntry (line 33) which sets the MediaResource property on the serialized ODataEntry.

Finally the custom serializer is wired into the configuration:

1
2
3
4

config.Formatters.InsertRange(0,
    ODataMediaTypeFormatters.Create(
        new ODataPackageDefaultStreamAwareSerializerProvider(),
        new DefaultODataDeserializerProvider()));

This whole mess of code nets us a modest customization in the Atom output from WebApi OData:

1
2
3
4
5
6
7

<entry>
  <id>http://example/api/odata/Packages(Id='Lucene.Net.Linq',Version='3.2.55')</id>
  <content
    type="application/zip"
    src="http://example/api/packages/Lucene.Net.Linq/3.2.55/content"/>
  <!-- snip -->
</entry>

This enables the NuGet client to download packages.

1
2
3
4
5
6
7
8
9
10

public void MapDataServiceRoutes(HttpConfiguration config)
{
    var builder = new ODataConventionModelBuilder();

    var entity = builder.EntitySet<ODataPackage>("Packages");
    entity.EntityType.HasKey(pkg => pkg.Id);
    entity.EntityType.HasKey(pkg => pkg.Version);

    // snip
}

This is enough for our OData feed to render edit and self links for each individual entity in a form like:

http://localhost/odata/Packages(Id='Sample',Version='1.0.0')

But if we navigate to this URL, instead of getting just this one entity by key, we get back the entire entity set.

To get the correct behavior, first we need an override on our PackagesODataController that gets an individual entity instance by key:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

public class PackagesODataController : ODataController
{
    public IMirroringPackageRepository Repository { get; set; }

    public IQueryable<ODataPackage> Get()
    {
        return Repository.GetPackages().Select(p => p.ToODataPackage()).AsQueryable();
    }

    public IHttpActionResult Get(
        [FromODataUri] string id,
        [FromODataUri] string version)
    {
        var package = Repository.FindPackage(id, version);

        return package == null
          ? (IHttpActionResult)NotFound()
          : Ok(package.ToODataPackage());
    }
}

However, out of the box WebApi OData doesn’t know how to bind composite key parameters to an action such as this, since the key is comprised of multiple values.

We can fix this by creating a new routing convention that binds the stuff inside the parenthesis to our route data map:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

public class CompositeKeyRoutingConvention : IODataRoutingConvention
{
    private readonly EntityRoutingConvention entityRoutingConvention =
        new EntityRoutingConvention();

    public virtual string SelectController(
        ODataPath odataPath,
        HttpRequestMessage request)
    {
        return entityRoutingConvention
          .SelectController(odataPath, request);
    }

    public virtual string SelectAction(
        ODataPath odataPath,
        HttpControllerContext controllerContext,
        ILookup<string, HttpActionDescriptor> actionMap)
    {
        var action = entityRoutingConvention
            .SelectAction(odataPath, controllerContext, actionMap);

        if (action == null)
        {
            return null;
        }

        var routeValues = controllerContext.RouteData.Values;

        object value;
        if (!routeValues.TryGetValue(ODataRouteConstants.Key,
          out value))
            {
              return action;
            }

        var compoundKeyPairs = ((string)value).Split(',');

        if (!compoundKeyPairs.Any())
        {
            return null;
        }

        var keyValues = compoundKeyPairs
            .Select(kv => kv.Split('='))
            .Select(kv =>
              new KeyValuePair<string, object>(kv[0], kv[1]));

        routeValues.AddRange(keyValues);

        return action;
    }
}

This class decorates a standard EntityRoutingConvention and splits the raw key portion of the URI into key/value pairs and adds them all to the routeValues dictionary.

Once this is done the standard action resolution kicks in and finds the correct action overload to invoke.

This routing convention was adapted from the WebApi ODataCompositeKeySample project.

Here we see another difference between WebApi OData and WCF Data Services. In WCF Data Services, the framework handles generating a query that selects a single instance from an IQueryable. This limits our ability to customize how finding an instance by key is done. In WebApi OData, we have to explicitly define an overload that gets an entity instance by key, giving us more control over how the query is executed.

This distinction might not matter for most projects, but in the case of NuGet.Lucene.Web, it enables a mirror-on-demand capability where a local feed can fetch a package from another server on the fly, add it to the local repository, then send it back to the client as if it was always there in the first place.

To customize this in WCF Data Services required significant back flips.

OData Controller

1
2
3
4
5
6
7
8
9
10

public class PackagesODataController : ODataController
{
    public IMirroringPackageRepository Repository { get; set; }

    [Queryable(PageSize = 100, HandleNullPropagation = HandleNullPropagationOption.False)]
    public IQueryable<ODataPackage> Get()
    {
        return Repository.GetPackages().Select(p => p.ToODataPackage()).AsQueryable();
    }
}

This controller exposes an entity set of package metadata. The underlying entity type IPackage has some complex types that OData doesn’t play well with, so the types are decorated with ToODataPackage which flattens and simplifies the metadata into primitive types like strings, bools and ints.

The QueryableAttribute exposes some very useful settings that allow us to enable or disable advanced querying options like $select, $expand, $orderby and others that can be used to lock down the endpoint to avoid overly expensive or invalid queries from being executed.

HandleNullPropagation is a much welcomed offering that wasn’t easily tweakable before. It tells the engine whether or not the underlying query execution engine can handle null values in expressions. The default is to modify query expressions dynamically to put null-check conditions around all comparisons and operations like ToLowerCase(), Contains() and such. This is necessary for LINQ to Collections, but generally not necessary for other LINQ implementations.

Null Propagation seems like a prudent default, but it can cause performance problems. David Ebbo wrote about an issue the NuGet Gallery team encountered. If your LINQ provider doesn’t need it, you should definitely turn it off.

Configuration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

public void MapDataServiceRoutes(HttpConfiguration config)
{
    var builder = new ODataConventionModelBuilder();

    var entity = builder.EntitySet<ODataPackage>("Packages");
    entity.EntityType.HasKey(pkg => pkg.Id);
    entity.EntityType.HasKey(pkg => pkg.Version);

    var conventions = ODataRoutingConventions.CreateDefault()
        .Select(c => (IODataRoutingConvention)
          new ControllerAliasingODataRoutingConvention(
            c, "Packages", "PackagesOData"));

    config.Routes.MapODataRoute(
        RouteNames.Packages.Feed,
        ODataRoutePath,
        builder.GetEdmModel(),
        new DefaultODataPathHandler(),
        conventions);
}

This is mostly a vanilla configuration but already there are two modifications.

First, the Packages entity set has a composite key that uniquely identifies each entity instance, comprised of a package ID and version.

The next tweak is that we’re customizing the default list of OData routing conventions, wrapping each one with ControllerAliasingODataRoutingConvention.

My project already has a PackagesController that inherits from ApiController. I wanted to put OData related methods in a separate controller and name it PackagesODataController, but still have my entity set be named Packages.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

public class ControllerAliasingODataRoutingConvention : IODataRoutingConvention
{
    private readonly IODataRoutingConvention delegateRoutingConvention;
    private readonly string controllerAlias;
    private readonly string targetControllerName;

    public ControllerAliasingODataRoutingConvention(
        IODataRoutingConvention delegateRoutingConvention,
        string controllerAlias,
        string targetControllerName)
    {
        this.delegateRoutingConvention = delegateRoutingConvention;
        this.controllerAlias = controllerAlias;
        this.targetControllerName = targetControllerName;
    }

    public string SelectController(ODataPath odataPath, HttpRequestMessage request)
    {
        var controller = delegateRoutingConvention.SelectController(odataPath, request);
        return string.Equals(controller, controllerAlias,
                StringComparison.OrdinalIgnoreCase)
            ? targetControllerName
            : controller;
    }

    public string SelectAction(
        ODataPath odataPath,
        HttpControllerContext controllerContext,
        ILookup<string, HttpActionDescriptor> actionMap)
    {
        return delegateRoutingConvention.SelectAction(
          odataPath, controllerContext, actionMap);
    }
}

This is a basic decorator that renames a standard controller name with a desired target name.

Summary

At this point we have a WebApi endpoint that speaks OData. We can enumerate packages, filter, order and page. We’re still pretty far from something that a NuGet client can speak to though.

The NuGet core team has more or less acknowledged that OData is overkill for the types of operations the client needs (list, search, find updates), and they’re even planning on migrating away from OData in v3 of their HTTP api.

Unfortunately (or fortunately?) NuGet has achieved wide adoption using the existing api, so it will take years for everyone to update their clients once this new api has shipped.

In the mean time, we’re stuck with OData.

To make the most of the situation, I wanted to try out the new WebApi integrated OData packages. These packages allow us to use the same base classes and infrastructure as regular WebApi controllers. They also enable us to build and deploy self-hosted applications to decouple us from IIS.

WebApi OData has seen lots of features get implemented and adds supports for newer protocol versions of OData. However, there are a few things that the NuGet clients require that are not built in. This series will go over configuring WebApi OData to create a NuGet compatible package feed.

Required Features

OData is a huge standard and NuGet uses only a subset of capabilities. Here’s a list of what we’ll need to support:

• Entity Sets
• Composite Keys
• Actions
• Default Streams
• $count meta-action
• $filter
• $orderby
• Paging with $top and $skip

Most of these capabilities are built into WebApi OData. The ones that aren’t will be covered in the following articles.

I’m leaving it here for my own reference.

This recipe dumps the key and value fields in an array of NameObjectCollectionBase+NameObjectEntry, though it would probably work with similar key/value tuples.

1
2
3
4
5

.foreach /pS 16 /ps 1 (addr {!da 3ffc0c460}) {
  .if ($sicmp("${addr}", "null") != 0) {
    !do poi(${addr}+8); !do poi(${addr}+10)
  }
}

Note: the sample code above is formatted for readability. Remove line breaks before using it in windbg, which doesn’t seem to handle line breaks in commands well.

The DumpArray command in SOS does not have a -short flag like DumpHeap does, so the foreach skips the first 16 tokens, then skips a single token for each item processed. This effectively selects array item addresses from the output.

Inside the foreach block, I check that the address is not null, then dump the fields at offsets 8 and 16 (0x10 in hex).

The new guidance is to not use the “Enable NuGet Package Restore” option in Visual Studio, but instead to let it happen automatically.

Unfortunately, many developers will probably miss this guidance if they’re already used to restoring packages as part of their workflow.

The option offered when right-clicking the Solution in Visual Studio gives no indication that the option is deprecated, and the text implies that if you don’t activate the option, packge restore will not be enabled.

These are minor annoyances that will hopefully be addressed as more users adopt NuGet 2.7 and later clients.

One thing the guidance fails to address completely is how an automated build system, like a Continuous Integration server, should approach restoring packages.

The section that covers this topic explains how packages can be restored using the NuGet comand line client prior to building the solution, but where are you supposed to get nuget.exe from?

There are 3 approaches you can take to address this:

1. Assume the build server is configured to have a recent nuget.exe on the PATH
2. Check nuget.exe into your source repository
3. Fetch the latest nuget.exe from nuget.org on the fly.

The now obsolete “Enable NuGet Package Restore” option did a mix of (2) and (3). It would look for nuget.exe in the .nuget folder, and if it isn’t found and some options are enabled, it will download it from the web.

How can we continue using this feature while moving towards the new One True Way of Package Restore?

We can do it by co-opting parts of NuGet.targets in our own build script.

First, layout:

1
2
3
4
5

project/
project/IntegratedBuild.proj
project/NuGet.targets
project/source/
project/source/Foo.sln

IntegratedBuild.proj

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="12.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <SolutionDir>$(MSBuildProjectDirectory)\source\</SolutionDir>
  </PropertyGroup>

  <ItemGroup>
    <Solution Include="$(SolutionDir)*.sln"/>
  </ItemGroup>

  <Target Name="Build" DependsOnTargets="RestoreSolutionPackages;BuildSolution"/>

  <Target Name="RestoreSolutionPackages" DependsOnTargets="DownloadNuGetCommandLineClient">
    <Exec Command="$(NuGetCommand) restore $(SolutionDir) -NonInteractive -Source http://www.nuget.org/api/v2/"/>
  </Target>

  <Target Name="BuildSolution">
    <MSBuild Projects="@(Solution)"/>
  </Target>

  <Import Project="NuGet.targets"/>
</Project>

NuGet.targets

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <NuGetToolsPath>$(MSBuildThisFileDirectory)</NuGetToolsPath>
  </PropertyGroup>

  <PropertyGroup>
    <NuGetExePath Condition=" '$(NuGetExePath)' == '' ">$(NuGetToolsPath)\NuGet.exe</NuGetExePath>

    <NuGetCommand Condition=" '$(OS)' == 'Windows_NT'">"$(NuGetExePath)"</NuGetCommand>
    <NuGetCommand Condition=" '$(OS)' != 'Windows_NT' ">mono --runtime=v4.0.30319 $(NuGetExePath)</NuGetCommand>
  </PropertyGroup>

  <Target Name="DownloadNuGetCommandLineClient">
    <DownloadNuGet OutputFilename="$(NuGetExePath)" Condition=" !Exists('$(NuGetExePath)')" />
  </Target>

  <UsingTask TaskName="DownloadNuGet" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll">
    <ParameterGroup>
      <OutputFilename ParameterType="System.String" Required="true" />
    </ParameterGroup>
    <Task>
      <Reference Include="System.Core" />
      <Using Namespace="System" />
      <Using Namespace="System.IO" />
      <Using Namespace="System.Net" />
      <Using Namespace="Microsoft.Build.Framework" />
      <Using Namespace="Microsoft.Build.Utilities" />
      <Code Type="Fragment" Language="cs">
          <![CDATA[
          try {
              OutputFilename = Path.GetFullPath(OutputFilename);

              Log.LogMessage("Downloading latest version of NuGet.exe...");
              WebClient webClient = new WebClient();
              webClient.DownloadFile("https://www.nuget.org/nuget.exe", OutputFilename);

              return true;
          }
          catch (Exception ex) {
              Log.LogErrorFromException(ex);
              return false;
          }
      ]]>
      </Code>
    </Task>
  </UsingTask>
</Project>

Make sure to add NuGet.exe to your .gitignore or equivalent scm configuration.

It’s a little unfortunate that this boilerplate has to be copied from one project to another. However, since we’re bootstrapping NuGet we have a chicken and egg problem that prevents the targets from being shared in a package.

I hope this helps developers get the most out of NuGet’s latest approach to restoring packages while avoiding needing to check NuGet.exe into source control and keep it up to date.

This won’t work with xbuild on Mono since it does not support CodeTaskFactory, but if you are using mono you could add an alternate target that uses curl or wget to perform the download. Assuming curl or wget are already on your PATH. Sigh.

A simpler approach if you know curl is on the system is to use a different scripting language, that effectively does

1
2
3

curl -o nuget.exe https://www.nuget.org/nuget.exe
./nuget.exe restore source -NonInteractive
msbuild source\*.sln

But why use 3 lines of script when you can have a 100 lines of XML?

I think Phil is trying to solve the same problem lots of .NET developers run into. Let’s say you have a web project that is using the wonderful Lucene.Net.Linq package and you want to fix a bug or add a feature to the library, and test it in your web project without needing to manually copy DLLs around from one solution to another.

This is a workflow I use constantly, and the tool I use to minimize friction during development is Slimjim.

Slimjim, in a nut, is a command line tool that generates Visual Studio solution files. The killer feature is that it will also modify csproj files to replace assembly references with project references. This way you can have a solution with all the projects you want to work with, and when you hit <ctrl>+<shift>+B to build the solution, everything builds in the right order and your top level project gets the newly compiled code without you needing to do any extra work.

An Example

Most of my open source work recently has been on Klondike, a fast NuGet package server you can host on your private network.

This project has actually exploded into lots of sub-projects, each of which correlates to one or more NuGet packages available on nuget.org.

Getting the Source

First, we need to clone all of the repositories that have code we want to work on.

mkdir workspace
cd workspace
git clone https://github.com/themotleyfool/Klondike
git clone https://github.com/themotleyfool/NuGet.Lucene
git clone https://github.com/themotleyfool/Lucene.Net.Linq
git clone https://github.com/themotleyfool/AspNet.WebApi.HtmlMicrodataFormatter

I said there were a lot of projects right? For fun let’s pull in some 3rd party open source code too. That way we can get debug builds and browse the code to better understand it.

git clone https://git01.codeplex.com/nuget
svn checkout https://svn.apache.org/repos/asf/lucene.net/tags/Lucene.Net_3_0_3_RC2_final/ Lucene.Net

And note in some cases you may want to checkout a tag or branch other than master. For nuget:

cd nuget
git checkout Release-2.7.2
cd ..

Getting Slimjim

Slimjim is on nuget.org, so you can install it locally by doing:

nuget install slimjim

And now we have our utility in ./Slimjim.1.0.3/tools/Slimjim.exe

Generating a Solution

Running Slimjim without any flags will create a Solution that includes all csproj projects found under the current directory.

There may be projects you don’t care about that you don’t want cluttering your workspace. For example, Nuget has lots of projects and I only depend on Nuget.Core.

Since I’m working on Klondike, I’ll tell Slimjim to target that project and include anything it depends on, recursively.

./Slimjim.1.0.3/tools/Slimjim.exe -t Klondike -a -c

The -a switch says to include efferent dependencies, and the -c switch says to modify the csproj files to convert assembly references to project references.

If we look at one of the csproj files, we can see an example of this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

diff --git a/source/Klondike/Klondike.csproj b/source/Klondike/Klondike.csproj
index 049d8eb..df61e4b 100644
--- a/source/Klondike/Klondike.csproj
+++ b/source/Klondike/Klondike.csproj
@@ -45,10 +45,16 @@
       <SpecificVersion>False</SpecificVersion>
       <HintPath>..\packages\Antlr.3.5.0.2\lib\Antlr3.Runtime.dll</HintPath>
     </Reference>
-    <Reference Include="AspNet.WebApi.HtmlMicrodataFormatter, Version=2.0.0.0, Culture=neutral, processorArchitecture=MSIL">
-      <SpecificVersion>False</SpecificVersion>
-      <HintPath>..\packages\AspNet.WebApi.HtmlMicrodataFormatter.2.0.0\lib\net40\AspNet.WebApi.HtmlMicrodataFormatter.dll</HintPath>
-    </Reference>
+    <ProjectReference Include="C:\Projects\workspace\AspNet.WebApi.HtmlMicrodataFormatter\source\AspNet.WebApi.HtmlMicrodataFormatter\AspNet.WebApi.HtmlMicrodataFormatt+      <Project>{794ACAF9-D25C-4571-B801-9FC2CF86C1B4}</Project>
+      <Name>AspNet.WebApi.HtmlMicrodataFormatter</Name>
+      <SlimJimReplacedReference>
+        <Reference Include="AspNet.WebApi.HtmlMicrodataFormatter, Version=2.0.0.0, Culture=neutral, processorArchitecture=MSIL">
+          <SpecificVersion>False</SpecificVersion>
+          <HintPath>..\packages\AspNet.WebApi.HtmlMicrodataFormatter.2.0.0\lib\net40\AspNet.WebApi.HtmlMicrodataFormatter.dll</HintPath>
+        </Reference>
+      </SlimJimReplacedReference>
+    </ProjectReference>
     <Reference Include="Common.Logging, Version=2.1.2.0, Culture=neutral, PublicKeyToken=af08829b84f0328e, processorArchitecture=MSIL">

Remember not to commit this or you’ll have a bad time!

Fixing some NuGet things

This is where we’ll hit some friction. I’d like to improve Slimjim to handle some of these problems, but the good news is you only have to do this once.

If any of your projects are using NuGet’s package restore feature, your newly generated solution probably won’t build. This is because NuGet expects to have some resources available relative to the $(SolutionDir) property, and our Solution file is in a directory that doesn’t have these tools.

I usually copy them from one of the projects:

cp -R Klondike/source/.nuget .

Additionally, assembly hint paths are hard-coded in the csproj files, usually like:

<HintPath>..\packages\SomeLib.1.0\lib\net40\SomeLib.dll</HintPath>

However, when you build from the solution file in workspace, NuGet will restore packages relative to this directory, so the csproj files won’t be able to see them.

The simple fix is to build each project individually so it will restore packages to the right place. After each individual repo’s solution builds, we’ll be able to build from our Slimjim solution.

The more complex solution is to use NTFS junctions to make sure each repo is using the same package directory.

mkdir packages
cd Lucene.Net.Linq/source
mklink /j packages ..\..\packages

Doing actual work

Now we have a Solution with the projects we want to work on and it should build successfully from the command line using msbuild.exe.

We can open the Solution in Visual Studio, write code, build, test with IIS or IIS Express, run unit tests, write more unit tests, and all the other good stuff we like doing.

Releasing

Once we’ve made changes we’re happy with, we have to undo the changes Slimjim made to our csproj files before we can commit:

./Slimjim.1.0.3/tools/Slimjim.exe -t Klondike -a -u

The -u switch is the correlary to -c, restoring assembly references that we had turned into project references before.

Now we can commit/push/pull request, push packages to nuget.org, then update packages in the projects that need them.

Looking ahead

This workflow is not perfect. It requires some manual setup and upkeep whenever we want to pull changes and keep everything in sync. It introduces a danger in that you might accidentally commit stuff to your csproj. If you update any nuget packages while working in your generated Solution, the hint paths will point to the wrong packages folder.

I’d like to explore how Slimjim can address some of these problems. It would also be nice if nuget was a little more dynamic about hint paths.

In the mean time, Slimjim makes it possible to write features across disparate projects without needing to push new binaries or manually copy DLL files around.

Git submodules

Coming back to what @haacked wanted to do, it would be possible to treat your workspace folder as a new git repo and include each repo within as a submodule. This could reduce some friction for teams that want to work using a shared pattern.

WebApi also includes the concept of an ApiExplorer, a built in service that knows about all the routes, templates and parameters an application is configured to handle. It even supports attaching human friendly documentation.

With these built in perks, WebApi gets you 95% of the way there. What it doesn’t do out of the box, unfortunately, is something that it says right on the box. It doesn’t do web.

HTTP, ReST and the Web

Writing an application that speaks json over http does not automatically give you something that can be called ReST. To get to ReST nirvana your api needs to speak using hypermedia, providing the client with not just data, but links and forms that can do stuff with the data.

If you aren’t familiar with this distinction, here are some great resources:

• Richardson Maturity Model
• Building Hypermedia APIs with HTML (talk by Jon Moore)
• Slide decks by Stefan Tilkov
• Exist In the Web - Not on it slide deck by Amy Palamountain

Microdata

Jon Moore’s work is particularly intriguing because he shows what you can already do with Microdata.

Microdata boils down to 3 attributes that you stick on elements in your html5 markup, and they’re all you need to give semantic information to a client. You can nest properties into a scope, describe the type of data being represented, and even nest scopes with scopes recursively.

Jon’s talks show how a web client that speaks Microdata can already start on sites like ebay.com, submit a form and find and manipulate resources without knowing anything ahead of time about what resources are there and what URLs are used to access them.

Update: In the (long) time between drafting this post and publishing it, Microdata has died. The spirit of this post remains intact, but I’ll probably be looking into switching to RDFa Lite.

WebApi + Microdata

On the one hand we have a .net framework for writing HTTP apis that has pluggable support for different content types. On the other hand we have html5 and Microdata. Think they could be friends?

I’m excited to announce HtmlMicrodataFormatter, a new open source (Apache Public License) project available on nuget.org and hosted on GitHub.

When you install this package into your WebApi project, it adds two features.

First, if you enter the URL for one of your api methods into a web browser, you get (relatively) nice, human friendly html5 instead of the wall of xml you would otherwise be greeted with.

Second, and perhaps more importantly, if you browse to ~/api, you will be greeted by an html5 page that includes links and forms for all the routes that are registered with ApiExplorer.

You can fill in the inputs and submit forms to test them out right in the browser. If your routes are templated, a javascript event handler will process the URI template and update the form action just before the form is submitted.

Turn on xml documentation for your project, rebuild, and now all the documentation comments you attach to your controllers and actions appear on the forms.

Why it Matters

Obviously, getting a free html form generated for each action can be a productivity boon for developers who are tired of writing

curl -X POST -H 'Accept: application/json' \
  --data 'param1=value1&param2=value2' \
  http://localhost:49497/api/users/fred'

But it’s more than that. Make your client project use html5 and instead of hard-coding URLs for each thing the client wants to do, the client starts at /api, finds the form it wants, fills it in and submits it. The Microdata attributes in the response, along with links and forms, give the client the pieces it needs to drive state, just like Roy T. Fielding intended when ReST was first described.

Do you have third party developers who consume your api? Now you don’t have to give them a separate document or website that describes (out of band) how to use your api. Just link them to yourwebsite/api, and they can read the documentation and discover what they can do without guessing or following brittle conventions.

Putting the web into WebApi removes the tight coupling between client and server. Now you can change your routes, rename parameters, add parameters, and the client will automatically see those changes reflected in the forms and links they get when they enter your api. No more /api/v1/users through /api/v23/users. No more api version headers.

Next Steps

HtmlMicrodataFormatter is pretty young and there are many ideas left to flesh out. For example, making it easy to add links and forms to the output of controller actions, or improving how xml documentation is collected and converted to html, or making it easier to link from resources to related resources with appropriate rel attributes. The list goes on.

I hope you like HtmlMicrodataFormatter, and if you do, maybe get involved and help improve it by submitting feedback, bug reports, enhancements or pull requests to the GitHub project.

Similar to Jon Moore’s python based microdata client, work has started on a .net dynamic client which I hope to share soon.

Get the source and see examples at https://github.com/themotleyfool/AspNet.WebApi.HtmlMicrodataFormatter.

Since I’ve been working on writing a NuGet web application, I have had the opportunity to become intimately familiar with the web api and its wrinkles.

I was going to title this post “Fun with the NuGet REST Api”, but then I had to change it because what NuGet does is pretty far from RESTful. The documentation for Tab Completion Endpoints openly acknowledges this, stating, “We currently don’t look at the Accept header or do lots of other proper HTTP API stuff”. Caveat emptor.

There have been plenty of well written posts about what REST is and why most things that speak json or xml over HTTP are not Restful.

1. Clarifying REST from @kellabyte
2. Representational State Transfer (REST) Roy Fielding’s doctoral thesis
3. REST APIs must be hypertext-driven

WCF Data Services and REST

NuGet has used WCF Data Services (OData) since inception to do the heavy lifting. OData kind of sort of does a decent job of adhering to RESTful constraints and using hyperlinks that give hints to clients about collections, entities and functions that can be invoked on them.

Query the API Root:

curl http://nuget.org/api/v2/

1
2
3
4
5
6
7
8
9

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<service xml:base="http://nuget.org/api/v2/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:app="http://www.w3.org/2007/app" xmlns="http://www.w3.org/2007/app">
  <workspace>
    <atom:title>Default</atom:title>
    <collection href="Packages">
      <atom:title>Packages</atom:title>
    </collection>
  </workspace>
</service>

You can see there’s a collection of Packages, and you even get an href that tells you where it is.

Unfortunately it does not tell you about the magic $metadata RPC call that gets you more information about the entities contained in the Packages collection, and what other functions are available:

curl 'http://nuget.org/api/v2/$metadata'

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<edmx:Edmx Version="1.0" xmlns:edmx="http://schemas.microsoft.com/ado/2007/06/edmx">
  <edmx:DataServices xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" m:DataServiceVersion="2.0">
    <Schema Namespace="NuGetGallery" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns="http://schemas.microsoft.com/ado/2006/04/edm">
      <EntityType Name="V2FeedPackage" m:HasStream="true">
        <Key>
          <PropertyRef Name="Id" />
          <PropertyRef Name="Version" />
        </Key>
        <Property Name="Id" Type="Edm.String" Nullable="false" m:FC_TargetPath="SyndicationTitle" m:FC_ContentKind="text" m:FC_KeepInContent="false" />
        <Property Name="Version" Type="Edm.String" Nullable="false" />
        <Property Name="Authors" Type="Edm.String" Nullable="true" m:FC_TargetPath="SyndicationAuthorName" m:FC_ContentKind="text" m:FC_KeepInContent="false" />
        <snip/>
      </EntityType>
      <EntityContainer Name="FeedContext_x0060_1" m:IsDefaultEntityContainer="true">
        <EntitySet Name="Packages" EntityType="NuGetGallery.V2FeedPackage" />
        <FunctionImport Name="Search" EntitySet="Packages" ReturnType="Collection(NuGetGallery.V2FeedPackage)" m:HttpMethod="GET">
          <Parameter Name="searchTerm" Type="Edm.String" Mode="In" />
          <Parameter Name="includePrerelease" Type="Edm.Boolean" Mode="In" />
        </FunctionImport>
        <FunctionImport Name="FindPackagesById" EntitySet="Packages" ReturnType="Collection(NuGetGallery.V2FeedPackage)" m:HttpMethod="GET">
          <Parameter Name="id" Type="Edm.String" Mode="In" />
        </FunctionImport>
      </EntityContainer>
    </Schema>
  </edmx:DataServices>
</edmx:Edmx>

This xml vomit says that in addition to the hard-coded, out-of-band RPC API that is OData, you can also invoke the Search and FindPackagesById functions, how to invoke them and what parameters they take.

At some point the NuGet developers must have gotten sick of using WCF Data Services, because the endpoints for uploading a package, deleting a package, and the tab completion endpoints all are implemented without integration with WCF Data Services. That means that these operations are not discoverable by a client, so if the client wants to invoke them it has to know their locations.

The Evolution of a Public API

When the client has no way of discovering available resources, that means the server can never change the namespace of those resources, or risk breaking older clients.

SalesForce has at least 26 namespaced versions of some services which causes some problems that @kellabyte discusses on a twitter thread.

NuGet in its current form uses /api/v2 for most HTTP API calls, and has managed to stay on v2 for a while now. We’ve seen some new features get added without forcing everyone to move over to v3 yet. The v1/v2 crutch should never have been introduced, but since it is we’re stuck with it.

Dumb Servers and Dumber Clients

Since the server won’t tell the client how to invoke certain operations, the client must make assumptions and NuGet does exactly this. Badly.

The tab completion endpoints are predictably dumb. If you deploy a NuGet feed at http://example.com/ and point Visual Studio to it, the PowerShell console will try to go to http://example.com/api/v2/package-ids. If you deploy the same feed at http://example.com/nuget, and point Visual Studio to that, the PowerShell console will still try to go to http://example.com/api/v2/package-ids. Note how the /nuget/ part of the namespace vanished. This will generally result in a 404.

Try to push or delete a package and things get more confusing. If you do “nuget push Foo.nupkg -source http://example.com/”, instead of blindly appending api/v2 to the URI, the client first does a GET request on the root URI. If that GET request results in a 301/302 Redirect, the client will follow the redirect and use that location to push the package. However, if the root URI returns some other response code, the client reverts to dumbly hard-coding the path to api/v2/package.

But if you push to a URI that is not the root URI, the client listens to you and pushes to whatever URI you told it.

What all of this means is that it is surprisingly hard to write a web application that complies with all the weird shit that the NuGet client does. If you tell Visual Studio that the package feed is at http://example.com/api/v2, then NuGet will push to http://example.com/api/v2. But if you tell NuGet to publish to http://example.com/ then it will publish to http://example.com/api/v2/package.

That means you need redundant routes to catch both cases.

Example Routes

For my application, here are the routes I came up with that seem to keep the NuGet client happy, as long as you don’t deploy it as a child application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

public static void MapApiRoutes(HttpRouteCollection routes)
{
    // Serves up HTML to browsers that accept text/html
    routes.MapHttpRoute(RouteNames.Home,
                        "",
                        new { controller = "Home" },
                        new { acceptHeader = new AcceptHtmlConstraint() });

    routes.MapHttpRoute(RouteNames.PackageDownload,
                        "api/v2/package/{id}/{version}/content",
                        new { controller = "Packages", action = "DownloadPackage" });

    routes.MapHttpRoute(RouteNames.PackageInfo,
                        "api/v2/package/{id}/{version}",
                        new {controller = "Packages", action = "GetPackageInfo", version = ""},
                        new { httpMethod = new HttpMethodConstraint(HttpMethod.Get) });

    routes.MapHttpRoute(RouteNames.PackageApi,
                        "api/v2/package/{id}/{version}",
                        new { controller = "Packages", id = "", version = "" },
                        new { httpMethod = new HttpMethodConstraint(HttpMethod.Put, HttpMethod.Post, HttpMethod.Delete) });

    routes.MapHttpRoute(RouteNames.TabCompletionPackageIds,
                        "api/v2/package-ids",
                        new { controller = "TabCompletion", action = "GetMatchingPackages", maxResults = 30, includePrerelease = false });

    routes.MapHttpRoute(RouteNames.TabCompletionPackageVersions,
                        "api/v2/package-versions/{packageId}",
                        new {controller = "TabCompletion", action = "GetPackageVersions", includePrerelease = false});
}

public static void MapDataServiceRoutes(RouteCollection routes)
{
    var dataServiceHostFactory = new DataServiceHostFactory();

    // Maps OData to root of application (NOT /api/v2)
    var serviceRoute = new ServiceRoute("", dataServiceHostFactory, typeof(PackageDataService))
        {
            Defaults = RouteNames.PackageFeedRouteValues,
            Constraints = RouteNames.PackageFeedRouteValues
        };

    routes.Add(RouteNames.PackageFeed, serviceRoute);
}

Next

I would love to see NuGet adopt more RESTful practices with regards to its HTTP API, but having had one pull request declined already, I’m not sure how eager I’ll be in trying to bring about this change.

I came across the Cornell University Ergonomics Department’s research summarization on Sitting and Standing at Work a few years ago. The article starts off by describing some scary correlations between glucose metabolism, heart disease and kidney disease with sitting for prolonged periods of time, then goes to say that regular aerobic exercise does not seem to counteract some of the negative affects.

The recommended activity is to take a short, 2 minute break every 20-30 minutes and move around. The theory is that taking a short walk or doing other light exercise increases circulation and encourages the metabolism to burn blood sugar instead of storing it as fat.

Workrave

Since I work in a c#/.net shop, my workstation is a Windows PC, so I found a program called Workrave. It also runs on Linux, but if you’re a Mac user there are programs that do similar things too.

Workrave is designed primarily to prevent a different ailment: Repetitive Strain Injury (RSI). To that end, the program aims to remind the user to take “micro breaks” every 10-15 minutes to let the wrists and fingers rest or give them a nice stretch. Workrave also has an optional daily limit feature that will tell you when you’ve been at the keyboard for too many hours in a day, and politely ask you to drop what you’re doing until tomorrow.

Personally, I haven’t had much trouble with wrist pain, so I don’t use the micro-break and daily limit features. But Workrave also has a “Rest break” timer that you can customize to remind you to take a break.

For my purposes, Workrave is really just a glorified countdown timer, but it has a pretty friendly notification system that gets your attention but doesn’t completely interrupt your work when the time comes due. It also has a nice option to postpone or skip a rest break when it comes due.

One important caveat about Workrave: since it was created to prevent RSI, the program only runs the timer by default when it detects keyboard and mouse activity. So if you spend lots of time reading code, browsing, reading e-mail and the like, Workrave will go significantly longer than the time limit you set before asking you to take a break. This is easily fixed however by enabling the “Reading mode” option, which causes the timer to run continuously regardless of whether you are reading or typing and mousing.

Workrave also has built-in suggested exercises you can try during breaks that help relax your eyes, stretch the neck and things like that. These are pretty useful, but it’s paramount to get out of the chair and get your blood flowing. I largely ignore the built in exercises and just go for a short walk.

Breaks

The trouble you inevitably run into is that getting little reminders to take breaks can be distracting. Programmers, like anybody else, are most productive when they achieve that state of mind known as flow, where attention to the task at hand is so focused that even the passing of time fades away. The notification UI is subtle enough that I find I can often postpone a rest break without losing much focus, but the trouble is that the whole point is to get up and move away from the computer at least every 30 minutes.

When I really get into the zone, I still struggle sometimes with tearing myself away, and I end up hitting postpone repeatedly. But whenever I do take a break I find that I can come back and settle back into my work quickly without having to warm up again. Plenty has been written about the negative affects of “context switching” between various tasks, but taking a short break actually lets you keep noodling on the same task while you take a brief walk around. Sometimes taking your eyes off the code lets you think about a problem from a different angle. I’ve actually found it beneficial to take these breaks while debugging or troubleshooting a tough problem that I’ve been stuck on.

Conclusion

Sitting all day is bad for you. Working out after work doesn’t change that. Twice an hour, get up and move around. It’s good for your health and productivity.

Series 1

1. NuGet Perf Problems, part I
2. Nuget Perf Problem, Part II-Importing To RavenDB
3. NuGet Perf, Part III-Displaying the Packages page
4. NuGet Perf, Part IV-Modeling the packages
5. NugGet Perf, Part V-Searching Packages
6. NuGet Perf, Part VI AKA how to be the most popular dev around
7. NuGet Perf, Part VII AKA getting results is only half the work
8. NuGet Perf, Part VIII: Correcting a mistake and doing aggregations

Series 2

1. NuGet Perf, The Final Part - Load Testing - Setup
2. NuGet Perf, The Final Part - Load Testing - The Tests
3. NuGet Perf, The Final Part - Load Testing - The Results
4. NuGet Perf, The Final Part - Load Testing - Results ^ 2
5. NuGet Perf, The Final Part - Load Testing - Source Code

This series piqued my interest given that I’ve been working on Lucene.Net.Linq and integrating it with NuGet.Server. I haven’t worked directly with RavenDB, but I have used some products like Octopus Deploy which uses RavenDB. It seems like a pretty cool product and I don’t have anything against it. However, there are some problems with the blog series that might lead the casual reader to be misled about what has been accomplished, especially with regards to declarations of victory and some misleading performance numbers.

I know the blog series is only meant to show off RavenDB and give potential users an idea of what their code might look like, but nonetheless some hefty claims are made about performance in comparison to the Entity Framework / SQL Server solution in use at nuget.org. Let’s take a look.

Pick a Schema, Any Schema Will Do

Of course if you’re going to model data in a new persistence layer, it makes perfect sense to simplify or improve the way your data is stored and indexed for later retrieval. Ayende points out some problems with the dense, non-semantic way some fields are stored in nuget, such as dependency versions and tags. He uses these as examples that illustrate how RavenDB can handle nested collections of strings and complex objects.

That’s great and all, but if we’re redesigning the schema, we’ve already changed the rules. Nuget.org is having some trouble specifically because of a poorly designed schema that requires too many joins and some other complex hoop-jumping to execute some frequently used queries.

Furthermore, regardless of how the data is stored behind the server, NuGet uses an OData API to expose its packages to clients. If you change the schema, you have to do it in a way that remains compatible with that client API, or the millions of installed instances suddenly stop working.

Semantic Version Sort vs. Lexical Sort

One of the primary ways that packages are sorted in NuGet is by version. Unfortunately, the complex nature of semantic versions, such as 0.9-alpha or 1.0.5, makes it impossible to sort them correctly using a basic lexical sort. For example, 1.0-alpha should get sorted before 1.0, and 1.2 should get sorted before 1.10. With lexical sort they do not.

This problem was pointed out in a comment on Part IV but left unresolved.

Review

In using NuGet as an example for RavenDB, the following features are not addressed:

1. Providing a backwards-compatible OData endpoint
2. Sorting correctly by package version
3. Storing and retrieving package contents (only package metadata is addressed)
4. Adding new packages into the index
5. Keeping track of download counters (per-package and total)

Without addressing adding new packages and tracking downloads, the data store is effectively read-only, meaning that caches never get invalidated and there are no writes to disk. Based on this criteria, is it really fair to compare performance benchmarks between this sample and a real, production system?

The Load Tests

In the next part of the series, the sample system is put under some load to see what kind of performance RavenDB can provide when there are concurrent users. The Tests goes into some detail about the load testing plan.

Reviewing the load test, we have another obvious problem: only a handful of sample queries are being used. When the number of search queries is low, say, under 10, the system under load can simply cache the results from the previous time the query was executed and return that to the user. Since there isn’t any variance in which columns are used to sort, what page is retrieved, etc., the system under load doesn’t have to think very hard at all.

Combine the simplicity of the test plan with the fact that the system under test is not doing any writes at all, and you might as well be slinging static html at that point. I guess it does validate, if nothing else, that the caching built into RavenDB works.

One indicator that the load test isn’t really hitting any critical thresholds is that Pages/Sec grows pretty linearly as User Load ramps up. If the system hit such a threshold, one would expect Pages/Sec to peak at a certain rate and (ideally) sustain that rate as concurrency continues to grow.

Summary

To reiterate, I’m not trying to bash RavenDB, which I’m sure is an awesome product and probably really would perform very effectively under load similar to that experienced by nuget.org. However, casual observers may come away with unrealistic expectations after reading the performance results offered up on ayende.com.

Why Do I Care

If Ayende had picked virtually any example other than NuGet packages to do a blog series on, I probably would have read along and not thought much about any of it. But having worked on NuGet.Server + Lucene.Net.Linq for several months earlier this year, the subject matter is near and dear to my heart.

In future posts I’ll get into some more compare and contrast on RavenDB vs. Lucene.Net.Linq and maybe do a load test of my own of our custom NuGet.Server builds.

The Before Times

Octopus started a beta phase in late 2011, so what were we doing before? Since I started working in an asp.net shop in 2006, I’ve been surprised by the lack of robust deployment tools for the ecosystem.

The Web Deployment Projects system provided by Microsoft has always seemed like a non-starter for me. When you have mature processes like version control, continuous integration, automated testing, and quality assurance, how does it make sense to hand the keys over to a developer running Visual Studio to click “Deploy” on their desktop? What guarantee do you have that the code has been checked in, that it builds, that the tests pass, that QA signed off?

Even if you only give trusted users permission to deploy, I’ve never understood how it makes sense that you would need Visual Studio to deploy your projects. I mean, Visual Studio is for development. It isn’t even needed by a build server to compile your code. But now QA (or whoever is allowed to perform deployments) needs to have Visual Studio installed somewhere and checkout the code and build it themselves? So much for automation. So much for predictability. So much for repeatability.

So it was with a lack of viable alternatives that I started writing a deployment tool in 2007. The tool was named Bazooka (because it “shoots” software onto servers), and we used it for 5 years. We never released it as open source because it made several assumptions particular to us and we didn’t take the time to clean it up.

Bazooka was a web application that watched a directory for release candidates to show up as they were prepared by our build server. Each release candidate contained a deployment descriptor that had some meta data and a list of components that could be deployed to servers that matched certain roles like WebServer or AppServer.

Bazooka used xml configuration for everything. The deployment descriptor was xml, the server and environment configuration was xml, the permissions were xml, and the log files that recorded what had been deployed where were also xml. This made Bazooka pretty slow as deployment logs piled up, but it beat the heck out of doing anything more heavy-weight with sql.

Bazooka was, in reality, just a web interface to MSBuild. When you went through the deployment wizard, Bazooka simply executed MSBuild in the directory specified by each component with some properties saying which server to deploy to, which target to execute, what the environment was, etc.

We could have easily kept improving Bazooka, but we had actual code to write for our business, so Bazooka had some weaknesses that we never really addressed. For example, managing servers and environments was done by hand-editing the xml files. We didn’t build a UI for that. Also, it was pretty tough to answer simple questions like, “What was the most recent deployment of my project to the staging environment?”

But it worked as a fine stand-in for 5 years until someone built something better.

Octopus Trial

It was fun playing with Octopus during the beta period and seeing another approach to a web based deployment tool. In the beginning there were plenty of things Bazooka had that Octopus didn’t (yet), but development has been pretty rapid with Octopus delivering frequent releases.

The one time where we asked if using Octopus was right was when Paul Stovell, the creator/developer of Octopus Deploy, took a few months to rewrite the persistence layer, converting it from Entity Framework to RavenDB. It was the right decision to make and we’re happy in the long run, but we found ourselves in a lurch waiting for some important features and bugfixes. We came out fine on the other side though.

We also found that using a local file-based NuGet feed with Octopus doesn’t scale very well, and switching to NuGet Server provided no benefit either. This inspired us to create a custom fork of NuGet Server that uses Lucene.Net and Lucene.Net.Linq to provide a scalable, lightning fast internal feed for Octopus.

The Switch

As we integrated some pilot projects with Octopus, we slowly stopped using Bazooka and eventually turned off Bazooka integration. Some quick stats of our Octopus configuration today:

• 48 projects
• 18,080 release candidates
• 1,405 deployments

Octopus has done a decent job managing our high demands. We have experienced some slow page loads here and there, but Paul has been very responsive about troubleshooting and optimizing these.

Reusing Deployment Scripts

Since we already had a highly automated deployment system, we wanted to preserve our exising capabilities while finding ways to improve the system.

One problem that Octopus doesn’t solve for you is how to share deployment scripts across projects. By default, Octopus will execute deployment scripts contained in each project, but there isn’t a built-in or standard way to reuse common functionality.

One of the first projects we set up in Octopus we named OctopusScripts. This project consists of a collection of PowerShell modules that we want to be available everywhere. When we deploy the project, the deployment scripts install the modules into a standard location where PowerShell will probe for them. Then from other projects, we can simply start a script with:

1

Import-Module SmokeTest

Reuse Moar

Moving most of our scripts into PowerShell modules was working great, but we started to notice that our Deploy.ps1 scripts still looked awfully repetitious.

All of our web projects follow the same basic deployment recipe:

1. PreDeploy
   1. Disable machine in load balancer
   2. Create IIS site and app pool definitions if missing
   3. Update site host-header bindings as needed
2. Let Octopus update the document root to point to the new application
3. PostDeploy
   1. Execute smoke tests against the server and abort if any URLs return a non-200 response
   2. Enable machine in load balancer

Of course we have different configurations and different URLs to use for smoke tests for each project. We ended up creating a configuration based, modular script collection so that each project simply needs to include a stub:

1
2

Import-Module Fool-Octopus
Invoke-OctopusDeploymentTasks

The Invoke-OctopusDeploymentTasks function looks at whatever variable and configuration are present and figure out which steps to execute. The same scripts are used for Windows Service type projects and others too, and they figure out based on conventions if they need to run web steps, create services, etc.

If the stub is missing from a project (because why duplicate the stub?) our build scripts automatically insert a stub Deploy.ps1, PreDeploy.ps1 and PostDeploy.ps1.

We think we’re about as DRY as we can get with regards to our deployment scripts.

Ad-hoc PowerShell

One thing we’d like to see that hasn’t made it into Octopus yet is the concept of ad-hoc powershell scripts. Basically, we want to be able to run some arbitrary scripts during a deployment, only once. It doesn’t need to run once on each machine being deployed to, just once and then done. There’s a story card on Paul’s Trello board that we’re looking forward to. In the mean time, we’ve been emulating this behavior by deploying a small package to a dummy server and letting the script run there.

We mostly want this feature to simplify tasks such as sending email/other notifications when deployments are beginning or completed. It might also be useful for a green/blue style deployment model where the load balancer needs to toggle just once after the servers have been updated.

Without the Ad-hoc feature, one of the stumbling points we run into is sometimes forgetting to check the “Force redeployment” checkbox that Octopus leaves unchecked. When we forget, some steps get skipped leading to confusing results.

Looking Ahead

Because of the level of automation we integrated with Octopus, our business is able to deploy software more frequently and more reliably than ever before. In the coming months and years, we look forward to seeing improvements in Octopus features that will help us with cloud deployments to AWS or Azure. Octopus has definitely filled a gap in our deployment capabilities, allowing us to deliver value to our business quickly, iteratively and predictably.

When we started using this appraoch, we were still limited in a few ways:

• Everyone needs to integrate with the newest code
• Difficult to patch an old version of a dependency
• Cascading failures on the build server

Even though we broke the ProjectReference rats nest, we still had an implicit dependency on various shared code. It all had to be checked out and built in the right order.

Binary Package Management

The next logical step was to further decouple our shared code by packaging it up and publishing those packages. If we could do that, we could decide when to upgrade dependencies on a product by product basis.

There are two package managers in the .net ecosystem: OpenWrap and NuGet.

When we started shopping around, OpenWrap had been around longer and seemed to be a better choice. There’s a comparison of the products on stackoverflow.

We worked with OpenWrap for over 6 months and during that time started to find some problems around integration with Visual Studio and ReSharper. OpenWrap wants to manage dependencies per solution, and we have many cases where we want to control dependencies at a per project level. We also started to notice that NuGet was getting new versions released on a fairly regular schedule, while OpenWrap 2.0 was in unstable beta limbo for over a year.

Around the same time we started playing with Octopus Deploy for deploying our code. Since Octopus uses NuGet packages for deployment, we figured it would make sense to standardize on one package management system for both deployments and dependency management. It’s true that those are separate problem spaces, but having less build scripts is always a good thing.

Thoughts on NuGet

Conventions

NuGet has several conventions that make it easy to create simple packages that others can reference. You can share assemblies and content easily, and when you want to customize anything there are some powershell extension points you can hook into.

One problem we run into is that when building packages, sometimes there’s a NuGet convention we want to customize or suppress, and often we can’t.

For example, if you create a nuspec and place it adjacent to a csproj file, NuGet will look at the project and automatically inject metadata and content into the package. For some things, you can override this behavior with explicit specifications in the nuspec, but the behavior can be surprising and confusing.

Dependency Scoping

NuGet supports the concept of transitive dependencies… sort of. If you install package A, and A depends on package B, NuGet will go find a version of B and install it while installing A. However, NuGet doesn’t do any record keeping to remember that B is a transitive dependency. To your project, A and B appear simply as direct dependencies.

There may be cases where A depends on B at runtime, but consumers of A shouldn’t need to code against B at design time.

There may be other cases where B is an optional dependency for A, and A can be used without it.

Since NuGet doesn’t have a concept of scope, it only has one simplistic approach to dealing with transitive dependencies: treat them just like direct dependencies.

Upgrade Behavior

When you ask NuGet to update a specific package, it will first look for updates to transitive dependencies that the package depends on. This may seem obvious or desirable to some, but personally I find it confusing. You can control this behavior with the -IgnoreDependencies flag in the Package Management Console, but oddly you don’t get that option in the command line nuget.exe or from the Visual Studio GUI Package Manager.

Package Feed Performance

We use continuous integration, and every successful build produces “release candidate” versions of packages. We generate 50 to 100 packages a day.

Using the simple NuGet UNC share quickly failed to scale, so next we tried NuGet.Server and found that it doesn’t perform well either.

NuGet Gallery seemed like overkill with its SQL Server requirement, so I started optimizing NuGet.Server. This project ended up taking quite a while, but the good news is that the fruits of the labor are now open source on GitHub at https://github.com/themotleyfool/NuGet.

For more information about that project, see my previous post.

Refactoring Applications and Shared Code

We try to use Semantic Versioning to communicate breaking changes in the packages we publish, so sometimes when we want to use a refactoring tool like Change Method Signature or Use Base Class it would be nice to have application and shared code loaded into a single instance of Visual Studio.

We created a tool called SlimJim that generates these Solution files on the fly.

If you create a Solution with application code and shared library code, ReSharper will be smart enough to apply refactoring tools across the projects even though ProjectReference style references are not being used.

However, Visual Studio won’t know the correct order to build projects in, and won’t automatically copy outputs from shared libraries over to applications.

We extended SlimJim to convert assembly references to project references and back to address this limitation.

Conclusion

In terms of capability and maturity, we’re in a much better place than we were a few years ago. However, we still have a ways to go in terms of productivity and workflow.

NuGet has helped us move in the right direction and we hope to see further enhancements and even contribute some more of our own as we develop them.

In the beginning, there was one repository and it held all the projects for The Motley Fool, and it was good. There were around a dozen asp.net web projects, a smattering of service and console apps, and a bunch of class libraries to hold shared code. There was one Solution (sln) to rule them all.

As time went on, we found that there are downsides to the one-giant-solution approach to .net development:

• Big Ball of Mud
• Slow builds
• Tight coupling
• Configuration hell
• Hard to release different applications on different schedules

Typically our larger applications would be split into several projects following a typical N-tier layered architecture:

• Web
• Service
• Domain
• Data Access

Despite our attempts to encapsulate data access and domain logic behind the service project, code ended up leaking out to the point where domain projects were using types and methods from unrelated domain projects. Cats and dogs were sleeping together.

Around this time Steven Bohlen presented a talk to the Washington DC Alt.NET User Group titled “Domain Driven Design Implementation Patterns in .NET”. While some of us were already familiar with concepts of DDD, this talk lit a spark for us to try fixing our big ball of mud.

In late 2010 we started to make some changes. Instead of having one giant repository, shared code would be split out into separate repositories. We also took this opportunity to introduce a new project organization and architecture.

We established one repository to hold utility code, broken into specific class libraries:

• Fool.Abstractions - similar in spirit to System.Web.Abstractions; adds interfaces and wrappers to various FCL types that lack them
• Fool.Lang - similar in spirit to Jakarta Commons Lang; adds general utility classes and methods not found elsewhere
• Other projects that extend 3rd party class libraries to make them easier for us to work with in standardized ways.

Then we established another repository to hold Domain Driven, er, Domains. For example, many of our applications and web sites deal with stock market data, so one of our business domains is Quotes. In the Quotes Domain we have these projects:

• Fool.Quotes - contains service interfaces and value types; serves as an API to the domain
• Fool.Quotes.Core - contains domain logic, models, and entities; serves as a private implementation
• Fool.Quotes.Web.Api - exposes Fool.Quotes interfaces over a RESTful web API

The key to keeping our domains distinct and decoupled is to keep Core projects private. While Core is required at runtime, it should never be referenced at compile time. To bridge the gap, we use Dependency Injection to provide concrete implementations.

Domains may depend on other domains provided that they consume each other through the API project. That way entities and business logic are kept focused on their own concerns and don’t leak out to other problem areas where they don’t fit.

Gluing It Together

Having projects split into different repositories and different solutions meant that we couldn’t simply have one mega Solution that includes everything. That’s by design, so good on us. But this introduces a problem in that we still need to reference code from our utility projects and DDD projects in our applications. The first solution we came up with to handle this problem was to use the AssemblyFolders registry to have our libraries appear in the Add Reference dialog. Then to solve the runtime dependency on our private Core assemblies, we install those to the GAC so they can be loaded using reflection by our IoC container.

This approach worked fine, mostly. But we encountered some downsides after using it for a while:

• Need to have all library code checked out and built on each development machine
• No built-in way to manage different versions of the same dependency
• GAC considered harmful
• Hard to debug build errors and runtime errors

Using Continuous Integration means we’re producing new builds dozens of times a day, so it isn’t practical for us to manage different assembly versions for each build. Like most shops, we leave our assembly versions at 1.0.0.0 despite injecting actual version information into the AssemblyInformationalVersion attribute.

In order to support parallel development, we needed to find a more flexible way of managing dependencies, and at this point we started to look at binary package management.

One drawback to this migration is that we no longer have all projects in one Visual Studio solution, making it hard to work on writing a new feature or fixing a bug in a Framework Project and testing that change in an Application Project. Where we used to be able to do this seemlessly, we now have to jump through some hoops:

1. Change Framework Project
2. Build Framework Project
3. Manually copy the dll to the Application Project
4. Change Application Project to use new code (if necessary)
5. Build Application Project

Once the code changes are deemed acceptable, additional work is necessary to commit/push changes to other developers:

1. Commit/Push Framework Project
2. Wait for build server to publish a NuGet package
3. Use NuGet to update Framwork project dependency in Application Project
4. Commit/Push Application Project

This multistep process obviously slows down development significantly. My mission is to find a way to cut out some of these manual steps so our developers can get back to writing and refactoring code instead of wrestling with tools.

SlimJim

A colleague of mine, Aaron Togerson, started a project a while back called SlimJim. SlimJim is a Visual Studio Solution Generator. I’ve always liked the phrase “Solution Generator” despite the fact that Visual Studio solution files have little to do with solving actual problems.

Anyway, SlimJim is a command-line tool that analyzes c# projects and generates a .sln file that includes a target project (such as a Framework Project) and all projects that depend on it. This tool enables us to quickly generate Solutions that allow us to have Framework Projects and Application projects open in a single instance of Visual Studio. Since SlimJim generates these files pretty quickly, we never check them into source control.

Using SlimJim in conjunction with ReSharper we gain the ability to navigate through the code and apply refactoring tools to rename types and methods, change method signatures, introduce parameters, and all the other fun stuff.

The Pitfall

It’s great that ReSharper knows how to do this, but the problem is when you hit F6 or click Build Solution, and you get a bunch of compilation errors. What’s going on? First, NuGet injects assembly references into the Application Project instead of injecting project references. Visual Studio uses project references to determine the build order. So because there is no project reference from Application Project pointing to Framework Project, the order in which the projects are built will be undefined. The second problem is that even if the build order could be corrected, there is nothing in place that copies the build output from Framework Project over to where Application Project will find it.

We could solve both problems by converting the c# project files for Application Projects to replace <Reference/> items with <ProjectReference/> items. This is how Visual Studio decides what order projects are built in and how outputs from one project get copied to another.

The problem is that we don’t want to accidentally commit changes like this. We only want them to apply to local development. Accidentally checking such changes in will certainly break the build and take us back to the bad old days of slow builds.

Attempt 1

To avoid changing c# project files, the first approach I took was to try to do some MSBuild magic and hope Visual Studio would get in on the game. MSBuild provides some hooks that enable us to manipulate Item Groups dynamically. First, the InitialTargets attribute can be used to execute some targets before the requested targets are executed. Second, we can do some analysis of the solution file being built and the project references to try to remove <Reference/> items and add <ProjectReference/> items dynamically.

Note: this example uses tasks from the MSBuild Community Tasks project.

This implementation works great in MSBuild on the command line when invoked on a c# project. If you specify a SolutionPath and BuildingInsideVisualStudio, MSBuild will faithfully build the Framework Projects before building the Application Project.

However, it doesn’t work in Visual Studio. Evidently when Visual Studio loads a solution, it either doesn’t execute InitialTargets or it uses some other mechanism to extract <ProjectReference/> items from each project.

Attempt 2

If we can’t dynamically tell Visual Studio about project references, I thought maybe we could just build the projects in the right order ourselves. MSBuild already does this when building outside Visual Studio provided the BuildProjectReferences property is not false.

So I modified my MSBuild script to try to do that:

Once again, this works great in MSBuild outside of Visual Studio. When you load up the solution in VS and hit build, the Output pane says encouraging things like “MyApp is building C:\Projects\MyFramework\src\MyFramework\MyFramework.csproj”.

Except it doesn’t. I couldn’t figure out why, but somehow Visual Studio seems to know that we’re trying to trick it, and it refuses to actually build the project. It pretends to, but it doesn’t. I thought this might have to do with In-Process Compilers and other optimizations described in the Visual Studio MSBuild Integration article, but attempting to disable UseHostCompilerIfAvailable has no effect.

I gave up on this strategy becuase even if it helped to build projects in the right order, it doesn’t copy build outputs from one project to the other.

Attempt 3

Once again, Visual Studio proved to be too enterprisey for us to customize. The only proof of concept I was able to make work was the dumb one: converting c# projects and risking that the converted versions get checked in. We can solve the accidental commit issue with a pre-commit hook, so maybe it isn’t all bad as long as we can seemlessly convert back and forth.

Since we already have a tool that analyzes c# project references, it makes sense to extend SlimJim to mangle c# projects while analyzing them.

My fork of SlimJim now includes two new switches: --convert and --unconvert.

Suppose SlimJim finds two projects, MyApp and MyFramework, and MyApp.csproj looks like this:

1
2
3
4
5
6
7

<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup>
    <Reference Include="MyFramework, Version=1.0.0.0, Culture=neutral, PublicKeyToken=3c369c070579152a, processorArchitecture=MSIL">
      <HintPath>..\packages\MyFramework\lib\net40\MyFramework.dll</HintPath>
    </Reference>
  </ItemGroup>
</Project>

Using --convert, SlimJim will change MyApp.csproj to look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13

<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup>
    <ProjectReference Include="C:\projects\MyFramework\src\MyFramework\MyFramework.csproj">
      <Project>{673A97DA-4302-48C9-BFF0-A57FD7DBA93F}</Project>
      <Name>MyFramework</Name>
      <SlimJimReplacedReference>
        <Reference Include="MyFramework, Version=1.0.0.0, Culture=neutral, PublicKeyToken=3c369c070579152a, processorArchitecture=MSIL">
          <HintPath>..\packages\MyFramework\lib\net40\MyFramework.dll</HintPath>
        </Reference>
      </SlimJimReplacedReference>
    </ProjectReference>
  </ItemGroup>
</Project>

And using --uncovert will put it back. The original assembly reference is preserved in the markup, moved out of the way so MSBuild and Visual Studio will ignore it, but making it easy for us to restore it without losing metadata like HintPath and SpecificVersion.

As long as we remember to --unconvert before committing, this new feature should provide a great benefit.

Recent versions of Lucene.Net.Linq added support for the Unit of Work pattern.

The standard Unit of Work has methods like registerNew and registerDeleted, but I decided to use more generic names that make the interface appear more like a simple collection of documents.

1
2
3
4
5
6
7
8
9
10

public interface ISession<T> : IDisposable
{
  IQueryable<T> Query();
  void Add(params T[] items);
  void Delete(params T[] items);
  void Delete(params Query[] items);
  void DeleteAll();
  void Commit();
  void Rollback();
}

The Delete method that takes one or more Query objects is a case special to Lucene.Net. This is an escape hatch for when you want to delete one or more documents without first retrieving them. For example, you may want to delete all documents that match a query like type:person. I have mixed feelings about this escape hatch, because it smells like a Leaky Abstraction. On the other hand, since it is an overload to a more abstract method, I think it makes sense from a performance standpoint.

Anyway, one notable method pair is missing from the interface: registerDirty and registerClean. That’s because the session does some book keeping behind the scenes to automatically detect dirty documents when the session is committed.

This makes using Lucene.Net.Linq as a Repository dead simple, and makes the code look nice and clean in the client:

1
2
3
4
5
6
7
8
9
10

public void ChangeMailingAddress(string name, string newAddress)
{
    var session = provider.OpenSession<SampleDocument>();

    using (session)
    {
      var person = session.Query().Single(p => p.Name == name);
      person.MailingAddress = newAddress;
    }
}

Document Tracking

So how does it work?

When you ask for an IQueryable from session, the session attaches an instance of IRetrievedDocumentTracker to the queryable before returning it. That internal interface has only one method call:

1

void TrackDocument(T item, T hiddenCopy);

When the LuceneQueryExecutor starts returning results one item at a time, if it detects that a document tracker has been attached, it makes two copies of each result. One is returned to the client and the other is passed only to the tracker as a hidden copy. This gives the library the ability to compare pristine objects that came from the index unmodified with ones that may have been modified with the client.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

for (var i = skipResults; i < hits.ScoreDocs.Length; i++)
{
  var doc = hits.ScoreDocs[i].doc;
  var score = hits.ScoreDocs[i].score;
  
  var item = ConvertDocument(searcher.Doc(doc), score);

  if (tracker != null)
  {
      var copy = ConvertDocument(searcher.Doc(doc), score);
      tracker.TrackDocument(item, copy);
  }

  itemHolder.Current = item;
  yield return projector(itemHolder.Current);
}

Finally, when the session is committed, all tracked documents are compared with their hidden copies (using reflection) to detect which documents were modified. Modified documents are written back to the index.

Eventual Consistency

The concepts of eventual consitency describe what expectations a client may have about the ordering and visibility of writes in a distributed asynchronous persistence system. Werner Vogels has written about different consistency definitions.

Since the unit of work in Lucene.Net.Linq is implemented by an interface named ISession, it would have been nice to provide Session Consistency. It’s open to some interpretation if this behavior is achieved or not. Since the underlying Lucene engine will not make changes visible to an IndexReader until a commit happens and the IndexReader is reopened, it is difficult or impossible to allow a client to see the effects of adding, deleting or modifying a document in subsequent queries within the same session. However, this problem can be side-stepped by simply calling Commit between making changes and subsequent queries. Calling Commit explicitly may have drawbacks if the client will make further changes and wishes for those changes to appear atomically. Knowing the details of when staged changes become visible will help clients to make changes smartly.