Code rant

New Blog At mikehadlow.com

noreply@blogger.com (Mike Hadlow) — 2021年7月06日 11:41:00 +0000

This is my last post here at Code Rant. From now I will be posting at mikehadlow.com. I've written a post on my new blog here explaining the reasons. Thanks for visiting Code Rant and please take a moment to look at my new blog.

C# preprocessor directive symbols from the dotnet build command line via DefineConstants

noreply@blogger.com (Mike Hadlow) — 2020年8月31日 14:36:00 +0000

Invoking the C# compiler directly allows one to pass in symbols for the preprocessor via a command option (-define or -d). But it's not at all obvious how to do this with the dotnet build command. There is no 'define' flag, so how do you do it?

Let me first show you how this works using the C# compiler directly:

Create a new file 'Program.cs' with this code:

using System;
namespace CscTest
{
 class Program
 {
 static void Main(string[] args)
 {
#if FOO
 Console.WriteLine("Hello FOO!");
#else
 Console.WriteLine("NOT FOO!");
#endif
 }
 }
}

Now compile it with CSC:

>csc -d:FOO Program.cs

And run it:

>Program
Hello FOO!

Happy days.

It is possible to do the same thing with dotnet build, it relies on populating the MSBuild DefineConstants property, but unfortunately one is not allowed to access this directly from the command line:

If you invoke this command:

dotnet build -v:diag -p:DefineConstants=FOO myproj.csproj

It has no effect, and somewhere deep in the diagnostic output you will find this line:

The "DefineConstants" property is a global property, and cannot be modified.

Instead one has to employ a little indirection. In your csproj file it is possible to populate DefineConstants. Create a project file, say 'CscTest.csproj', with a DefineConstants PropertyGroup element with the value FOO:

<Project Sdk="Microsoft.NET.Sdk">
 <PropertyGroup>
 <OutputType>Exe</OutputType>
 <TargetFramework>netcoreapp3.1</TargetFramework>
 <DefineConstants>FOO</DefineConstants>
 </PropertyGroup>
</Project>

Build and run it with dotnet run:

>dotnet run .
Hello FOO!

The csproj file is somewhat like a template, one can pass in arbitrary properties using the -p flag, so we can replace our hard coded FOO in DefineConstants with a property placeholder:

<Project Sdk="Microsoft.NET.Sdk">
 <PropertyGroup>
 <OutputType>Exe</OutputType>
 <TargetFramework>netcoreapp3.1</TargetFramework>
 <DefineConstants>$(MyOption)</DefineConstants>
 </PropertyGroup>
</Project>

And pass in FOO (or not) on the command line. Unfortunately it now means building and running as two individual steps:

>dotnet build -p:MyOption=FOO .
...
>dotnet run --no-build
Hello FOO!

And all is well with the world. It would be nice if the MSBuild team allowed preprocessor symbols to be added directly from the command line though.

Restoring from an Azure Artifacts NuGet feed from inside a Docker Build

noreply@blogger.com (Mike Hadlow) — 2020年8月04日 11:21:00 +0000

If you are using Azure DevOps pipelines to automate building your .NET Core application Docker images, it's natural to also want to use the DevOps Artifacts NuGet feed for your internally hosted NuGet packages. Unfortunately there is much confusion and misinformation about how to authenticate against the Artifacts NuGet feed. While researching this topic I found various sources saying that you needed to install the NuGet credential provider as part of the docker build, and then set various environment variables. I followed this route (excerpt from an example below), even to the extent of creating a custom Docker image for all our dotnet builds with the credential provider already installed.

ARG PAT
RUN wget -qO- https://raw.githubusercontent.com/Microsoft/artifacts-credprovider/master/helpers/installcredprovider.sh | bash
ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true
ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS "{\"endpointCredentials\": [{\"endpoint\":\"https://pkgs.dev.azure.com/jakob/_packaging/DockerBuilds/nuget/v3/index.json\", \"password\":\"${PAT}\"}]}"

The technique is to install the credential provider, then configure it with the DevOps Artifacts endpoint and a Personal Access Token (PAT), which you can generate by going to your user settings from the DevOps UI:

After wasting over a day on this, I was then very surprised indeed to find that a colleague was restoring from the same DevOps Artifacts feed on a locally hosted TeamCity server, simply by providing the PAT as the NuGet API-Key! They hadn't installed the NuGet credential provider, so according to the Microsoft documentation it shouldn't work?

I tried it myself. The PAT does indeed work as a NuGet API-Key. A slight further complication is that the 'dotnet restore' command doesn't have an API-Key switch, so the next easiest thing is to simply use a nuget.config file as follows:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
 <packageSources>
 <add key="DevOpsArtifactsFeed" value="your-devops-artifacts-nuget-source-URL" />
 </packageSources>
 <packageSourceCredentials>
 <DevOpsArtifactsFeed>
 <add key="Username" value="foo" />
 <add key="ClearTextPassword" value="your-PAT" />
 </DevOpsArtifactsFeed>
 </packageSourceCredentials>
</configuration>

Replace the place-holders with your Artifacts NuGet feed URL and your PAT. The Username is not considered by Artifacts feed and can be any string. Copy the above configuration into a file named 'nuget.config' and run create a Dockerfile like this:

FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build
WORKDIR /app
# copy source code, nuget.config file should be placed in the 'src' directory for this to work.
COPY src/ .
# restore nuget packages
RUN dotnet restore --configfile nuget.config
# build
RUN dotnet build
# publish
RUN dotnet publish -o output
# build runtime image
FROM mcr.microsoft.com/dotnet/core/runtime:3.1 AS runtime
WORKDIR /app
COPY --from=build /app/output/ ./
# 
ENTRYPOINT ["your/entry/point"]

This is the simplest thing that will possibly work. But you really shouldn't hard code secrets such as your PAT into your source control system. Very conveniently, the dotnet restore command will do environment variable replacement in the nuget.config file, so you can replace your hard-coded PAT with a reference to an ENV var and then pass that to docker build: In your nuget.config file:

 <packageSourceCredentials>
 <DevOpsArtifactsFeed>
 <add key="Username" value="foo" />
 <add key="ClearTextPassword" value="%NUGET_PAT%" />
 </DevOpsArtifactsFeed>
 </packageSourceCredentials>

In your Dockerfile:

ARG NUGET_PAT
ENV NUGET_PAT=$NUGET_PAT

Your docker build command:

docker build -t my-image --build-arg NUGET_PAT="your PAT" .

I hope this short post saves somebody from the many hours that I wasted on this. I also hope that Microsoft updates their documentation!

A Framework to DotNet Core Conversion Report

noreply@blogger.com (Mike Hadlow) — 2020年4月15日 16:04:00 +0000

An experience report of converting a large microservice platform from .NET Framework to dotnet core.

Background

For the last year or so I’ve been working with company that maintains a significant trading platform built in .NET. The architecture consists of a number of Windows Service components that communicate using RabbitMQ with EasyNetQ. These are all backend components that at the top level communicate with various clients via a web API maintained by a different team. The infrastructure is hosted in the company’s own data center with a CI/CD software process featuring BitBucket, Team City, and Octopus, a pretty standard .NET delivery pipeline.

Motivation

Our motivation for porting to dotnet core was essentially twofold: to keep the technology platform up to date, and to be in a position to exploit new developments in application platforms, specifically to take advantage of container technology, such as Docker, and container orchestrators, such as Kubernetes.

Microsoft is, on the whole, very good at supporting their technology for the long term; there are many companies with VB6 applications still running, for example, and the .NET Framework will undoubtedly be supported on Windows for years to come. However there are significant costs and risks in supporting legacy software platforms, such as: difficulty in using newer technologies and protocols because libraries aren’t available for the legacy platform; difficulty hiring and retaining technology staff who will fear that their skills are not keeping up to date with the market; and the increasing cost over time of porting to a newer platform as year on year the gap with the legacy technology widens. There is a danger that at some point in the future the legacy platform will become unsupportable, but the technology gap is so wide that the only feasable solution is a very expensive re-write.

Software infrastructure has experienced a revolution in the last few years. I’ve written before why I think that containerization is a game changer, especially for distributed microservice architectures such as ours. It has the potential to significantly reduce risks and costs and increase flexibility. For all Microsoft’s efforts, Windows containers are still a platform that one should use with caution; all the maturity is with Linux containers. We are very keen to exploit the opportunities of Docker and Kubernetes, and so this has a prerequisite that our software can run on Linux. It provides the second strong incentive for our move to dotnet core.

Process

Analysis

A dotnet core application can only consume dotnet core or dotnet standard dependencies, so the first task is to understand the dependency tree; what are the projects, NuGet packages, and system assemblies that the application relies upon, and which assemblies do these rely on in turn. Once we have that picture, we can work from the leafs down to the trunk; from the top-level dependencies down to the application itself. For third party NuGet packages we have to make sure that a dotnet standard version is available. For libraries internal to the organisation, we have to add each one to our list of projects that we will need to convert to dotnet standard.

I used my own tool: AsmSpy to help with this. It was originally designed to report on assembly version conflicts, but since it already built an internal dependency graph, it was a relatively simple extension to add a visualizer to output the graph as a tree view. To do this, simply add the -tr option:

asmspy.exe <path to application executable> -tr

At the end of the analysis process, we should have a list of NuGet packages to be checked for dotnet standard versions, our internal libraries that need to be converted to dotnet core, and our applications/services that need to be converted to dotnet core. We didn’t have any problems with base class libraries or frameworks because our services are all console executables that communicate via EasyNetQ, so the BCL footprint was very light. Of course you will have a different experience if your application uses something like WCF.

Converting Projects to dotnet Standard and Core

Some early experiments we tried with converting .NET Frameworks to dotnet Standard or Core in place, by modifying the .csproj files, did not go well, so we soon settled on the practice of creating entirely new solutions and projects and simply copying the .cs files across. For this Git Worktree is your very good friend. Worktree allows you to create a new branch with a new working tree in a separate directory, so you can maintain both your main branch (master for example), and your conversion branch side by side. The project conversion process looks something like this:

Create a new branch in a new worktree with the worktree command: git worktree add -b core-conversion <path to new working directory>
In the new branch open the solution in Visual Studio and remove all the projects.
Delete all the project files using explorer or the command line.
Create new projects, copying the names of the old projects, but using the dotnet Standard project type for libraries, ‘Class Library (.NET Standard)’, and the dotnet Core project type for services and applications. In our case all the services were created as ‘Console App (.NET Core)’. For unit tests we used ‘xUnit Test Project (.NET Core)’, or ‘MSTest Test Project (.NET Core)’, depending on the source project test framework.
From our analysis (above), add the project references and NuGet packages required by each project.
Copy the .cs files only from the old projects to the new projects. An interesting little issue we found was that old .cs files were still in the repository despite being removed from their projects. .NET Framework projects enumerate each file by name (the source of many a problematic merge conflict) but Core and Standard projects simply use a wildcard to include every .cs file in the project directory, so a compile would include these previously deleted files and cause build problems. Easily fixed by deleting the rogue files.
Once all this is done the solution should build and the tests should all pass.
NuGet package information is now maintained in the project file itself, so for your libraries you will need to copy that from your old .nuspec files.
One you are happy that the application is working as expected, merge your changes back into your main branch.

You have now successfully converted your projects from .NET Framework to dotnet core and standard. Read on if you want to take advantages of the new dotnet Core frameworks available, and for ideas about build and deployment pipelines.

Taking advantage of new dotnet core frameworks

At this point we need to make a strategic decision about how far we want to take advantage of the new hosting, dependency-injection, configuration, and logging frameworks that now come out-of-the-box with dotnet core. We may decide that we will simply use standard versions of all our existing frameworks. In our case we had: TopShelf for windows service hosting, Ninject for DI, System.Configuration for configuration, and log4net and NLog for logging, but we decided to replace all these with their Generic Host equivalents from the Microsoft.Extensions.* namespaces.

Framework NuGet Package	`Microsoft.Extensions.*` equivalent
TopShelf	Microsoft.Extensions.Hosting.WindowsServices
Ninject	Microsoft.Extensions.DependencyInjection
System.Configuration	Microsoft.Extensions.Configuration
log4net	Microsoft.Extensions.Logging

The APIs of the existing 3rd party frameworks differ from the equivalent Microsoft.Extensions.* frameworks, so some refactoring is required to replace these. In the case of TopShelf and Ninject, the scope of this refactoring is limited; largely to the Program.cs file and the main service class for TopShelf, and to the NinjectModules where service registration occurs for Ninject. This makes it relatively painless to do the substitution. With Ninject, the main issue is the limited feature set of Microsoft.Extensions.DependencyInjection. If you make widespread use of advanced container features, you’ll find yourself writing a lot of new code to make the same patterns work. Most of our registrations were pretty straightforward to convert.

Replacing log4net with Microsoft.Extensions.Logging is a bit more of a challenge since references to log4net, especially the ILog class and its methods, were spread liberally throughout our codebase. Here we found that the best refactoring method was to let the type system do the heavy lifting, using the following steps:

Uninstall the log4net NuGet package. The build will fail with many missing class and method exceptions.
Create a new interface named ILog with namespace log4net, now the build will fail with just missing method exceptions.
Add methods to your Ilog interface to match the missing log4net methods (for example void Info(object message);) until you get a clean build.
Now use Visual Studio’s rename symbol refactoring to change your ILog interface to match the Microsoft.Extensions.Logging ILogger interface and its methods to match ILogger's methods. For example rename void Info(object message); to void LogInformation(string message);.
Rename the namespace from log4net to Microsoft.Extensions.Logging. This is a two step process because you can’t use rename symbol to turn one symbol into three, so rename log4net to some unique string, then use find and replace to change it to Microsoft.Extensions.Logging.
Finally delete your interface .cs file, and assuming you’ve already added the Microsoft.Extensions.Hosting NuGet package and its dependencies (which include logging), everything should build and work as expected.

Configuration is another challenge. Gone are our old friends App.config and System.Configuration.ConfigurationManager to be replaced with a new configuration framework, Microsoft.Extensions.Configuration. This is far more flexible and can load configuration from various sources, including JSON files, environment variables, and command line arguments. We replaced our App.config files with appsettings.json, and refactored our attributed configuration classes into POCOs and used the IConfigurationSection.Bind<T>(..) method to load the config. An easier and more streamlined process than the clunky early 2000’s era System.Configuration. At a later date we will probably move to loading environment specific configuration from environment variables to better align with the Docker/k8s way of doing things.

Changes to our build and deployment pipeline

As I mentioned above, we use a very common combination of BitBucket, Team City, and Octopus to host our build and deployment pipeline. We follow a continuous delivery style deployment process. Any commit to a BitBucket Git repository immediately triggers a build, test and package process in Team City, which in turn triggers Octopus to deploy the package to our development environment. We then have to manually use the Octopus UI to release to first our QA environment and then to Production. Although our ultimate aim, and a prime motivation for the transition to Core, is to move to Docker and Kubernetes, we needed to be able to build and deploy using our existing tooling for the time being. This proved to be pretty straightforward. The changes were in three main areas:

Using the dotnet tool: The build and test process changed from using NuGet, MSBuild and xUnit, to having every step, except the Octopus trigger, run with the dotnet tool. This simplifies the process. One very convenient change is how easy it is to version the build with the command line switch /p:Version=%build.number%. We also took advantage of the self-contained feature to free us from having to ensure that each deployment target had the correct version of Core installed. This is a great advantage.
JSON configuration variables: We previously used the Octopus variable substitution feature to inject environment specific values into our App.config files. This involved annotating the config file with Octopus substitution variables, a rather fiddly and error prone process. But now with the new appsettings.json file we can use the convenient JSON configuration variable feature to do the replacement, with no need for any Octopus specific annotation in our config file.
Windows service installation and startup: Previously, with TopShelf, installing our windows services on target machines was a simple case of calling ourservice.exe install and ourservice.exe start to start it. Although the Microsoft.Extensions.Hosting framework provides hooks into the Windows service start and stop events, it doesn’t provide any facilities to install or start the service itself, so we had to write somewhat complex powershell scripts to invoke SC.exe to do the installation and the powershell Start-Service command to start. This is definitely a step backward.

Observations

The conversion of our entire suite of services from .NET Framework to Core turned out to be a bigger job than we at first expected. This was mainly because we took the opportunity to update our libraries and services to replace our 3rd party NuGet packages with the new Microsoft.Extensions.* frameworks. This was a significant refactoring effort. Doing a thorough analysis of your project and its dependencies before embarking on the conversion is essential. With large scale distributed applications such as ours, it’s often surprising how deep the organisations internal dependency graph goes, especially if, like me, you are converting large codebases which you didn’t have any input into writing. With the actual project conversion I would highly recommend starting with new projects rather than trying to convert them in place. This turned out to be a far more reliable method.

DotNet Core is a complete ground up reinvention of the .NET tooling and frameworks, and the 20 year difference shows in many places. The tooling is modern, as are the frameworks, and although there’s plenty to argue about with the individual decisions the team have made, on the whole it’s a large step forward. This was apparent in many ways during the conversion process, with many things be simpler and easier than with the old .NET Framework. Having the entire SDK surficed through a single command line tool (the dotnet command), making automated build processes so much easier, is probably the most prominent example. I for one am very pleased we were able to take the effort to make the change.

Why Containers are a Game Changer for Software Development

noreply@blogger.com (Mike Hadlow) — 2019年1月29日 14:47:00 +0000

I originally wrote this piece as of part of a paper evaluating container technology for a client.

This document describes container technology, best represented by Docker. Containerization is a game changing technology that’s experiencing rapid adoption. Some measures have around 25% of companies now using Docker in some form (https://www.datadoghq.com/docker-adoption/). Containers can dramatically simplify the software development process, allowing companies to be more agile and lower the cost of building and maintaining large software systems. This document looks at how containers fit within the general evolution of software systems.

A brief history of software

The history of software development is a story of successive rounds of abstraction and commodification. If you can treat a class of something (a computer, a network or a peripheral) as a black box with a consistent API, it enables common industry wide tooling and commodification.

In the early years of computing software was written for a particular version of hardware. Each program would would take complete control of the machine, use the processor’s physical instruction set, directly address physical memory and have intimate knowledge of the locations and capabilities of any devices attached to the machine. This meant that a program written for one model of machine would not work on a different model. Machines were typically sold with a dedicated software suite, which meant that the same classes of software had to be written repeatedly for each machine. In the early days of home computers it was typical for a word processor, for example, to come in different versions for all the major machines on the market and with drivers for a range of popular printers. If your printer wasn’t included it wouldn’t work.

To solve this problem and allow a single program to run on a variety of machines, operating systems were created to provide an abstraction layer over the underlying hardware. So long as a piece of software was designed to run on the operating system of your computer, it worked. The operating system also isolated the program from variations in peripheral hardware. You no longer had to care about what particular printer was attached to the computer because an operating system driver provided a common abstract printer API regardless of the actual hardware model. As operating systems evolved they provided not only isolation from the hardware, but also isolation from other programs running on the same computer with innovations such as protected memory and pre-emptive multitasking. With the adoption of an operating system as a common platform, the thing it abstracted, the hardware, became a commodity. This lead to dramatic cost reductions and economies of scale, both for hardware and software.

The same adoption and standardisation also occurred with networking. TCP/IP became the standard which allowed computer systems to be connected world wide and HTTP has become a standard for sharing data globally. This has allowed software solutions to serve customers at a massive scale.

As software that runs on commodified platforms became more complex, various mechanisms evolved to make software more modular and reusable. Collections of modular software ‘libraries’ could be brought together to create more powerful applications in less time. Software environments also evolved to include runtimes to relieve programmers from the need to manage memory and to further abstract the program from its environment. Software systems also evolved to be composed of multiple processes running on multiple machines to better aid scalability and resilience. Various services and infrastructure tools such as web servers and databases provided off-the-shelf capabilities to further aid software development.

The complexity of the modern software environment

All these libraries, services and infrastructure have to be correctly configured for the software to run. This is often a semi-manual, complex, time consuming and error prone task. When multiple pieces of software run on a single machine there can often be complex and damaging interactions between conflicting library and tool versions. The complexity of provisioning environments, installing tools and libraries of the correct version, opening the correct ports and configuring connections, especially when this is done in different environments with differing network topologies, a fertile environment for human error.

Once in production, these complex systems need to be monitored, managed and audited. This introduces additional tooling and configuration, adding yet another vector for misconfiguration and error.

Also the difficulty of coordinating teams of software developers who create complex software systems requires the formalisation and automation of the software development process. This introduces new tools, such as build and deployment systems that must also be configured correctly for the software to be successfully delivered into production. This configuration work is also often manual, fragile and error prone, and since a single toolset is often shared by many teams and components, it creates significant friction when introducing new services, libraries or tools.

Because the delivery and runtime environments are maintained and versioned separately from source code, this introduces risk and friction. Services often share both environments and delivery processes, meaning that upgrades and changes have to be coordinated. In a worse-case scenario separate teams may be tasked with maintaining the runtime infrastructure and the delivery process, escalating any change to a large scale organisational issue. Often the overwhelming task of synchronising software and environment upgrades means that they are done infrequently and with a great deal of ceremony and risk.

Virtual machines don’t really help here. They can make the work of technical operations easier; they decouple an entire operating system environment from hardware and make it easy to replicate and move environments around hardware infrastructure. However, VMs make very little difference to software developers. The software pipeline and runtime environment is still maintained and configured separately from the the software source code itself.

The stage has been set for another round of abstraction, this time the abstraction is the interface between the operating system, the userland environment and the network topology that the software is built and runs within.

Containerisation is the technology that provides this abstraction and solves many of the problems described above. Containers provide a scripted per-process runtime user environment that is maintained alongside the source code. The software build process and target network topology of a large software system is also defined in container and composition/orchestration scripts. Because the scripts are maintained by developers on a per-process (per service) basis and are maintained under source control alongside the service’s source code, the software describes the environment that builds it and that it runs in, and this description is versioned with the software. Effectively it reverses the usual hierarchy and allows each component to own it’s environment and delivery process. The environment for a component is identical regardless of whether it’s running on the developer’s machine, in a test environment or in production and removes much of the risk of configuring the software pipeline and runtime environment described above. This idea of extending Git workflow to operations is known as GitOps. (see https://www.weave.works/blog/what-is-gitops-really). In the same way that operating systems removed the need for software to care about hardware, so containers allow the software environment to be described without having to know or care about the specific operating system environment and the physical network.

Conclusion

Docker and its various orchestration options offer game changing performance increases for large software organisations. It provides a single, integrated, scripted, scalable platform for both the software delivery pipeline and production operations. It’s experiencing fast adoption and will soon be as standard a part of IT infrastructure as VMs are currently. Any software organisation of reasonable scale should now be seriously looking at a path for adoption.

Decoupling, Architecture and Teams

noreply@blogger.com (Mike Hadlow) — 2018年11月06日 15:54:00 +0000

This article discusses the relationship in software development between code organisation and social organisation. I discuss why software and teams do not scale easily, lessons we can learn from biology and the internet, and show how we can decouple software and teams to overcome scaling problems.

The discussion is based on my 20 years experience of building large software systems, but I’ve also been very impressed with the book Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations by Nicole Forsgren, Jez Humble and Gene Kim, which provides research data to back up most of the assertions that I make here. It’s a highly recommended read.

Software and software teams do not scale.

It’s a very common story, the first release of a product, perhaps written by one or two people, often seems remarkably easy. It might only provide limited functionality, but it is written quickly and fulfils the customer’s requirements. Customer communication is great because the customer is usually in direct communication with the developers. Any defects are quickly fixed and new features can be added quite painlessly. After a while though the pace slows. Version 2.0 takes a little longer than expected, it’s harder to fix bugs and new features don’t seem to come out quite so easily. The natural response to this is to add new developers to the team, yet each extra person added to the team seems to reduce productivity. As the software ages and grows in complexity it appears to atrophy. In extreme cases, organizations can find themselves running on software that’s hugely expensive to maintain and that it seems almost impossible to change. There are negative scale effects. The problem is that you don’t have to make ‘mistakes’ for this to happen, it’s so common that one could almost say that it’s a ‘natural’ property of software.

Why is this? There are two reasons, code related and team related. Neither code nor teams scale well.

As a codebase grows, it becomes harder for a single person to understand. There are fixed human cognitive limits and while it is possible for a single individual to maintain a mental model of the details of a small system, once it gets past a certain size, it grows larger than the cognitive range of a single person. Once a team grows past five or more people it’s almost impossible for one person to stay up to speed with how all parts of the system work. When no one person understands the complete system, fear reigns. In a tightly coupled large system it’s very hard to know the impact of any significant change since the consequences are not localised. Developers learn to work in a minimum-impact style of work-arounds and duplication rather than factoring out commonalities and creating abstractions and generalisations. This feeds back into system complexity, further amplifying these negative trends. Developers stop feeling any ownership of code they don’t really understand and are reluctant to refactor. Technical debt increases. It also makes for unpleasant and unsatisfying work and encourages ‘talent evaporation’, where the best developers, those who can more easily find work elsewhere, move on.

Teams also don’t scale. As the team grows, communication gets harder. The simple network formula comes into play:

c = n(n-1)/2
(where n is the number of people and c is the number of communication channels)

Number of team members	Number of communication channels
1	0
2	1
5	10
10	45
100	4950

The communication and coordination needs of the team rise geometrically as the team size increases. It’s very hard for a single team to stay a coherent entity over a certain size and the natural human social tendency to split into smaller groups will lead to informal sub-groups forming even if there is no management input. Peer level communication becomes difficult and will naturally be replaced by emergent leaders and top-down communication. Team members change from being equal stakeholders in the system to directed workers. Motivation suffers and there is a lack ownership driven by the diffusion of responsibility effect.

Management often intervenes at this stage and formalises the creation of new teams and management structures to organize them. But whether formal or informal, larger organisations find it hard to keep people motivated and actively engaged.

It’s typical to blame poorly skilled developers and bad management for these scaling pathologies, but that’s unfair, scale issues are a ‘natural’ property of growing and aging software, it’s what always happens unless you spot the problem early, recognise the inflexion point and work very hard to mitigate it. Software teams are constantly being created, the amount of software in the world is constantly growing and most software is small scale, so it’s quite common for a successful and growing product to have been created by a team that has no experience of large-scale software development. Expecting them recognise the inflexion point when the scale issues start to bite and to know what to do about it is unrealistic.

Scaling lessons from nature

I recently read Geoffrey West’s excellent book Scale. He talks about the mathematics of scale in biological and social-economic systems. His thesis is that all large complex systems obey fundamental scaling laws. It’s a fascinating read and very much recommended. For the purposes of this discussion I want to focus on his point that many biological and social systems scale amazingly well. Take the basic mammal body plan. We share the same cell types, bone structure, nervous and circulatory system of all mammals. Yet the difference in size between a mouse and a blue whale is around 10^7. How does nature use the same fundamental materials and plans for organisms of such hugely different scales? The answer appears to be that evolution has discovered fractal branching networks. This can be seen quite obviously if you consider a tree. Each small part of the tree looks like a small tree. The same is true for our mammalian circulatory and nervous systems, they are branching fractal networks where a small part of your lungs or blood vessels looks like a scaled down version of the whole.

Can we take these ideas from nature and apply them to software? I think there are important lessons that we can learn. If we can build large systems which have smaller pieces that look like complete systems themselves, then it might be possible to contain the pathologies that affect most software as it grows and ages.

Are there existing software systems that scale many orders of magnitude successfully? The obvious answer is the internet, a global software system of millions of nodes. Subnets do indeed look and work like smaller versions of the whole internet.

Attributes of decoupled software.

The ability to decouple software components from the larger system is the core technique for successful scaling. The internet is fundamentally a decoupled software architecture. This means that each node, service or application on the network has the following properties:

Obeys a shared communication protocol.
Only shares state via a clear contract with other nodes.
Does not require implementation knowledge to communicate.
Versioned and deployed independently.

The internet scales because it is a network of nodes that communicate over a set of clearly defined protocols. The nodes only share their state via the protocol, the implementation details of one node do not need to be understood by the nodes communicating with it. The global internet is not deployed as a single system, each node is separately versioned and deployed. Individual nodes come and go independently of each other. Obeying the internet protocol is the only thing that really matters for the system as a whole. Who built each node, when is created or deleted, how it’s versioned, what particular technologies and platforms it uses are all irrelevant to the internet as a whole. This is what we mean by decoupled software.

Attributes of decoupled teams.

We can scale teams by following the similar principles:

Each sub-team should look like a complete small software organization.
The internal processes and communication of the team should not be a concern outside the team.
How the team implements software should not be important outside the team.
Teams should communicate with the wider organisation about external concerns: common protocols, features, service levels and resourcing.

Small software teams are more efficient than large ones, so we should break large teams into smaller groups. The lesson from nature and the internet is that the sub-teams should look like a single, small software organisations. How small? Ideally one to five individuals.

The point that each team should look like a small independent software organisation is important. Other ways of structuring teams are less effective. It’s often tempting to split up a large team by function, so we have a team of architects, a team of developers, a team of DBAs, a team of testers, a deployment team and an operations team, but this solves none of the scaling problems we talked about above. A single feature needs to be touched by every team, often in an iterative fashion if you want to avoid waterfall style project management - which you do. The communication boundaries between these functional teams become a major obstacle to effective and timely delivery. The teams are not decoupled because they need to share significant internal details in order to work together. Also the interests of the different teams are not aligned: The development team is usually rewarded for feature delivery, the test team for quality, the support team for stability. These different interests can lead to conflict and poor delivery. Why should the development team care about logging if they never have to read the logs? Why should the test team care about delivery when they are rewarded for quality?

Instead we should organise teams by decoupled software services that support a business function, or a logical group of features. Each sub-team should design, code, test, deploy and support their own software. The individual team members are far more likely to be generalists than specialists because a small team will need to share these roles. They should focus on automating as much of the process as possible: automated tests, deployment and monitoring. Teams should choose their own tools and decide for themselves how to architect their systems. While the organizational protocols that the system uses to communicate must be decided at an organization level, the choice of tools used to implement the services should be delegated to the teams. This very much aligns with a DevOps model of software organization.

The level of autonomy that a team has is a reflection of the level of decoupling from the wider organization. Ideally the organization should care about the features, and ultimately business value, that the team provides, and the cost of resourcing the team.

The role of the software architect is important in this style of organisation. They should not focus on the specific tools and techniques that teams use, or micro-manage the internal architecture of the services, instead they should concentrate on the protocols and interactions between the various services and the health of the system as a whole.

Inverse Conway: software organisation should model the target architecture.

How do decoupled software and decoupled teams align? Conway’s Law states that:

"organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations."

This is based on the observation that the architecture of a software system will reflect the team structure of the organization that creates it. We can ‘hack’ Conway’s law by inverting it; organize our teams to reflect our desired architecture. With this in mind we should align our decoupled teams with our decoupled software components. Should this be a one-to-one relationship? I think this is ideal, although it seems that it’s fine for a single small team to deliver several decoupled software services. I would argue that the scaling inflexion point for teams is larger than that for software, so this style of organisation seems valid. However, it’s important that the software components should remain segregated with their own version and deployment story even if some share the same team. We would like to be able to split the team if it grows too large, and being able to hand off various services to different teams would be a major benefit. We can’t do that if the services are tightly coupled or share process, versioning or deployment.

We should avoid having multiple teams work on the same components, this is an anti-pattern and is in some ways worse than having a single large team working on an oversize single codebase because the communication barriers between the teams leads to even worse feelings of lack-of-ownership and control.

The communication requirements between decoupled teams building decoupled software are minimised. Taking the example of the internet again, it’s often possible to use an API provided by another company without any direct communication if the process is simple and documentation sufficient. The communication should not require any discussion of software process or implementation, that is internal to the team, instead communication should be about delivering features, service levels, and resourcing.

An organisation of decoupled software teams building decoupled software should be easier to manage than the alternatives. The larger organization should focus on giving the teams clear goals and requirements in terms of features and service levels. The resource requirements should come from the team, but can be used by the organization to measure return on investment.

Decoupled Teams Building Decoupled Software

Decoupling software and teams is key to building a high performance software organisation. My anecdotal experience supports this view. I’ve worked in organisations where teams were segregated by software function or software layer or even where they’ve been segregated by customer. I’ve also worked in chaotic large teams on a single codebase. All of these suffer from the scaling problems discussed above. The happiest experiences were always where my team was a complete software unit independently building, testing and deploying decoupled services. But you don’t have to rely on my anecdotal evidence, the book Accelerate (described above), has survey data to support this view.

Visual Programming - Why it’s a Bad Idea

noreply@blogger.com (Mike Hadlow) — 2018年10月01日 13:56:00 +0000

Note. This post had a great response on Reddit with over 300 comments. I’ve added an update section to the end of this post to address some of the main criticisms.

A visual programming language is one that allows the programmer to create programs by manipulating graphical elements rather than typing textual commands. A well known example is Scratch, a visual programming language from MIT that’s used to teach children. The advantages given are that they make programming more accessible to novices and non-programmers. There was a very popular movement in the 1990’s to bring these kinds of tools into the enterprise with so called CASE tools, where enterprise systems could be defined with UML and generated without the need for trained software developers. This involved the concept of ‘round tripping’, where a system could be modelled visually, the program code would be generated from the models, and any changes to the code could be pushed back to the model. These tools failed to deliver on their promises and most of these attempts have now been largely abandoned.

So visual programming has failed to catch on, except in some very limited domains. This is fundamentally attributable to the following misconceptions about programming:

Textual programming languages obfuscate what is essentially a simple process.
Abstraction and decoupling play a small and peripheral part in programming.
The tools that have been developed to support programming are unimportant.

The first misconception holds that software development has significant barriers to entry because textual programming languages obfuscate the true nature of programming. The popularity of Scratch among educationalists plays to this misconception. The idea is that programming is actually quite simple and if we could only present it in a clear graphical format it would dramatically lower the learning curve and mental effort required to create and read software. I expect this misconception comes from a failure to actually read a typical program written in a standard textual programming language and imagine it transformed into graphical elements of boxes and arrows. If you do this it soon becomes apparent that a single line of code often maps to several boxes and since it’s not untypical for even a simple program to contain hundreds of lines of code, this translates into hundreds or even thousands of graphical elements. The effort to mentally parse such a complex picture is often far harder than reading the equivalent text.

The solution for most visual programming languages is to make the ‘blocks’ represent more complex operations so that each visual element is equivalent to a large block of textual code. Visual workflow tools are a particular culprit here. The problem is that this code needs to be defined somewhere. It becomes ‘property dialogue programming’. The visual elements themselves only represent the very highest level of program flow and the majority of the work is now done in standard textual code hidden in the boxes. Now we have the worst of both worlds, textual programming unsupported by modern tooling. The properly dialogues are usually sub-standard development environments and enforce a particular choice of language, usually a scripting language of some kind. Visual elements can’t be created except by experienced programmers, or understood except by reading their underlying code, so most of the supposed advantages of the visual representation are lost. There’s an impedance mismatch between the visual ‘code’ and the textual code, and the programmer has to navigate the interface between the two, often spending more effort on conforming to the needs of the graphical programming tool than solving the problem at hand.

Which bring us to the second misconception, that abstraction and decoupling are peripheral concerns. Visual programming makes the assumption that most programs are simple procedural sequences, somewhat like a flowchart. Indeed, this is how most novice programmers imagine that software works. However, once a program gets larger than a quite trivial example, the complexity soon overwhelms the novice programmer. They find that it’s very hard to reason about a large procedural code base and often struggle to produce stable and efficient software at scale. Most of the innovation in programming languages is an attempt to manage complexity, most commonly via abstraction, encapsulation and decoupling. All the type systems and apparatus of object-oriented and functional programming is really just an effort to get this complexity under control. Most professional programmers will be continually abstracting and decoupling code. Indeed, the difference between good and bad code is essentially how well this has been done. Visual programming tools rarely have efficient mechanisms to do this and essential trap the developer in an equivalent of 1970’s BASIC.

The final misconception is that visual programmers can do without all the tools that have been developed over the decades to support programming. Consider the long evolution of code editors and IDEs. Visual Studio, for example, supports efficient intellisense allowing the look-up of thousands of APIs available in the base class library alone. The lack of good source control is another major disadvantage of most visual programming tools. Even if they persist their layout to a textual format, the diffs make little or no sense. It’s very hard to do a ‘blame’ on a large lump of XML or JSON. Things that make no difference to the functional execution of the program, such as the position and size of the graphical elements, still lead to changes in the metadata, which make it harder still to parse a diff. Textual programming languages have learnt to separate units of code into separate source files, so a change in one part of the system is easy to merge with a change in another. Visual tools will usually persist as a diagram per file which means that merges become problematic, made harder still when the semantic meaning of the diff is difficult to parse.

In conclusion, the advantages given for visual programming tools, that they make the program easier to create and understand are almost always a mirage. They can only succeed in the simplest of cases and at best result in the suboptimal situation where the visual elements are simply obfuscating containers for textual code.

Update...

I was probably wrong to use a screen-shot of Scratch and use it as the primary example in my first paragraph. I’m not an educator and I don’t really have an opinion about Scratch’s effectiveness as a teaching tool. Many people say that they find it enormously useful in teaching programming, especially to children. Anything that introduces more people to the wonderful and exciting world of programming is only to be celebrated. I really didn’t intend this post as a criticism of Scratch specifically, it was simply the visual programming system that I thought the largest number of people would have heard of.

Another counter example cited on Reddit were static structure tools, such as UI designers, database schema designers, or class designers. I agree that they can be very useful. Anything that helps to visualise the structure of data or the large scale structure of a program is a bonus. These are never enough on their own though. The ultimate failure of 90’s tools such as Power Builder that attempted to build on graphical visualisations to create a fully code-free development environment attest to this.

What I Learnt Creating Guitar Dashboard: SVG, TypeScript and Music Theory.

noreply@blogger.com (Mike Hadlow) — 2018年9月14日 13:57:00 +0000

Guitar Dashboard is a side project I’ve been working on occasionally over the past two years. It’s an open source web application (you can find it at http://guitardashboard.com/ and the code at https://github.com/mikehadlow/gtr-cof). It’s intended as an interactive music theory explorer for guitarists that graphically links theoretical concepts, such as scales, modes and chords to the guitar fretboard. It evolved out my my own attempts, as an amateur guitarist, to get a better understanding of music theory. It includes an algorithmic music theory engine that allows arbitrarily complex scales and chords to be generated from first principles. This gives it far more flexibility than most comparable tools. Coming at music theory from the point of view of software developer, and implementing a music theory rules engine, has given me a perspective that’s somewhat different from most traditional approaches. This post outlines what I’ve learnt, technically and musically while building Guitar Dashboard. There are probably things here that are only interesting to software developers, and others only of interest to musicians, but I expect there’s a sizable group of people, like me, who fit in the intersection of that Venn diagram and who will find it interesting.

Why Guitar Dashboard?

Guitar dashboard’s core mission is to graphically and interactively integrate music theory diagrams, the chromatic-circle and circle-of-fifths, with a graphical representation of the fretboard of a stringed instrument. It emerged from my own study of scales, modes and chords over the past three or four years.

I expect like many self taught guitarists, my main aim when I first learnt to play at the age of 15 was to imitate my guitar heroes, Jimmy Page, Jimi Hendrix, Steve Howe, Alex Lifeson and others. A combination of tips from fellow guitarists, close listening to 60’s and 70’s rock cannon, and a ‘learn rock guitar’ book was enough to get me to a reasonable imitation. I learnt how to play major and minor bar chords and a pentatonic scale for solos and riffs. This took me happily through several bands in my 20s and 30s. Here’s me on stage in the 1980’s with The Decadent Herbs.

I was aware that there was a whole school of classical music theory, but it didn’t at first appear to be relevant to my rock ambitions, and any initial attempts I tried at finding out more soon came to grief on the impenetrable standard music notation and vocabulary, and the very difficult mapping of stave to fretboard. I just couldn’t be bothered with it. I knew there were major and minor scales, I could even play C major on my guitar, and I’d vaguely heard of modes and chord inversions, but that was about it. In the intervening years I’ve continued to enjoy playing guitar, except these days it’s purely for my own amusement, but I’d become somewhat bored with my limited range of musical expression. It wasn’t until around four years ago on a train ride, that a question popped into my head, "what is a ‘mode’ anyway?"

In the intervening decades since my teenage guitar beginnings the internet had happened, so while then I was frustrated by fusty music textbooks, now Wikipedia, immediately to hand on my phone, provided a clear initial answer to my ‘what is a mode question’, followed soon after by a brilliant set of blog posts by Ethan Hein, a music professor at NYU. His clear explanations of how scales are constructed from the 12 chromatic tones by selecting certain intervals, and how chords are then constructed from scales, and especially how he relates modes to different well known songs, opened up a whole new musical world for me. I was also intrigued by his use of the circle-of-fifths which led me to look for interactive online versions. I found Rand Scullard’s excellent visualisation a great inspiration. At the same time in my professional work as a software developer I’d become very excited by the possibilities of SVG for interactive browser based visualisations and realised that Rand’s circle-of-fifths, which he’d created by showing and hiding various pre-created PNG images, would be very easy to reproduce with SVG, and that I could drive it from an algorithmic music engine implemented from the theory that Ethan Hein had taught me. The flexibility offered by factoring out the music generation from the display also meant that I could easily add new visualisations, the obvious one being a guitar fretboard.

My first version was pretty awful. Driven by the hubris of the novice, I’d not really understood the subtleties of note or interval naming and my scales sometimes had duplicate note names amongst other horrors. I had to revisit the music algorithm a few times before I realised that intervals are the core of the matter and the note names come out quite easily once the intervals are correct. The algorithmic approach paid off though; it was very easy to add alternative tunings and instruments to the fretboard since it was simply a case of specifying a different set of starting notes for each string, and any number of strings. Flipping the nut and providing a left-handed fretboard were similarly straightforward. I more recently added non-diatonic scales (access them via the ‘Scale’ menu). This also came out quite easily since the interval specification for the original diatonic scale is simply a twelve element Boolean array. Unfortunately the note naming issue appears again, especially for non-seven-note-scales. Moving forward, it should be relatively easy to add a piano keyboard display, or perhaps, to slay an old demon, a musical stave that would also display the selected notes.

For an introduction to Guitar Dashboard, I’ve created a video tour:

So that’s Guitar Dashboard and my motivation for creating it. Now a brief discussion of some of the things I’ve learnt. First some technical notes about SVG and TypeScript, and then some reflections on music theory.

The awesome power of SVG.

The visual display of Guitar Dashboard is implemented using SVG.

SVG (Scalable Vector Graphics) is an "XML-based vector image format for two-dimensional graphics with support for interactivity and animation." (Wikipedia). All modern browsers support it. You can think of it as the HMTL of vector graphics. The most common use case for SVG is simple graphics and graphs, but it really shines when you introduce animation and interactivity. Have a look at these blog posts to see some excellent examples.

I was already a big fan of SVG before I started work on Guitar Dashboard and the experience of creating it has only made me even more enamoured. The ability to programmatically build graphical interactive UIs or dashboards is SVG’s strongest, but most underappreciated asset. It’s gives the programmer, or designer, far more flexibility than image based manipulation or HTML and CSS. The most fine grained graphical elements can respond to mouse events and be animated. I used the excellent D3js library as an interface to the SVG elements but I do wonder sometimes whether it was an appropriate choice. As a way of mapping data sets to graphical elements, it’s wonderful, but I did find myself fighting it to a certain extent. Guitar Dashboard is effectively a data generator (the music algorithm) and some graphs (the circles and the fretboard), but the graphs are so unlike most D3js applications, that it’s possible I would have been better off just manipulating the raw SVG or developing my own targeted library.

Another strength of SVG is the tooling available to manipulate it. Not only is it browser native, which also means that it’s easy to print and screen-shot, but there are also powerful tools, such as the open source vector drawing tool, Inkscape that make it easy to create and modify SVG documents. One enhancement that I’m keen to include in Guitar Dashboard is a ‘download’ facility that will allow the user to download the currently rendered SVG as a file that can be opened and modified in Inkscape or similar tools. Imagine if you want to illustrate a music theory article, or guitar lesson, it would be easy to select what you want to see in Guitar Dashboard, download the SVG and then edit it at will. You could easily just cut out the fretboard, or the circle-of-fifths, if that’s all you needed. You could colour and annotate the diagrams in any way you wanted. Because SVG is a vector graphics format, you can blow up an SVG diagram to any size without rasterization. You could print a billboard with a Guitar Dashboard graphic and it would be completely sharp. This makes it an excellent choice for printed materials such as textbooks.

TypeScript makes large browser based applications easy.

Creating Guitar Dashboard was my first experience of writing anything serious in TypeScript. I’ve written plenty of Javascript during my career, but I’ve always found it a rather unhappy experience and I’ve always been relieved to return to the powerful static type system of my main professional language C#. I’ve experimented with Haskell and Rust which both have even stronger type systems and the experience with Haskell of '"if it compiles it will run" is enough to make anyone who might have doubted the power of types a convert. I’ve never understood the love for dynamic languages. Maybe for a beginner, the learning curve of an explicit type system seems quite daunting, but for anything but the simplest application, its lack means introducing a whole class of bugs and confusion that simply don’t exist for a statically typed language. Sure you can write a million unit tests to ensure you get what you think you should get, but why have that overhead?

Typescript allows you to confidently create large scale browser based applications. I found it excellent for making Guitar Dashboard. I’m not sure I am writing particularly good Typescript code though. I soon settled into basing everything around interfaces, enjoying the notion of structural rather than nominal typing. I didn’t use much in the way of composition and there’s no dependency injection. Decoupling is achieved with a little home made event bus:

 export class Bus<T> {
 private listeners: Array<(x:T)=>void> = [];
 private name: string;
 constructor(name: string) {
 this.name = name;
 }
 public subscribe(listener: (x:T)=>void): void {
 this.listeners.push(listener);
 }
 public publish(event: T): void {
 //console.log("Published event: '" + this.name + "'")
 for (let listener of this.listeners) {
 listener(event);
 }
 }
 }

A simple event bus, is just a device to decouple code that wants to inform that something has happened from code that wants to know when it does. It’s a simple collection of functions that get invoked every time an event is published. The core motivation is to prevent event producers and consumers from having to know about each other. There’s one instance of Bus for each event type.

Each of the main graphical elements is its own namespace which I treated like stand alone modules. Each of which subscribe to and raise typed events via a Bus instance. I only created classes when there was an obvious need, such as the Bus class above and the NoteCircle class which has two instances, the chromatic-circle and circle of fifths. I didn’t write any unit tests either, although now I think the music module algorithm is complex enough that it’s really crying out for them. Guitar Dashboard is open source, so you can see for yourself what you think of my Typescript by checking it out on GitHub.

Another advantage of TypeScript is the excellent tooling available. I used VS Code which itself is written in TypeScript and which supports it out-of-the-box. The fact that VS Code has been widely adopted outside of the Microsoft ecosystem is a testament to its quality as a code editor. It came top in the most recent Stack Overflow developer survey. I’ve even started experimenting with using it for writing C# and it’s a pretty good experience.

What I learnt about music.

Music is weird. Our ears are like a serial port into our brain. With sound waves we can reach into our cerebral cortex and tweak our emotions or tickle our pleasure senses. A piece of music can take you on a journey, but one which bares no resemblance to concrete reality. Music defines human cultures and can make and break friendships; people feel that strongly about it. But fundamentally it’s just sound waves. It greatly confuses evolutionary psychologists. What possible survival advantage does it confer? Maybe it’s the human equivalent of the peacock’s tail; a form of impressive display; a marker of attendant mental agility and fitness? Who knows. What is true is that we devote huge resources to the production and consumption of music: the hundreds of thousands of performers; the huge marketing operations of the record companies; the global business of producing and selling musical instruments and the kit to record it and play it back. The biggest company in the world, Apple, got its second wind from a music playback device and musical performers are amongst the most popular celebrities.

But why do our brains favour some forms of sound over others? What makes a melody, a harmony, a rhythm, more or less attractive to us? I recently read a very good book on this subject, The Music Instinct by Philip Ball. The bottom line is that we have no idea why music affects us like it does, but that’s unsurprising given that the human brain is still very much a black box to science. It does show, however, that across human cultures there are some commonalities: rhythm, the recognition of the octave, where we perceive two notes an octave apart as being the same note, and also something close to the fifth and the third. It’s also true that music is about ratios between frequencies rather than the frequencies themselves, with perhaps the exception of people with perfect pitch. The more finely grained the intervals become, the more cultures diverge, and it’s probably safe to say that the western twelve tone chromatic scale with its ‘twelfth root of two’ ratio is very much a technical innovation to aid modulation rather than something innate to the human brain. Regardless of how much is cultural or innate, the western musical tradition is very much globally dominant. Indeed, it’s hard buy a musical instrument that isn’t locked down to the twelve note chromatic scale.

However, despite having evolved a very neat, mathematical and logical theory, western music suffers from a common problem that bedevils any school of thought that’s evolved over centuries, a complex and difficult vocabulary and a notation that obfuscates rather than reveals the structure of what it represents. Using traditional notation to understand music theory is like doing maths with Roman numerals. In writing the music engine of guitar dashboard, by far the most difficult challenges have been outputting the correct names for notes and intervals.

This is a shame, because the fundamentals are really simple. I will now explain western music theory in four steps:

Our brains interpret frequencies an octave apart as the same ‘note’, so we only need to care about the space between n and 2n frequencies.
Construct a ratio such that applying the ratio to n twelve times gives 2n. Maths tells you that this must be the 12th root of 2. (first described by Simon Stevin in 1580). Each step is called a semitone.
Start at any of the twelve resulting notes and jump up or down in steps of 7 semitones (traditionally called a 5th) until you have a total of 7 tones/notes. Note that we only care about n to 2n, so going up two sets of 7 semitones (or two 5ths) is the same as going up 2 semitones (a tone) (2 x 7 – 12 = 2. In music all calculations are mod 12). This is a diatonic scale. If you choose the frequency 440hz, jump down one 7-semitone step and up 5, you have an A major scale. Up two 7-semitone steps and down four gives you A minor. The other five modes (Lydian, Mixolydian, Dorian, Phrygian and Locrian) are just different numbers of up and down 7-semitone steps.
Having constructed a scale, choose any note. Count 3 and 5 steps of the scale (the diatonic scale you just constructed, not the original 12 step chromatic scale) to give you three notes. This is a triad, a chord. Play these rhythmically in sequence while adding melody notes from the scale until you stumble across something pleasing.

That, in four simple steps, is how you make western music.

OK, that’s a simplification, and the most interesting music breaks the rules, but this simple system is the core of everything else you will learn. But try to find this in any music textbook and it simply isn’t there. Instead there is arcane language and confusing notation. I really believe that music education could be far simpler with a better language, notation and tools. Guitar Dashboard is an attempt to help people visualise this simplicity. Everything but the fretboard display is common to all musical instruments. It’s only aimed at guitarists because that’s what I play and it also helps that guitar is the second most popular musical instrument. The most poplar, piano, would be easy to add. Piano Dashboard anyone?

The Possibilities of Web MIDI With TypeScript

noreply@blogger.com (Mike Hadlow) — 2018年9月05日 11:17:00 +0000

If you’ve ever had any experience with music technology, or more specifically sequencers, keyboards or synthesisers, you will have come across MIDI (Musical Instrument Digital Interface). It’s used to send note and controller messages from musical devices, such as keyboards or sequencers which are used to play and record music, and devices that produce sounds, such as samplers or synthesizers. It’s pure control information, for example, "play a c# in the 3rd octave with a velocity of 85", there’s no actual audio involved. It dates back to the early 1980s, when a group of musical instrument manufacturers such as Roland, Sequential Circuits, Oberheim, Yamaha and Korg got together to define the standard. It soon lead to a huge boom in low cost music production and the genesis of new musical styles. It’s no accident that rap and electronic dance music date from the mid to late 80’s.

Web MIDI is a new W3C specification for an API to allow browser applications to access MIDI input and output devices on the host machine. You can enumerate the devices, then choose to listen for MIDI messages, or format and send your own messages. It’s designed to allow applications to consume and emit MIDI information at the protocol level, so you receive and send the actual raw message bytes rather the API providing the means to play MIDI files using General MIDI for example. Don’t let this put you off though, the protocol is very simple to interpret as I’ll demonstrate later.

The potential for a large new class of browser based musical applications is huge. The obvious examples are things like browser based sequencers and drum machines emitting MIDI messages and synthesizers and samplers on the consuming side using Web Audio, another interesting new standard. But it goes much wider than that, the MIDI protocol is ideally suited to any real-time parameter control. It’s already widely used for lighting rigs and special effects in theatrical productions for example. Also because it’s such an established standard, there is all kinds of cheaply available hardware controller interfaces full of knobs and buttons. If you’ve got any application that requires physical control outside the range of keyboard/mouse/trackpad, it might be a solution. Imagine a browser based application that allowed you to turn knobs on a cheap MIDI controller to tweak the parameters of a mathematical visualisation, or some network based industrial controller, or even as new input for browser based games. The possibilities are endless.

I’m going to show a simple TypeScript example. I’m currently working on a TypeScript application that consumes MIDI and I couldn’t find much good example code so I’m hoping this might help. I’m using the type definitions from here: https://www.npmjs.com/package/@types/webmidi.

The entry point into the new API is a new method on navigator, requestMIDIAccess. This returns a Promise<MIDIAccess> that you can use to enumerate the input and output devices on the system. Here I’m just looking for input devices:

window.navigator.requestMIDIAccess()
 .then((midiAccess) => {
 console.log("MIDI Ready!");
 for(let entry of midiAccess.inputs) {
 console.log("MIDI input device: " + entry[1].id)
 entry[1].onmidimessage = onMidiMessage;
 }
 })
 .catch((error) => {
 console.log("Error accessing MIDI devices: " + error);
 });

I’ve bound my onMidiMessage function to the onmidimessage event on every input device. This is the simplest possible scenario, it would be better to provide an option to your user to choose the device they want to use. This allows us to process MIDI events as they arrive from MIDI devices.

MIDI events arrive as byte arrays with a length of 1 to 3 bytes. The first byte is always the ‘status’ byte. The four most significant bits are the status type. Here we’re only concerned with note on (9) and off (8) messages. The four least significant bytes tell us the MIDI channel. This allows up to 16 different devices, or voices to be controlled by a single controller device. If you ignore the channel, as we’re doing here, it’s known as OMNI mode. For note on/off messages, the second byte is the note number and the third is the velocity, or how loud we want the note to sound. The note number describes the frequency of the note using the classical western chromatic scale; good luck if you want to make Gamelan dance music! The notes go from C0 (around 8hz) to G11 (approx 12543hz). This is much wider than a grand piano keyboard and sufficient for the vast majority of applications. See the code for how to convert the note number to name and octave. See this page and the Wikipedia page for more details.

In this example we filter for on/off messages, then write the channel, note name, command type and velocity to the console:

let noteNames: string[] = ["C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B"];
function onMidiMessage(midiEvent: WebMidi.MIDIMessageEvent): void {
 let data: Uint8Array = midiEvent.data;
 if(data.length === 3) {
 // status is the first byte.
 let status = data[0];
 // command is the four most significant bits of the status byte.
 let command = status >>> 4;
 // channel 0-15 is the lower four bits.
 let channel = status & 0xF;
 console.log(`$Command: ${command.toString(16)}, Channel: ${channel.toString(16)}`);
 // just look at note on and note off messages.
 if(command === 0x9 || command === 0x8) {
 // note number is the second byte.
 let note = data[1];
 // velocity is the thrid byte.
 let velocity = data[2];
 let commandName = command === 0x9 ? "Note On " : "Note Off";
 // calculate octave and note name.
 let octave = Math.trunc(note / 12);
 let noteName = noteNames[note % 12];
 console.log(`${commandName} ${noteName}${octave} ${velocity}`);
 }
 }
}

Here’s the output. I’m using Vmpk (Virtual MIDI Piano Keyboard) to play the notes. You’ll also need a MIDI loopback device such as loopMIDI if you want to connect software devices, but it should be plug and play with a hardware controller:

So there we have it. MIDI is now very easy to integrate into a browser based application. I’ve demonstrated this with just a few lines of code. It opens up possibilities for a new class of software and not for just musical applications. It’s going to be very interesting to see what people do with it.

Configure AsmSpy as an external tool in Visual Studio

noreply@blogger.com (Mike Hadlow) — 2018年1月18日 11:19:00 +0000

AsmSpy is a tool I wrote a few years ago to view assembly version conflicts. Despite the fact that it started as a single page of code command line application, it’s been one of my more successful open source efforts. I still use it all the time, especially now with the ‘forking’ of .NET into Framework and Core and spreading use of dotnet standard, both good things IMHO, but not without the occasional assembly version head scratcher.

Today I want to show how easy it is to integrate AsmSpy into Visual Studio as an ‘external tool’.

First download AsmSpy from the GitHub repository. If you download the zip file, you’ll see that it’s merely a stand alone exe that you can run from the command line.

In VS select External Tools from the ‘Tools’ menu.

Now configure AsmSpy as follows:
Title: AsmSpy
Command: The path to where you’ve put the AsmSpy.exe file.
Arguments: $(BinDir) - this points AsmSpy at the output directory of the currently selected project.
Initial Directory: $(ProjectDir)
Use Output Window: checked. – this ensures that the output from AsmSpy will go to Visual Studio’s output window.

Now you can select a project in Solution Explorer and go to Tools –> AsmSpy. AsmSpy will run against the build output of your project and you can view Assembly version conflicts in the Visual Studio output window.

Running The KestrelHttpServer On Linux With CoreCLR

noreply@blogger.com (Mike Hadlow) — 2016年1月14日 11:43:00 +0000

Being both a long-time .NET developer and Linux hobbyist, I was very excited about the recent ‘go live’ announcement for CoreCLR on Linux (and Windows and Mac). I thought I’d have a play with a little web server experiment on an Amazon EC2 instance. To start with I tried to get the KesteralHttpServer sample application working which wasn’t as easy as I’d hoped, so this post is a note of the steps you currently need.

So first create a new Ubuntu Server 14.04 AMI:

With a t2.micro instance type:

Next log in and update:

ssh -i .ssh/mykey.pem ubuntu@-the-ip-address
...
sudo apt-get update
sudo apt-get upgrade

Currently there are two different sets of instructions for installing CoreCLR on Linux. The first one I found (I think linked from Scott Hanselman’s blog) shows how to use the standard Debian package manager to install the new ‘dotnet’ comand line tool. Apparently Kestrel will not currently work with this. The second set of instructions use the existing ‘dnvm’, ‘dnu’ and ‘dnx’ tools. These do work, but you need to get the latest unstable RC2 version of CoreCLR, like this:

First install the Dot Net Version Manager tool:

sudo apt-get install unzip curl
curl -sSL https://raw.githubusercontent.com/aspnet/Home/dev/dnvminstall.sh | DNX_BRANCH=dev sh && source ~/.dnx/dnvm/dnvm.sh
source /home/ubuntu/.dnx/dnvm/dnvm.sh

Next install the latest unstable (thus the '-u' flag) CoreCLR:

sudo apt-get install libunwind8 gettext libssl-dev libcurl4-openssl-dev zlib1g libicu-dev uuid-dev
dnvm upgrade -u -r coreclr

At the time of writing this installed 1.0.0-rc2-16357

You also need to follow the instructions to build the libuv library from source. This sounds hairy, but it worked fine for me:

sudo apt-get install make automake libtool curl
curl -sSL https://github.com/libuv/libuv/archive/v1.8.0.tar.gz | sudo tar zxfv - -C /usr/local/src
cd /usr/local/src/libuv-1.8.0
sudo sh autogen.sh
sudo ./configure
sudo make
sudo make install
sudo rm -rf /usr/local/src/libuv-1.8.0 && cd ~/
sudo ldconfig

Next get the KestrelHttpServer source code:

git clone https://github.com/aspnet/KestrelHttpServer.git

Restore the Kestrel packages by running dnu restore in the root of the repository:

cd KestrelHttpServer
dnu restore

Next navigate to the sample app and restore the packages there too:

cd samples/SampleApp/
dnu restore

Now you should be able to run the sample app by typing:

dnx web

Voilà!

There’s obviously some way to go before this is a straightforward out-of-the-box experience. The team should also try and unify their getting started instructions because there are various different conflicting pages floating around. The Kestrel team were very helpful though in getting this working. Now to do something with my new found Linux web server.

Learn To Code, It’s Harder Than You Think

noreply@blogger.com (Mike Hadlow) — 2015年12月04日 13:28:00 +0000

TL;DR: All the evidence shows that programming requires a high level of aptitude that only a small percentage of the population possess. The current fad for short learn-to-code courses is selling people a lie and will do nothing to help the skills shortage for professional programmers.

This post is written from a UK perspective. I recognise that things may be very different elsewhere, especially concerning the social standing of software developers.

It’s a common theme in the media that there is a shortage of skilled programmers (‘programmers’, ‘coders’, ‘software developers’, all these terms mean the same thing and I shall use them interchangeably). There is much hand-wringing over this coding skills gap. The narrative is that we are failing to produce candidates for the "high quality jobs of tomorrow". For example, this from The Telegraph:

"Estimates from the Science Council suggest that the ICT workforce will grow by 39 per cent by 2030, and a 2013 report from O2 stated that around 745,000 additional workers with digital skills would be needed to meet demand between now and 2017.

Furthermore, research by City & Guilds conducted last year revealed that three quarters of employers in the IT, Digital and Information Services Sector said that their industry was facing a skills gap, while 47 per cent of employers surveyed said that the education system wasn’t meeting the needs of business."

Most commentators see the problem as being a lack of suitable training. Not enough programmers are being produced from our educational institutions. For example, here is Yvette Cooper, a senior Labour party politician, in The Guardian:

"The sons and daughters of miners should all be learning coding. We have such huge advantages because of the world wide web being invented as a result of British ingenuity. We also have the English language but what are we doing as a country to make sure we are at the heart of the next technology revolution? Why are we not doing more to have coding colleges and technical, vocational education alongside university education?"

There is also a common belief in the media that there are high barriers to entry to learning to code. This from the Guardian is typical:

"It’s the must-have skill-set of the 21st century, yet unless you’re rich enough to afford the training, or fortunate enough to be attending the right school, the barriers to learning can be high."

So the consensus seems to be that high barriers to entry and a lack of accessible training mean that only a rich and well educated elite have access to these highly paid jobs. The implication is that there is a large population of people for whom programming would be a suitable career if only they could access the education and training that is currently closed to them.

In response, there are now a number of initiatives to encourage people to take up programming. The UK government created ‘Year of Code’ in 2014:

The message is "start coding this year, it’s easier than you think." Indeed the executive director of Year of Code, Lottie Dexter, said in a Newsnight interview that people can "pick it up in a day". Code.org, a "non-profit dedicated to expanding participation in computer science education", says on its website, "Code.org aims to help demystify that coding is difficult".

So is it really that easy to learn how to code and get these high paying jobs? Is it really true that anyone can learn to code? Is it possible to take people off the streets, give them a quick course, and produce professional programmers?

What about more traditional formal education? Can we learn anything about training programmers from universities?

Given the skills shortage one would expect graduates from computer science courses to have very high employment rates. However, it seems that is not the case. The Higher Education Statistics Agency found that computer science graduates have "the unwelcome honour of the lowest employment rate of all graduates." Why is this? Anecdotally there seems to be a mismatch between the skills the students graduate with and those that employers expect them to have. Or more bluntly, after three years of computer science education they can’t code. A comment on this article by an anonymous university lecturer has some interesting insights:

"Every year it's the same - no more than a third of them [CS students] are showing the sort of ability I would want in anyone doing a coding job. One-third of them are so poor at programming that one would be surprised to hear they had spent more than a couple of weeks supposedly learning about it, never mind half-way through a degree in it. If you really test them on decent programming skills, you get a huge failure rate. In this country it's thought bad to fail students, so mostly we find ways of getting them through even though they don't really have the skills."

Other research points to similar results. There seems to be a ‘double hump’ in the outcome of any programming course between those who can code and those who can’t.

"In particular, most people can't learn to program: between 30% and 60% of every university computer science department's intake fail the first programming course."

Remember we are talking about degree level computing courses. These are students who have been accepted by universities to study computer science. They must be self selecting to a certain extent. If the failure rate for programming courses is so high amongst undergraduates it would surely be even higher amongst the general population - the kinds of candidates that the short ‘learn to code’ courses are attempting to attract.

Let’s look at the problem from the other end of the pipeline. Let’s take successful professional software developers and ask them how they learnt to code. One would expect from the headlines above that they had all been to expensive, exclusive coding schools. But here again that seems not to be the case. Here are the results of the 2015 Stack Overflow developers survey. Note that this was a global survey, but I think the results are relevant to the UK too:

Only a third have a computer science or related degree and nearly 42%, the largest group, are self taught. I have done my own small and highly unscientific research on this matter. I run a monthly meet-up for .NET developers here in Brighton, and a quick run around the table produced an even more pronounced majority for the self-taught. For fun, I also did a quick Twitter poll:

76% say they are self taught. Also interesting were the comments around the poll. This was typical:

Even programmers with CS degrees insist that they are largely self taught. Others complained that it was a hard question to answer since the rate of change in the industry means that you never stop learning. So even if you did at some point have formal training, you can’t rely on that for a successful career. Any formal course will be just a small element of the continual learning that defines the career of a programmer.

We are left with a very strange and unexpected situation. Formal education for programmers seems not to work very well and yet the majority of those who are successful programmers are mostly self taught. On the one hand we seem to have people who don’t need any guided education to give them a successful career; they are perfectly capable of learning their trade from the vast sea of online resources available to anyone who wants to use it. On the other hand we have people who seem unable to learn to code even with years of formal training.

This rather puts the lie to the barriers to entry argument. If the majority of current professional software developers are self taught, how can there be barriers to entry? Anyone with access to the internet can learn to code if they have the aptitude for it.

The evidence points to a very obvious conclusion: there are two populations: one that finds programming a relatively painless and indeed enjoyable thing to learn and another that can’t learn no matter how good the teaching. The elephant in the room, the thing that Yvette Cooper, the ‘year of code’ or ‘hour of code’ people seem unwilling to admit is that programming is a very high aptitude task. It is not one that ‘anyone can learn’, and it is not easy, or rather it is easy, but only if you have the aptitude for it. The harsh fact is that most people will find it impossible to get to any significant standard.

If we accept that programming requires a high level of aptitude, it’s fun to compare some of the hype around the ‘learn to code’ movement with more established high-aptitude professions. Just replace ‘coder’ or ‘coding’ with ‘doctor’, ‘engineer’, ‘architect’ or ‘mathematician’.

"You can pick up Maths in a day."
Start surgery this year, it’s easier than you think!
skyscraper.org aims to help demystify that architecture is difficult.
"The sons and daughters of miners should all be learning to be lawyers."

My friend Andrew Cherry put it very well:

Answer: only one: software development. You want to be a doctor? Go to medical school for seven years.

Accepting that aptitude is important for a successful career in programming, we can approach the ‘shortage’ problem from a different angle. We can ask how we can persuade talented people to choose programming rather than other high-aptitude professions. The problem is that these individuals have a great deal of choice in their career path and, as I’m going to explain, programming has a number of negative social and career attributes which make them unlikely to choose it.

There’s no doubt that software development is a very attractive career. It’s well paid, mobile, and the work itself is challenging and rewarding. But it has an image problem. I first encountered this at university in the 1990’s. I did a social science degree (yes I’m one of those self taught programmers). Socially, us arts students looked down on people studying computer science, they were the least cool students on the campus - mostly guys, with poor dress sense. If anyone considered them at all it was with a sense of pity and loathing. When towards the end of my degree, I told my then girlfriend, another social science student, that I might choose a career in programming, she exclaimed, "oh no, what a waste. Why would you want to do that?" If you did a pop-quiz at any middle-class gathering in the UK and asked people to compare, say, medicine, law, architecture or even something like accountancy, with software development, I can guarantee that they would rate it as having a lower social status. Even within business, or at least more traditional businesses, software development is seen as a relatively menial middle-brow occupation suitable for juniors and those ill-qualified for middle management. Perversely, all these courses saying ‘learn to code, it’s easy’ just reinforce the perception that software development is not a serious career.

There’s another problem with software development that’s the flip side of the low barriers to entry mentioned above, and that is there is no well established entry route into the profession. Try Googling for ‘how to become a doctor’, or ‘how to become a lawyer’ for example:

There are a well established series of steps to a recognised professional qualification. If you complete the steps, you become a recognised member of one of these professions. I’m not saying it’s easy to qualify as a doctor, but there’s little doubt about how to go about it. Now Google for ‘how to become a software developer’, the results, like this one for example, are full of vague platitudes like ‘learn a programming language’, ‘contribute to an open source project’, ‘go to a local programming group’. No clear career path, no guarantees about when and if you will be considered a professional and get access to those high-paying jobs of the future.

Now take a high-aptitude individual who has done well at school and finds demanding intellectual tasks relatively straightforward, and offer them a choice: on the one hand, here is a career, let’s take medicine for example, you follow these clearly enumerated steps, which are demanding but you are good at passing exams, and at the end you will have a high-status, high paying job. Or, how about this career: go away, learn some stuff by yourself, we’re not sure exactly what; try and get a junior, low status job, and just learn more stuff – which you can work out somehow – and work your way up. No guarantees that there’s a well paying job at the end of it. Oh, and, by the way, the whole world will think you are a bit of a social pariah while you are about it. Which would you choose?

So could software development follow the example of older professions and establish a professional qualification with high barriers to entry? There are attempts to do this. The British Computer Society (BCS) calls itself ‘the chartered institute for IT’ and seeks establish professional qualifications and standards. The problem is that it’s comprehensively ignored by the software industry. Even if you could get the industry to take a professional body seriously, how would you test people to see if they qualified? What would be on the exam? There are very few established practices in programming and as soon as one seems to gain some traction it gets undermined by the incredibly rapid pace of change. Take Object Oriented programming for example. In the 2000’s, it seemed to be establishing itself as the default technique for enterprise programming, but now many people, including myself, see it as a twenty year diversion and largely a mistake. How quickly would programming standards and qualifications stay up to date with current practice? Not quickly enough I suspect.

However, my main point in this post has been to establish that programming is a high-aptitude task, one than only some people are capable of doing with any degree of success. If the main point of a professional qualification is filter out people who can’t code, does it really matter if what is being tested for is out of date, or irrelevant to current industry practices? Maybe our tentative qualification would involve the completion of a reasonably serious program in LISP? A kind of Glass Bead Game for programmers? The point would be to find out if they can code. They can learn what the current fads are later. The problem still remains how to get industry to recognise the qualification.

In the meantime we should stop selling people a lie. Programming is not easy, it is hard. You can’t learn to code, certainly not to a standard to get a well-paid-job-of-the-future, in just a few weeks. The majority of the population can not learn to code at all, no matter how much training they receive. I doubt very much if the plethora of quick learn-to-code courses will have any impact at all on the skills shortage, or the problem of unskilled low pay and unemployment. Let’s stop pretending that there are artificial barriers to entry and accept that the main barrier to anyone taking it up is their natural aptitude for it. Instead let’s work on improving the social status of the software industry – I think this is in any case happening slowly – and also work on encouraging talented young people to consider it as a viable alternative to some of the other top professions.

Partial Application in C#

noreply@blogger.com (Mike Hadlow) — 2015年9月10日 11:02:00 +0000

My recent post, C# Program Entirely With Static Methods, got lots of great comments. Indeed, as is often the case, the comments are in many ways a better read than the original post. However, there were several commenters who claimed that C# does not have partial application. I take issue with this. Any language that supports higher order functions, that is, functions that can take functions as arguments and can return functions, by definition, supports partial application. C# supports higher order functions, so it also supports partial application.

Let me explain.

Let’s start by looking at partial application in F#. Here’s a simple function that adds two numbers (you can type this into F# interactive):

>let add a b = a + b;;

Now we call use our ‘add’ function to add two numbers, just as we’d expect:

> add 3 4;;
val it : int = 7

But because F# supports partial application we can also do this:

> let add3 = add 3;;
> add3 4;;
val it : int = 7

We call add with a single argument and it returns a function that takes a single argument which we can then use to add three to any number.

That’s partial application. Of course, if I try this in C# it doesn’t work:

Red squiggly line saying "delegate Func has two parameters but is invoked with one argument.

Case proven you say: C# does not support partial application!

But wait!

Let’s look again at the F# add function. This time I’ll include the response from F# interactive:

> let add a b = a + b;;
val add : a:int -> b:int -> int

This shows us the type of the add function. The important bit is: "a:int –> b:int –> int". This tells us that ‘add’ is a function that takes an int and returns a function that takes an int and returns an int. It is not a function with two arguments. F# is a restrictive language, it only has functions with single arguments. That is a good thing. See Mark Seemann’s post Less is More: Langauge Features for an in depth discussion of how taking features away from a language can make it better. When people say "F# supports partial application" what they really mean is that "F# functions can only have one argument." The F# compiler understands the syntax ‘let add a b = ...’ to mean "I want a function that takes a single argument and returns a function that takes another single argument."

There’s nothing to stop us from defining our C# function with the same signature as our F# example. Then we can partially apply it in the same way:

There you are: partial application in C#. No problem at all.

"But!" You cry, "That’s weird and unusual C#. I don’t want to define all my functions in such a strange way." In that case, let me introduce you to my friend Curry. It’s not a spicy dish of South Asian origin but the process of turning a function with multiple arguments into a series of higher order functions. We can define a series of overloaded Curry extension methods:

We can then use them to turn ‘ordinary’ C# functions with multiple arguments into higher-order functions which we can partially apply:

Thinking more about Mark Seemann’s blog post, it would be an interesting exercise to start to take features away from C# whilst keeping syntactic changes to a minimum. If we took away multiple function arguments, classes, interfaces, nullable types, default mutability etc, would we end up with a subset language that would be perfect for functional programming, but still familiar to C# developers? You would of course lose backward compatibility with existing C# code, so the incentive to do it isn’t that great, but it’s a fascinating thought experiment.

C#: Program Entirely With Static Methods

noreply@blogger.com (Mike Hadlow) — 2015年8月07日 14:13:00 +0000

OK, that’s a provocative title to get your attention. This post is really about how one can move to a more functional programming style and remove the need for much of the apparatus of object-oriented programming, including interfaces and classes. In this post, I’m going to take some typical object-oriented C# code and refactor it in a more functional style. I’ll show that the result is more concise and easier to test.

Over the past couple of years I’ve noticed that my C# coding style has changed drastically under the influence of functional programming. Gone are interfaces and instance classes to be replaced by static methods, higher-order functions and closures. It’s somewhat ironic since I spent many years as a cheerleader for object-oriented programming and I considered static methods a code smell.

I guess if I look at my programming career, it has the following progression:

Procedural –> Object-Oriented –> Functional

The OO phase now looks like something of a detour.

C# has all the essential features you need for functional programming – higher-order functions, closures, lambda expressions – that allow you to entirely ditch the OO programming model. This results in more concise, readable and maintainable code. It also has a huge impact on unit testing, allowing one to do away with complex mocking frameworks, and write far simpler tests.

Introducing our object oriented example

Let’s look at an example. First I’ll introduce a highly simplified OO example, a simple service that grabs some customer records from a data-store, creates some reports and then emails them. Then I’ll show the same code refactored in a more functional style using delegates and higher-order static methods.

Let’s look at the object-oriented example first:

Well written object-oriented code is compositional. Concrete classes depend on abstractions (interfaces). These interfaces are consumed as dependencies by classes that rely on them and are usually injected as constructor arguments. This is called Dependency Injection. It’s good practice to compose object instances in a single place in the application - the composition root - usually when the application starts up, or on a significant event, such as an HTTP request. The composition can be hand coded or handed off to an IoC container. The constructed graph is then executed by invoking a method on the root object. This often occurs via an application framework (such as MVC or WebApi) rather than being explicitly invoked by user code.

We are going to get some customer records, create some reports and then email them to our customers. So first we need three interfaces: a data access abstraction, a report building abstraction, and an emailing abstraction:

And here are the implementations. This is not a real program of course, I’ve just coded some dummy customers and the emailer simply writes to the console.

Now we have our service class that depends on the three abstractions and orchestrates the reporting process:

As you can see, we inject the dependencies as constructor arguments, store them in class properties, then invoke methods on them in the code in the RunCustomerReportBatch method. Some people like to store the dependencies in class fields instead. That’s a matter of choice.

Our composition root composes the ReportingService with its dependencies and then returns it for the program to invoke. Don’t forget this is a highly simplified example. Composition is usually far more complex:

To write a unit test for the reporting service we would typically use either hand-crafted mocks, or some kind of mocking framework. Here’s an example unit test using XUnit and Moq:

We first create mocks for ReportingService’s dependencies with the relevant methods stubbed, which we inject as constructor arguments. We then invoke ReportingService and verify that the emailer was invoked as expected.

So that’s our object-oriented example. It’s typical of much well constructed C# code that you will find in the wild. It’s the way I’ve been building software for many years now with much success.

However, this object-oriented code is verbose. About a third of it is simply OO stuff that we have to write repeatedly and mechanically rather than code that is actually solving our problem. This boilerplate includes: the class’ properties (or fields) to hold the dependencies; the assigning of constructor arguments to those properties; writing the class and constructor. We also need complex mocking frameworks simply to test this code. Surely that’s a smell that’s telling us something is wrong?

Enlightenment

Enlightenment begins when you realise that the dependencies and method arguments can actually just be seen as arguments that are applied at different times in the application’s lifecycle. Consider a class with a single method and a single dependency:

We could equally represent this as a static method with two arguments:

But how do we partially apply these arguments? How do we give ‘DoThing’ the IDependency argument at composition time and the ‘string arg’ at the point where it is required by the application logic? Simple: We use a closure. Anything taking a dependency on ‘DoThing’ will ask for an Action<string>, because that is the signature of the ‘Do’ method in our ‘Thing’ class. So in our composition root, we ‘close over’ our previously created IDependency instance in a lambda expression with the signature, Action<string>, that invokes our DoThing static method. Like this:

So the interface is replaced with the built-in Action<T> delegate, and the closure is effectively doing the job of our ‘Thing’ class, the interface’s implementation, but with far fewer lines of code.

Refactoring to functional

OK. Let’s go back to our example and change it to use this new insight. We don’t need the interface definitions. They are replaced by built in delegate types:

ICustomerData becomes Func<IEnumerable<Customer>>

IEmailer becomes Action<string, string>

IReportBuilder becomes Func<Customer, Report>

The classes are replaced with static methods:

Our ReportingService is also replaced with a single static method that takes its dependencies as delegate arguments:

Composition looks like this:

This is functionally equivalent to the object-oriented code above, but it has 57 lines of code as opposed to 95; exactly 60% of the original code.

There’s also a marked simplification of the unit test:

The requirement for a complex mocking framework vanishes. Instead we merely have to set up simple lambda expressions for our stubs. Expectations can be validated with closed over local variables. It’s much easier to read and maintain.

Moving to a functional style of programming is certainly a huge departure from most C# code that you find in the wild and can initially look a little odd to the uninitiated. But it has many benefits, making your code more concise and easier to test and reason about. C# is, surprisingly, a perfectly adequate functional programming language, so don’t despair if for practical reasons you can’t use F#.

The complete code example for this post is on GitHub here: https://github.com/mikehadlow/FunctionalDemo

C#: How to Record What Gets Written to or Read From a Stream

noreply@blogger.com (Mike Hadlow) — 2015年6月05日 10:54:00 +0000

Streams are a very nice abstraction over a read/write loop. We can use them to represent the contents of a file, or a stream of bytes to or from a network socket. They make it easy to read and write large amounts of data without consuming large amounts of memory. Take this little code snippet:

Example.txt may be many GB in size, but this operation will only ever use the amount of memory configured for the buffer. As an aside, the .NET framework’s Stream class’s default buffer size is the maximum multiple of 4096 that is still smaller than the large object heap threshold (85K). This means it likely to be collected at gen zero by the garbage collector, but still gives good performance.

But what if we want to log or view the contents of Example.txt as it’s copied to the output file? Let me introduce my new invention: InterceptionStream. This is simple class that inherits and decorates Stream and takes an additional output stream. Each time the wrapped stream is read from, or written to, the additional output stream gets the same information written to it. You can use it like this:

I could just as well have wrapped the input stream with the InterceptionStream for the same result:

You can use a MemoryStream if you want to capture the log in memory and assign it to a string variable, but of course this negates the memory advantages of the stream copy since we’re now buffering the entire contents of the stream in memory:

Here is the InterceptionStream implementation. As you can see it’s very simple. All the work happens in the Read and Write methods:

Inject DateTime.Now to Aid Unit Tests

noreply@blogger.com (Mike Hadlow) — 2015年5月28日 10:10:00 +0000

If you have logic that relies on the current system date, it's often difficult to see how to unit test it. But by injecting a function that returns DateTime.Now we can stub the current date to be anything we want it to be.
Let's look at an example. Here we have a simple service that creates a new user instance and saves it in a database:

 public class UserService : IUserService
 {
 private readonly IUserData userData;
 public UserService(IUserData userData)
 {
 this.userData = userData;
 }
 public void CreateUser(string username)
 {
 var user = new User(username, createdDateTime: DateTime.UtcNow);
 userData.SaveUser(user);
 }
 }

Now if I want to write a unit test that checks that the correct created date is set, I have to rely on the assumption that the system date won't change between the creation of the User instance and the test assertions.

 [TestFixture]
 public class UserServiceTests
 {
 private IUserService sut;
 private IUserData userData;
 [SetUp]
 public void SetUp()
 {
 userData = MockRepository.GenerateStub<iuserdata>();
 sut = new UserService(userData);
 }
 [Test]
 public void UserServiceShouldCreateUserWithCorrectCreatedDate()
 {
 User user = null;
 // using Rhino Mocks to grab the User instance passed to the IUserData stub
 userData.Stub(x => x.SaveUser(null)).IgnoreArguments().Callback<user>(x =>
 {
 user = x;
 return true;
 });
 sut.CreateUser("mike");
 Assert.AreEqual(DateTime.UtcNow, user.CreatedDateTime);
 }
 }

But in this case, probably because Rhino Mocks is doing some pretty intensive proxying, a few milliseconds pass between the user being created and my assertions running.

Test 'Mike.Spikes.InjectingDateTime.UserServiceTests.UserServiceShouldCreateUserWithCorrectCreatedDate' failed: 
 Expected: 2015年05月28日 09:08:18.824
 But was: 2015年05月28日 09:08:18.819
 InjectingDateTime\InjectDateTimeDemo.cs(75,0): at Mike.Spikes.InjectingDateTime.UserServiceTests.UserServiceShouldCreateUserWithCorrectCreatedDate()

The solution is to inject a function that returns a DateTime:

 public class UserService : IUserService
 {
 private readonly IUserData userData;
 private readonly Func<datetime> now;
 public UserService(IUserData userData, Func<datetime> now)
 {
 this.userData = userData;
 this.now = now;
 }
 public void CreateUser(string username)
 {
 var user = new User(username, createdDateTime: now());
 userData.SaveUser(user);
 }
 }

Now our unit test can rely on a fixed DateTime value rather than one that is changing as the test runs:

 [TestFixture]
 public class UserServiceTests
 {
 private IUserService sut;
 private IUserData userData;
 // stub the system date as some arbirary date
 private readonly DateTime now = new DateTime(2015, 5, 28, 10, 46, 33);
 [SetUp]
 public void SetUp()
 {
 userData = MockRepository.GenerateStub<iuserdata>();
 sut = new UserService(userData, () => now);
 }
 [Test]
 public void UserServiceShouldCreateUserWithCorrectCreatedDate()
 {
 User user = null;
 userData.Stub(x => x.SaveUser(null)).IgnoreArguments().Callback<user>(x =>
 {
 user = x;
 return true;
 });
 sut.CreateUser("mike");
 Assert.AreEqual(now, user.CreatedDateTime);
 }
 }

And the test passes as expected.
In our composition root we inject the current system time (here as UTC):

 var userService = new UserService(userData, () => DateTime.UtcNow);

This pattern can be especially useful when we want to test business logic that relies on time passing. For example, say we want to check if an offer has expired; we can write unit tests for the case where the current (stubbed) time is both before and after the expiry time just by injecting different values into the system-under-test. Because we can stub the system time to be anything we want it to be, it makes it easy to test time based busines logic.

A Simple Nowin F# Example

noreply@blogger.com (Mike Hadlow) — 2015年4月22日 10:54:00 +0000

In my last post I showed a simple F# OWIN self hosted server without an application framework. Today I want to show an even simpler example that doesn’t reference any of the Microsoft OWIN libraries, but instead uses an open source server implementation, Nowin. Thanks to Damien Hickey for pointing me in the right direction.

The great thing about the Open Web Interface for .NET (OWIN) is that it is simply a specification. There is no OWIN library that you have to install to allow web servers, application frameworks and middlewear built to the OWIN standard to communicate. There is no interface that they must implement. They simply need to provide an entry point for the OWIN application delegate (better know as the AppFunc):

 Func<IDictionary<string , object>, Task>

For simple applications, where we don’t need routing, authentication, serialization, or an application framework, this means we can simply provide our own implementation of the AppFunc and pass it directly to an OWIN web server.

Nowin, by Boris Letocha, is a .NET web server, built directly against the standard .NET socket API. This means it should work on all platforms that support .NET without modification. The author claims that it has equivalent performance to NodeJS on Windows and can even match HttpListener. Although not ready for production, it makes a compelling implementation for simple test servers and stubs, which is how I intend to use it.

To use any OWIN web server with F#, we simply need to provide an AppFunc and since F# lambdas have an implicit cast to System.Func<..> we can simply provide the AppFunc in the form:

 fun (env: IDictionary<string, obj>) -> Task.FromResult(null) :> Task

Let’s see it in action. First create an F# console application and install the Nowin server with NuGet:

 Install-Package Nowin

Now we can host our Nowin server in the application’s entry point:

 [<entrypoint>]
 let main argv = 
 use server = 
 Nowin.ServerBuilder
 .New()
 .SetEndPoint(new IPEndPoint(IPAddress.Any, port))
 .SetOwinApp(fun env -> Task.FromResult(null) :> Task)
 .Build()
 server.Start() 
 printfn "Server listening on http://localhost:%i/ \nhit <enter> to stop." port
 Console.ReadLine() |> ignore
 0

Of course this server does nothing at all. It simply returns the default 200 OK response with no body. To do any useful work you need to read the OWIN environment, understand the request and create a response. To make this easier in F# I’ve created a simple OwinEnvironment type with just the properties I need. You could expand this to encompass whatever OWIN environment properties you need. Just look at the OWIN spec for this.

 type OwinEnvironment = {
 httpMethod: string;
 requestBody: Stream;
 responseBody: Stream;
 setResponseStatusCode: (int -> unit);
 setResponseReasonPhrase: (string -> unit)
 }

Here is a function that takes the AppFunc environment and maps it to my OwinEnvironment type:

 let getOwinEnvironment (env: IDictionary<string , obj>) = {
 httpMethod = env.["owin.RequestMethod"] :?> string;
 requestBody = env.["owin.RequestBody"] :?> Stream;
 responseBody = env.["owin.ResponseBody"] :?> Stream;
 setResponseStatusCode = 
 fun (statusCode: int) -> env.["owin.ResponseStatusCode"] <- statusCode
 setResponseReasonPhrase = 
 fun (reasonPhrase: string) -> env.["owin.ResponseReasonPhrase"] <- reasonPhrase
 }

Now that we have our strongly typed OwinEnvironment, we can grab the request stream and response stream and do some kind of mapping. Here is a function that does this. It also only accepts POST requests, but you could do whatever you like in the body. Note the transform function is where the work is done.

 let handleOwinEnvironment (owin: OwinEnvironment) : unit =
 use writer = new StreamWriter(owin.responseBody)
 match owin.httpMethod with
 | "POST" ->
 use reader = new StreamReader(owin.requestBody)
 writer.Write(transform(reader.ReadToEnd()))
 | _ ->
 owin.setResponseStatusCode 400
 owin.setResponseReasonPhrase "Bad Request"
 writer.Write("Only POST requests are allowed")

Just for completeness, here is a trivial transform example:

 let transform (request: string) : string =
 sprintf "%s transformed" request

Now we can re-visit our console Main function and pipe everything together:

 [<entrypoint>]
 let main argv = 
 use server = 
 Nowin.ServerBuilder
 .New()
 .SetEndPoint(new IPEndPoint(IPAddress.Any, port))
 .SetOwinApp(fun env -> 
 env 
 |> getOwinEnvironment 
 |> handleOwinEnvironment 
 |> endWithCompletedTask)
 .Build()
 server.Start() 
 printfn "Server listening on http://localhost:%i/ \nhit  to stop." port
 Console.ReadLine() |> ignore
 0

The endWithCompletedTask function, is a little convenience to hide the ugly synchronous Task return code:

 let endWithCompletedTask = fun x -> Task.FromResult(null) :> Task

So as you can see, OWIN and Nowin make it very easy to create small web servers with F#. Next time you just need a simple service stub or test server, consider doing something like this, rather that using a heavyweight server and application framework such as IIS, MVC, WebAPI or WebForms.

You can find the complete code for the example in this Gist https://gist.github.com/mikehadlow/c88e82ee98619f22f174:

Basic OWIN Self Host With F#

noreply@blogger.com (Mike Hadlow) — 2015年4月16日 11:49:00 +0000

I’m still very much an F# noob, but yesterday I thought I’d use it to write a little stub web service for a project I’m currently working on. I simply want to respond to any POST request to my service. I don’t need routing, or any other ‘web framework’ pieces. I just wanted to use the Microsoft.AspNet.WebApi.OwinSelfHost package to create a little web service that runs inside a console program.

First create a new F# console project. Then install the self host package:

 Microsoft.AspNet.WebApi.OwinSelfHost

Note that this will also install various WebApi pieces which we don’t need here, so we can go ahead and uninstall them:

 uninstall-package Microsoft.AspNet.WebApi.OwinSelfHost
 uninstall-package Microsoft.AspNet.WebApi.Owin
 uninstall-package Microsoft.AspNet.WebApi.Core
 uninstall-package Microsoft.AspNet.WebApi.Client

My requirement is to simply take any POST request to the service, take the post body and transform it in some way (that’s not important here), and then return the result in the response body.

So first, here’s a function that takes a string and returns a string:

 let transform (input: string) =
 sprintf "%s transformed" input

Next we’ll write the OWIN start-up class. This needs to be a class with a single member, Configuration, that takes an IAppBuilder:

 open Owin
 open Microsoft.Owin
 open System
 open System.IO
 open System.Threading.Tasks
 type public Startup() = 
 member x.Configuration (app:IAppBuilder) = app.Use( ... ) |> ignore

We need something to pass into the Use method on IAppBuilder. The Use method looks like this:

 public static IAppBuilder Use(
 this IAppBuilder app,
 Func<IOwinContext, Func<Task>, Task> handler
 )

So we need a handler with the signature Func<IOwinContext, Func<Task>, Task>. Since F# lambdas cast directly to Func<..> delegates, we simply use lots of type annotations and write a function which looks like this:

 let owinHandler = fun (context:IOwinContext) (_:Func) -> 
 handleOwinContext context; 
 Task.FromResult(null) :> Task

Note that this is running synchronously. We’re just returning a completed task.

Now lets look at the handleOwinContext function. This simply takes the IOwinContext, grabs the request, checks that it’s a ‘POST’, and transforms the request stream into the response stream using our transform function:

 let handleOwinContext (context:IOwinContext) =
 use writer = new StreamWriter(context.Response.Body)
 match context.Request.Method with
 | "POST" -> 
 use reader = new StreamReader(context.Request.Body)
 writer.Write(transform(reader.ReadToEnd()))
 | _ ->
 context.Response.StatusCode <- 400
 writer.Write("Only POST")

Now all we need to do is register our Startup type with the OWIN self host in our Program.Main function:

open System
open Microsoft.Owin.Hosting
[]
let main argv = 
 let baseAddress = "http://localhost:8888"
 use application = WebApp.Start<Startup.Startup>(baseAddress)
 Console.WriteLine("Server running on {0}", baseAddress)
 Console.WriteLine("hit <enter> to stop")
 Console.ReadLine() |> ignore
 0

And we’re done. Now let’s try it out with the excellent Postman client, just run the console app and send a POST request to http://localhost:8888/:

Full source code in this Gist.

The Lava Layer Anti-Pattern

noreply@blogger.com (Mike Hadlow) — 2014年12月15日 17:15:00 +0000

TL:DR Successive, well intentioned, changes to architecture and technology throughout the lifetime of an application can lead to a fragmented and hard to maintain code base. Sometimes it is better to favour consistent legacy technology over fragmentation.

An ‘anti-pattern’ describes a commonly encountered pathology or problem in software development. The Lava Layer (or Lava Flow) anti-pattern is well documented (here and here for example). It’s symptoms are a fragile and poorly understood codebase with a variety of different patterns and technologies used to solve the same problems in different places. I’ve seen this pattern many times in enterprise software. It’s especially prevalent in situations where the software is large, mission critical, long-lived and where there is high staff turn-over. In this post I want to show some of the ways that it occurs and how it’s often driven by a very human desire to improve the software.

To illustrate I’m going to tell a story about a fictional piece of software in a fictional organisation with made up characters, but closely based on real examples I’ve witnessed. In fact, if I’m honest, I’ve been several of these characters at different stages of my career. I’m going to concentrate on the data-access layer (DAL) technology and design to keep the story simple, but the general principles and scenario can and do apply to any part of the software stack.

Let’s set the scene...

The Royal Churchill is a large hospital in southern England. It has a sizable in-house software team that develop and maintain a suite of applications that support the hospital’s operations. One of these is WidgetFinder, a physical asset management application that is used to track the hospital’s large collection of physical assets; everything from beds to CT scanners. Development on WidgetFinder was started in 2005. The software team that wrote version 1 was lead by Laurence Martell, an developer with may years experience building client server systems based on VB/SQL Server. VB was in the process of being retired by Microsoft, so Laurence decided to build WidgetFinder with the relatively new ASP.NET platform. He read various Microsoft design guideline papers and a couple of books and decided to architect the DAL around the ADO.NET RecordSet. He and his team hand coded the DAL and exposed DataSets directly to the UI layer, as was demonstrated in the Microsoft sample applications. After seven months of development and testing, Version 1 of WidgetFinder was released and soon became central to the Royal Churchill’s operations. Indeed, several other systems, including auditing and financial applications, soon had code that directly accessed WidgetFinders database.

Like any successful enterprise application, a new list of requirements and extensions evolved and budget was assigned for version 2. Work started in 2008. Laurence had left and a new lead developer had been appointed. His name was Bruce Snider. Bruce came from a Java background and was critical of many of Laurence’s design choices. He was especially scornful of the use of DataSets: "an un-typed bag of data, just waiting for a runtime error with all those string indexed columns." Indeed WidgetFinder did seem to suffer from those kinds of errors. "We need a proper object-oriented model with C# classes representing tables, such as Asset and Location. We can code gen most of the DAL straight from the relational schema." He asked for time and budget to rewrite WidgetFinder from scratch, but this was rejected by the management. Why would they want to re-write a two year old application that was, as far as they were concerned, successfully doing its job? There was also the problem that many other systems relied on WidgetFinder’s database and they would need to be re-written too.

Bruce decided to write the new features of WidgetFinder using his OO/Code Gen approach and refactor any parts of the application that they had to touch as part of version 2. He was confident that in time his Code Gen DAL would eventually replace the hand crafted DataSet code. Version 2 was released a few months later. Simon, a new recruit on the team asked why some of the DAL was code generated, and some of it hand-coded. It was explained that there had been this guy called Lawrence who had no idea about software, but he was long gone.

A couple of years went by. Bruce moved on and was replaced by Ina Powers. The code gen system had somewhat broken down after Bruce had left. None of the remaining team really understood how it worked, so it was easier just to modify the code by hand. Ina found the code confusing and difficult to reason about. "Why are we hand-coding the DAL in this way? This code is so repetitive, it looks like it was written by an automation. Half of it uses DataSets and the other some half baked Active Record pattern. Who wrote this crap? If you hand code your DAL, you are stealing from your employer. The only sensible solution is an ORM. I recommend that we re-write the system using a proper domain model and NHibernate." Again the business rejected a rewrite. "No problem, we will adopt an evolutionary approach: write all the new code DDD/NHibernate style, and progressively refactor the existing code as we touch it." Many months later, Version 3 was released.

Mandy was a new hire. She’d listened to Ina’s description of how the application was architected around DDD with the data access handled by NHibernate, so she was surprised and confused to come across some code using DataSets. She asked Simon what to do. "Yeah, I think that code was written by some guy who was here before me. I don’t really know what it does. Best not to touch it in case something breaks."

Ina, frustrated by management who didn’t understand the difficulty of maintaining such horrible legacy applications, left for a start-up where she would be able to build software from scratch. She was replaced by Gordy Bannerman who had years of experience building large scale applications. The WidgetFinder users were complaining about it’s performance. Some of the pages took 30 seconds or more to appear. Looking at the code horrified him: Huge Linq statements generating hundreds of individual SQL requests, no wonder it was slow. Who wrote this crap? "ORMs are a horrible leaky abstraction with all kinds of performance problems. We should use a lightweight data-access technology like Dapper. Look at Stack-Overflow, they use it. They also use only static methods for performance, we should do the same." And so the cycle repeated itself. Version 4 was released a year later. It was buggier than the previous versions. Gordy had dismissed Ina’s love of unit testing. It’s hard to unit test code written mostly with static methods.

Mandy left to be replaced by Peter. Simon introduced him to the WidgetFinder code. "It’s not pretty. A lot of different things have been tried over the years and you’ll find several different ways of doing the same thing depending on where you look. I don’t argue, just get on with trawling through the never ending bug list. Hey, at least it’s a job."

This is a graphical representation of the DAL code over time. The Y-axis shows the version of the software. It starts with version one at the bottom and ends with version four at the top. The X-axis shows features, the older ones to the left and the newer ones to the right. Each technology choice is coloured differently. red is the hand-coded RecordSet DAL, blue the Active Record code gen, green DDD/NHibernate and Yellow is Dapper/Static methods.

Each new design and technology choice never completely replaced the one that went before. The application has archaeological layers revealing it’s history and the different technological fashions taken up successively by Laurence, Bruce, Ina and Gordy. If you look along the Version 4 line, you can see that there are four different ways of doing the same thing scattered throughout the code base.

Each successive lead developer acted in good faith. They genuinely wanted to improve the application and believed that they were using the best design and technology to solve the problem at hand. Each wanted to re-write the application rather than maintain it, but the business owners would not allow them the resources to do it. Why should they when there didn’t seem to be any rational business reason for doing so? High staff turnover exacerbated the problem. The design philosophy of each layer was not effectively communicated to the next generation of developers. There was no consistent architectural strategy. Without exposition or explanation, code standing alone needs a very sympathetic interpreter to understand its motivations.

So how should one mitigate against Lava Layer? How can we approach legacy application development in a way that keeps the code consistent and well architected? A first step would be a little self awareness.

We developers should recognise that we suffer from a number of quite harmful pathologies when dealing with legacy code:

We are highly (and often overly) critical of older patterns and technologies. "You’re not using a relational database?!? NoSQL is far far better!" "I can’t believe this uses XML! So verbose! JSON would have been a much better choice."
We think that the current shiny best way is the end of history; that it will never be superseded or seen to be suspect with hindsight.
We absolutely must ritually rubbish whoever came before us. Better still if they are no longer around to defend themselves. There’s a brilliant Dilbert cartoon for this.
We despise working on legacy code and will do almost anything to carve something greenfield out of an assignment, even if it makes no sense within the existing architecture.
Rather than try to understand legacy code, how it works and the motivations that created it, we throw up our hands in despair and declare that the whole thing needs to be rewritten.

If you find yourself suggesting a radical change to an existing application, especially if you use the argument that, "we will refactor it to the new pattern over time." Consider that you may never complete that refactoring, and think about what the application will look like with two different ways of doing the same thing. Will this aid those coming after you, or hinder them? What happens if your way turns out to be sub-optimal? Will replacing it be easy? Or would it have been better to leave the older, but more consistent code in place? Is WidgetFinder better for having four entirely separate ways of getting data from the database to the UI, or would it have been easier to understand and maintain with one? Try and have some sympathy and understanding for those who came before you. There was probably a good reason for why things were done the way they were. Be especially sympathetic to consistency, even if you don’t necessarily agree with the design or technology choices.

Hire Me

noreply@blogger.com (Mike Hadlow) — 2014年7月03日 14:22:00 +0000

I’m on a sales drive. I want to move away from daily-rate contracting, and focus on full-lifecycle project delivery. I’ve created a new website to help market myself http://mikehadlow.com/. I’m looking for customers who want software written to a specification for a fixed price.

I’ve been working in IT since 1996, although I’ve played with computers and programming since I was a teenager. Except for the first two years when I had a permanent job, I’ve worked as a daily or hourly rate contractor, with just the occasional foray into fixed-price project work. Looking at my CV I can count 17 different organizations that I’ve worked for during that time. Some of them where large companies where I was just a small part of a large team. For example, I was one of over a hundred contractors at one particular public sector project. Others were tiny local Brighton companies where I was often the end-to-end developer for a complete system. I’ve had a variety of roles, from being a travelling troubleshooter, driving around the country fixing installs of one particularly nasty system, a bug-fixer for months on end on a huge mission critical system, and a plug-n-play C# programmer on a whole range of different projects. More recently, for the last five years or so, I’ve mostly been hired in an ‘architect’ role. What this means is somewhat vague, but it usually encompasses giving higher-level strategic design direction and getting involved in team structure, process design, and planning. All this experience has given me some very strong opinions about what makes a successful software project. I hope that’s pretty obvious to anyone reading this blog. It’s also given me the confidence to take responsibility for the entire project lifecycle.

During this time I’ve also occasionally done fixed-price projects. The largest of these was a customer relationship management system for a pharmaceutical company, this was a six month project which I worked with a DBA to deliver. I’ve also built a property management system for a legal practice, and a complete eCommerce system that I’ve also maintained for the last six years. I always enjoyed these projects the most. It’s very satisfying to be able to deliver a working system to a client and see it really helping their business. The problem has been finding the work. I’m hoping now that the popularity of this blog and the success of EasyNetQ will provide enough of an audience for me that I’ll be able to do projects full time.

I want to move from being just an element of a project’s delivery, to being the person responsible for it. Taking responsibility means delivering to a price and time-scale and creating and managing the team that does it. So if you have a requirement for bespoke software and you need a safe pair of hands to deliver it, please get in touch with me at mike@suteki.co.uk and let’s talk.

Heisenberg Developers

noreply@blogger.com (Mike Hadlow) — 2014年6月06日 15:01:00 +0000

TL:DR You can not observe a developer without altering their behavior.

First a story.

Several years ago I worked on a largish project as one of a team of developers. We were building an internal system to support an existing business process. Initially things went very well. The user requirements were reasonably well defined and we worked effectively iterating on the backlog. We were mostly left to our own devices. We had a non-technical business owner and a number of potential users who gave us broad objectives, and who tested features as they became available. When we felt that piece needed refactoring, we spent the time to do it. When a pain point appeared in the software we changed the design to remove it. We didn’t have to ask permission to do any of things, so long features appeared at reasonable intervals, everyone was happy.

Then came that requirement. The one where you try to replace an expert user’s years of experience and intuition with software. What started out as a vague and wooly requirement, soon became a monster as we started to dig into it. We tried to push back against it, or at least get it scheduled for a later version of the software to be delivered at some unspecified time in future. But no, the business was insistent, they wanted it in the next version. A very clever colleague thought the problem could be solved with a custom DSL that would allow the users themselves to encode their business rules and he and another guy set to work building it. Several months later, he was still working on it. The business was frustrated by the lack of progress and the vaguely hoped for project delivery dates began to slip. It was all a bit of a mess.

The boss looked at this and decided that we were loose cannons and the ship needed tightening up. He hired a project manager with an excellent CV and a reputation for getting wayward software projects under control. He introduced us to ‘Jira’, a word that strikes fear into the soul of a developer. Now, rather than taking a high level requirement and simply delivering it at some point in the future, we would break the feature into finely grained tasks, estimate each of the tasks, then break the tasks into finer grained tasks if the estimate was more than a day’s work. Every two weeks we would have a day long planning meeting where these tasks were defined. We then spent the next 8 days working on the tasks and updating Jira with how long each one took. Our project manager would be displeased when tasks took longer than the estimate and would immediately assign one of the other team members to work with the original developer to hurry it along. We soon learned to add plenty of contingency to our estimates. We were delivery focused. Any request to refactor the software was met with disapproval, and our time was too finely managed to allow us refactor ‘under the radar’.

Then a strange thing started to happen. Everything slowed.

Of course we had no way to prove it because there was no data from ‘pre-PM’ to compare to ‘post-PM’, but there was a noticeable downward notch in the speed at which features were delivered. With his calculations showing that the project’s delivery date was slipping, our PM did the obvious thing and started hiring more developers, I think they were mostly people he’d worked with before. We, the existing team had very little say in who was hired, and it did seem that there was something of a cultural gap between us and the new guys. Whenever there was any debate about refactoring the code, or backing out of a problematic feature, the new guys would argue against it, saying it was ‘ivory tower’, and not delivering features. The PM would veto the work and side with the new guys.

We became somewhat de-motivated. After loosing an argument about how things should be done more than a few times, you start to have a pretty clear choice: knuckle down, don’t argue and get paid, or leave. Our best developer, the DSL guy, did leave, and the ones of us arguing for good design lost one of our main champions. I learnt to inflate my estimates, do what I was told to do, and to keep my imagination and creativity for my evening and weekend projects. I found it odd that few of my new colleagues seemed to actually enjoy software development, the talk in our office was now more about cars than programming languages. They actually seemed to like the finely grained management. As one explained to me, "you take the next item off the list, do the work, check it in, and you don’t have to worry about it." It relieved them of the responsibility to make difficult decisions, or take a strategic view.

The project was not a happy one. Features took longer and longer to be delivered. There always seemed to be a mounting number of bugs, few of which seemed to get fixed, even as the team grew. The business spent more and more money for fewer and fewer benefits.

Why did it all go so wrong?

Finely grained management of software developers is compelling to a business. Any organization craves control. We want to know what we are getting in return for those expensive developer salaries. We want to be able to accurately estimate the time taken to deliver a system in order to do an effective cost-benefit analysis and to give the business an accurate forecast of delivery. There’s also the hope that by building an accurate database of estimates verses actual effort, we can fine tune our estimation, and by analysis find efficiencies in the software development process.

The problem with this approach is that it fundamentally misunderstands the nature of software development. That it is a creative and experimental process. Software development is a complex system of multiple poorly understood feedback loops and interactions. It is an organic process of trial and error, false starts, experiments and monumental cock-ups. Numerous studies have shown that effective creative work is best done by motivated autonomous experts. As developers we need to be free to try things out, see how they evolve, back away from bad decisions, maybe try several different things before we find one that works. We don’t have hard numbers for why we want to try this or that, or why we want to stop in the middle of this task and throw away everything we’ve done. We can’t really justify all our decisions, many them are hunches, many of them are wrong.

If you ask me how long a feature is going to take, my honest answer is that I really have no idea. I may have a ball-park idea, but there’s a long-tail of lower-probability possibilities, that mean that I could easily be out by a factor of 10. What about the feature itself? Is it really such a good idea? I’m not just the implementer of this software, I’m a stake holder too. What if there’s a better way to address this business requirement? What if we discover a better way half way through the estimated time? What if I suddenly stumble on a technology or a technique that could make a big difference to the business? What if it’s not on the road map?

As soon as you ask a developer to tell you exactly what he’s going to do over the next 8 days (or worse weeks or months), you kill much of the creativity and serendipity. You may say that he is free to change the estimates or the tasks at any time, but he will still feel that he has to at least justify the changes. The more finely grained the tasks, the more you kill autonomy and creativity. No matter how much you say it doesn’t matter if he doesn’t meet his estimates, he’ll still feel bad about it. His response to being asked for estimates is twofold: first, he will learn to put in large contingencies, just in case one of those rabbit-holes crosses his path; second, he will look for the quick fix, the hack that just gets the job done. Damn technical debt, that’s for the next poor soul to deal with, I must meet my estimate. Good developers are used to doing necessary, but hard to justify work ‘under the radar’, they effectively lie to management about what they are really doing, but finely grained management makes it hard to steal the time in which to do it.

To be clear, I’m not speaking for everyone here. Not all developers dislike micromanagement. Some are more attracted to the paycheck than the art. For them, micromanagement can be very attractive. So long as you know how to work the system you can happily submit inflated estimates, just do what you’re told, and check in the feature. If users are unhappy and the system is buggy and late, you are not to blame, you just did what you were told.

Finely grained management is a recipe for ‘talent evaporation’. The people who live and breathe software will leave – they usually have few problems getting jobs elsewhere. The people who don’t like to take decisions and need an excuse, will stay. You will find yourself with a compliant team that meekly carries out your instructions, doesn’t argue about the utility of features, fills in Jira correctly, meets their estimates, and produces very poor quality software.

So how should one manage developers?

Simple: give them autonomy. It seems like a panacea, but finely grained management is poisonous for software development. It’s far better to give high level goals and allow your developers to meet them as they see fit. Sometimes they will fail; you need to build in contingency for this. But don’t react to failure by putting in more process and control. Work on building a great team that you can trust and that can contribute to success rather than employing rooms full of passive code monkeys.

JSON Web Tokens, OWIN, and AngularJS

noreply@blogger.com (Mike Hadlow) — 2014年4月29日 10:44:00 +0000

I’m working on an exciting new project at the moment. The main UI element is a management console built with AngularJS that communicates with a HTTP/JSON API built with NancyFX and hosted using the Katana OWIN self host. I’m quite new to this software stack, having spent the last three years buried in SOA and messaging, but so far it’s all been a joy to work with. AngularJS makes building single page applications so easy, even for a newbie like me, that it almost feels unfair. I love the dependency injection, templating and model binding, and the speed with which you can get up and running. On the server side, NancyFx is perfect for building HTTP/JSON APIs. I really like the design philosophy behind it. The built-in dependency injection, component oriented design, and convention-over-configuration, for example, is exactly how I like build software. OWIN is a huge breakthrough for C# web applications. Decoupling the web server from the web framework is something that should have happened a long time ago, and it’s really nice to finally say goodbye to ASP.NET.

Rather than using cookie based authentication, I’ve decided to go with JSON Web Tokens (JWT). This is a relatively new authorization standard that uses a signed token, transmitted in a request header, rather than the traditional ASP.NET cookie based authorization.

There are quite a few advantages to JWT:

Cross Domain API calls. Because it’s just a header rather than a cookie, you don’t have any of the cross-domain browser problems that you get with cookies. It makes implementing single-sign-on much easier because the app that issues the token doesn’t need to be in any way connected with the app that consumes it. They merely need to have access to the same shared secret encryption key.
No server affinity. Because the token contains all the necessary user identification, there’s no for shared server state – a call to a database or shared session store.
Simple to implement clients. It’s easy to consume the API from other servers, or mobile apps.

So how does it work? The JWT token is a simple string of three ‘.’ separated base 64 encoded values:

Here’s an example:

eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyIjoibWlrZSIsImV4cCI6MTIzNDU2Nzg5fQ.KG-ds05HT7kK8uGZcRemhnw3er_9brQSF1yB2xAwc_E

The header and payload are simple JSON strings. In the example above the header looks like this:

{ "typ": "JWT", "alg": "HMACSHA256" }

This is defined in the JWT standard. The ‘typ’ is always ‘JWT’, and the ‘alg’ is the hash algorithm used to sign the token (more on this later).

The payload can be any valid JSON, although the standard does define some keys that client and server libraries should respect:

{
 "user": "mike",
 "exp": 123456789
}

Here, ‘user’ is a key that I’ve defined, ‘exp’ is defined by the standard and is the expiration time of the token given as a UNIX time value. Being able to pass around any values that are useful to your application is a great benefit, although you obviously don’t want the token to get too large.

The payload is not encrypted, so you shouldn’t put sensitive information it in. The standard does provide an option for encrypting the JWT inside an encrypted wrapper, but for most applications that’s not necessary. In my case, an attacker could get the user of a session and the expiration time, but they wouldn’t be able to generate new tokens without the server side shared-secret.

The token is signed by taking the header and payload, base 64 encoding them, concatenating with ‘.’ and then generating a hash value using the given algorithm. The resulting byte array is also base 64 encoded and concatenated to produce the complete token. Here’s some code (taken from John Sheehan’s JWT project on GitHub) that generates a token. As you can see, it’s not at all complicated:

/// <summary>
/// Creates a JWT given a payload, the signing key, and the algorithm to use.
/// </summary>
/// <param name="payload">An arbitrary payload (must be serializable to JSON via <see cref="System.Web.Script.Serialization.JavaScriptSerializer"/>).</param>
/// <param name="key">The key bytes used to sign the token.</param>
/// <param name="algorithm">The hash algorithm to use.</param>
/// <returns>The generated JWT.</returns>
public static string Encode(object payload, byte[] key, JwtHashAlgorithm algorithm)
{
 var segments = new List<string>();
 var header = new { typ = "JWT", alg = algorithm.ToString() };

 byte[] headerBytes = Encoding.UTF8.GetBytes(jsonSerializer.Serialize(header));
 byte[] payloadBytes = Encoding.UTF8.GetBytes(jsonSerializer.Serialize(payload));

 segments.Add(Base64UrlEncode(headerBytes));
 segments.Add(Base64UrlEncode(payloadBytes));

 var stringToSign = string.Join(".", segments.ToArray());

 var bytesToSign = Encoding.UTF8.GetBytes(stringToSign);

 byte[] signature = HashAlgorithms[algorithm](key, bytesToSign);
 segments.Add(Base64UrlEncode(signature));

 return string.Join(".", segments.ToArray());
}

Implementing JWT authentication and authorization in NancyFx and AngularJS

There are two parts to this: first we need a login API, that takes a username (email in my case) and a password and returns a token, and secondly we need a piece of OWIN middleware that intercepts each request and checks that it has a valid token.

The login Nancy module is pretty straightforward. I took John Sheehan’s code and pasted it straight into my project with a few tweaks, so it was just a question of taking the email and password from the request, validating them against my user store, generating a token and returning it as the response. If the email/password doesn’t validate, I just return 401:

using System;
using System.Collections.Generic;
using Nancy;
using Nancy.ModelBinding;
using MyApp.Api.Authorization;

namespace MyApp.Api
{
 public class LoginModule : NancyModule
 {
 private readonly string secretKey;
 private readonly IUserService userService;

 public LoginModule (IUserService userService)
 {
 Preconditions.CheckNotNull (userService, "userService");
 this.userService = userService;

 Post ["/login/"] = _ => LoginHandler(this.Bind<LoginRequest>());

 secretKey = System.Configuration.ConfigurationManager.AppSettings ["SecretKey"];
 }

 public dynamic LoginHandler(LoginRequest loginRequest)
 {
 if (userService.IsValidUser (loginRequest.email, loginRequest.password)) {

 var payload = new Dictionary<string, object> {
 { "email", loginRequest.email },
 { "userId", 101 }
 };

 var token = JsonWebToken.Encode (payload, secretKey, JwtHashAlgorithm.HS256);

 return new JwtToken { Token = token };
 } else {
 return HttpStatusCode.Unauthorized;
 }
 }
 }

 public class JwtToken
 {
 public string Token { get; set; }
 }

 public class LoginRequest
 {
 public string email { get; set; }
 public string password { get; set; }
 }
}

On the AngularJS side, I have a controller that calls the LoginModule API. If the request is successful, it stores the token in the browser’s sessionStorage, it also decodes and stores the payload information in sessionStorage. To update the rest of the application, and allow other components to change state to show a logged in user, it sends an event (via $rootScope.$emit) and then redirects to the application’s root path. If the login request fails, it simply shows a message to inform the user:

myAppControllers.controller('LoginController', function ($scope, $http, $window, $location, $rootScope) {
 $scope.message = '';
 $scope.user = { email: '', password: '' };
 $scope.submit = function () {
 $http
 .post('/api/login', $scope.user)
 .success(function (data, status, headers, config) {
 $window.sessionStorage.token = data.token;
 var user = angular.fromJson($window.atob(data.token.split('.')[1]));
 $window.sessionStorage.email = user.email;
 $window.sessionStorage.userId = user.userId;
 $rootScope.$emit("LoginController.login");
 $location.path('/');
 })
 .error(function (data, status, headers, config) {
 // Erase the token if the user fails to login
 delete $window.sessionStorage.token;

 $scope.message = 'Error: Invalid email or password';
 });
 };
});

Now that we have the JWT token stored in the browser’s sessionStorage, we can use it to ‘sign’ each outgoing API request. To do this we create an interceptor for Angular’s http module. This does two things: on the outbound request it adds an Authorization header ‘Bearer <token>’ if the token is present. This will be decoded by our OWIN middleware to authorize each request. The interceptor also checks the response. If there’s a 401 (unauthorized) response, it simply bumps the user back to the login screen.

myApp.factory('authInterceptor', function ($rootScope, $q, $window, $location) {
 return {
 request: function (config) {
 config.headers = config.headers || {};
 if($window.sessionStorage.token) {
 config.headers.Authorization = 'Bearer ' + $window.sessionStorage.token;
 }
 return config;
 },
 responseError: function (response) {
 if(response.status === 401) {
 $location.path('/login');
 }
 return $q.reject(response);
 }
 };
});

myApp.config(function ($httpProvider) {
 $httpProvider.interceptors.push('authInterceptor');
});

The final piece is the OWIN middleware that intercepts each request to the API and validates the JWT token.

We want some parts of the API to be accessible without authorization, such as the login request and the API root, so we maintain a list of exceptions, currently this is just hard-coded, but it could be pulled from some configuration store. When the request comes in, we first check if the path matches any of the exception list items. If it doesn’t we check for the presence of an authorization token. If the token is not present, we cancel the request processing (by not calling the next AppFunc), and return a 401 status code. If we find a JWT token, we attempt to decode it. If the decode fails, we again cancel the request and return 401. If it succeeds, we add some OWIN keys for the ‘userId’ and ‘email’, so that they will be accessible to the rest of the application and allow processing to continue by running the next AppFunc.

using System;
using System.Collections.Generic;
using System.Threading.Tasks;

namespace MyApp.Api.Authorization
{
 using AppFunc = Func<IDictionary<string, object>, Task>;

 /// <summary>
 /// OWIN add-in module for JWT authorization.
 /// </summary>
 public class JwtOwinAuth
 {
 private readonly AppFunc next;
 private readonly string secretKey;
 private readonly HashSet<string> exceptions = new HashSet<string>{ 
 "/",
 "/login",
 "/login/"
 };

 public JwtOwinAuth (AppFunc next)
 {
 this.next = next;
 secretKey = System.Configuration.ConfigurationManager.AppSettings ["SecretKey"];
 }

 public Task Invoke(IDictionary<string, object> environment)
 {
 var path = environment ["owin.RequestPath"] as string;
 if (path == null) {
 throw new ApplicationException ("Invalid OWIN request. Expected owin.RequestPath, but not present.");
 }
 if (!exceptions.Contains(path)) {
 var headers = environment ["owin.RequestHeaders"] as IDictionary<string, string[]>;
 if (headers == null) {
 throw new ApplicationException ("Invalid OWIN request. Expected owin.RequestHeaders to be an IDictionary<string, string[]>.");
 }
 if (headers.ContainsKey ("Authorization")) {
 var token = GetTokenFromAuthorizationHeader (headers ["Authorization"]);
 try {
 var payload = JsonWebToken.DecodeToObject (token, secretKey) as Dictionary<string, object>;
 environment.Add("myapp.userId", (int)payload["userId"]);
 environment.Add("myapp.email", payload["email"].ToString());
 } catch (SignatureVerificationException) {
 return UnauthorizedResponse (environment);
 }
 } else {
 return UnauthorizedResponse (environment);
 }
 }
 return next (environment);
 }

 public string GetTokenFromAuthorizationHeader(string[] authorizationHeader)
 {
 if (authorizationHeader.Length == 0) {
 throw new ApplicationException ("Invalid authorization header. It must have at least one element");
 }
 var token = authorizationHeader [0].Split (' ') [1];
 return token;
 }

 public Task UnauthorizedResponse(IDictionary<string, object> environment)
 {
 environment ["owin.ResponseStatusCode"] = 401;
 return Task.FromResult (0);
 }
 }
}

So far this is all working very nicely. There are some important missing pieces. I haven’t implemented an expiry key in the JWT token, or expiration checking in the OWIN middleware. When the token expires, it would be nice if there was some algorithm that decides whether to simply issue a new token, or whether to require the user to sign-in again. Security dictates that tokens should expire relatively frequently, but we don’t want to inconvenience the user by asking them to constantly sign in.

JWT is a really nice way of authenticating HTTP/JSON web APIs. It’s definitely worth looking at if you’re building single page applications, or any API-first software.

A Contractor’s Guide To Recruitment Agencies

noreply@blogger.com (Mike Hadlow) — 2014年4月16日 13:53:00 +0000

I haven’t contracted through an agency for a long time, but I thought I’d write up my experiences from almost ten years of working as an IT contractor for anyone considering it as a career choice.

IT recruitment agencies provide a valuable service. Like any middle-man, their job is to bring buyers and sellers together. In this case the buyer is the end client, the company that needs a short-term resource to fill a current skills gap. The seller is you, the contactor offering the skill. The agency needs to do two things well: market intelligence - finding clients in need of resources and contractors looking to sell their skills; and negotiation – negotiating the highest price that the client will pay, and the lowest price that the contractor will work for. The agency’s income is a simple formula:

(client rate – contractor rate) * number of contractors placed.

Minimize the contractor rate, maximize the client rate, and place as many contractors as possible. That’s success.

Anyone with a phone can set themselves up as a recruitment agency. There are zero or low startup costs. The greatest difficultly most agencies face is finding clients. Finding contractors is a little easier, as I’ll explain presently. Having a good relationship with a large corporate or government client is a gold standard for any agency. Even better if that relationship is exclusive. Getting a foot in the door with one of these clients is very difficult, usually some long established, large agency has long ago stitched up a deal with someone high-up. But any company or organization in need of a contractor is a potential client, and agencies spend inordinate amounts of time in the search for names they can approach with potential candidates.

As I said before, finding contractors is somewhat easier. There are a number of well known websites, Jobserve is the most common one to use in the UK, so it’s merely a case of putting up a job description and waiting for the CVs to roll in. The agent will try to make the job sound as good as possible to maximize the chances of getting applications within the limits of the client’s job spec.

An ideal contractor for an agency is someone who the client wants to hire, and who is willing to work for the lowest possible rate, and who will keep the client happy by turning up every day and doing the work that the client expects. Since agencies take an on-going percentage of the daily rate, the longer the contract lasts the better. The agency will attempt to do some filtering to ‘add value’, but since few agencies have any real technology knowledge, this mainly consists of matching keywords and years-of-experience. Anyone with any experience of talking to agencies will know how frustrating it can be, "Do you know any ASPs?" "No, they don’t want .NET, they want C#." I’m not making those quotes up. Ideally they will want to persuade the client that they have some kind of exclusive arrangement with ‘their’ contractors and that the client would not be able to hire them through anyone else. It can be very embarrassing for them if the client receives your CV through a competing agency as well as theirs.

The job hunt. How you should approach it.

Let’s say you’re a competent C# developer, how should you approach landing your dream contract role? The obvious first place to look are the popular jobsites. Do a search for C# contracts in your local area, or further afield if you’re willing to travel. Scan the job listings looking for anything that looks like it vaguely fits. Don’t be too fussy at this stage, you want to increase your chances by applying for as many jobs as possible. Once you’ve got a list of jobs it’s worth trying to see if you can work out who the company is. If you can make a direct contract with the client, so much the better. Don’t worry about feeling underhand, agencies do this to each other all the time, it’s part of the game.

Failing a direct contact, the next step is to email your CV to the agency. Remember they’ll be trying to match keywords, so it’s worth customizing your CV to the job advert. Make sure as many keywords as possible match those in the advert, remembering of course that you might have to back up your claims in an interview.

The next step is usually a short telephone conversation with the recruiter. This call is the beginning of the negotiations with the recruiter. Negotiating is their full time job, they are usually very good at it. Be very wary. Your attitude is that you are a highly qualified professional who is somewhat interested in the role, but it’s by no means the only possibility at this stage. Whatever you do, don’t appear desperate. Remember at this stage you are an unknown quantity. Most contractors a recruiter comes into contact with will be duds (there’s no barriers to entry in our profession either), and they will initially be suspicious of you. Confidently assert that you have all the experience you mention in your CV, and that, of course, you can do the job. There is no point in getting into any technical discussion with the recruiter, they simply won’t understand. Remember: match keywords and experience. At this stage, even if you’ve got doubts about the job, don’t express them, just appear keen and confident.

Sometimes there’s a rate mentioned on the advert, at other times it will just say ‘market rates’, which is meaningless. If the agent doesn’t bring up rates at this point, there’s no need to mention them. At this stage you are still an unknown quantity. Once the client has decided that they really want you, you are gold, and in a much stronger bargaining position. If there’s a rate conversation at the pre interview stage, try to stay non-committal. If there’s a range, say you’ll only work for the top number.

They may ask you for references. Your reply should be to politely say that you only give references after an interview. It’s a common trick to put an imaginary job on a jobsite then ask applicants for references. Remember, an agency’s main difficulty is finding clients and the references are used as leads. If you give them references you will never hear from them again, but your previous clients will be hounded with phone calls.

Another common trick is to ask you where else you are applying. They are looking for leads again. Be very non-committal. They may also ask you for names of people you worked for at previous jobs, this is just like asking for references, you don’t need to tell them. Sometimes it’s worth have a list of made up names to give out if they’re very persistent.

Next you will either hear back from the agent with an offer of an interview, or you won’t hear from them at all. No agency I’ve ever had contact with bothered to call me giving a reason why an interview hadn’t materialized. If you don’t hear from them, move on with applying for the next job. Constantly calling the agency smacks of desperation and won’t get you anywhere. There are multiple possible reasons that the interview didn’t materialize, the most common being that the job didn’t exist in the first place (see above).

At all times be polite and professional with the agent even if you’re convinced they’re being liberal with the truth.

If you get an interview, that’s good. This isn’t a post about interviewing, so let’s just assume that you were wonderful and the client really wants you. You’ll know this because you’ll get a phone call from the agent congratulating you on getting the role. You are now a totally different quantity in the agent’s eyes, a successful candidate, a valuable commodity, a guaranteed income stream for as long as the contract lasts. Their main job now is to get you to work for as little as possible while the client pays as much as possible. If you agreed a rate before the interview, now is their chance to try and lower it. You may well have a conversation like this: "I’m very sorry John, but the client is not able to offer the rate we agreed, I’m afraid it will have to be XXX instead." Call their bluff. Your answer should be: "Oh that’s such a shame, I was really looking forward to working with them, but I my minimum rate is <whatever you initially agreed>. Never mind, it was nice doing business with you." I guarantee they will call you back the next day telling you how hard they have been working on your behalf to persuade the client to increase your rate.

If you haven’t already agreed a rate, now is the time to have a good idea of the minimum you want to work for. Add 30% to it. That’s your opening rate with the agent. They will choke and tell you there’s no way that you’ll get that. Ask them for their maximum and choke in return. Haggle back and forth until you discover what their maximum is. If it’s lower than your minimum, walk away. You may have to walk away and wait for them to phone you. Of course you’ve got to be somewhere in the ballpark of the market rate or you won’t get the role. Knowing the market rate is tricky, but a few conversations with your contractor mates should give you some idea.

Once the rate has been agreed and you start work your interests are aligned with the agent. You both want the contract to last and and you both want to maintain a good relationship with the client. The agency should pay you promptly. Don’t put up with late or missing payments, just leave. Usually a threat to walk off site can work wonders with outstanding invoices. Beware, at the worst some agents can be downright nasty and bullying. I’ve been told that would never work in IT again by at least two different characters. It’s nice to see how that turned out. Just ignore bullies, except to make a note that you will never work for their agency again.

Agencies are a necessary evil until you have built up a good enough network and reputation that you don’t need to use them any more. Some are professional and honest, many aren’t, but if you understand their motivations and treat anything they say with a pinch of salt, you should be fine.

A Docker ‘Hello World' With Mono

noreply@blogger.com (Mike Hadlow) — 2014年4月03日 13:22:00 +0000

Docker is a lightweight virtualization technology for Linux that promises to revolutionize the deployment and management of distributed applications. Rather than requiring a complete operating system, like a traditional virtual machine, Docker is built on top of Linux containers, a feature of the Linux kernel, that allows light-weight Docker containers to share a common kernel while isolating applications and their dependencies.

There’s a very good Docker SlideShare presentation here that explains the philosophy behind Docker using the analogy of standardized shipping containers. Interesting that the standard shipping container has done more to create our global economy than all the free-trade treaties and international agreements put together.

A Docker image is built from a script, called a ‘Dockerfile’. Each Dockerfile starts by declaring a parent image. This is very cool, because it means that you can build up your infrastructure from a layer of images, starting with general, platform images and then layering successively more application specific images on top. I’m going to demonstrate this by first building an image that provides a Mono development environment, and then creating a simple ‘Hello World’ console application image that runs on top of it.

Because the Dockerfiles are simple text files, you can keep them under source control and version your environment and dependencies alongside the actual source code of your software. This is a game changer for the deployment and management of distributed systems. Imagine developing an upgrade to your software that includes new versions of its dependencies, including pieces that we’ve traditionally considered the realm of the environment, and not something that you would normally put in your source repository, like the Mono version that the software runs on for example. You can script all these changes in your Dockerfile, test the new container on your local machine, then simply move the image to test and then production. The possibilities for vastly simplified deployment workflows are obvious.

Docker brings concerns that were previously the responsibility of an organization’s operations department and makes them a first class part of the software development lifecycle. Now your infrastructure can be maintained as source code, built as part of your CI cycle and continuously deployed, just like the software that runs inside it.

Docker also provides docker index, an online repository of docker images. Anyone can create an image and add it to the index and there are already images for almost any piece of infrastructure you can imagine. Say you want to use RabbitMQ, all you have to do is grab a handy RabbitMQ images such as https://index.docker.io/u/tutum/rabbitmq/ and run it like this:

docker run -d -p 5672:5672 -p 55672:55672 tutum/rabbitmq

The –p flag maps ports between the image and the host.

Let’s look at an example. I’m going to show you how to create a docker image for the Mono development environment and have it built and hosted on the docker index. Then I’m going to build a local docker image for a simple ‘hello world’ console application that I can run on my Ubuntu box.

First we need to create a Docker file for our Mono environment. I’m going to use the Mono debian packages from directhex. These are maintained by the official Debian/Ubuntu Mono team and are the recommended way of installing the latest Mono versions on Ubuntu.

Here’s the Dockerfile:

#DOCKER-VERSION 0.9.1
#
#VERSION 0.1
#
# monoxide mono-devel package on Ubuntu 13.10

FROM ubuntu:13.10
MAINTAINER Mike Hadlow <mike@suteki.co.uk>

RUN sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -q software-properties-common
RUN sudo add-apt-repository ppa:directhex/monoxide -y
RUN sudo apt-get update
RUN sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -q mono-devel

Notice the first line (after the comments) that reads, ‘FROM ubuntu:13.10’. This specifies the parent image for this Dockerfile. This is the official docker Ubuntu image from the index. When I build this Dockerfile, that image will be automatically downloaded and used as the starting point for my image.

But I don’t want to build this image locally. Docker provide a build server linked to the docker index. All you have to do is create a public GitHub repository containing your dockerfile, then link the repository to your profile on docker index. You can read the documentation for the details.

The GitHub repository for my Mono image is at https://github.com/mikehadlow/ubuntu-monoxide-mono-devel. Notice how the Docker file is in the root of the repository. That’s the default location, but you can have multiple files in sub-directories if you want to support many images from a single repository.

Now any time I push a change of my Dockerfile to GitHub, the docker build system will automatically build the image and update the docker index. You can see image listed here: https://index.docker.io/u/mikehadlow/ubuntu-monoxide-mono-devel/

I can now grab my image and run it interactively like this:

$ sudo docker pull mikehadlow/ubuntu-monoxide-mono-devel
Pulling repository mikehadlow/ubuntu-monoxide-mono-devel
f259e029fcdd: Download complete 
511136ea3c5a: Download complete 
1c7f181e78b9: Download complete 
9f676bd305a4: Download complete 
ce647670fde1: Download complete 
d6c54574173f: Download complete 
6bcad8583de3: Download complete 
e82d34a742ff: Download complete 

$ sudo docker run -i mikehadlow/ubuntu-monoxide-mono-devel /bin/bash
mono --version
Mono JIT compiler version 3.2.8 (Debian 3.2.8+dfsg-1~pre1)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
 TLS: __thread
 SIGSEGV: altstack
 Notifications: epoll
 Architecture: amd64
 Disabled: none
 Misc: softdebug 
 LLVM: supported, not enabled.
 GC: sgen
exit

Next let’s create a new local Dockerfile that compiles a simple ‘hello world’ program, and then runs it when we run the image. You can follow along with these steps. All you need is a Ubuntu machine with Docker installed.

First here’s our ‘hello world’, save this code in a file named hello.cs:

using System;

namespace Mike.MonoTest
{
 public class Program
 {
 public static void Main()
 {
 Console.WriteLine("Hello World");
 }
 }
}

Next we’ll create our Dockerfile. Copy this code into a file called ‘Dockerfile’:

#DOCKER-VERSION 0.9.1

FROM mikehadlow/ubuntu-monoxide-mono-devel

ADD . /src

RUN mcs /src/hello.cs
CMD ["mono", "/src/hello.exe"]

Once again, notice the ‘FROM’ line. This time we’re telling Docker to start with our mono image. The next line ‘ADD . /src’, tells Docker to copy the contents of the current directory (the one containing our Dockerfile) into a root directory named ‘src’ in the container. Now our hello.cs file is at /src/hello.cs in the container, so we can compile it with the mono C# compiler, mcs, which is the line ‘RUN mcs /src/hello.cs’. Now we will have the executable, hello.exe, in the src directory. The line ‘CMD ["mono", "/src/hello.exe"]’ tells Docker what we want to happen when the container is run: just execute our hello.exe program.

As an aside, this exercise highlights some questions around what best practice should be with Docker. We could have done this in several different ways. Should we build our software independently of the Docker build in some CI environment, or does it make sense to do it this way, with the Docker build as a step in our CI process? Do we want to rebuild our container for every commit to our software, or do we want the running container to pull the latest from our build output? Initially I’m quite attracted to the idea of building the image as part of the CI but I expect that we’ll have to wait a while for best practice to evolve.

Anyway, for now let’s manually build our image:

$ sudo docker build -t hello .
Uploading context 1.684 MB
Uploading context 
Step 0 : FROM mikehadlow/ubuntu-monoxide-mono-devel
 ---> f259e029fcdd
Step 1 : ADD . /src
 ---> 6075dee41003
Step 2 : RUN mcs /src/hello.cs
 ---> Running in 60a3582ab6a3
 ---> 0e102c1e4f26
Step 3 : CMD ["mono", "/src/hello.exe"]
 ---> Running in 3f75e540219a
 ---> 1150949428b2
Successfully built 1150949428b2
Removing intermediate container 88d2d28f12ab
Removing intermediate container 60a3582ab6a3
Removing intermediate container 3f75e540219a

You can see Docker executing each build step in turn and storing the intermediate result until the final image is created. Because we used the tag (-t) option and named our image ‘hello’, we can see it when we list all the docker images:

$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
hello latest 1150949428b2 10 seconds ago 396.4 MB
mikehadlow/ubuntu-monoxide-mono-devel latest f259e029fcdd 24 hours ago 394.7 MB
ubuntu 13.10 9f676bd305a4 8 weeks ago 178 MB
ubuntu saucy 9f676bd305a4 8 weeks ago 178 MB
...

Now let’s run our image. The first time we do this Docker will create a container and run it. Each subsequent run will reuse that container:

$ sudo docker run hello
Hello World

And that’s it.

Imagine that instead of our little hello.exe, this image contained our web application, or maybe a service in some distributed software. In order to deploy it, we’d simply ask Docker to run it on any server we like; development, test, production, or on many servers in a web farm. This is an incredibly powerful way of doing consistent repeatable deployments.

To reiterate, I think Docker is a game changer for large server side software. It’s one of the most exciting developments to have emerged this year and definitely worth your time to check out.

Docker: Bulk Remove Images and Containers

noreply@blogger.com (Mike Hadlow) — 2014年4月01日 10:24:00 +0000

I’ve just started looking at Docker. It’s a cool new technology that has the potential to make the management and deployment of distributed applications a great deal easier. I’d very much recommend checking it out. I’m especially interested in using it to deploy Mono applications because it promises to remove the hassle of deploying and maintaining the mono runtime on a multitude of Linux servers.

I’ve been playing around creating new images and containers and debugging my Dockerfile, and I’ve wound up with lots of temporary containers and images. It’s really tedious repeatedly running ‘docker rm’ and ‘docker rmi’, so I’ve knocked up a couple of bash commands to bulk delete images and containers.

Delete all containers:

sudo docker ps -a -q | xargs -n 1 -I {} sudo docker rm {}

Delete all un-tagged (or intermediate) images:

sudo docker rmi $( sudo docker images | grep '<none>' | tr -s ' ' | cut -d ' ' -f 3)