Sam Merrell Tinkerer. Parent. ADHD. Developer.


Automate project suckage away

How much tedious work is involved in setting up and developing individual projects at your work? In my first post I wrote about how much tedious work is involved in setting up a new machine, now let’s dig a little deeper to the project level.

Automation at the project level is very much in line with Zach Holman’s talk on automating machine setup. In his talk he mentions that each project has it’s own setup scripts to get things up and running quickly. While working on a project at work, I’ve come to realize how important automation is inside of individual projects as well. Automating processes inside a project can help in many ways. First, it makes us developers happy. We no longer have to keep doing the same boring thing over and over, and second it means the tedious task is done consistently and without error instead of by an error-prone manual process.

Windows Workflow Foundation…

The project at work that my team and I have been working on is called the IVR. It is the first thing a retiree interacts with when they call into our call center. Unfortunately the project for the IVR uses Windows Workflow Foundation which is… terrible to work with. Since it is a Microsoft product, the decision was to make the configuration and designing of the workflow using a GUI. Through that GUI the workflows are generated in a relatively unflattering XML representation. These are the .xoml files and the xoml rules files.

To allow for dynamically changing how the call flow works, we’ve loaded the xoml files and the grammars into the database. The xoml files control the overall flow of someone calling in and the grammars are used to recognize phrases or dial tones the person calling in says or dials (i.e. Say foo to go to bar or press 1). This works well for us having these files loaded into the database since we get a lot of flexibility in changing the IVR without redeploying. Unfortunately for us, Windows Workflow Foundation’s editor of the xoml files screws them up if you open them. The problem is in declaring what assembly the worfklows use. It adds a specific version of the assemblies for the xoml files and doesn’t fully declare the right namespace for the rules file.

This makes it extremely tedious to modify any of our workflows because you have one of two options. Either you just don’t use the editor and modify the xml directly or you deal with the changes the editor makes and you go back and fix it later. Either way really sucks since it involves so much tedious work. Fortunately for us, my coworker Bob got tired of fixing the files over and over by hand.

Rake to the rescue

What Bob did was to make a couple rake tasks that go through every xoml file and rule file and uses a regex to remove the specific assembly versions, and add the missing declarations from the rule files. This has saved us so much time and headache from deploying the new changes and having it crash because it was looking for a specific assembly version.

Not only that, but he added a rake task to add the new xoml and rule file declarations into a sql script so we can easily run the script and insert it into the database saving us even more time. The great thing about this is that it has saved us a significant amount of time and greatly reduced our frustration with the project.

Automation FTW

Even just these little bits of automation have significantly improved the ease of working on the project. We’ve already talked about next steps on making things even easier to use. The biggest improvement being a way to automatically deploy the sql scripts we’ve generated and deploy the dlls to our dev server. This experience has really shown me how much nicer development can be with seemingly simple automation tasks to reduce the tedious, manual work that can be involved with some projects.

Shell Scripting and Git Fun

This afternoon I got the urge to whip up a quick script to help me when using git and a GitHub hosted repository. Quite often I want to look at the GitHub page of a project I’m working on when I’m sitting on the command line in the repository. So I figured, how could I go about making it easier to open that URL without having to find the link in Chrome or on GitHub’s site. Shell scriping to the rescue!

I realized with a little bit of text manipulation, I could get the git remote show command to help me out. So I took the output

$ git remote show origin
* remote origin
  Fetch URL:
  Push  URL:
  HEAD branch: master
  Remote branches:
    gh-pages tracked
    master   tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':

And grepped it on Fetch URL leaving me with Fetch URL: From there I cut the first 14 characters using cut -c 14- leaving me with just the git url I was looking for. With just the URL left, I got working on crafting a Sed command to manipulate the URL to what I needed, coming up with this:

sed -r -e 's/\w+@/http:\/\//' -e 's/\.git$//'
-e 's/\.com:/.com\//' -e's/git:/http:/'

This command just replaces the git@ part with http instead, I then stripped off the .git and .com: with .com/. producing the desired Awesome!

Once I had the URL I wanted, it became pretty simple to open up the url using my default browser. I simply call start and gave it the url.

OSX Support

After getting this all working on my Windows box at work, I had to see if this would work on OS X. Being confident in my skills I just copied the file over to my Mac, dropped it in the bin and happily tried to run it… Until it blew up on me.

First, I had to remember that on Windows you use start but on OS X you need to use open. That was easy enough though, I just added a quick check to see if I’m on OS X or Windows and set a variable opencmd to the right thing. But then the painful part hit me.

This is when I learned the painful difference between Gnu-based tools and BSD tools. Sed was very unhappy with the parameters I was trying to pass into it. Specifically the -r parameter. The script above uses extended regex to get the special characters of \w and +. Unfortunately BSD Sed uses the -E flag while Gnu Sed uses the -r flag.

Fortunately this is pretty easy to work around as I store the extendedRegex flag in my script and set it depending on the OS I’m on.

The most annoying difference between Gnu Sed and BSD sed though, is the support of Regular Expression Metacharacters. In BSD sed, you do not get the metacharacters, so \w must become [[:space:]]. So now I know a painful difference between Gnu Sed regex extensions and BSD regex extensions. After I replaced the metacharacters I was using with their full version, I ran the script again and it worked! So I now have an easy way to open a GitHub repo from the command line.

I still have many imrovements I could make to this script, but I’m pretty happy with how it turned out after putting about a half hour of work into it. Below is the file in its entirety and you can find it on GitHub as well. Feel free to borrow the script and use it if you find it useful. I’d love to get some feedback on ways to improve what I have so far.

Chocolatey machine setup

How well automated is your machine setup process? After watching Zach Holman’s excellent video Unsucking Your Teams Development Environment I got to thinking about the setup process we have at Extend Health (who is hiring, come work with us!). While we’ve always talked about smoothing out the process of setting up a new machine, we’ve never actually gotten around to it. That is, until now.

For GitHub, they use what they call The Setup which is a Ruby script that uses Puppet to manage all their machines. When I had first looked at Puppet a couple years ago, Windows support was not an option. It appears that this has changed. While we investigate using Puppet, my friend Kyle and I have begun setting up a system that uses Chocolatey and straight PowerShell to set up our Windows boxes.

So what exactly is Chocolatey? Chocolatey is a package manager designed for Windows and is modeled somewhat like apt-get for Linux. Essentially it uses the NuGet infrastructure to deliver software such as Vim or Skype. This makes Chocolatey an attractive option because like NuGet, you can host your own feed of software that can be installed on your machine.

Once we decided we’d go with Chocolatey we needed to set up a test base. So we got ourselves set up with a vanilla Windows 7 VM. This works great since we can play around with setting up a brand new Windows install and then restore when we want to try it again. This also helped expose one of our first problems in setting up a new machine. Chocolatey requires .NET 4.0 to be installed before it will work.

While I’m admittedly not the biggest fan of PowerShell, when it comes to scripting Windows to do what you want, it can definitely come in handy. Kyle came across a very handy PowerShell module called PSWindowsUpdate. Once we got that script working and how to continue our script using elevated permissions we were off. Here’s the line we got directly from the PSWindows update example.

Get-WUInstall -ServiceID 9482f4b4-e343-43b6-b170-9a65bc822c77 -Title ".NET Framework 4" -AcceptAll

The last thing we were able to do this Friday was get PSWindows update to install .NET 4.0 and Chocolatey! Exciting news for us because now we should be able to take a brand new machine and install everything we need to write code at Work.

So where do we take it from here? We’ve got many plans for our own setup script but the first step is to make the initial setup from a new machine to getting the right software installed. We’ve gotten .NET 4.0 out of the way but now we need to see what other snags we may run into. We also need to see how we can get Visual Studio 2010 and SQL Server 2012 installed. From there we’d also like to start making profiles so QA and our DB teams can also plug into what we’re building.

Longer term, we’d also like to keep investigating what Puppet may be able to provide and start trying out options there, but for now this is a great step forward in automating the suck out of our setup process.