Another Chapter?

2008-09 – GeoEstimator is born/takes off.

05/2011 – Pictometry acquires GeoEstimator (press release)

01/2013 – Pictometry merges with EagleView (press release)

01/2014 – Verisk acquires Eagleview (press release)

……?????

Filed under:Jobs, Meanderings

Javascript Mathematics Libraries… GO!

I needed to look at the performance of multiple different Javascript Math libraries, specifically around Matrices. So, I’ve created numerous jsperf.com tests to compare them on different items. I’d originally started with 3 libraries:

Sylvesterjs: http://sylvester.jcoglan.com
Numericjs: http://www.numericjs.com/
Google Closure: http://closure-library.googlecode.com/svn/docs/class_goog_math_Matrix.html

After learning to use the libraries, comparing the performance, and just generally using them I parred it down to 2: Sylvester & Numeric. Sylvester is on the far OOP side of design, fluent interface, pretty good looking and readable code. Numeric is on the completely opposite end of the world with all static calls, no types just arrays, very functional looking, with code that is nearly IMPOSSIBLE to read. Google Closure on the other hand is the worst of both worlds. It’s OOP but the different libraries don’t play nice with each other (e.g. Vec3, Matrix) while it’s not easy to read the code and rather tedious in general. So, most of my later tests leave out gc.

Identity Matrix: http://jsperf.com/numeric-vs-sylvester-for-3d-matrix-operations/2
4×4 * 4×1 Multiplication: http://jsperf.com/numeric-vs-sylvester-vs-closure-matrix-multiplication
Single 3D Transformation: http://jsperf.com/sylvester-vs-numeric-vs-closure-for-3d-transformation/2
500 3D Rotates (T,R,T) w/ Unrolled: http://jsperf.com/numeric-vs-sylvester-vs-closure-for-unrolled
1000 3D Rotates (T,R,T) w/ Unrolled: http://jsperf.com/numeric-vs-sylvester-vs-closure-for-unrolled/2
1000 3D Rotates (T,R,T) w/ Unrolled not building Matrix: http://jsperf.com/numeric-vs-sylvester-vs-closure-for-unrolled/3
Row Access: http://jsperf.com/numeric-vs-sylvester-on-col-row-access
Col Access: http://jsperf.com/numeric-vs-sylvester-on-col-row-access/2
4×4 & 3×3 Inversion: http://jsperf.com/numeric-vs-sylvester-4×4-3×3-inversion
Equality: http://jsperf.com/sylvester-vs-numeric-vector-equality

When I got to the Row access, I thought things were going to be ‘similar’. MAN, were they not. Numeric kicked Sylvesters butt, like 100 to 1 style. I decided to look into it a bit and found that quite a bit happens on Matrix/Vector creation in Sylvester. Inputs are checked, arrays are copied/sliced/created, and all around good stuff when dealing with outside input. By simply removing all of this ‘stuff’ and simply creating a Vector and settings it’s elements to a known ‘good’ array from another matrix I was able to improve Sylvesters time by ~10x. That still left it years behind Numeric on row access so I dug deeper. Turns out it’s simple prototypal object creation that’s slowing Sylvester down. The vector constructor does absolutely nothign code-wise, simply instantiates an object. Simply putting ‘new Vector();’ in the numeric tests slowed down them down put them right on par with Sylvester. Leads me to believe that the speed of complex object creation is a big part of what is perceived as ‘slowness’ in Sylvester.

Simplified Constructor: http://jsperf.com/simplifying-sylvester-constructors

At the end of the day I’d recommend Numericjs for ‘raw performance’ and Sylvester for about everything else. Sylvester is easier to understand, has more implementation around common 3D constructs, is readable, and all round ‘nicer’. Numericjs cuts ALL cruft, is horrible to read, not nice at all, but is fast. Did I mention lots of variables named x,y,z (and not x, y, z coordinates) and functions named foo, dim, and other such worthless names? Did I mention that a LOT of the code is actually creating javascript on the fly with string concatenation?

Filed under:Coding, javascript, Tools

Defining ‘Technical Debt’, breaking down agile, and how it all fits together

Here are some links to the slidedeck and other items of interest in relation to my talk on May 16th, 2013 at the Indy Software Artisans meetup.

My Slidedeck:
Technical Debt & Craftsmanship

Slides for most of the speakers at the Agile Indy Conference:
http://www.agileindyconf.com/2013/04/agileindy-2013-slides/

Ward Cunninghams Correction on Technical Debt:
http://www.youtube.com/watch?v=pqeJFYwnkjE

Ron & Chet giving the same talk @ COHAA:
http://www.youtube.com/watch?v=phFSVZMoZ7E

Cynefin Framework overview:
http://www.youtube.com/watch?v=N7oz366X0-8

Filed under:agile, indysa, presentations

Monitoring MySQL replication with Nagios.

Fork me on GitHub
This isn’t a post about setting it all up. That would require quite a bit more space/time than I have as Nagios is fairly complex. I just wanted to let people know about some changes I’ve made to an existing package.

I found the check-mysql-all scripts on google code that already contained a number of nice scripts for monitoring with Nagios. What it didn’t support was the mk-heartbeat command in it’s check_maatkit command. In my github repo I’ve added this support. We’ll see if it ever makes it into the google code repo. Not having easy push/pull requests sucks! (maybe google code does if I used it?)

Handling Master/Slave replication servers in a Zend MVC app

Fork me on GitHub

Increasingly, our mysql db has been the bottleneck on performance in our application. After researching, looking around, weighing the options and all that jazz we decided to look at adding read-only slaves to our existing HA (drbd style) mysql server. When looking for solutions in Zend that would help to take advantage of this scenario I found a few projects out there but all they appeared to be were simple data structures to create, store, and retrieve Zend_Db_Adapters. This would require a code change everyplace the database is used to get either a master (update,insert,delete) or a slave (read).

I wanted plug and play, so I created the new Zend_Db_Adapter_Replication. It simply takes 2 adapters, anything implementing Zend_Db_Adapter_Abstract should do. From there it uses the correct connection depending on what operation you are running. If the update/insert/delete functions are called it passes them through to the master. If select is called it goes to slave. If the more generic query is called it looks at the query to determine which connection to use. If you start a transaction, it assumes you want master, and it forces everything else that happens inside that transaction to happen on the master.

This was enough for me to simply replace our current db in the Zend_Registry with the Replication db and everything ‘just worked’. You can watch munin/nagios/mysqladmin and the distribution looks perfect.

At the time of writing there’s no way to get an individual connect simply because I’ve not needed it. There’s also no way to force it one way or the other besides using transactions, again because I’ve not needed it.

So need it now? Where is it? Get the code on github!

Filed under:Coding, Databases, PHP

behat – BDD for PHP

I’ve looked at implementing acceptance testing style tests for our website with cucumber/ruby in the past but never pulled the trigger due to the fact that I live in a PHP world at work. Any ruby setup code that was written for the step definitions would most likely be duplicate code that we already have written and support in PHP.

Recently I became aware of a BDD framework, nearly identical to cucumber, for PHP named behat. It uses a PHP cloned gherkin parser so the features themselves are nearly identical to cucumber

Feature: User sessions
In order to access their account
As a user
I need to be able to log into and out of the website

Scenario: Login
Given I am on "/account/login"
And I should see "Login"
When I fill in "username" with "username"
And I fill in "password" with "passwd"
And I press "Login"
Then I should be on "/account/my"
And I should see "Hello Mark Garringer"

Behind the scenes the step definitions are similar. Due to language limitations the regular expressions for matching are in doc blocks on methods rather than method names but very workable. I’m not sure I’m a fan of the more inheritance based nature of their step definitions but it does the job and doesn’t get in the way.

Much like cucumber out of the box it doesn’t do web, but instead you use a driver that does web. In this case it is Mink (by the exact same people). Mink provides headless (javascript-less) browsers for quick tests as well as drivers for selenium and sahi for testing javascript enabled pages.

All in all, what testing I have done, has proved very promising. The documentation is pretty good although the ‘google factor’ seems fairly low. Finding examples and help outside of the website and project resources is fairly hard. I’ve implemented some rather in depth tests putting it through it’s paces and it has worked like a champ. It suffers from the same javascript timing issues that other such frameworks suffer from but it does what it says, and does it well.

Most importantly it can output in junit format so integrating it with our build server and getting feedback was a cinch!

Behat FTW!!

Filed under:Coding, PHP, TDD, Tools

Changing Default Column Values

A note to self so I remember when next I need this:

First, I don’t want to wade into the argument about where default values should, in the database or in the code (see this and this for example).

I left a default value off of a number of fields when I first created them and now I wish that I hadn’t. I found absolutely nothing in the guides on changing defaults and googling turned up nothing as well. Finally digging through the documentation on ActiveRecord migrations I turned up paydirt.

http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/Table.html#method-i-change_default

[ruby]t.change_default(:qualification, ‘new’)
t.change_default(:authorized, 1)[/ruby]

or a more complete example

[ruby]
def self.up
change_table users do |o|
o.change_default :active, true
end
end
[/ruby]

Filed under:Breadcrumb, Ruby

Starting Ruby on Rails: what platform…

Notice: I understand this is nearing religious boundaries, these are just my feelings from my experience..

I’ve been playing around with Ruby on Rails (RoR) for sometime now on the side, looking to get to know it so any ‘new’ projects at work can utilize it’s goodness. The reason for this could be a whole other post, but for time sake we’ll just say that for the last couple years on a PHP/Zend project I’ve found myself striving to implement things that RoR rails provides out of the box. The following is just my experience in learning RoR.

    Windows 7

In my playing I started out on my Windows 7 based desktop and the Windows 7 install on my dual-boot laptop as this was what I was in most of the time for work. My dev environment was pretty simple:

  • Windows
  • Emacs 23.?
  • Multiple modes, flymake, ecb, other emacs goodness
  • git bash (as a terminal)
  • Ruby 1.9.2 w/Devkit (rubyinstsaller.org)
  • Rails 3 w/ various gems

I will say that it was easy to setup on Windows, no problems there. Some issues that I accounted to Windows were actually issues with most tutorials still being Rails2 at the time and the inclusion of bundler and the need for bundle exec in Rails3.

The reason I would NOT recommend Windows for Rails development is SPEED. If I’d gotten into more depth as far as gems and such it sounds like there might have been compatibility issues as well but I stopped well short of that due to speed. Running 10 unit tests (rspec) would take as much as 30 seconds OR MORE on Windows. Those same tests on other platforms would take less 5 seconds or less depending on circumstances. Starting console or server was just as painfully slow. I’m running an i7 quad core with 6gb of RAM, so it’s not like it was the machine lacking; it’s the most powerful of all 3 machines.

The other negative was no RVM on Windows. I don’t jump ruby versions much but having separate gemsets has turned out to be awesome. I’ve since heard of a windows copy called pik, but I can’t vouch for it.

    OSX

I also have a mac mini at my desk for previous projects I’ve done for iPhone apps. Setting up my ‘standard’ environment in OSX was actually somewhat of a problem. Turns out that a number of the tools I use are baked into OSX and either upgrading them or replacing them is near impossible. Luckily, as it seems OSX is the platform of choice for RoR development, most of these problems have been overcome. One such tool is RVM. It’s an awesome tool in itself but it’s almost required on OSX if you want to use anything beyond the OSX baked in ruby 1.8.7 from 2009.

Once up and running OSX performed VERY nicely, especially after coming off of the very slow Windows. Having a better command line and more/better tools there is a definite plus as well.

Another negative that I’ve since found deals with the MacBook Pro and it’s keyboard layout. I setup my dev environment on my wife’s MBP and I find it very trying. If you’re an emacs user Ctrl & Alt are more important than the home row to you and your coding. Well, on the MBP, they put the FN key out there where every other keyboard I’ve used puts the Ctrl key and it is MESSING with me. I’ve tried getting used to it but can’t. Fortunately I think you can remap the keyboard, if I could only talk my wife into it…

    Ubuntu

My laptop has 2 hdd’s and currently it’s setup to dual boot between Windows 7 on one drive and Ubuntu Linux on the other drive. I think this is going to change to be a full Ubuntu install with a VM of windows on the second drive very soon. Setup was much the same as on windows: Emacs, Rails 3, Rails 1.9, etc. The only addition was RVM as with the OSX.

I’d definitely recommend Ubuntu (or linux) for a development environment for Ruby. Easy to install and setup. Very speedy (compared to windows) for running tests, console, and server. Linux in general can have some problems from time to time. Being all open, sometimes things just don’t work as nicely together as you’d want. But it tends to give you may more flexibility.

    Conclusion
  1. Ubuntu (Linux)
  2. OSX
  3. Windows

Windows would be the one ‘do not recommend’. It’s doable but painful! The other two have there problems but Ubuntu wins for two reasons: I like Linux, and it’s almost certainly what your app is going to be running on in production anyways! Also, OSX has a rather high barrier to entry (cost), while Ubuntu has become almost stupidly simple to install and the inclusion of an ‘App Store’ makes getting all the big stuff together a breeze. Plus you can try it out inside a VM (like VirtualBox) in like an hour or two for free.

Granted, my post is based off a number of assumptions like using Emacs for an IDE. If I was a TextMate user I’d almost be forced to use OSX. I’d guess there are F/OSS editors that have tried to mimic TextMate you could get on Linux but TextMate itself is OSX.

From Left to Right: Mac Mini (2 screens), Windows 7 desktop, Dual-Boot laptop

But you don’t have to take my word for it!

Mark

Filed under:Coding, Meanderings, Ruby

Never implement INotifyPropertyChanged again

I hate every time I am working on something and I have to implement INotifyPropertyChanged.  My DRY-dey sense tingles.  Not only am I forced to not use auto-properties (1st DRY violation), I’m forced to fire the event in each setter (2nd DRY violation), and specify the name of the property that is getting set, inside of that property’s setter (3rd DRY violation).  That much WET (read: not-DRY), for something so simple leaves me a little grumpy.

I’ve been on this quest before, to simplify this a bit, but it was still a little hackety, and limiting.

This time, I set out to do it right.

I’ll spare you most of the technical details, but it’s backed by Castle’s DynamicProxy project, and there’s some integration with StructureMap to make it super easy, though you don’t really have to use StructureMap if you don’t want to.  [note:  I’ll probably add more container support as I find time.  If you have a specific need, let me know, or submit a patch.]

Here are the codez to show it in action:

Basics

Using it for a class with an interface

// note the attribute goes on the interface, not the class

[AutoNotify]

public interface IFoo

{

    string Value { get; set; }

}

 

public class Foo : IFoo

{

    public string Value { get; set; }

}

 

Using it for a class

[AutoNotify(Fire = FireOptions.OnlyOnChange)]

public class Foo

{

    // note for autonotify to work, the property must be virtual

    public virtual string Value { get; set; }

}

 

The previous example shows how to get the event to fire only when the value is different also.  It defaults to always firing, whether the value changes or not.  It’s also important to note that your properties need to be virtual so the calls to the setter can be intercepted.

Dependent Properties

Sometimes (usually) you’ve got calculated properties that need to fire the notified event too, these usually turn into WET mess as well.  We’ve got the problem solved, and you’ve got a few different options, hopefully one of them suits you.

Dependency Map – DependsOn

You specify the type that defines the DependencyMap on the attribute, and then set up your dependencies in that type’s constructor.  This style is somewhat influenced by the FluentNHibernate API.

[AutoNotify(DependencyMap = typeof(ProjectDependency))]

public class Project

{

    public virtual string Name { get; set; }

    public virtual string[] Files { get; set; }

    public virtual int FileCount { get { return Files.Length; } }

}

 

class ProjectDependency : DependencyMap<Project>

{

    public ProjectDependency()

    {

        Property(x => x.FileCount).DependsOn(x => x.Files);

    }

}

 

Dependency Map – Updates

If you’d rather express your dependency the other way around, that’s fine too.  The two are equivalent.

[AutoNotify(DependencyMap = typeof(ProjectDependency))]

public class Project

{

    public virtual string Name { get; set; }

    public virtual string[] Files { get; set; }

    public virtual int FileCount { get { return Files.Length; } }

}

 

class ProjectDependency : DependencyMap<Project>

{

    public ProjectDependency()

    {

        Property(x => x.Files).Updates(x => x.FileCount);

    }

}

 

Dependency Map – UpdatesWith

If you want to stick with an auto-property, and leave the calculated property logic somewhere else, you can hook it in via your dependency map too.  This example, again, is equivalent to the previous two.

[AutoNotify(DependencyMap = typeof(ProjectDependency))]

public class Project

{

    public virtual string Name { get; set; }

    public virtual string[] Files { get; set; }

    public virtual int FileCount { get; set; }

}

 

class ProjectDependency : DependencyMap<Project>

{

    public ProjectDependency()

    {

        Property(x => x.Files).Updates(x => x.FileCount).With(p => p.Files.Length);

    }

}

 

DependsOn Attribute

If you don’t like any of those options and are looking for something a little more simple, maybe you’ll like this one.  Just specify which things your property depends on in an attribute.  You lose your 100% static typing help, but it’s more concise.

[AutoNotify]

public class Project

{

    public virtual string Name { get; set; }

    public virtual string[] Files { get; set; }

 

    [DependsOn("Files")]

    public virtual int FileCount { get { return Files.Length; } }

}

 

Containers and otherwise

Hooking it into StructureMap

There are a couple conventions you can use to hook into StructureMap.  There is the attribute convention (which is what you’re seeing above), and there is a generic predicate convention that you can use any predicate logic.  Below you can see the attribute one getting hooked in.

var container = new Container(config => config.Scan(scanConfig =>

{

    scanConfig.With(new AutoNotifyAttrConvention());

    scanConfig.TheCallingAssembly();

    scanConfig.WithDefaultConventions();

}));

 

var project = container.GetInstance<Project>();

Using it without StructureMap

If you’re using another container, or no container at all, but want to use some other factory or something, you can do that too.  This example is for something with an interface.  It’s very similar to do the same for a concrete class… you just don’t instantiate the object first.  You also have an opportunity to hook into the dependent property structure here as well with the DependencyMap parameter.

var foo = new Foo();

var notifiableFoo = Notifiable.MakeForInterface(

    typeof(IFoo),

    foo,

    FireOptions.Always,

    new ProxyGenerator(),

    DependencyMap.Empty);

 

Assert.That(notifiableFoo is INotifyPropertyChanged);

 

Whew, done

So… that’s a lot of ‘how to’, but hopefully it’ll be somewhat complete introduction to get you working with it.  I really don’t see much of any reason to ever implement INotifyPropertyChanged ever again (unless you are in an environment where you can’t use DynamicProxy).  It can automatically be done for you from now on.

The code is up on github, and there is a gem up on rubygems if you’re using nu or noodle+bundler.  Fork it, send me a patch, use it, send feedback, etc.  I hope you love it!

Syntax highlighting for technical presentations

Ever wanted a quick/easy/automated way to get syntax highlighted code from your editor of choice into PowerPoint?

EVERY time I do a technical presentation I need this.  Usually I resort to taking a screenshot, or finding an “export to html” type plugin for the editor I’m using at the time (Visual Studio, Vim, IntelliJ IDEA, etc.) and then try to get that somehow into my slides.

The problem I usually run into, is that while I like coding on a dark background with a 14pt font, that’s not usually that great for presentations.  So I switch to my “presentation settings” that has a light background and bigger font size, and then switch back once I’m done taking screenshots or outputting to html. [ugh, what a pain… terribly manual].

Then what happens when I spot a bug in code that’s in PowerPoint, or I want to add a comment, or I need to change the syntax highlighting theme of the entire presentation because it doesn’t go well with the lighting in the room.  UGH!  What a disaster.

Basically, it’s always an uphill struggle, and it really bothered me… so I decided to fix it once and for all.

Solution

I’m going to solely discuss PowerPoint (for Windows or Mac) here.  I don’t have a copy of iWork, and I’m too cheap to buy one.  The reason I have Office for the Mac, is because I got it for free… KeyNote/Mac zealots: feel free to be zealous, but don’t hate on me because I’m frugal… unless you want to buy me a copy of iWork.

First, download: http://www.andre-simon.de/doku/highlight/en/highlight.html

Windows Solution

The highlight tool can output many formats, but the important one for Windows is RTF.

So something like:

highlight < infile > outfile –-rtf --syntax=rb  --style=vim --font=”Lucida Console” --font-size=18

This will  take the source code file ‘infile’ and syntax color it as Ruby to an RTF file and output it to ‘outfile’.  The text will be 18pt Lucida Console and syntax highlighted with the “vim” color scheme.

There are lots of themes included, you can ask highlight for help (highlight --help), and it’ll tell you all the options available, as well as all the options available for output, and for languages that it supports.

Next, in PowerPoint, do “Insert Object” on your target slide, and choose “Create from file”.  Make sure you check the “Link” checkbox before pressing OK.

Now, whenever your source changes, re-run your command line, then you can choose “Update Link” on your embedded object in PowerPoint, or if you close and then re-open PowerPoint, it’ll give you the option to update all your links.

Mac Solution

My version of PowerPoint for Mac doesn’t like being able to link to RTF files, but it does allow linking to images.

Highlight can output to SVG, but my PowerPoint doesn’t like that either.

Now download inkscape, if you haven’t already.  It will allow you to rasterize the SVG into a PNG, which PowerPoint does like.  You can export your PNG from inkscape from the command line like this:

<path to inkscape> –export-png=<png file> <svg file>

The path to my inkscape command line runner is at:

/Applications/Inkscape.app/Contents/Resources/bin/inkscape

It turns out that inkscape doesn’t like external CSS files, which is what highlight gives you with your SVG, so you can merge your CSS file into your SVG file with a little script.  The other cool bit about SVG, is you can tweak it, if you want, since it’s just XML.

Now, in PowerPoint, choose “Insert Picture” and browse to your newly generated PNG.  Make sure the “Link to File” checkbox is checked.

I wrote a script to do this for the latest presentation I did on dependency injection.  You can take a look at the script here [source.rb].   It’s Ruby, and you can see where I merge the CSS and SVG files together, and where I mess with the line spacing as well (I wasn’t happy with the default line spacing, so I tweaked it to my preference).  All the source/images/PowerPoint for that presentation are available here [dependencies presentation] if you want to check them out to see how the whole process works.

Now, when I need to change the font and syntax theme throughout my entire presentation because it doesn’t match the lighting in the room, it’s super-simple: I change the configuration, re-run my script, update PowerPoint, and chill.

Filed under:dev, presenting, Ruby