Rob Conery's Blog: Riding The Geek Rewrite Wave

Hanalei, Hawaii 9/2/2010

438 Posts and Counting

Riding The Geek Rewrite Wave

Monday, January 21, 2008 - On the plane tonight (I'm in Redmond this week) I got to thinking about writing "Maintainable Code" and what it means in light of our fad-centric industry which churns itself every 18 months or so with a new language or platform version. We all inherit projects and are confronted with Geek Lust to rewrite everything in our own image - but what is the motivator for you when you do this? My question to you is: how often do you justify a rewrite (or upgrade) with a platform rev or language change? Good Clients Up until a little over a year ago, I'd had the same client for the close to 9 years: KLA-Tencor. It all started when I wrote a small ASP-classic site for them in 1997 that evolved into a pretty robust Commerce/Fulfillment app in late 2005. You might think this is crazy, but after the initial rev, I rewrote the site 3 times:

First Rev: 1997 (basic data-driven brochure site
Second Rev: 1999. ECommerce application and backend sales-support
Third Rev: 2002: Upgraded site to .NET 1.1 and added web-service support
Fourth Rev: 2004: Upgraded site to .NET 2.0 and refactored just about everything.

Four revs in 9 years is just about average for the industry (from what I've seen), and thinking on this really made me wonder if most developers consider platform rev/technology shift (and that of the project their working on) when architecting their site? I Am A Geek. And So Can You. We build, we change, we look for better ways, and once found, said better ways are hoisted up on the shoulders of the masses and paraded around the blogosphere. We're fad-prone (like any industry) and one minute we might be throwing around words like "Paradigm Shift" and "Service Oriented Architecture" and the next we throw in the words "Separation of Concerns" and "Loosely Coupled" to spice up the conversation. I had a good giggle the other day talking to Mr. TDD Himself (Steve Harman) on Skype, and the conversation was pretty damn funny. Steve left a comment on my blog about the new stuff we're doing with SubSonic:

Rob, will there be a testing story for the Repositories? That is. how would I go about mocking them out for unit tests so as to avoid hitting the db?

It's a good question. My reply wasn't so professional:

@Steve: if you like TDD so much why don't you just marry it :p

You have to understand that I love Steve. No, I LERRRV Steve. In fact I love him so much I stole him from right out of Haack's nose and made him a main committer for the new SubSonic MVC stuff (Steve is Phil's right-hand with SubText). Booya! Anyway - I made sure to follow up the conversation with Steve on Skype to get his ideas, and I asked him what he thought we should do to make some of our new stuff more testable (we have a whole mess of non-virtual, non-interfacable static methods as part of our Data Factory which are not easy to test). He came up with a great plan (paraphrased, geekiness his):

You could crank up an IoC containter and use Dependency Injection...

My reply was my usual not-so-professional-smart-assiness:

Ooooh! I like that idea! And look at all the buzzwords I can blog about!

I hope you don't read this as me being snarky - I couldn't help myself. Steve is such a good guy and he's shown me some great tricks regarding TDD and tools. But see, when you're nice to me, my claws come out and I rip your best shirt while coughing up stuff on your best shoes. I can't help it... it's my nature. When Do You Tell A Client To Rewrite? So as I sit there tonight on my plane, watching a Horrible Movie with The Rock dressed up as a cheerleader, I got to thinking about Steve and our conversation (Steve used to be in Glee Club and on his highschool Drill Team). Unit Testing is something you learn about in school, but let's face it - no one in the MS space took it seriously until NUnit came out a few years back - it's a "wave" that's hitting it's stride right now (and a very good wave it is) and I'm wondering just how many people will rewrite an old application to be more testable? Even in the case where one might be working just fine? If you were handed an ASP Classic site (yes, there are many) - let's say ... ohhh... Starbucks - would you push them to rewrite it? If so, would a complete set of Unit Tests be part of the cost? A client said to me once (over beers):

It doesn't matter how good the developer tools get, you guys will always find ways to invent work to fill your time and bill me for it.

Interesting point and, from a client's perspective, hard to argue with. I didn't tell him I needed Unit Tests 5 years ago - why are they so critical now? Again: I'm not criticizing TDD or Unit Testing. Just using it as an example of something that wasn't commonly done just a few years back. Honestly - please don't flame me :). I love Unit Testing. It loves me too. I think. Anyway... When do you council a client on a rewrite? What sorts of issues come into the conversation in terms of cost/benefit? Have rewrites worked for you and has your client been happy? I'd really appreciate your feedback on this...

Ads by The Lounge

Ayende Rahien - Monday, January 21, 2008 - Rob, I am more than willing to use the latest tools to give them what they wanted 5 years ago. I bet that I can beat everything that they had 5 years ago as well. It is the case that we are required to supply more and more as time pass. As for when I tell a customer that they need a re-write, when it will cost more to add a feature than to write it from scratch. Or that the ability to understand what is going on is lost.

RhysC - Monday, January 21, 2008 - I have only added tests into OLD code that i am up to arms in or code that has already been "refactored more than once" and business rules need to more concrete, ie be defined by a test. Retro fitting code for no business reason wouldn't fly well with to may of the guys signing my cheques.

Steven Harman - Monday, January 21, 2008 - Rob, I checked into getting hitched to TDD but the laws here in Ohio aren't as liberal and open-minded as those on the west coast, and certain areas of the eastern seaboard. :)

Karthik Hariharan - Monday, January 21, 2008 - There's a lot of factors involved in deciding to re-write. In my mind, one of the biggest factors is moving to a platform with more flexibility for third party components and integration with other apps. It's rare that apps today don't talk to one another in some form. Even closed loop internal apps often have communication with other apps on the system. If you don't have access to a strong service layer then you that will become a sticking point in the future. The other big one is whether people exist at a reasonable rate who can continue to maintain your older tech application. At some point the cost/benefit of sticking with your existing technology shifts in favor of those who still work on it. I've often see clients overpay the original engineers who wrote their system 10 years ago just because they know they'll never find anyone else to work on the COBOL-based system and they better hang onto those engineers for dear life.

Michael Kimsal - Monday, January 21, 2008 - Good points brought up. This was a question I used to face a lot, and there's no easy answer. The biggest indicator to me as to whether to 'upgrade' someone or not was whether any new work was needed. A project could roll along quite nicely on its previous setup and as long as no changes were required, I'd just let it roll. I knew eventually that when changes were required, if they were anything substantial, more work would be required to bring it 'up to code' so to speak - to 'modern' standards. Thinking out loud here, people don't often rebuild old buildings from scratch just for the sake of it, but retrofit bits and pieces to keep them up to current safety codes. However, at some point the cost of doing that outweighs the cost of a teardown and rebuild (or, in some cases, just a teardown). There's probably some deep meaningful comment I can wrench from that comparison, but I'm too tired to try right now.

Jarle Nygård - Monday, January 21, 2008 - A complete rewrite of a major application/website is not something that should be taken on lightly. We rewrote our application from Win32/MFC to ASP.Net when the application was 8-9 years old and the customer demand for a Web based solution was "loud enough". in your example (Starbucks) I'd recommend that new functions be added as ASP.net pages and recommend moving the most error prone/oldest and/or most complex modules over as time and money allowed it. Bu to rewrite a working application, just for the sake of rewriting is just madness. If the argument is better performance, less bugs or better flexibility; then the client can decide if the ROI is good enough to allow a rewrite.

In the end I think ROI should be the guiding principle.

Tim Hardy - Monday, January 21, 2008 - I can speak mainly to large, SoA applications with ongoing development. It's already been said that it's a rather complex cost/benefit analysis. What I've tried to do is raise visibility into the costs of code and the process of writing code that is not testable. How much time has been spent "fixing" bad code? How much bad PR or customer experience has been incurred and what has that cost the company? How much difficulty/time is the result of trying to extend that code, and what is the quality of the result?

In many cases it's a systemic issue. While it's not feasible to sit down and simply start over with an enterprise application, at some point, processes have to be put in place to make sure all new code has a much higher level of quality, maintainability, extensibility - all the "bilities". It's a matter of "stopping the bleeding" before going back and re-writing existing code, and I think TDD is the best way to do this.

Scott Peterson - Monday, January 21, 2008 - More often than not, my reasons are based on the aforementioned "it costs to much to add so lets start over." However, sometimes there are other factors at work. For example, we have a perfectly good dBase IV database app that, in 1997, was getting a bit long in the tooth. We used this app internally to collect and manage a client's data and generate reports. In order to remain competitive (the contract is re-bid every 3 years), we needed to come up with a better solution. So, we sat down and designed an app that, at the time, was a pretty slick VB6 app with an Access 97 back end. Now, that app is a bit long in the tooth (it has received minor upgrades), but the contract is different so there isn't the same pressure. What is pusing us now? It's our app, so we have to convince the client to pay or cover it ourselves. How do we justify a re-write to .NET with SQL Server Express? They want to know if we will give them this same song and dance in 3 years. At least now we can say that VB6 will no longer be supported, but it's often a hard sell to convince a client to re-write something that's working just fine, because in the end, they have basically the same thing.

Shawn Oster - Monday, January 21, 2008 - I do a lot of Windows development though not all of it is with Microsoft tools. I still use CodeGear's Delphi on a daily basis for native Win32 and it's the only way to go when you're going native. Because I'm not isolated in the Microsoft world I'd known about XP (extreme) development, Agile, TDD, "Getting Real", iterations, units, blah blah for almost half a decade so it's funny to hear it called "a fad".

Perhaps because I'm a Windows developer, not a "Microsoft" one, I've used unit testing for at least 5 or 6 years solidly. Starting any major project without unit tests just seems strange (and dare I say stupid?). It's not even an extra billable item, it's just "development". They are intrinsically linked in my mind. It's like charging extra if you're going to use a MVC approach vs. WebForms.

As much as I love TDD I'd never rearchitect an application/site just to make it more testable *unless* there were actual issues, such as junior developers that kept breaking things, constant "wonkiness" or just a general uneasiness that "hmm, I wonder if I just broke something". Then you need the tests just as sanity check that things are actually getting fixed vs. playing plug the dam with your finger. That's really what tests are about, not having to worry that you're new fancy QueryExSquared just broke your previous Query98.

I would try to sneak as many tests into a "it's working just fine" system as possible though.

Rob Conery - Monday, January 21, 2008 - @Shawn: Indeed - Unit Testing is a pretty old concept but you have to agree that it didn't gain much popularity in the MS world until Microsoft Test came along. NUnit's been around for a very long time as well and I'm not saying it's anything new - but it is a "fad" in a sense in the MS realm :).

The main theme I'm reading here is that people would rewrite if the cost to expand was too much. Lemme ask another question:

Do you often say that of your own code? If so, what is it in response to? A platform change, mindset change?

How do you justify to your client that the rewrite will "cost" more and in what terms?

Shawn Oster - Monday, January 21, 2008 - @Rob: I actually have no idea about the hardcore MS testing adoption rate. I thought it was the reverse, that NUnit is what made people sit up and take notice while Microsoft Test was considered something of a toy. Shows how much I know about the Microsoft-centric development isolation chamber :)

Any of my code that is more than a few months old tends to become "Past Shawn's Code", not actually *my* code so I look at it the same way I do someone else's, that is, "I could do it better, who the hell was this guy anyway?" :)

My motivations for wanting to rewrite my own stuff tends to either be that I want to bring it inline with my other projects so it's easier to move between them (like rewriting a classic ASP in ASP.NET only because then everything will be in .NET) or because something has come along that makes the hacks I was attempting previously obsolete, like how I was trying to fake MVC using WebForms or how I used to fake Master pages in 1.1 and then 2.0 offered a "real" way to do it.

Then of course there are those coding epiphanies that totally alter how you view development and everything prior looks like the equivalent of stick figures and you want to obliterate any trace of how you used to do things :)

I try to get the client to make the choice by putting it as plainly as possible in terms of their dollars: I can spend 40 hours a month adding various new features to your website/application or I can spend 160 hours in just *this* month and you'll spend only 2 to 4 hours a month after that. I basically use the same justification with them in terms of money as I do to myself in terms of time and ease. That helps keep me honest as well since sometimes I just want to rewrite the app because it'd be better for me, not them, which is unfair.

Craig - Monday, January 21, 2008 - I usually never recommend a complete re-write for a client simply because it introduces too many risks and the clients tend to want to change things during a re-write so it really isn't a re-write. Let's assume for a second that the project is still supported on it's current platform I simply follow continuous refactoring with the newer tools/features whatever. So for example - a project I'm working on now has 3 generations of "upgrade" going on. Any new features and general maintenance of the project is done in the latest technology. In this case MVC and Linq2SQL. The project has a DAL which is 95% built by a ORM/Generator and the remaining 5% was a custom tool for DAL generation (initial write). If nothing new comes out within the next year or so the project would eventually be all Linq and MVC but by only working on little pieces at a time I'm able to avoid feature creep and also keep the code pretty stable. however ..... if I client wants to pay for a new "buzz" word ... so be it.... I'm only human, I think ;)

Robin - Tuesday, January 22, 2008 - I like this article: http://www.laputan.org/mud/mud.html This seems to be what I have ended up with whilst developing our in-house CMS! I am trying to convince the powers that be to go for a re-write using the ASP.Net MVC framework when it is available. However two factors worry me: 1. How to prevent myself from developing another "Big ball of mud"? 2. Given that the answer to 1. is probably something like (but not exclusively) TDD, how do I estimate timescales when programming in a way I have no experience of? Bearing in mind: I am the only developer at the company, so anytime I am not spending on expanding the "big ball of mud" is not going to be adding to the business in the short term. Suggestions welcome!

John Sequeira - Wednesday, January 23, 2008 - There are three types of projects I've worked on: - shipping products with a reasonably long lifetime ( 4 years) - internal business apps with little chance of substantial upgrades (the majority of what I do) - potentially long-lived but rapidly evolving web app/sites The cost/benefit on testing is different for each case, but I've only found a payoff on testing for the first case. Category #3 really needs to systematically monitor uptime more than anything else. It's a type of testing but not really in the extreme sense. Most of what I do falls into category #2, and it's always a sprint to finish. Testing doesn't help to deliver the first version of the app bug free for a certain level of app complexity ( which most biz apps fall under). I think the tricky thing here is that in promoting MVC you're implicitly promoting class hierarchies of business objects, which imply an investment in that class hierarchy over many generations of code, which implies that an investing in testing is worthwhile. You could have one (MVC) without the other (a decently complicated class hierarchy as your model), but the kind of folks who really like to MVC don't consider do it that way.

terryg - Thursday, January 24, 2008 - Their was an article not too long ago called Guerilla Refactoring where I found my favorite quote of the year. "First, however, we need to go to the customer and tell him, "The architecture of this app is not unlike something akin to roadkill. It compiles only during the waxing phase of the moon and the fact that it runs at all has restored my faith in God. I know you've been using it relatively successfully for several months but we'll need several tens of thousands of dollars from you to refactor large portions of it. When we're done, there will be no discernible difference to the application from your perspective but we in the IT department will sure feel better about ourselves."

Riding The Geek Rewrite Wave

Related