ASP.NET MVC: MVC Storefront, Part 3 – Pipes and Filters

In this screencast I tackle some recent feedback on parts 1 and 2, and I also dive into the Pipes and Filters pattern I’m implementing on top of the Repository.

I also have popped the code up on CodePlex and you’re welcome to download the source, such as it is .

Previously, On The MVC Storefront

Part 1: Architectural Discussion and Overview. I cover the initial architecture here, in part 1, where I discuss the Repository Pattern with Ayende Rahien and Steve Harman. I also ask Phil Haack what’s first: The Test Chicken or the Model Egg.
Part 2: The Repository Pattern. I walk through our Repository implementation and write out the first main set of unit tests. I also structure up the initial service layer methods and interfaces.

Lots To Say
There’s a lot here in this screencast – I really wanted to capture the whole TDD "vibe" while at the same time making the screencast watchable. I didn’t want to go over 15 minutes, so I ended up editing a whole lot to keep the flow moving. It might seem a bit choppy – but I’m hoping you’ll be able to stop and go back when needed.

I’ve also upped my font size, and changed the encoding for better quality.

Hopefully you can see the process I went through clearly while coding … that’s my goal here.

You can watch the screencast here.

CodePlex
The source is on CodePlex under MSPL (Microsoft Permissive License), which basically makes it just like a starter kit – take it and run with it .

You can access the project site here.

Your Feedback Is Critical
I’m taking this pretty seriously and I really do want to hear from you if you have any questions. What I’d like to ask of you, up front, is that if you have a suggestion – please be very clear with me with respect to what the issue is and what you were expecting/would like to have seen.

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.

This entry was posted on Thursday, April 10th, 2008 at 4:37 am and is filed under MVC Storefront. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Comments to “ASP.NET MVC: MVC Storefront, Part 3 – Pipes and Filters”

Martin says:

April 10, 2008 at 5:16 am

Awesome work Rob. I’ve been struggling to get my head around TDD for a while now and this is really helping me.

Thank you.

Ray says:

April 10, 2008 at 6:03 am

Neat! Thanks Rob, keep em coming

James says:

April 10, 2008 at 6:41 am

This is great, I’m in a simliar position to Martin. I’m really interested to see how this system develops in an agile/TDD fashion. I’m looking forward to see how you build out and test the actual repository (and UI) layer.

By the way, though it doesn’t effect anything, your loopIndex is not being incremented in GetProducts() in TestCatalogRepository.

Cheers Rob.

Perry says:

April 10, 2008 at 9:08 am

Great series, please keep it up!

For the TDD purists out there that don’t like Rob’s approach, remember that one of the things that TDD says is to take steps that *you* are comfortable with and if they are too big then you can revert to smaller steps.

Anyway I had a question about the Pipes & Filters pattern. This pattern appears to return the complete resultset form the repository (eventually DB) and then filters out the values locally using the WithXXX() extension. This does not seem very efficient especially since we can easily add the filters with LINQ to SQL??

Andrew says:

April 10, 2008 at 9:42 am

This is great stuff Rob, and really timely, we’re looking at starting to try and go TDD at work and the repository pattern you’ve been using is new to us and we like what we’re seeing (previously we used the interface as the definition and talked direct to the implementation, rather than using a intermediate service).

The only question that’s coming up for me at the moment is, with the Extension Methods, if you were using a proper db backing, and had thousands of products – wouldn’t you be pulling back every product from the db and then filtering for one result?

(Using the last example where you’re returning a single).

Given we’d be doing this in a scenario where we could have millions of results, wouldn’t that be overkill to select everything from the db to return a filtered result?

I might be thinking ahead of where you’re heading but yeah, that’s on my mind after watching the filter/pipe example.

Thanks though, fantastic examples and it’s great to watch someone building an application so openly from scratch like this!

Andrew says:

April 10, 2008 at 9:45 am

Thank you Perry for asking as well!

I didn’t refresh between dowloading/watching and posting, so you may have beaten me to that one by a bit, so I apologise about asking for a second time

Justin Etheredge says:

April 10, 2008 at 9:48 am

Rob, I think you owe the community an apology…for creating such great screencasts! Nice work, keep it up man.

Bill Kempf says:

April 10, 2008 at 10:27 am

First, two those worried about the design pulling too much data, I don’t believe this does. The Repository always provides the user with in memory results, but everything prior to that works on IQueryable objects that use delayed evaluation. In the case where he asks for a single, LINQ to SQL should be able to execute a single query to the DB.

Rob, I think you’re still going to upset the purists. Purists believe in "one assertion per test", and you’re not following that. Of course, that makes me wonder what they do for specific cases such as testing to verify that construction sets members to expected values? It’s those sorts of details that someone new to TDD trying to follow advice from experts is left wondering.

Steven Harman says:

April 10, 2008 at 11:12 am

Bill,

re:"One assertion per test." For me that means only making one _logical_ assertion per test. As long as the asserts are all verifying a single expected behavior/result… it’s all good. So its not about only making one call to Assert.XXX() per test.

Actually, if the later were the case you wouldn’t ever want to directly call Assert.XXX when using mocks to do interaction-based testing as the mock framework will make Assert calls under the hood when verifying the interactions occurred.

Tomek says:

April 10, 2008 at 11:21 am

This is getting better and better…

My question ties back to Perry’s and Andrew’s, extremely nice syntax but what about the performace when large result sets come into play?

John Smoltz says:

April 10, 2008 at 11:28 am

Um, not to poopoo this stuff, but shouldn’t you be working on getting SubSonic 2.1 out? You’re getting paid for that too

Marcus McConnell says:

April 10, 2008 at 11:55 am

Nice work! I’d be careful that this doesn’t turn into the TDD Show with Rob Conery. Maybe it’s time to step back to a higher level and say "download the code for the details of all these tests."

Josh Stodola says:

April 10, 2008 at 12:11 pm

Cool, looks like this isn’t in Silverlight. Thanks! Downloading @ 29 kb/s

Stephen says:

April 10, 2008 at 12:30 pm

I’m really enjoying the screen casts (and the debate following them). Having never done TDD, its interesting to see the purpose of it, I guess the point is that your testing methods push your progress, rather than tests you may see them more as objectives, with tests to prove you have achieved a given objective..

One thing I noticed here is that there are logical and data driven tests, and while the compilation and interfaces seems to do most of logical proof, I’m not sure how useful the data driven tests are..

For example, you have a test to prove that 5 products are returned, this is of course a test that is proving your test data works, but other than that, I’m not sure what it proves?

In the scenario that my application compiles, what should my tests be trying to prove? that I’m getting non-nulls from certain invocation, or that ranges are ’sane’, or that I do or don’t get side effects (since tests should be about proving something should ‘break’ from certain scenarios, just as much as it should work for others)..

But then all these tests seem to then be reliant on test data.. great- my test catalog does return me products, what does that prove for my live catalog?

I’m very interested in what TDD is supposed to do, I currently ‘design’ by starting interfaces and designing them to do expected data.. I judge my application logically works by the fact I’ve got from A -> B without writing any working code and that it actually compiles..

Am I inadvertently heading towards TDD without realizing it? or is TDD all about runtime tests that check to see if test data actually does flow through your logic the way you expected?

And again, if thats what it is, what good is test data other than catching those little ‘gotcha!’s’? or is it about catching all those little gotcha’s at the same time as you test your logic, in order that they drive each other to create a cleaner solution?

Luke Duf says:

April 10, 2008 at 1:15 pm

I appreciate that you’re trying to get the TDD right but I’m hoping that part of the development process is deemphasized in future parts of the series. I’m interested in seeing how you’re using MVC, LINQ, filters, etc. and the constant flipping back-and-forth with the tests is distracting.

It seems like in this part, you were so worried about getting green lights that you didn’t have time to talk about LINQ’s delayed evaluation and that left some folks scratching their heads.

Jim Bonnie says:

April 10, 2008 at 1:16 pm

Rob,

Thanks for posting the code, I look forward to following along.

I am curious as to where the requirements come from. They seem to be coming from your head. Does TDD provide for any guidance as to how to break requirements up or keep track of them.

Dave says:

April 10, 2008 at 1:43 pm

I too would like to know about that scalability of this pattern. I have always just returned exactly what I need from a DB and then worked on that resulting set. So far in this project, we don’t know what data access we will be using and ideally with this pattern it could be anything. If I decide to use an old school strored proc and ADO way to access my data, would I have to return all of the data and filter it with this design pattern?

David Alpert says:

April 10, 2008 at 1:44 pm

i appreciate this series immensely, and think that #3 hits what i’ve read of TDD right on the nose; test-first-mandates-everything.

agreeing with Luke Duf, now that this aspect of TDD has been demonstrated so well, perhaps we could skip the try-to-build-a-project-that-we-know-won’t-compile step in your exposition and go right from "here’s the test" (test-first) to "here’s the code that fails" (red) to "here’s the code that passes" (green).

Now where i’m really curious about is how your app architects it’s separation of concerns.

Specifically, i scratch my head a little bit to see you coding your model in an assembly called "Data". After working on several open-source projects i am inclined to isolate the model in a "Core" assembly that can be consumed by different UI environments.

I then expect to see a "Data" assembly that contains the IRepository and implementation-agnostic services, while perhaps a "Data.Linq2SQL" assemble that contains my implmentation.

I’m new to this Linq2SQL thing so my architectural instincts may be off. I want to use Linq and love the fluency of your Filter examples, but i’m nervous about wiring my database too tightly to my model.

Mind you, i’m also concerned about wasting perf time executing left-hand-right-hand code to copy out of my repository into memory and sync them back down again after.

What’s a non-NHibernate-using developer to do?

I’m really looking forward to watching your example grow a database implementation and a front-end UI and how the pieces are wired together to maintain separation of concerns .

Keep up the great work, Rob. Love this blog and ASP.NET MVC and can’t wait to hear more about MVC-flavoured SubSonic!

Dave says:

April 10, 2008 at 1:45 pm

I forgot to add, it seems like the linq code is not needed if I use the ado way and only return a filtered set, so why would I have it? Seems like a lot of overhead for nothing. Maybe I am missing the boat here, so correct me if I am wrong. I love the casts, keep them coming.

Allan N says:

April 10, 2008 at 2:17 pm

Rob, Love the TDD stuff. Great to see someone working their way through it, responding to feedback, etc. Mirrors the real world for many of us also struggling to put into practice all of the ‘theory’ we read. I know the series isn’t intended to be exclusively about TDD, but it’s been the best part for me (although the filters and pipes stuff is tres cool as well).

Had the same questions about where the tests come from. Why test for count of elements returned, really? Why not test whether specific elements are returned? What are the expected elements, and why do we expect them? I think there is an underlying design point or requirement being tested, but it doesn’t seem to be that obvious going into building the tests. I struggle with the whole chicken/egg thing – how do you know what to test without starting top down from some basic user requirements and subsequent design to fulfill them? I think this process has happened (you have an picture of the application you want to build and some ideas about it’s architecture). Just not sure how that fits into the whole TDD process. Do we need a test that says ‘customer can buy a widget?’ at the very beginning? That feels artificial somehow, as the test won’t _really_ pass until we’ve got 50% of the app built

Not really looking for answers , just sharing that a lot of us (well me anyways) struggle with how to ‘do it right’.

Rob Conery says:

April 10, 2008 at 2:55 pm

RE the resultset and perf – if you go TDD this is something you address as needed . But I can tell you that a neat feature of Linq To Sql is the delayed exeucution – it will only execute the query when you iterate or send something "ToList()" or "ToDictionary".

The tests that many are asking for here WRT the db come in the form of Integration Tests – Phil is urging me to be sure of pattern here and in the next webcast I’ll do just that.

Keep it coming – all this feedback is what I need!

Rob Conery says:

April 10, 2008 at 3:31 pm

@Perry:

>>>This pattern appears to return the complete resultset form the repository (eventually DB) and then filters out the values locally using the WithXXX() extension.<<<

IQueryable is just a Linq statement, and with Linq To Sql that statement is only fired when you iterate over it using a foreach or ToList().

@Bill Kempf:

>>>I think you’re still going to upset the purists. Purists believe in "one assertion per test", and you’re not following that.<<<

That’s OK I don’t think I can butter everyone’s bread. The test I think you’re talking about, however, is the Product test and you have to Assert the whole thing or else it’s pointless.

Another place I didn’t show that I went the purist route was when I created the Filters – step one should have been to make the NULL test pass, which means all i needed to do was return "qry" – this to me is silly. I don’t think "faking myself out" is going to get me anywhere….

@Jon Smoltz:

>>>Um, not to poopoo this stuff, but shouldn’t you be working on getting SubSonic 2.1 out? You’re getting paid for that too <<<

I thought I saw someone lurking outside! I’ll be sure to get that TPS cover sheet out with it . If you’re curious about what we’re doing with 2.1 (notice there’s a WE there) – just check our repository for activity. Two checkins this morning and we’re right on target for 2.1 beta 3.

No need to worry about SubSonic – I’ve been doing this for a while .

@Marcus:

>>>I’d be careful that this doesn’t turn into the TDD Show with Rob Conery. Maybe it’s time to step back to a higher level<<<

I’m sort of just letting this roll out the way it does. I knew the TDD stuff would be a bigger issue at the outset and I hear you WRT to focusing on the code a bit more – but TDD is fascinating and many folks are very curious.

@Stephen:

>>>this is of course a test that is proving your test data works, but other than that, I’m not sure what it proves<<<

That’s it really – but in that test passing, you validate all the other tests that use GetProducts() which are assuming the structure is correct.

>>>But then all these tests seem to then be reliant on test data.. great- my test catalog does return me products, what does that prove for my live catalog<<<

Nothing at the moment – the deal here is that I’m testing my Repository and some logic on my service (going very slowly). When I switch over to my live DB, I’ll create Integration Tests to be sure that perf is acceptable, and so on (more on that in the next screencast).

@Jim Bonnie:

>>>I am curious as to where the requirements come from. They seem to be coming from your head. Does TDD provide for any guidance as to how to break requirements up or keep track of them.<<<

Out of my head … sort of. I obviously don’t have a client (not yet anyway – someday it will be the community I spose) so I’m modeling this after the CSK.

TDD doesn’t speak to requirements, per se, but Agile does. The idea is you create a circle from a square, slowly. Simple requirements, refined along the way, allowing for change and inspiration.

@David Alpert:

>>>now that this aspect of TDD has been demonstrated so well, perhaps we could skip the try-to-build-a-project-that-we-know-won’t-compile step<<<

Yes – I’ll be speeding it up .

>>>Specifically, i scratch my head a little bit to see you coding your model in an assembly called "Data"<<<

"Physical" location and "Separation of Concerns" are not related. At this point, from a purist TDD perspective, I’d have everything in one assembly until the tests mandated otherwise . I sort of leap-frogged that though – to me

Dave says:

April 10, 2008 at 4:08 pm

Rob, you mention with linq to sql this will scale because of the delayed execution. But what about other technologies? I thought part of the reason for the repository pattern was so that the repository could be switched easily. It sounds like you are tying to linq specifically. Am I missing something? In my tests, linq to SQL is pretty darn slow compared to using more traditional data access methods. It would be nice to see something like this developed with more than one data access technology. Everything coming from MS right now is linq and not everyone is using linq because of the perf issues.

The Other Steve says:

April 10, 2008 at 4:08 pm

This is the coolest thing ever.

Rob Conery says:

April 10, 2008 at 4:24 pm

@Dave:

>>>I thought part of the reason for the repository pattern was so that the repository could be switched easily<<<

It’s more about testing. I agree that using Linq here limits the choices – but that’s for now. In the next 6 months I’m going to be we have a lot more access technologies that work with it.

>>>In my tests, linq to SQL is pretty darn slow compared to using more traditional data access methods<<<

Interesting – I’d like to see those tests. I did some Linq To SQL benchmark tests where I put it up against SubSonic and SqlClient. Without "tuning" it ran just fine – in fact it ran faster than SQL Client if you "tuned it" by defining and caching your query.

I pretty much regard any data access method (NHib, SubSonic, ADO, LinqToSQL) as the same.

If I do this right – it won’t matter . You’ll be able to pull data from Amazon:

blog.wekeroad.com/…/linq-how-to-use

Bill Kempf says:

April 10, 2008 at 4:27 pm

Steven,

Thanks. I’m in the same boat as Rob here. New to TDD. Your explanation makes perfect sense to me, but it’s not the impression I’ve gotten from reading up on the topic of "one assertion per test". I’m going to assume I’m just reading the wrong things, or reading too much into them, because your explanation rings true to me.

Bill Kempf says:

April 10, 2008 at 4:30 pm

Oh, and just to illustrate what I meant about what I’ve read indicating it truly is supposed to mean one assertion per test: http://www.artima.com/weblogs/viewpost.jsp?thread=35578.

Aaron Jensen says:

April 10, 2008 at 4:52 pm

First off Rob, thanks for these. It takes a strong person to take Bellware’s "hard love", ignore it, then manage to have a conversation.

>>> Another place I didn’t show that I went the purist route was when I created the Filters – step one should have been to make the NULL test pass, which means all i needed to do was return "qry" – this to me is silly. I don’t think "faking myself out" is going to get me anywhere…. <<<

Perhaps it was the null test that was silly then and it could have just been skipped entirely. The point of only writing code the test mandates that is so that you don’t end up adding a bunch of code that isn’t tested or designed with testing in mind.

TDD should drive the design, if you write one test that tests 1 thing and then go write a method that does 10 things, you’re not doing TDD entirely.

I’ll be the first to admit I cheat, sometimes I write multiple tests at once, or I write code for a test that I’m about to write. I’m only a purist in speech, not in practice. I try to not cheat much though, I can see a difference in rhythm and code when I do.

I think that whoever said in these comments that everyone finds their own path in TDD is spot on. That said, I think the purist way should be taught and explained, then people can judge for themselves what they want to do or not.

Conrad says:

April 10, 2008 at 4:57 pm

Rob, thanks for working through the TDD process for us TDD newbies. Keep these screencasts coming.

adminjew@gmail.com says:

April 10, 2008 at 5:40 pm

R0b gr8 stuff

Rob – eCommerce

Microsoft – Northwind

Lance Fisher says:

April 10, 2008 at 8:40 pm

Nice screencast, and thanks for showing some of the nitty-gritty of TDD!

I’m wondering if the extension methods filters are really buying you anything. Is it any better to use:

return _repository.GetProducts().WithID(id).Single();

Rather than:

return (from p in _repository.GetProducts()

where p.CategoryID == categoryID

select p).ToList();

I suppose it depends on your mindset when you are querying data. If you want to think like you keep running your data through filters to remove the items you don’t want, then the extension methods might make more sense. However, I see using LINQ in the Service as being more flexible, and still very nice syntax. Plus, you don’t have any extra methods.

jonx says:

April 10, 2008 at 8:47 pm

Rob, very undertating wawy to learn new things along with you…

I was wondering one thing… Shall we try to build the project and then run the tests for each and every new thing you add? This makes the development procedure look a lot like trial and error Is this just for the sake of the demonstration and usualy you will create whole "classes" before trying to build/test your code…

Shaneo says:

April 10, 2008 at 9:14 pm

@Rob Conery (in your @Dave response)

>>>Interesting – I’d like to see those tests. I did some Linq To SQL benchmark tests where I put it up against SubSonic and SqlClient. Without "tuning" it ran just fine – in fact it ran faster than SQL Client if you "tuned it" by defining and caching your query.

This "tuning" concept could be a very worthy blog post. Any takers? If not, any quick resource you could point me to and perhaps I could write a post about it myself?

Aaron Jensen says:

April 10, 2008 at 9:29 pm

Jonx,

The practice is called Test Driven Development or Test Driven Design. It’s not something done for demonstration purposes, it’s something done to aid you in writing testable, tested, and well-designed code.

Here’s a wikipedia on it: en.wikipedia.org/…/Test-driven_dev

Lance Fisher says:

April 10, 2008 at 9:35 pm

Oops, I grabbed the wrong line of code for the sample. The first block should be:

return _repository.GetProducts()

.WithCategory(categoryID).ToList();

Is that any better than using the LINQ right in the Service? Seems like just another way to write it.

Dave says:

April 10, 2008 at 9:51 pm

Shaneo says:

Thursday, April 10, 2008

@Rob Conery (in your @Dave response)

This "tuning" concept could be a very worthy blog post. Any takers? If not, any quick resource you could point me to and perhaps I could write a post about it myself?

– I will try to dig up my test code or write some more. I will grant him that my tests are about 6 months old so things may have changed. Butat that time linq was about 700 times slower than sps and data reader. I also don’t see how it could ever be faster tning or not. At the end of the day it is an abstraction layer on top of ado and generates parameterized ado queries and then executes them. So, all things considered if I "Tune" ado the same way I tune Linq, how could it be faster, ever? I will say that if it gets close it really won’t matter and I can set about trying to get the db guys to allow something other than sps. Currently, we only allow sps and they all have security applied to limit our exposure to threats. I think that is why I would really like to see some samples without linq, not everyone will be able to use it. MS does this witheach new technology, all of their samples are done in whatever make the language look slick even if not real-world. I was hoping here that we would look at doing something other thant linq to sql or perhaps writing two repositories one with the slick new stuff and one with plain old business objects that we return from sps. Then we could even benchmark both and prove me wrong that linq is slow . =)

jnw says:

April 10, 2008 at 10:16 pm

Great, and cannot wait to see more coming out!! You mentioned that you had LinqtoSql in mind, but, does Linq-to-Sql already has the repository pattern built-in?

So, instead of saying:

Category sub = new Category();

…

sub.ParentID = parentCategory.ID;

in your TestCatalogRepository class, one could say:

Category sub = new Category();

…

parentCategory.SubCategories.Add(sub);

By using the existing repository in Linq-to-Sql, you not only update parentCategory.SubEvents (collection) and the sub.ParentID simultaneously, but also eliminate the need to most of the query/filter code here. For example, the CatalogService.GetCategories() already exists as the Category property of the repository. To list all subcategories, simply say: aCategory.SubCategories, and so on… (e.g. aCategory.Products).

To be clear, I am not sure all the pros and cons involved, and don’t meant to suggest one over another. One problem I have now (so you know my question come from suffering) is that the the built-in repository doesn’t work without the database itself (or I haven’t found out how to use it yet). The result? My unit-tests are sl…o…o…o…w.

I really like the fact that I almost never need to deal with category.ID with Linq-to-Sql. Just have the foreign key relation set up, and I can use objects exclusively (well, almost).

Again, thanks a lot for the post, and I am looking forward to hear opinions back from the whole group!

Shaneo says:

April 10, 2008 at 11:16 pm

@Dave

You raise some very good points. At my workplace, we still hand code our entire data layer using stored procedures. Some of the more clever ones have created slick code generators to create much of their data access code based off the stored procedures.

Myself, I’ve only recently begun playing around with Linq to SQL (@home) and I’m really liking it for the ease of use standpoint.

Since I’m still hooked on sprocs to some degree, I really like the fact that I can drag and drop my sprocs onto my tables (in the dbml) and have my data layer generated instantly. A quick transform to business entity objects using the IRepository pattern, and I’m away.

The "tuning" comment caught my attention as I’m always interested in increasing performance

The Other Steve says:

April 11, 2008 at 1:03 am

Ok, I’ve got a question. How would I do a transaction with this pattern?

As an example, let’s say our commerce site actually keeps track of Quantity on Hand. So when I have a sale, I record a record in the Orders table, and then I do something to a ProductInventory table. I would want to wrap this all together as a single transaction.

It seems though that the Service layer, while it handles logic such as this… it is disconnected such from the Repository layer that there’s no way to control the two pieces of work together as one transaction.

I’m probably getting ahead of myself. I was trying to see how this would work with Linq to SQL, and then started thinking about saving data, and came up with that common scenario.

Erik says:

April 11, 2008 at 2:22 am

I like the pipeline / filtering concept a lot, but why don’t you expose that beyond your service layer? For example you turn ".GetProducts().WithID(id).Single();" into "GetProductByID(int id)" – why not just let callers into the service using the filters, so you don’t have to keep making new methods in your service for each new set of filters?

Evan says:

April 11, 2008 at 4:10 am

>> Another place I didn’t show that I went the purist route was when I created the Filters – step one should have been to make the NULL test pass, which means all i needed to do was return "qry" – this to me is silly. I don’t think "faking myself out" is going to get me anywhere….

Yes, most people skip that step in the real world BUT it has an important point behind it. I’ll just tell you so you don’t have to figure it out for yourself.

The point of making the null test is to show you that design can come in very, very, very small baby steps. While I, too, would skip the null test in practice, the technique of moving in very tiny steps works really well if/when you are working on a very, very complex piece of code.

So, in practice, my tests tend to vary with the amount of stuff I’m comfortable with (attempting to have no more than 15 minutes per Red, Green, Refactor cycle). If I’m having a hard time designing something, I shrink the scope of the test down to a managable, bite sized chunk.

Oh. And you’re good to skip the failed compile. I don’t think anyone does that in real practice either. Although it can be useful for teaching.

Learning the VS shortcut keys for Build and Run Unit Tests (for whatever your flavor is) can also help speed things up.

SOO..one of the big things with Agile is having a feedback loop. What’s your feedback on TDD so far? Did you find the green lights slightly addicting?

Rob Conery says:

April 11, 2008 at 4:40 am

On a plane today – sorry for the delay … responses below…

@Lance Fisher:

>>>I’m wondering if the extension methods filters are really buying you anything<<<

Reusability. Filters aren’t just WHERE statements – they can do anything, like grouping and transforms, calculations and abstraction. If you create them cleanly – you can stack as needed and have the logic in one place. In other words I don’t want to write your example LINQ code all over the place . DRY…

@Shaneo – using LinqToSql, you can set stored queries on the context that are, essentially, pre-compiled Expressions. I don’t have it in front of me – I’ll look it up for you.

@Dave- the load tests I ran involved loading up the entire Product table from Northwind 10,000 times. The variation using SubSonic/NHib/Linq/ADO was, literally, negligible.

You’re better off focusing on the real bottle necks instead of shaving nanoseconds. These would include using GZip compression (for instance), too much javascript (in 3rd party UI code or too much Ajax for instance), etc.

SPs don’t limit your exposure to threats, unless your DBA requires you to use explicit login credentials to fire each one (which may be your case, I don’t know). They’re also not any faster than straight up SQL . I don’t mean to tell your company how it should work – but you might find some freedom in NOT maintaining a barn full of SPs . Just a thought…

@jnw:

>>> does Linq-to-Sql already has the repository pattern built-in<<<

No – Linq To Sql has a DataContext and it’s a data-access tool which at the moment doesn’t fit well with TDD in that you can’t stub the data source (which is important – you don’t want to run tests on your DB).

The code you wrote works fine, but you can’t test the logic there without hitting the DB. If you hit the DB the test is considered an integration test. You don’t want some other thing to fail your test (like a bad connection or bad data) other than your Assertion.

What you’re doing works fine – it just doesn’t fit well with TDD.

@Other Steve:

>>>Ok, I’ve got a question. How would I do a transaction with this pattern?<<<

The TDD guy would say "why do you need one?" :). I’m not concerned YET with transactions – but I know I will be. You raise a good point RE the DataContext sort of "imposing itself" on the process. I know that I can use System.Transactions in the Service class – but, hopefully, I can come up with a better scenario. Not there yet.. but I will be .

@Erik:

>>>I like the pipeline / filtering concept a lot, but why don’t you expose that beyond your service layer<<<

This, in it’s rawest form, is actually a security risk. In essence it’s asking "why don’t you send out your raw SQL command" since IQueryable isn’t the results – it’s the command (essentially).

If I had a method in Services that returned orders (GetOrders()) for example and I allowed 3rd party vendors to hit that API, all it would take is one GetOrders.ToList() and you have a nightmare.

@Evan – thanks for the thoughts on Baby Steps but there comes a time when Baby Steps turn into crawling and … well to me it can go too far down the theory route. If I write a method called "GetMeABeer()" and I stub it out with NotImplemented, the next bit of code I write better be "return new ColdOne()" and NOT "return new PBR()" because that’s not what I want to do and is a waste of time . But your point is well taken.

RE feedback loop – the post is in the queue. I actually have a great story about an edited piece of the webcast where I realized I completely screwed up the TestCatalogRepository code and I caught with the most mundane, stupid test…

tgmdbm says:

April 11, 2008 at 10:59 am

I love these screencasts.

I’ve done a bit of TDD before so I find myself cringing a bit along with the purists out there when something goes "wrong". But then i remember you’re not teaching TDD you’re _learning_ it so i forgive you .

I’m really glad there’s someone like you out there spreading the TDD love.

It would be nice (now we know*) to just see the test, then see the code, without seeing the failed build. Or even if we saw a few tests for the same method all in one go and then jumped to the completed code. We should be able to take for granted (for the most part) that you’re using TDD.

* http://www.youtube.com/…/watch

jnw says:

April 11, 2008 at 4:22 pm

Hi, Rob,

I know everybody said that we shouldn’t hit DB for business-logic testing, and I don’t want my "teeth go bad:)" so I do testing often. So far, my tests refresh (delete and create) all tables about 100 times, and it takes about 5 minutes to finish.

Besides speed, however, everything else works in TDD fashion (RGR), and I start with domain model strictly (DB schema is generated automatically).

I guess my current approach do work for small projects or somebody just started with TDD+LinqtoSql, but I am also eager to learn your approach! Actually, I spent quit a lot time learning NHibernate before even considering LinqtoSql, only stopped because I didn’t find a good repository model example then! So good luck to this project, and I am sure it will save lots people’s time.

Best wishes,

Ridge says:

April 11, 2008 at 6:14 pm

@Rob

I just sent you code that abstracts out the datacontext…

jnw says:

April 11, 2008 at 6:37 pm

Hi, Ridge,

I wonder if you could send me the code as well, or make it public??? It will be tremendously helpful to the community!

All the best (jwang000-at-gmail-dot-com)

Josh Stodola says:

April 11, 2008 at 9:03 pm

Fantstic screencast, Rob. Keep ‘em coming. Any chance of making the first two available for download in .wmv?

Josh Stodola says:

April 11, 2008 at 9:11 pm

Nevermind, yo. I found the links. Thanks!! PS: Your blog doesn’t remember my comment fields.

Dave says:

April 13, 2008 at 1:11 am

@Dave- the load tests I ran involved loading up the entire Product table from Northwind 10,000 times. The variation using SubSonic/NHib/Linq/ADO was, literally, negligible.

Rob, do you have the code you could post for this? I also did something similar, but with the orders table in Northwind and I found with a run of 10,000 selects, ado Stored procs took 8 secs, ado dynamic sql was 8 seconds , Linq to sqlys 18 seconds and subsonic was 32 seconds. Sothat seemed more than negligable. Maybe I am missing something in my methodology. As for our company, yes SPs have different security and limit what can be viewed, changed etc by the web user. Something by the way that microsoft has said was essential in the past. If we don’t do that and the webserver is compromised, then we do open up additional security holes. Yes? It just seems like yesterday’s best practices on security are somehow now not valid that we have this ORM tool. The security concerns have not changed, so why have the recommendations? Does RAD outweigh the risk, I guess that can only be answered by the individual company, but I see this opening some holes for new developers that do not understan security and are taking this as the way to develop applications.

pn says:

April 14, 2008 at 8:20 pm

Hi Rob,

I’ve noticed that you’re always returning IList<Category> or something<Category>, and this is something i’ve been struggling with… how, with linq, do you get an ISomething ? i’ve tried to put ISomething in xml mapping and in GetTable<ISomething> without success (of course it doesn’t know what type to create and it complaints about finding a public parameterless constructor)… with GetTable<Something>().Cast<ISomething>, then you have the problem for inserts, updates and deletes… any thoughts about this you can share ?

thanks,

Martin Nyborg says:

April 17, 2008 at 5:41 am

Cant wait, when will there be a new screencast?

Christian Schiffer says:

April 18, 2008 at 5:05 am

Rob, these are great webcasts, I am anxiously awaiting part 4,5,6,7,8…

Yvan says:

April 21, 2008 at 6:24 pm

Thanks Rob!

You shine again with that brillant walkthrough…

I am eager to see the rest of it.

alberto says:

April 29, 2008 at 3:45 pm

Hi Rob. I’m trying to catch up with your video series.

I really liked how you explained filters.

I still don’t like very much how you name (and test) your code when you are retrieving data from the repository (i.e. hardcoding the number of elements both in the Assert and in the test name).

Rob Conery says:

April 29, 2008 at 3:48 pm

Hi Alberto – the reason I can get away with the hardcoding stuff is because I control, completely, the Repository (TestCatalogRepository). This is a "stub" and since it’s code-based and predictable, I can test against it nicely without having to worry about DB Connections and weird data.

alberto says:

April 30, 2008 at 1:42 pm

Yeah, I know. But I would get a collection and insert it in the Repository and then compare data retrived from it with data in the collection. Feels better to me than hardcoding expectations, and you don’t have to worry to rewrite those tests because you added some more data to your Repository later.

Rob Conery says:

April 30, 2008 at 3:55 pm

>>>But I would get a collection and insert it in the Repository<<<

Isn’t the repository supposed to give me data? Moreover – isn’t this what I’m doing in a sense?

>>>Feels better to me than hardcoding expectations<<<

That’s not what I’m doing – I’m hardcoding the data. I expect that the stuff I do with the data will work.

>>>and you don’t have to worry to rewrite those tests because you added some more data to your Repository later<<<

Which is precisely why I don’t want to use a database

alberto says:

May 2, 2008 at 12:18 pm

I’m afraid I’m still not making my point clear.

You are following a TDD approach and the point of it is using test to drive your design. Which requirement says "each category contains 5 products"? If that is not part of your requirements, IMHO your test is wrong. It just happens to be green, but it is just checking very specific data in your repository, not it’s general behavior. It won’t work if you work with a real Repository or even if, as a said, you happen to change the data *in your fake repository* to support/test new functionality. That’s a very weak (and wrong, I think) test.

In summary, you are imposing an artificial restriction that is not part of the system/requirements.

Rob Conery says:

May 2, 2008 at 3:37 pm

@alberto: the point of the test is to make sure that my TestCatalogRepository is providing Products on each Category. This wouldn’t be a valid test with *real* data.