Rob Conery

SubSonic 3.0 Repository Template Update

Rob Conery — Thu, 13 Nov 2008 21:11:38 GMT

Got a fun email from Will Gant today:

Since everything else is going my way (sorta like Burger king), now comes the "can I have fries with that?" question. Would it be possible for the Add, Delete, and Update methods on the repository class to have overloads that take IEnumerable<T>?

And I sort of started to kick myself a bit - this should have seemed obvious, and I should have done it from the start, but I didn't and I'm not too sure why. This is the goodness that is Open Source!

Will sent me a quick patch, and I added some spice to it and now we have a batch-updatable (read: one connection, one sql execution) Add/Update/Delete for the Repository which you can download here (I'll include this in subsequent releases).

Hopefully this underscores a bit more just how much control you have in this!

To show you how this might work, here's a quick spike using the Chinook database (our new test db):

        [TestMethod]
        public void Update_Should_Update_En_Masse() {

            ChinookRepository<Customer> repo = new ChinookRepository<Customer>();

            var customers = repo.GetAll().ToList();
            int index=0;
            foreach (var c in customers) {
                c.Company = "Company" + index.ToString();
                index++;
            }

            repo.Update(customers);
        }

Just to re-iterate, the Update() method here takes in the List<Customer> that is created from the repo.GetAll().ToList() call. This will not work if you leave it as IQueryable. I'm not sure why - but the bottom line is that I tried and the changes weren't honored as I think IQueryable doesn't allow you to assign values to it - it's just an Expression manager.

The Update method will create one single SQL query and execute it as a batch - one call! I was going to wrap the ExecuteBatch() method in a transaction but decided against it for now - however if you want to do that, you can edit the templates as you need :).

SubSonic 3.0 Preview 2

Rob Conery — Thu, 13 Nov 2008 02:48:35 GMT

Over the last week I've made massive improvements and upgrades to SubSonic 3.0 and the templates that it comes with. I haven't been this obsessive and inspired since I created SubSonic in the first place, and I have to tell you I am more than stoked to see this pull together.

The Foundation
I'll be really concise with this post - there's so much here ... it will be hard but I really want to keep to the point. SubSonic 3.0 is a Layer Cake which is focused on openness and transparency. Here are the layers:

Core: Data Access and SQL Translation. This is the part that executes commands against the database, and translates queries (SubSonic's or Linq's) into SQL.
Surface: Working with your database. This layer "surfaces" your database and implements "IQuerySurface" (I couldn't think of a better name). In short, it exposes every table as IQueryable<T>, every Stored Procedure as an executable method, and also provides you with Aggregation queries (Count, Sum, Min, Max, etc).
Implementation: Your Favorite Data Pattern. This can be Repository, ActiveRecord... whatever. I've created (with this release) an IRepository pattern for you that I think covers 99% of the needs out there (you'll see in a moment) and also allows for some nice testability.

Of these, the only thing that is locked down is part 1 - The Core (as part of our DLL). The rest is built using T4 templates that you add to your project. So if you hear me constantly say "you can change this as you need to in the templates" - well it's because you can. These aren't hard to learn, and are created using plain old C# code so changing/altering as you need should be easy.

In fact I was explaining to Steve Harman yesterday that if you wanted to create a full-blown Unit Of Work ORM, you could. You could also create an ActiveRecord pattern too. Really - it's that flexible.

The "Surface"
There's probably a better name for this - but it seems to capture what I'm after: a simple, easy to understand way to work with your database. This template (called "QuerySurface" - which used to be "Provider") exposes the tables in your database as IQueryable<T> so you can work against them using Linq:

            Northwind.DB db = new DB();
            var result = from p in db.Products
                         where p.CategoryID==5
                         select p;

            foreach (var item in result) {
                Console.WriteLine(item.ProductName);
            }

I can go off here and show you all the ways to query with Linq. But you've seen em before and hopefully you won't notice the difference with SubSonic. If you've used Linq To Sql, this will look extremely familiar to you. However know that my approach differs from Linq To Sql in that there is no change tracking and need to have a "context". The DB, in this sense, is simply a wrapper.

One thing you might not have seen with Linq To Sql is the ability to run Updates and Inserts, which I've always missed and have now implemented with SubSonic 3.0:

            db.Update<Products>().Set(
                x => x.Discontinued == false, 
                x => x.ReorderLevel == 100)
               .Where(x=>x.Category==5)
               .Execute();

        db.Insert.Into<Region>(x => x.RegionID, x => x.RegionDescription)
          .Values(6, "Hawaii")
          .Execute();

I know a lot of people don't like the Lambda thing - you can still use our old query tool (with structs or strings in place of these lambdas) if you like.

This one's a personal favorite - deleting has become a one-line operation:

            db.Delete<Region>(x => x.RegionID == 6).Execute();

This next thing is what I consider to be a "Killer Feature" - a batch query (or as NHibernate calls it, a "Future" query). One thing that always gets me about ORMs is having to manage the connections that it makes to a database. Wouldn't it be nice to lob a bunch of IQueryable queries into a batch querier and have it make only one call to the DB, in one execution step? If you think so, I have good news for you:

    db.Queue(from p in db.Products where p.CategoryID == 5 select p);
    db.Queue(from p in db.Products where p.CategoryID == 7 select p);
    db.Queue(from p in db.Products where p.CategoryID == 9 select p);
    IDataReader rdr=db.ExecuteBatch();

With this release I've included a new "BatchQuery" class that takes either one of SubSonic's queries or an IQueryable query, parses the SQL and pulls out the parameters into one, big batched SQL statement. It then will execute as a pass-through, or return a multi-result set. Yes, I will do my best to support transactions this way in the future :).

If you don't like working with DataReaders, never fear, one of the new members of SubSonic's Extension classes (DatabaseExtensions) has a new method for IDataReader called "ToList<T>()" - which will take a reader and shove the results into a "List<T>".

So to complete this example (making sure to reference SubSonic.Extension.DatabaseExtensions):

    db.Queue(from p in db.Products where p.CategoryID == 5 select p);
    db.Queue(from p in db.Products where p.CategoryID == 7 select p);
    db.Queue(from p in db.Products where p.CategoryID == 9 select p);

    List<Products> result1 = null;
    List<Products> result2 = null;
    List<Products> result3 = null;


    using (IDataReader rdr = qry.ExecuteReader()) {
        result1 = rdr.ToList<Products>();
        
        if (rdr.NextResult())
            result2 = rdr.ToList<Products>();

        if (rdr.NextResult())
            result3 = rdr.ToList<Products>();

    }

One SQL statement, one execution step, 3 typed lists. I like it :).

Believe it or not you probably shouldn't work "this close" to your database. This is where the 3rd tier comes in.

The Implementation
Hopefully you can see what I'm trying to do here - give YOU the power. By way of example I've created an IRepository<T> interface that's part of SubSonic:

    namespace SubSonic {
        public interface IRepository<T> {
    
            IQueryable<T> GetAll();
            PagedList<T> GetPaged(int pageIndex, int pageSize);
            IQueryable<T> Find(Expression<Func<T, bool>> expression);
            void Add(T item);
            int Update(T item);
            int Delete(T item);
            int Delete(object key);
            int Delete(Expression<Func<T, bool>> expression);
    
        }
    }

One of the templates you'll see in the download below is "Repository.TT", which is an implementation of this interface that you can use right now if you like. One day I might make an ActiveRecord one as well - but for now you can use this code like this:

            NorthwindRepository<Region> repo = new NorthwindRepository<Region>();
            Region r = repo.Find(x => x.RegionDescription == "Hawaii").SingleOrDefault();
            r.RegionDescription = "Hawaii FIVE O";
            repo.Update(r);

Again, one of my very favorites is "Delete":

            NorthwindRepository<Region> repo = new NorthwindRepository<Region>);
            repo.Delete(x=>x.RegionID==6);

You can also use the item, or a primary key:

            repo.Delete(6);
            //OR
            Region r=repo.Find(x=>x.RegionID==6).SingleOrDefault();
            rept.Delete(r);

Let's Watch a Video!
A lot of people have asked for a "Rob special" video, and not to disappoint I've created one to show you how to get started (props to whomever knows the tune). It's 4 minutes and covers how to use SubSonic 3.0 (and its templates) with a Console App. You can also use this with any other app EXCEPT for a Web Site - for some reason T4 doesn't like those:

(Double-click for full screen, or you can download the WMV here).

The Download

You can download Preview 2 here. I've fixed/tweaked/changed a lot of stuff, including:

Fixed an issue with Columns not generating. I think this was because I wasn't closing connections properly, and some databases were case-sensitive. I've tested this on 2008 and 2005 and it works well.
Made every class Partial
Changed tables from Fields to Properties
Fixed a paging bug with the Linq translator
Renamed some TT files. Provider has become "QuerySurface" and Connection.TT is now called "_Utility.tt".
Renamed the no-code, required TT files to have underscore
Implemented MANY of the changes suggested in this post by jguerts (thanks!)

As always- if you have any questions please let me know. If they're long ones, please email me at microsoft (robcon) or my full name at Gmail.

Cheers!

SubSonic 3.0 Preview 1: Linq Has Landed

Rob Conery — Sat, 08 Nov 2008 02:51:40 GMT

I've been working a lot over the last few months on our next rev of SubSonic, and I think I have something that's good enough to issue as a preview. Please understand that this is a preview in every sense of the word, and may shatter into a zillion pieces and make you want to pour your Red Bull over my head. This is bleeding edge edge crazy talk. It's everything bad that wakes you up screaming at night, and quite possibly is the coolest thing I've ever made... or somewhere in between. Just know that if you download and play with it, you're in for a ride... maybe. Or maybe it will shine like a million suns at the dawn of man...

SubSonic 3.0 Is...
A complete rewrite. The culmination of the things I've learned over the last year as I've (once again) Changed Everything and shared some wonderful Koolaid with my Alt.NET buddies :). Seriously - I've ripped everything out and redone it, with a focus on:

Simpler
Lighter
Simpler
Meaner
Simpler
Better
Faster
Simpler

(Not in that order - but generally I've placed an emphasis on simplicity). I've installed interfaces everywhere, and focused on making everything lightweight, testable, and injectable. I hope. We'll see.

Overall I've had exactly 5 false starts. I literally have 6 folders on my hard drive right now, each named "SubSonic 3.0 [X]" where "X" is the iteration number. Sometimes it's a swear word - but I won't go there now. Linq is hard - no really - just ask Ayende. It's not a matter of smarts - it's just really hard to try to fit Linq in... trust me. I was hoping originally that I could just create an Expression parser and build stuff using our core bits... but umm... that didn't happen.

One of the blogger people I've been following as I've been working through all of this is Matt Warren, the Zeus of Linq To Sql. He has a great series of posts on how to implement IQueryable, and when he got to post number 9 I began to think I should just use his code wholesale (yes, I know it's cheap of me). He's now on post number 11, and I asked him a while back if I could just steal all of his code - and this is why Matt Warren is my personal Santa Claus - he said "yes - it's all under MS-PL anyway. Just forward the license".

So there it is - the SubSonic.Linq namespace is 99.999999% Matt Warren's code. I've tweaked it a touch to work with our core Query stuff, but everything else is him and his large brain. Thank him for this and have a read on what you get with this provider.

My philosophy while developing SubSonic 3.0 is this:

I really dig Linq as a language feature - I want to include it as a query tool
I don't like all the extra "shmekt" that we have right now for things like Code Generation and figuring out how to rename bad table names
I want to change our Provider Model - this can be much simpler
I want to give you a lot more power to make code you want to use
I want to use POCOs
I want Linq to work where it should, SubSonic to work where it should, and seemlessly make them work together
If I do this right, you can build on top of SubSonic, to do what YOU want to do.

There it is. And I want it to be stupid simple.

SubSonic 3.0 Is Not...
I think this is as important as what I want to do. So, what I don't want to do is:

Take over as the new Linq To Sql. I know this is weird timing, given the latest grumblings happening in .NET space, but believe me Linq is scarey, and translating Expressions to SQL is not easy. It's really, really hard (see below) and if I didn't like Linq so much (and IQueryable) I would have kicked it to the curb.
Create another ORM. SubSonic's not an ORM believe it or not. There is no "mapping" ability really - I spose you can say it's "ORM-y". It's a data access tool for sure - I'll leave the mapping to you, which is why I decided to use Linq cause it makes is very simple.
Suggest in any way that we're done. There's some work to do here :) so this is for you to play with sort of like a sleeping cat (which could scratch you) or a pocket full of kryptonite. It's preview and may splode.
Backwards Compatible. Yes I know this will bum a lot of people out but it's the only way I can really rev this thing properly. Eric and I stressed over this, but the issue is that language/tooling features are letting us do something completely different and if it helps trim the API, bloat, and footprint than that's what we do. Very sorry about this.

Onward! To Some Code!

The Setup
To use SubSonic 3.0 you'll need to do exactly 3 things (assuming you're using .NET 3.5, which is required):

Create project references to SubSonic, System.Data, and System.Core (if you don't have it already)
Make sure you have a connection string (at least one), with the providerName set appropriately (usually it's System.Data.SqlClient)
Drop in the T4 templates, edit the connection string in "Connection.tt", and watch it generate your action.

No providers, no funkiness. That's it!

Generated Code
There are fiveT4 templates as of right now (this will probably change), and they are:

Connection.tt - this is a non-code template (utility) that gets included with the other templates and does things like connect to your DB to get a list of tables and SPs, etc.
Classes.tt - this T4 template generates 1 class per table in your DB using POCOs
Provider.tt - this class wraps your Database in a class and adds IQueryable<T> fields as well as some factory methods for SubSonic queries (more below).
SPs.tt - wraps your Stored Procedures and "appends" them to your provider class so you can call them as methods
Structs.tt - wraps the columns and tables in your database so you can refer to them without using strings.

There's a lot more you can do here - and that leads me to the next thought...

This Aint No ORM Disco
The "Big Guy" ORMs (NHibernate, EF, Linq To Sql) keep track of an object's state and interacts with the DB in what's known as a "Unit of Work". This can be great sometimes, a PITA others. I've realized that ORMs are a slippery slope in this regard, and no one can really agree on what they should really do, when, and why.

So I've decided to focus one step below and try to make data access and manipulation silly simple. That way, in the future, maybe I (or you!) can work up some cool ORM templates to sit on top of the core SubSonic bits, and we can all share and have a party.

Just know that my focus, for right now, is to make working with the DB bloody crazy simple. The more complicated stuff can come later (eager loading, unit of work, etc) in the form of T4 templates.

Enough Already - Show Me Some Code!
The meat of the system is worked up by Provider.tt. This is a "wrapper" for your database and allows you to work with IQueryable and SubSonic at the same time:

namespace Northwind{
    public partial class DB{
    
        public IDataProvider DataProvider = new DbDataProvider("Northwind");
        public  DbQueryProvider provider;

        public Query<CustomerDemographics> CustomerDemographics;
        public Query<Region> Region;
        public Query<Employees> Employees;
        public Query<Categories> Categories;
        public Query<Customers> Customers;
        public Query<Shippers> Shippers;
        public Query<Suppliers> Suppliers;
        public Query<EmployeeTerritories> EmployeeTerritories;
        public Query<Order_Details> Order_Details;
        public Query<CustomerCustomerDemo> CustomerCustomerDemo;
        public Query<Territories> Territories;
        public Query<Orders> Orders;
        public Query<Products> Products;        
        
        public Select Select() {
            return new Select(DataProvider);
        }
        public Insert Insert() {
            return new Insert(DataProvider);
        }        
        public Update<T> Update<T>() where T:new(){
            return new Update<T>(DataProvider);
        }
        public Delete Delete(){
            return new Delete(DataProvider);
        }

There's more that's generated here - but I'll get to that in a second. To use IQueryable, you do what you normally do with Linq To Sql:

    Northwind.DB db = new Northwind.DB();
    var products = from p in db.Products
                   where p.CategoryID == 5
                   select p;
    foreach (var p in products) {
        Console.WriteLine(p.ProductName);
    }

Which produces what you might think (using delayed execution):

Nothing really extraordinary about this - we've seen it before :). Most things are covered using the Linq provider that Matt started, but I'm sure there are some things missing - the good news there is that you have SubSonic's query tool to back you up if you get stuck (which is a lot of fun since you're not locked into one way of doing things now):

    var products2 = db.Select.From<Northwind.Products>()
        .Where(Northwind.ProductsTable.CategoryID)
        .IsEqualTo(5)
        .ExecuteTypedList<Northwind.Products>();

Note that you can also just use a string in the Where() statement.

And if you really get stuck, we've got your back so you can avoid all of the "Proprietary Object Noise" (with apologies to Jeff Atwood):

    var products3 = new CodingHorror("SELECT * FROM Products WHERE CategoryID=@id", 5)
        .ExecuteTypedList<Northwind.Products>();
    foreach (var p in products2) {
        Console.WriteLine(p.ProductName);
    }

I Like Lambdas
I've tried to rethink everything in terms of how the bits are put together so we can flex the changes in the language. To that end I've included lambdas where appropriate to add some type-safety etc. I know some people don't like em, but they're optional, as always.

    db.Insert.Into<Northwind.Region>(x => x.RegionID, x => x.RegionDescription)
        .Values(6, "Hawaii").Execute();

    object avg = db.Avg<Northwind.Products>(x => x.ProductID);

The latter example here is why I love lambdas so much - they can (if implemented properly) "terse up" your code nicely. You could use Linq on db.Products for this as well - but I like the terseness of this one line :).

All of our aggregates are now reachable from the db wrapper class:

    db.Avg<Northwind.Products>(x => x.ProductID);
    db.Sum<Northwind.Products>(x => x.ProductID);
    db.Count<Northwind.Products>(x => x.ProductID);
    db.Min<Northwind.Products>(x => x.ProductID);
    db.Max<Northwind.Products>(x => x.ProductID);
    db.Variance<Northwind.Products>(x => x.ProductID);
    db.StandardDeviation<Northwind.Products>(x => x.ProductID);

The best part is that if you don't like any of this, just change your template! It's all right there for you to tweak and alter to fit your project, which is a major goal of mine here: I want to give you the power to use SubSonic as you please. And if you do it better than us (which I'm sure you will) - perhaps you'll share. My focus is on the engine - you guys can hop it up with chrome and neon.

The biggest support issue we've had is with regards to our generated stuff. Specifically how we name objects. Our code generation is now completely transparent so if you don't like it - you can change the name of that table to be what you like!

Using System.Data.Common
At the core of the flexibility we now have is our use of System.Data.Common and the DataFactory pattern. If you don't know what this is, have a read here. The idea behind System.Data.Common is that we shouldn't have to rewrite all of our data access stuff if we happen to change database platforms. To that end the DataFactory pattern was created so that ADO data provider developers could write against a common API, and you could benefit.

We've used that to our advantage with SubSonic 3.0. So if you need to switch your DB provider, for instance, you can change from this:

    <connectionStrings>
        <add name="Northwind" 
        connectionString="server=.\SQLExpress;Integrated Security=true;database=northwind;" 
        providerName="System.Data.SqlClient"/>
    </connectionStrings>

To this (provided you have the DLL in your /bin):

    <connectionStrings>
        <add name="Northwind" 
        connectionString="server=localhost;user id=root;pwd=HAHA?;database=northwind;" 
        providerName="MySql.Data.MySqlClient"/>
    </connectionStrings>

There is no other setting - no "provider tweaks", no jiggity dances and prayers. The only assumption here is that you have the same DB on your target platform (which I do - I've ported Northwind to MySQL) and run:

Most database providers support ADO.NET 2.0 and the Data Factory. Some off the top of my head that I know of are MySQL, Oracle, and SqlLite and I'm sure there are others - I just don't know.

You can download SubSonic 3.0, Preview 1 here.

We're still actively working on it and I'm sure you'll encounter bugs, etc. Feel free to send me an email (see the README) and please try to let me know what you were doing in the context of Northwind, so I can repro the issue.

As always, please be patient.

Epilogue: SubSonic.Sugar is now SubSonic.Extensions.
Wanna have some fun? Have a look at the SubSonic.Extension methods that I've ported from the Sugar class and turned into applicable extensions. To use them, you'll need to make sure you reference the appropriate namespace:

    using SubSonic.Extensions.Strings;
    using SubSonic.Extensions.Dates;
    using SubSonic.Extensions.Files;
    using SubSonic.Extensions.Linq;
    using SubSonic.Extensions.Numbers;
    using SubSonic.Extensions.Objects
    using SubSonic.Extensions.Validation;

In there is some fun, and some damn handy utilities. Things like Date math, file creation and text loading, string pluralization, regex validation and matching (Validation's "Is" functions rock - things like "IsValidEmail" and "IsCreditCard" are super groovy).

I mention them last as they are, literally, sugar utilities to help you through your day, and help us with SubSonic. They're not the main point at all.

One Year At Microsoft

Rob Conery — Tue, 04 Nov 2008 06:38:04 GMT

I just got my notice that it's been one year here at the Blue Monster. Interesting, that. It's made me think about a lot of things with respect to my job - probably some things a typical one-year blue badge might not think.

My Boss Is a Rock Star
This is probably the biggest surprise for me. No, I'm not sucking up, though it may seem that way. It's just not... me to suck up to anyone :) and it's one reason I really like Shawn :). There are no "brown-nose" the boss kind of games with him, no power trips and no weird ego games. Hands-down he's the best boss I've ever worked for and believe it or not I didn't expect it when I was hired. Not because of him - but because of the caliber of people that Microsoft hires.

And he runs Iron Mans and is one hell of an athlete. I thought he'd be one of "those guys" - you know the super-athletic boss guys who eat rice cakes and don't drink and think video games are for fat girly-men. He's not any of those and, it turns out, I have less body-fat than he does - isn't that strange? :).

During my interview we had a very thoughtful conversation that roamed from Fibonacci numbers, to Kona Iron Mans, and then off to Public Enemy songs. I was pleasantly surprised to find that not only was Shawn a very engaging person, but he was really into what I wanted to do at Microsoft, and why. I don't know why I find that weird (in a good way). Turns out that most other people have a similar experience with their managers - and that's just the way it is there. I think that's groovy.

Culture as Concept
I work remotely, at my home office, and I don't get to involve myself regularly in the "cultural" things that happen at the company. Sometimes this is a good thing - other times it's bad. I'm able to retain the sense of "who I am" and really feel like I have my own voice. Other times I really feel like I'm missing out.

The neat things is that all I have to do is remote in to the network and cruise the intranet, or start up an IM with my compadres. Video conversations are easy, and everyone will make time to chat with you - even if they have no idea who you are.

Right now I'm working on getting an old computer setup in Phil's office with a permanent vide feed to my desk here. That way I'll be Phil's virtual office-mate and any time someone wants to drop by, they can. Seriously!

I get to fly out to Redmond every other month and every time I'm there I can't believe I work for a company that's this open, this exciting, and this much fun. Yes, the Koolaid is lovely, thank you. Believe it or not I actually mean this :).

Seriously, there is so much going on at a given time - so many decisions and ideas and personalities - that I literally giggle every time I swipe my badge at Building 42.

A feeling of smallness takes you over - seriously. I'm not an egotist and I hope you don't think I'm sitting here trying to name-drop, but it's really how I feel. Walking by a nameplate that says "Scott Guthrie" and having him wave as you walk by is ... well it's ridiculous. And it makes me laugh.

If I was there every day I might get used to it - rubbing shoulders with guys like ScottGu, Nikhil, Brad Abrams, Phil, and especially Eilon. But I'm not - so when I visit I feel like a kid visiting his big brothers off at college...

Contributing
Microsoft has a really cool way of allowing you to do what you do - it's called "here's your rope, how much you want?". I was told shortly after I was hired to "tell, don't ask and see if it flies" - and that's what I've been doing. Most everyone is receptive, but I've found that they look to you to just do what you do, and excel as you see fit. There is no "well let's see what so and so says" - they kind of want to see results first. If you fail - which I've done - people are quick to pull together to find out what went wrong and how to fix it.

I love that. I know that a lot of companies work that way - I've just never experienced it. Success and Failure usually means attribution to your boss in some form, and also making sure you align it with his boss's stuff - your typical Survivor politics. That doesn't work here, and I'm grateful for it.

Assessment
I feel like I've done well my first year. It's not an easy company to work for and that's not because of any weird authoritarian management policy or ticking off goals on a list. As I've mentioned - it's 100% up to me to succeed or fail, and I have to what it takes, completely, to navigate the waters and succeed.

What makes this particularly challenging is that I'm working with guys like Hanselman and Brad Wilson. Jon Lam, Jesse Liberty and Glenn Block. Eilon Lipton! People I greatly respect and whom I have to measure myself against (how do you do that?). I don't come to confidence easy, and I have a really hard time telling myself I belong here. But here I am indeed, and it's "do, or do not - there is no try".

I can safely say it's the best job I've ever had. You might have your opinions on my being "borged" and that's OK. I like the challenge. Ask Phil sometime how "borged" I am :), he'll tell you that I've passed my Koolaid off to him :).

Crazy Talk: Reducing ORM Friction

Rob Conery — Tue, 04 Nov 2008 03:08:28 GMT

Let's get this out of the way: I know you're going to think I'm nuts as you read this. You may "pfft" to what you're about to read - know that I know you're "pfft"-ing me. All I ask is that you consider what I'm about to suggest...

Development Friction
The term "friction" gets thrown around a lot in terms of development. It's what it sounds like: something you do or a process you undertake that slows you down as you crank out code. If you think on this for a second - when you're building an application what's the number one thing that slows you down (technically speaking. Running out of Red Bull doesn't count)?

For me it's the database "stuff". It's why I made SubSonic - I was tired of thinking about it and I wanted something faster and easier.

Tangent
This whole post (and thing I'm about to dive into) was dreamt up during a bike ride to the store. I thought a fun post idea was to "send a post back in time" and entitle it "Greetings from the Year 2012", and I would laugh at all the silly stuff I did in 2008. The top of that list, for me, is the continuing struggle we have with persisting data "properly". After 10+ years, the marriage of web and database is still arguing over the same old stuff. Can't we move on? Shouldn't this be easier by now?

You might have other sources of friction, or you might be saying "dude [MY ORM] roolz! LOLZ at U". I'll bet it does - but you still need to work with it (and your database) as you build out your site. Even if it takes you 1 minute to "update and regen" - you still have to mess with the DB and mess with mapping (if you do that).

I've talked with a lot of people over the last few weeks about this and asked them the same question:

In 5 years what do you think will finally be changed?

The answer, every single time, is a variation on "ORM's will finally work properly". What if I told you that you don't need to wait for 5 years for this? What if I told you to ditch your ORM and your database and focus on what's the most important thing: your application?

How Do You Do What You Do?
There are generally two camps of developers:

The guys who create a database and then build the app, and then update the database, and then change the app, etc.
The guys who write tests, create a model, update their database and remap their ORM, rinse and repeat.

If you're a TDD fan (or want to be) you might be interested in Domain-Driven Development (DDD). Yes, it's another one of those buzz-words but you might actually be a DDD person right now, and not even know it. Check it out:

"What it's all about is creating as simple a model as possible, one that still captures what's important for the domain of the application. During development the process could really be described as knowledge-crunching by the developers and domain experts together. The knowledge that is gained is put into the model."
(Nilsson, Applying Domain-driven Design and Patterns)

Yes, I'm quoting a DDD book. I've been absorbed :). The point here is that most of us have always worked this way - working closely with clients to understand their business, and making sure what we build is focused.

The split comes in with what system you work from: the database or the tests? Which is more appropriate in terms of building out an application domain? I'm hoping to convince you, today, to toss your database as you don't need it. Not yet anyway. Focus on your tests and your domain and I think you'll see that you can move a lot faster.

Database YAGNI (or the Database as a Feature)
YAGNI is the principle of "You Aint Gonna Need It", which essentially means "don't add it unless requirements/testing make you add it" and is one of the main drivers of TDD. It keeps code light and manageable, and keeps cruft out of your application.

I think this should apply to architectures as well - why implement, design, and build a database (with data access code) when the application doesn't need it? And yes, this is where the Crazy Talk kicks in.

Databases are pretty heavy information organization and retrieval systems. Even the lighter-weight ones are capable of powering some pretty high-level querying at very rapid speeds.

Do you need this when you're developing? Do you need to play "ORM-Catch up"? Probably not.

What if, in 10 years, the platform could just translate your model for you and save it properly without you thinking about it?

What if I told you that you can do that now? Well you can - and this is where you're gonna say "PFFT" and tell me I'm crazy. But as I said - I know you're going to say it... so just pretend I can hear you...

</CrazyTalk>

Off The Reservation with OODBs
Object Databases have been around for a long, long time. If you've never heard of these things, well they basically crumple your object up into binary (by serialization) and save it to disk for you to access later. If you'd like to read more, this is a great post on OODBs, what they are, and how they work. To summarize:

An OODBMS is the result of combining object oriented programming principles with database management principles. Object oriented programming concepts such as encapsulation, polymorphism and inheritance are enforced as well as database management concepts such as the ACID properties (Atomicity, Consistency, Isolation and Durability) which lead to system integrity, support for an ad hoc query language and secondary storage management systems which allow for managing very large amounts of data. The Object Oriented Database Manifesto [Atk 89] specifically lists the following features as mandatory for a system to support before it can be called an OODBMS; Complex objects, Object identity, Encapsulation , Types and Classes ,Class or Type Hierarchies, Overriding,overloading and late binding,Computational completeness , Extensibility, Persistence , Secondary storage management, Concurrency, Recovery and an Ad Hoc Query Facility.

Here's my thought for you: What if you used an OODB for development ONLY and implemented SQL Server later, when you know what you need to create.

You can't ditch your RDBMS entirely - never. However I will suggest that working on two systems at once, when you don't need to, is silly. You can do a lot more in terms of RAD development right now - and port to SQL later.

Doesn't it make more sense to create the database when, and only when, you need it? Hold that thought - I'm coming back to it.

DB4O - A Free OSS Object Database
I've been using DB4O a lot over the last few months and I really like it. The tutorials are very easy and getting up to speed is no problem at all. First, however, I'd like to go over some things you might be wondering about.

Let's say I have three objects in my model:

    public class Product {
        public string Name { get; set; }
        public Supplier Supplier { get; set; }
        public IList<Review> Reviews { get; set; }
    }

    public class Review {
        public string AuthorEmail { get; set; }
        public string Body { get; set; }
    }

    public class Supplier {
        public string Name { get; set; }

    }

You might be wondering about how these objects are persisted, and how integrity might be enforced. It's actually required that they enforce many of the same concepts as an RDBMS so that you don't make a mess out of your model storage.

For instance, if I create a Supplier for a Product, it will be stored as an independent Supplier that I can then assign to another Product. If I change that Supplier's name, it will get changed for both. It's a single object, and the OODB works with the idea of "pointers" in the same way that a regular database does. The thing here, however, is that there is no joining - the relationships implicit and understood. In this way, an OODB can actually (and often do) outperform their RDBMS counterparts.

But I'm not here to talk about perf and scaling - it doesn't matter for what I'm suggesting :).

If you're curious and want to play along at home, go and download DB4O from their website and install it. To use it you have to make a few references in your application (specifically Db4objects.Db4o.dll). You then need to write some wrapper code - but I've got that covered for you:

using Db4objects.Db4o;
using System.Web;
using System.IO;

public class DB4O {

    static readonly object padlock = new object();

    // static object container variable 
    static IObjectContainer _db = null;

    static string _dbPath = System.Configuration.ConfigurationManager.ConnectionStrings["ObjectStore"].ConnectionString;
    public static string DBPath {
        get {
            return _dbPath;
        }
        set {
            _dbPath = value;
        }
    }

    public static IObjectContainer Container {
        get {
            lock (padlock) {
                if (_db == null) {

                    //check to see if this is pointing to data directory
                    //change as you need btw
                    if (_dbPath.Contains("|DataDirectory|")) {

                        //we know, then, that this is a web project
                        //and HttpContext is hopefully not null...

                        _dbPath = _dbPath.Replace("|DataDirectory|", "");
                        string appDir = HttpContext.Current.Server.MapPath("~/App_Data/");
                        _dbPath = Path.Combine(appDir, _dbPath);
                    }

                    _db = Db4oFactory.OpenFile(_dbPath);
                }
                return _db;
            }
        }
    }

    public static void CloseContainer() {
        if (_db != null) {
            _db.Close();
        }
        _db = null;
    }

This code assumes that you

Have an App_Data directory and that you have a connection string in your Web.config like this one:

<add name="ObjectStore" connectionString="|DataDirectory|ObjectStore.yap"/>

Yes, that's a singleton you see there, and yes I know it's probably making you cringe. The reason we need to use a singleton is that the IObjectContainer locks the binary file where the data is kept. File locking and singletons might not sit well with you right now, but in this case is that the way I have this setup here is for a single user - ME - because I'm developing against it. If this were a live app I would be able to set a bunch of settings to make this thread-safe etc.

But it's not production - it's development only (have I mentioned this yet?) so you don't need to worry about singletons, perf, and scaling.

Great, so now that we have our container, let's store an object. This isn't really a test, but you've read a lot more crazy stuff so far, so this won't surprise you much. Consider this a spike please - or maybe pretend I'm asserting something:

[TestMethod]
public void ObjectRepo_Should_Store_Product() {
    Product p = new Product();
    p.Name = "test product";
    p.Supplier=new Supplier("Test supplier");
    DB.Container.Store(p);
}

Yes, it's that easy. But that's not the best part. I can add a DLL to the project that DB4O just released, called "Db4objects.Db4o.Linq" and it does what you might imagine, which is exremely cool:

[TestMethod]
public void ObjectRepo_Should_Return_Product() {
    var result = from Product p in DB.Container
                 where p.Name == "test product"
                 select p;

    Assert.AreEqual(1, result.Count());
}

Yes, that would be LINQ, working with an OODB container. We can also query a bit deeper, with no problems:

[TestMethod]
public void ObjectRepo_Should_Return_Product_By_Supplier() {
    var result = from Product p in DB.Container
                 where p.Supplier.Name == "Test supplier"
                 select p;

    Assert.AreEqual(1, result.Count());
}

And to illustrate my point above about object independence and integrity, I can also get the Supplier, independent of the Product:

[TestMethod]
public void ObjectRepo_Should_Return_Supplier() {
    var result = from Supplier s in DB.Container
                 select s;

    Assert.AreEqual(1, result.Count());
}

This, literally, is the tip of the iceberg. DB4O has support for transactions, indexing, and many different ways of querying to improve performance and usage. But I'm not talking about perf here :) - it doesn't matter, this is only for development.

Also it's worth noting that if I change the properties of Product around, it won't break. The changed property just won't get loaded - but everything else will. So you can change and alter as required and nothing breaks!

Real-World Application
What I'm suggesting is that you can create a IRepository<T> and then implement a nice ObjectRepository<T> to work with in your application. It's very simple - and yes, here's some more code for you:

using System;
using System.Collections;
using System.Linq;
using System.Linq.Expressions;

public interface IRepository<T> {

    IQueryable<T> GetAll();
    PagedList<T> GetPaged(int pageIndex, int pageSize);
    IQueryable<T> Find(Expression<Func<T, bool>> expression);
    void Save(T item);
    void Delete(T item);

}

Now, you can implement this quite nicely with DB4O:

using System;
using System.Collections;
using System.Linq;
using Db4objects.Db4o.Linq;


public class ObjectRepository<T> : IRepository<T> where T: class {

    /// <summary>
    /// Returns all T records in the repository
    /// </summary>
    public IQueryable<T> GetAll() {
        return (from T items in DB4O.Container
                select items).AsQueryable();
    }

    /// <summary>
    /// Returns a PagedList of items
    /// </summary>
    /// <param name="pageIndex">zero-based index to be used for lookup</param>
    /// <param name="pageSize">the size of the paged items</param>
    /// <returns></returns>
    public PagedList<T> GetPaged(int pageIndex, int pageSize) {
        var query=(from T items in DB4O.Container
                select items).AsQueryable();
        return new PagedList<T>(query,pageIndex,pageSize);        }

    /// <summary>
    /// Finds an item using a passed-in expression lambda
    /// </summary>
    public IQueryable<T> Find(System.Linq.Expressions.Expression<Func<T, bool>> expression) {
         return GetAll().Where(expression);
    }

    /// <summary>
    /// Saves an item to the database
    /// </summary>
    /// <param name="item"></param>
    public void Save(T item) {
        DB4O.Container.Store(item);
    }

    /// <summary>
    /// Deletes an item from the database
    /// </summary>
    /// <param name="item"></param>
    public void Delete(T item) {
        DB4O.Container.Delete(item);
    }

}

If you're wondering what a "PagedList" is - you can find out more here.

ORM Selection Made Easy
Suppose you didn't need to worry about your database as you build your application. Better yet, suppose you didn't need to worry about your ORM! This latter thought is actually a critical, critical item. When I suggested that you could wait until you're about to launch to actually implement a database, a friend of mine asked me "well if you do that, you might back yourself into such a complex model that your ORM won't handle it. What do you do then?".

And I said "BINGO".

Put another way, what you're getting by not worrying about your ORM (until you need it) is the freedom to develop your app without influence from your database. It is true that you can make a model that's too complex for your favorite ORM - but doesn't that mean your favorite ORM probably wasn't up to the task anyway? Isn't it much nicer to find that out the easy way?

In most cases you can probably just jump over to SQL very simply, just by replacing the reference to ObjectRepository to something like SqlRepository (this code is using Linq To Sql - but you can change this out with EF in the future - I'll try to update this later):

using System;
using System.Collections;
using System.Data.Linq;
using System.Linq;

public class SqlRepository<T> : IRepository<T> where T: class {

    NorthwindDB.DB _db = null;
    public SqlRepository(){
        _db=new NorthwindDB.DB();
    }
    /// <summary>
    /// Gets the table provided by the type T and returns for querying
    /// </summary>
    private Table<T> Table {
        get { return _db.GetTable<T>(); }
    }

    /// <summary>
    /// Returns all T records in the repository
    /// </summary>
    public IQueryable<T> GetAll() {
        return Table;
    }

    /// <summary>
    /// Returns a PagedList of items
    /// </summary>
    /// <param name="pageIndex">zero-based index to be used for lookup</param>
    /// <param name="pageSize">the size of the paged items</param>
    /// <returns></returns>
    public PagedList<T> GetPaged(int pageIndex, int pageSize) {
        return new PagedList<T>(Table, pageIndex, pageSize);           
    }

    /// <summary>
    /// Finds an item using a passed-in expression lambda
    /// </summary>
    public IQueryable<T> Find(System.Linq.Expressions.Expression<Func<T, bool>> expression) {
        return Table.Where(expression);
    }

    /// <summary>
    /// Saves an item to the database
    /// </summary>
    /// <param name="item"></param>
    public void Save(T item) {

        if (!Table.Contains(item)) {
            Table.InsertOnSubmit(item);
        }
        _db.SubmitChanges();
    }

    /// <summary>
    /// Deletes an item from the database
    /// </summary>
    /// <param name="item"></param>
    public void Delete(T item) {
        Table.DeleteOnSubmit(item);
        _db.SubmitChanges();
    }

}

Since we're wrapping everything in IRepository<T>, you can swap parts as needed. I'll be you were wondering why you might want to develop this way (I know I was a long time ago - who really ever swaps components anyway?) - but this is a good example of why you might want to decouple your system as much as possible.

Swapping out data stores like this is equivalent to turning your minivan into a ferrari if you want to drive the Autobahn, and back again when you need to get the kids from Soccer.

Am I nuts? If you think I am - please give me some details as I think this would make for an interesting discussion. Just don't tell the Alt.NET guys yet :).