Software Development Fundamentals, Part 2: Layered Architecture

This is part of a series of introductory guidelines for software development. It’s a continuation of the previous post about Dependency Injection.

One of the primary reasons to adopt Dependency Injection is that it is impossible to achieve a good layered architecture without it. Many people say that IoC container is only beneficial if you are doing test-driven-development. I disagree. I just don’t understand how you can even build a layered architecture at all without IoC.

First, let’s take a moment to define layered architecture. There is a common misconception about layered architecture, which is largely contributed by misleading diagrams in most textbooks and articles. This is how layered architecture often described in textbooks.

I think that’s rather deceptive. Unfortunately, beginner developers often take this diagram too literally. Each layer depends on the layers below it. Presentation layer depends on business-layer, and then both depend on data-access layer.

There is a huge flaw in that diagram. Data-access layer should be at the outmost part of application. Business layer should be at the core! But in this diagram, Data-access layer is becoming the core foundation of the whole application structure. It is becoming the critical layer. Any change on data-access layer will affect all other layers of the application.

Firstly, this architecture shows an incongruity. Data-access layer, in reality, can never be implemented without dependencies to business detail. E.g. DbUserRepository needs a dependency to User. So what often happens is developers start introducing a circular dependency between Business layer and DataAccess layer. When circular dependency happens between layers, it’s no longer a layered architecture. The linearity is broken.

Secondly, which is more important, this architecture tightly couples the whole application to infrastructure layer. Databases, Web-service, configurations, they are all infrastructure. This structure could not stand without infrastructure. So in this model, developers build the system starting from writing the infrastructure plumbing first (e.g. designing database-tables, drawing ERD diagram, environment configuration), then followed by writing the business code to support the infrastructure.

It’s a bit upside-down and not quite the best approach. If business-layer couples itself with infrastructure concerns, it’s doing way too much. Business layer should know nothing about infrastructure. It should be in the very core of the system. Infrastructure is only a plumbing to support the business-layer, not the other way around. Infrastructure details are most likely to change very frequently, so we definitely do not want to be tightly coupled to it.

Business layer is an area where you have the real meat of your application. You want it to be clean of any infrastructure concern. Development starts from domain-code, not data-access. You want to write the business code right away without setting up or even thinking about the plumbing. One of the guideline in previous post is that we want classes within business layer to be POCO. All classes in business layer should describe purely domain logic WITHOUT having any reference to infrastructure classes like data-access, UI, or configuration. Once we have all domain layer established, then we can start implementing infrastructure plumbing to support it. We get all our domain models baked properly first, before we start thinking about designing the database tables to persist them.

So this is a more accurate picture for layered architecture.

Infrastructure sits at the top of the structure. Domain layer is now the core layer of the application. It doesn’t have any dependency to any other layer. From code perspective, Domain project does not have any reference to any other project, or any persistence library. Databases, just like other infrastructure (UI, web-services), are external to the application, not the centre.

Thus, there is no such thing as "database application". The application might use database as its storage, but simply because we have some external infrastructure code in the neighbourhood implementing the interfaces. The application itself is fully decoupled from database, file-system, etc. This is the primary premise behind layered architecture.

Alistair Cockburn formalizes this concept as Hexagonal Architecture, so does Jeffrey Palermo as Onion Architecture, but they truly are merely formalized vocabularies to refer to the old well-known layered architecture wisdom.

Onion Architecture

To describe the concept, let’s switch the diagrams into onion-rings. Jeffrey Palermo describes bad layering architecture as follows:

In onion diagram, all couplings are toward the centre. The fundamental rule is that all code can only depend on layers more central, never on the layers further out from the core. This way, you can completely tear out and replace the skin of the onion without affecting the core. But you can’t easily take away the core without breaking all outer layers.

In the diagram above, the infrastructure sits in the core of the onion. It becomes an unreplaceable part of the system. The structure cannot stand without infrastructure layer in the core, and the business layer is doing too much by embracing the infrastructure.

This is the better architecture model:

UI and infrastructure sits right at the edge skin of the application. They are replaceable parts of the system. You can take the infrastructure layer completely and the entire structure stays intact.

Implementation

Take a look at the diagram again.

If you follow the previous post, we were building "resend PIN to email" functionality. AuthenticationService is in the domain layer at the core of application, and it does not have knowledge about SMTP client or database (or even where User information is stored). Notice that DbUserRepository and SmtpService are in outmost layer of the application, and they depend downward on the interfaces in the core (IUserRepository, IEmailService) and can implement them. Dependency Injection is the key.

public class AuthenticationService: IAuthenticationService
{
	IUserRepository userRepository;
	ICommunicationService communicationService;

	public UserAuthenticationService(IUserRepository userRepository, ICommunicationService communicationService)
	{
		this.userRepository = userRepository;
		this.communicationService = communicationService;
	}

	public void SendForgottenPassword(string username)
	{
		User user = userRepository.GetUser(username);
		if(user != null)
			communicationService.SendEmail(user.EmailAddress, String.Format("Your PIN is {0}", user.PIN));
	}
}

At runtime, IoC container will resolve the infrastructure classes that implement the service interfaces (IUserRepository, ICommunicationService) and pass them into UserAuthenticationService constructor.

Project Structure

My typical project structure looks like this in Visual Studio:

I separate each layer into its own project. One of the benefits is that it physically enforces linear dependencies. Each layer can only depend on the layer below it. If a developer introduces a circular dependency between Domain layer and Data-Access, Visual Studio (or Java’s Eclipse) won’t even compile. Here is how these projects work together within the architecture:

Every component depends only to the components in the layer below it. As you can see, the only project that has reference to System.Data and NHibernate library is Sheep.Infrastructure.Data. Likewise, the only project with System.Web.Services reference is Sheep.Infrastructure.BackEnds. And none of those infrastructure projects is referenced by any other part of the application (and this is enforced by Visual Studio). Therefore, they can be totally taken away and replaced without affecting the application core. On the flip side, the whole infrastructure layer is tightly coupled on the core layer.

Domain layer is POCO. It does not depend on any other project, and it does not contain any reference to any dll for particular technology. Not even to IoC framework. It contains just pure business logic code. In order to be executed, this assembly has to be hosted either within a system that can provide infrastructure implementation, or within test system that provides mock dependencies. Hence notice that we have 2 versions of top-level skins in the structure: Infrastructure and Test.

So what’s the benefit?

1. Domain layer is now clean of infrastructure concern. Developers actually write business code in human readable language, not a messy pile of SQL statements/NHibernate, socket programming, XML document, or other alien languages
2. You can write domain code straight from the start without system consideration or waiting for infrastructure implementation. I.e. you don’t need a database at early stage of development, which is invaluable because you eliminate the overhead of keeping your database schemas in sync with domain models that are still shaky and keep changing every few minutes.
3. The application core can be deployed for any client or any infrastructure, as long as they can provide the implementation for the service interfaces required by the domain layer.

Basically, it all describes Single Responsibility Principle (each class should only have 1 reason to change). In the classic diagram, being tightly coupled to data-access layer, domain layer needs to change because of some changes in infrastructure details. Whereas in layered architecture, you can completely tear apart the infrastructure layer at the skin, and replace with arbritary implementation without even leaving a dust on any domain class at the core. Thus there is only 1 reason the classes in domain layer need to change: when the business logic changes.

Having said that, this architecture is not necessarily the best approach for all kind of applications. For simple form-over-data applications, Active-Record approach is probably better. It couples the whole application with data-access concern, but it allows a rapid pace of data-driven development. However, for most non-trivial business applications, loose coupling is a very crucial piece for maintainability.

This whole post is actually an introduction to our next dicussion about Domain Driven Design in some later post.
To be continued in part 3, Object Relational Mapping

Software Development Fundamentals, Part I

Preamble

I’m going to start a post (or perhaps a series thereof) to discuss some basic practices and fundamental concepts in building software applications. It’s mainly intended for those just getting into software development and seeking some pragmatic introductory guidelines how to start writing good code and how to structure our application. It might also interest seasoned developers as we discuss some great OOS tools to support these good development practices, particularly those unfamiliar with .Net neighbourhood. I’d love to hear feedbacks, opinions, or suggestions about other practices you always put in your list of default development guidelines.

There are obviously other methodologies and architectural styles in building applications, but the ones covered here are actually some of the most essential practices that have been taken as the "default architecture" for typical LoB applications by about every developers ruling the earth today. This is the default architectural project template that I use liberally to start every LoB application development. These guidelines aren’t written in stone, but you will need a very strong reason to break the rules.

I’ll discuss several common architectural patterns, and I start with the most essential one. Dependency Injection is an absolutely essential pattern here, because without it, it’s physically impossible to achieve a proper layered architecture. More about Onion Architecture further down this post.

Dependency Injection

Application codes that are composed of hugely entangled components are extremely hard to maintain. So this is our first rule: C#/Java new keyword is a bitch You probably have heard plenty of times that new is evil and perhaps wondering what’s our problem with the seemingly humble innocent word.

To explain it, let’s start writing some code. Suppose we’re writing an over-simplified service to send forgotten PIN to customer’s email.

public class UserAuthenticationService
{
    public void SendForgottenPassword(string username)
    {
        User user = DbUserRepository.GetUser(username);
        if(user != null)
            SmtpService.SendEmail(user.EmailAddress, "PIN Reminder",
                String.Format("Hello, your PIN is {0}", user.PIN);
    }
}

This class has direct dependencies to UserRepository and EmailService. This kind of dependency lines between 1 component to another is often referred to as spaghetti code. There are 3 problems with this code:

1. This code cannot be worked on in isolation. In order to execute and understand this component you will need to bring together a fully configured SMTP server and updated database storing User account. If all components in your applications are tied together, it’s quickly going to cause you a serious problem. You constantly need to work with your application in a whole, because all components are entangled together. To change one component, you need to change the others. Developers brains aren’t scalable. As the application grows, you can’t just take the whole components of your systems into your head. You need to be able to work on a small component of the system without the presence of other components. You need to decouple every part of your system to a tiny workable unit.
2. Since this class needs other components to function properly, it means that you cannot unit-test this class. Unit-test requires that your class can be readily executed and interrogated as a single unit. Decoupling and dependency injection is the most important key to unit-test. More detail about unit-test in next post.
3. This code violates the natural structure of application layering. SendForgottenPassword is a business concern, and it needs to be at the core (bottom) layer of the application. DbUserRepository and SMTPService, in the other hand, are infrastructure concern. Infrastructure layer lives in outmost skin of applications. However, what we have here is a wrong direction of dependency. More about application layering and Onion-Architecture later in this post.

DbUserRepository and SmtpService are implemented as static classes. Here is our second rule: static class is an anti-pattern. Not always, but it’s a good rule of thumb.

Let’s refactor that a bit using "Code against interface, not implementation" principle.

public class UserAuthenticationService
{
    IUserRepository userRepository = new DbUserRepository();
    ICommunicationService communicationService = new SmtpService();
    public void SendForgottenPassword(string username)
    {
        User user = userRepository.GetUser(username);
        if(user != null)
            communicationService.Send(user.EmailAddress, String.Format("Your PIN is {0}", user.PIN);
    }
}

This is better. Now our method is decoupled from any concrete dependency. Instead, we set a contract. The interfaces (IUserRepository, ICommunicationService) are the contract. They merely define what operation you need to perform. We declare that we need to send an email to somebody. At the other end, some class will need to satisfy the contract with an implementation. In this case, SmtpService provides that implementation. Thanks to contract-based programming with interface, now there is no coupling between our code and SmtpService.

There is one problem though. Yes, it still requires direct dependencies to satisfy the new keyword. This is where Inversion-of-Control comes into play. Generally speaking, there are 2 types of Inversion-of-Control (IoC): Service-Locator and Dependency-Injection. Most IoC containers support both patterns.

Let’s now refactor this code to use Service-Locator pattern.

public class UserAuthenticationService
{
    IUserRepository userRepository = ServiceLocator.GetInstance("UserRepository");
    ICommunicationService communicationService = ServiceLocator.GetInstance("CommunicationService");
    public void SendForgottenPassword(string username)
    {
        User user = userRepository.GetUser(username);
        if(user != null)
            communicationService.SendEmail(user.EmailAddress, String.Format("Your PIN is {0}", user.PIN);
    }
}

A little bit better. ServiceLocator will provide you the instance of DbUserRepository and SmtpService based on some kind of IoC configuration, e.g. XML. Now this class is totally clean from any dependency. This class has low coupling. However, this is still not good enough.

1. This class actually has tight runtime coupling. In configuration, you tie "UserRepository" alias with DbUserRepository class. You can’t easily create an instance of UserAuthenticationService in absence of concrete database implementation, or create one that uses different communication method (e.g. using SMS instead of Email).
2. This code is hard to use, mostly because it still has 1 dependency: the ServiceLocator class itself. Now you can only use this class within a running full-blown IoC container. You can’t easily poke around this class in isolation. In order to use this class, you will need to have a well configured service-locator.

Let’s refactor this further into dependency injection.

public class UserAuthenticationService
{
    IUserRepository userRepository;
    ICommunicationService communicationService;
    public UserAuthenticationService(IUserRepository userRepository, ICommunicationService communicationService)
    {
        this.userRepository = userRepository;
        this.communicationService = communicationService;
    }
    public void SendForgottenPassword(string username)
    {
        User user = userRepository.GetUser(username);
        if(user != null)
            communicationService.SendEmail(user.EmailAddress, String.Format("Your PIN is {0}", user.PIN);
    }
}

All the dependencies are now injected from the constructor, hence constructor-injection.

This is much better. This class is now free of any dependency. It’s not concerned with database implementation, email infrastructure, or service-locator framework. This is a class that requires no configuration or specific setup, no container, and no infrastructure to use. It’s just a plain class. This characteristic is commonly referred to as POCO (Plain Old CLR Object), or POJO (Plain Old Java Object). So this is our third rule: POCO/POJO classes are good. Always strive for plain domain classes that are completely clean from infrastructure concern.

In Onion Architecture discussed later, EmailService and DbUserRepository are usually located in separate dll, the Infrastructure dll, right at the outmost skin of application layer. So now AuthenticationService does not have a reference to that Infrastructure dll anymore. It only defines interface contract. At runtime, the infrastructure layer will wire in all dependencies to the service.

 authenticationService = new AuthenticationService(dbUserRepositiory, emailService); 

Now you can easily substitute the communication method to sms-service, or you can also easily swap the user-repository with in-memory data cache or a fetch to LDAP server. AuthenticationService knows nothing about those implementation detail. It assumes nothing about application infrastructure. Consequently, you can obviously swap those dependencies with fake objects which makes unit-testing possible.

Managing Side Effect

With injected dependencies, we’re now very explicit about all dependencies that AuthenticationService requires. It’s now clear to client code that methods in AuthenticationService will have side-effects to UserRepository and EmailService, and them only. Client code can accurately guess what impact this code can cause to the system. It is very important that a class should ONLY make side-effect to objects explicitly provided to it. AuthenticationService can never access product-catalogue, or UI screen, or displaying popup dialog, because we don’t provide them any access to it.

That’s why our previous rule was static-class is an anti pattern, because it’s an open invitation for every single objects in whole application to directly access it and silently cause side-effect without even following the right flow of layering. E.g. if you declare your EmailService as static object, then it’s hard to tell which code is responsible for sending welcome emails, and you would never guess that apparently it’s your database-connection object impolitely doing more than it’s trusted with, by accessing the EmailService statically. More about managing side-effect in Command Query Separation principle

Problem

Unfortunately, there is one problem with this approach. Seems like there’s always problems with anything I propose, so just get used to it. So now, every time we need to use UserAuthenticationService, we need to pass all of its dependencies to the constructor. It’s tedious and violates the notion of abstraction. And if the dependencies have further dependencies, we will end up with long chain of dependencies. E.g.:

service = new UserAuthenticationService(
    new DbUserRepository(new DbConnection()),
    new SmtpService(new SmtpClient(new TcpConnection())));

It’s smelly. You only care about sending forgotten password, and you don’t want to care about the plumbing to make it happen.

This is how IoC container comes very handy. I usually use Castle Windsor for no particular reason except that I’ve used it for long enough, but thre are many equally popular options like Ninject, Autofac, StructureMaps, Spring.Net, and Unity.
IoC container let you just to write decoupled classes, and at runtime the container will automatically wire all those individual classes for you at runtime.

Zero Friction Development

A typical misguided complaint about using IoC container is that it’s holding up the pace of development, particularly due to its superfluous configuration. Not true. That was in old days when XML still ruled the planet.

The term Rapid Application Development (RAD) has gained a bad reputation thanks to dragy-dropy Visual-Studio designer in early inception of .Net. The term Zero Friction Development is now the new jargon for RAD. Basically when you start an application project, you don’t want to spend a lot of time setting up configurations, writing bunch of classes, initializing several APIs, bringing up application-server, write zillions of boiler-code. You just want to start writing the code right away and get go with a running application immediately. One thing I love about .Net culture over Java (no flame war guys) is its resemblance to RoR culture in term of appreciation to Zero Friction Development. We steal RoR wisdom of Convention over Configuration.

Everyone hates XML. It sucks. Luckily, you need none to setup your IoC container. In Windsor, I only need these lines to specify convention to configure the whole dependencies in the application project:

container.Register(
    AllTypes.Pick()
        .FromAssembly(typeof (Infrastructure.Data.Repository<>).Assembly)
        .WithService.Select(InNamespace("Sheep.Services")));

This code registers all implementation classes in Data assembly, and by convention uses any interface within Sheep.Service namespace as the service-interface. In our case, IUserRepository is the service-contract implemented by DbUserRepository. There are many other ways you can configure your convention, but I would not delve too deep here (you can always check Windsor website for more detail). The point is, this code allows rapid development.

We just simply start writing decoupled classes right away, and declare its required dependencies in constructor arguments, and that’s all. Your class knows nothing about the implementations at all. Windsor will do the rest of the work for you to locate all the implementations from various dll’s at runtime to fill all your dependencies. We end up with rapid and loosely coupled development model.

To be continued in part 2..

It seems like this writing about IoC went on longer than I expected, and I haven’t even touched how anything about our architecture and code structure, which is the main reason why this IoC is required. But I promised you that I was going to discuss Onion Architecture here to show IoC in action? Well, this is the forth rule for you: don’t trust what you read in a blog post. People lie in the Internet. It turns out I won’t do it in this post. It’s long enough to be it’s own post, so I’ll do it in the next post. Meanwhile I’ll appreciate any comment.

Just Hard Code Everything

We often forget the point why we extracted certain kind of application logic out into some sort of configuration XML or database.

Recently I worked on a system that has a dynamic screen flow for a sales process. This is very common functionality in CRM applications. Basically in the sales process, the user is presented with a series of questions (and sets of possible answers in dropdown/radio-button or free-text), and the flow of the questions varies depending on how the user answers the previous questions based on certain business rule. These end-to-end flows and rules are called a script.

So to accomodate this, the developers built a mechanism to store these rules and flows in a set of database tables. Everyone would get terrified if I suggest “just hardcode all of them!.” Typical reaction is that these business-logics are so volatile, subject to change every once in a while. We need XML configuration, or mastered in database, or build DSL. Hardcode just sounds insanely terrifying. They wanted some mechanism that allows easy changes.

So we’ve had this implemented as a set of database tables such like these (among others):

• SCRIPTS, storing several different scripts (representing an end-to-end flows for specific use-case)
• STEPS, represents each step (question/calculation) within a flow
• SCRIPT_STEPS, assign a step into part of a script. Few of its columns:
  • SCRIPT_ID
  • STEP_ID
• STEP_NAVIGATIONS, governs flow between steps of a script. Few of its columns are self-explanatory:
• SCRIPT_ID
• FROM_STEP_ID
• TO_STEP_ID
• CONDITION
• WIDGETS, specifies set of widgets need to be presented for each step. Not every step has UI presentation.
• WIDGET_ID
• WIDGET_TYPE (textbox, textlabel, dropdown, button, etc)
• LABEL
• VALUES
• VALIDATION
• STEP_WIDGETS, finally this is to link several widgets to each step
• STEP_ID
• WIDGET_ID

We put this configuration in database to allow easy change and restructuring. But look around, does that make it somewhat easy?

Reconfiguring the scripts is achieved by writing a series of insert and update SQL statements, changing these obscure columns and those flags in trial-error fashion. Each step/widget/property/navigation are represented in numeric IDs and flags. Even the simplest change to the business flow (e.g. adding a new step) takes careful examination of various tables, changing/adding data in numerous tables, some obscure data manipulation SQL over several tables full of magic numbers and flags to get all stars alligned perfectly. (I always need at least a pencil, a piece of paper, and strong determination just to make the slightest change to the navigation flow).
It feels like programming on punch-cards. And no unit-test is possible to make sure we’re doing it right.

Everyone seems to forget, “why did we build all these again”?
We often get trapped by ellusive purpose of building a configurable system with the premise of allowing runtime reconfiguration of the business logic. But in real practice, any change to business logic is a serious manner, requiring a proper release cycle. Runtime change to business flow while the users are still actively working will jeopardize system integrity. And most change-requests to the business logic need code changes anyway. And most importantly, all these configuration are CACHED at runtime! So we never really have any capability making immediate change to the script configuration in the database in the first place.

What’s the purpose of having a rule engine?
The main purpose of a configurable rule engine is NOT so that we can change it during runtime. It’s more to provide developers a Domain Specific Language to express the rule and business flow, translating business flow-chart into a working application, and probably allowing the domain-expert to read or verify the logic.
This is a good reason to take the application-logic out from the code, and put that into some sort of “readable” configuration. Some kind of xml structure, or DSL file, or database-backed authoring interface.

But this DB-based configuration we’re having right now, it doesn’t bring that value. We extract the logic into configurable database tables only to make it harder to express, almost impossible to read, and terribly tedious to change. It left us nothing.
Now, if you’re not providing DSL or authoring interface, you better off hardcode it. C# code is still way more readable and easier to write and change than bunch of numeric database collumns and tables.

Sometimes we need to step back a little bit, forget about neat configuration and just hardcode everything!

// ---- Step Definition ----
var askIfLinkedToHandset = script.CreateStep<bool>()
	.Ask("Do you want to link this product to a handset")
	.RadioButton("Yes", true)
	.RadioButton("No", false);

var askHandsetSerial = script.CreateStep<string>()
	.Ask("Please enter handset IMEI")
	.Textbox();
	.Validate(x=>x.Answer.Length == 20, "IMEI needs to be 20 digits");

var calculateFinance = script.CreateStep()
	.Execute(data=> data.FinanceAmount = CalculateHandsetFinance(askHandsetSerial.Answer));

var askOutrightAmount = script.CreateStep<Decimal>()
	.Ask("Finance amount for the handset is {0}. How much do you wish to pay outright?"
		, financeAmount)
	.Textbox();

var askBonusChoice = scripts.CreateStep<BonusOption>()
	.Ask("Please choose your preferred bonus option")
	.Dropdown(()=>bonusRepository.GetBonusOptionsForProduct(data.Product))
	.WithText(bonus=>bonus.Name);

// ---- Navigation flow ----
Navigate.From(askIfLinkedToHandset)
	.When(x=>x.Answer == true).GoTo(askHandsetSerial)
	.Else().GoTo(askBonusChoice);

Navigate.From(askHandsetSerial)
	.GoTo(calculateFinance);

Navigate.From(calculateFinance)
	.When(x=> x.Data.FinanceAmount == 0).GoTo(askBonusChoice))
	.When(x=> x.Data.FinanceAmount < x.Data.MinimumOutright).GoTo(fullOutrightPayment))
	.Else().GoTo(askOutrightAmount));

Navigate.From(askOutrightAmount)()
	.GoTo(askBonusChoice);

This is much easier to read! More importantly, it’s easier to change! It takes little effort to modify existing logic/calculation, or to radically swap over some navigation flows, or add bunch of new steps. We are dealing with concrete objects in OO fashion. Navigation flow is defined very clearly. It’s no longer a set of obscure numbers and flags scattered all across many normalized database tables. Shall domain-experts choose, they would prefer reading this source-code than the previous ultra-complicated DB config.

“RDBMS-based configurations are better in accomodating future change in business logic?” It does remove compilation step, but what’s so difficult about compiling this single .cs file? Is that really harder than writing database-migration SQL script?

Hardcoded logic like this is also much better in version-control. We can track the changes of the logics in CVS/SVN history. Alas what I have right now with our sophisticated DB configuration is just a series of long SQL update statements that keeps being added overtime as the business-flow evolves. Almost impossible to make out what business-rule has changed in each SQL update statement.

Finally, this piece of code is really easy to test. I can execute that navigation tree and easily poke it with various combination of data/answers and observe the flow of the steps.

Dealing with Long Running Processes

This question came up recently in .Net list. In many typical web applications, some requests may take some time to process. In this case, the user can ask the application to generate an Excel report which takes 5-10 minutes to process. Dealing this with ordinary synchronous model gives a bad user experience. The user will be left unserved with a blank screen on his browser while the request is being processed. This poor responsiveness can be improved by processing this request asynchronously. But is it the best solution?

Asynchonous processing might aleviate the problem, but it has a fundamental flaw.
When we open a new thread to perform the request asynchronously, this thread is occupied for as long as the request is being processed. So if we configure 25 maximum threads in the pool, now we can only serve 24 incoming requests. And when we eventually have 25 users requesting for Excel reports, we’re back with an unresponsive system. The application ceases to serve any user at all. Yes, worse than our synchronous counterpart, it’s unresponsive not only to the very few who actually request for Excel reports, but also to all ordinary users who only access simple pages.

A better solution? Depending on the size of the operation, if it takes more than few seconds, the best solution might be messaging. Typically, I would use messaging solution like NServiceBus or MassTransit.

Let’s take this into the way restaurants work. The staffs (threads) who take requests from the guests simply stick a piece of paper to the kitchen, ordering for a delicious “Excel-report” dinner, and then leave.
Behind the curtain, it’s then up to the chefs to push their arses around the kitchen and perform all necessary gymnastics to deliver the meal. The other staffs do not care. They just leave the paper then return to the front to serve the next customer. Meanwhile, the chefs can cook at their own pace. Even when the chefs are performing slow, they do not affect the performance of the wait-staffs who serves the customer.

In contrast, asynchronous processing (without messaging) is like having 25 generic staffs who serve as both roles: wait-staffs as well as chefs. Upon receving a meal order, 1 staff summons another staff to go to the kitcen and cook in the background, while the first staff stays in the front keeping the customers waited. However, after 25 requests, you’re running out of staffs. Everyone is busy cooking in the kitchen, and no one serves the customers, including those who are only requesting for menu and bills.

BDD: TDD Done Right

A recent discussion I had about TDD lately has drawn some interest about BDD, and I thought what I was going to respond would deserve its own post in this blog.

Anyone in TDD circle would agree that the very first thing one needs to know about Test-Driven-Development is: it has nothing to do with test. Doing TDD to replace other means of application tests is probably just as harmful as not having any test at all. What I keep hearing from times to times is people who start practicing TDD complaining that they’re still staring many bugs crawling around before their eyes, and they will start questioning the relevance of TDD in software development. In different occassions, I hear people using their keyboard and mouse on screens to do “unit-test”.

I think the word “test” in our profession is one of those unfortunate legacies we inherit from the past. It’s totally misleading. And there has been many attempts to fix this. New terms have been introduced, like “Design by Examples”, and “Behavior Driven Development”.
Behavior Driven Development is often claimed as an evolution from Test-Driven-Development, a sexy buzzword, a total paradigm shift that lets application code speaks in business language. And it’s suddenly becoming the greatest thing since sliced bread.

Well, the fact is, BDD is no other than our good old TDD. Or at least TDD done right, to be precise. BDD is not quite the evolution from TDD. It is TDD.

While some frameworks tackle BDD from completely different angle (e.g. NBehave), many BDD frameworks (e.g. SpecUnit, MSpec) are just a guideline of applying best practices in writing good xUnit tests. They are refinements of xUnit patterns (e.g. Four Phase Test). I use SpecUnit myself, and classes in SpecUnit actually inherit directly from nUnit.

One of the common pattern (anti-pattern?) in xUnit is Testcase Class per Class and Testcase Class Per Method, which usually leads to ineffective tests. BDD promotes Testcase Class Per User Story.

There are many BDD examples out there. But one remark came up during the discussion was that most examples about TDD/BDD around are usually centered around the framework and end-results, instead of the process itself. I think that’s a good point. The process might illustrate the key difference between the 2 approaches, and what BDD really means for us. I’ll try to cover some TDD process here, it’s going to be verbose. I’ll highlight the code changes between each steps.

At the risk of being totally boring, I’m going to use the ubiquotous shopping-cart story as an example.
Creating a test called ShoppingCartTest is perhaps too obviously baaaaadd.. So let’s make a test that’s “reasonably bad”, so common that lots of people actually do that in TDD, like AddingProductToShoppingCartTest. Then later on we’ll try to refactor this into BDD way.

Let’s say, for discussion sake, products that go into the shopping cart will be acquired from the inventory-system. Not realistic perhaps, but should be good enough for our purpose.

Test Driven Development

Let’s start with our first test-case:

[TestFixture]
public class AddingProductToShoppingCartTest
{
    [Test] public void ShouldContainAddedProductWithCorrectQuantity()
    {
        cart = new ShoppingCart(
                this.customer = new Customer(),
                this.inventoryService = mocks.Mock<InventoryService>());
        nokia = new MobileHandset(42342);
        customer.OrderHistory.Add(createPreviousOrder());

        inventoryService.Expect(x=>x.AcquireProduct(nokia, Args.AnyInt)).ThenReturn(true);

        cart.Add(nokia, 12);

        cart.GetLineFor(nokia).ShouldNotBeNull();
        cart.GetLineFor(nokia).Quantity.ShouldBe(20);
    }
}

Nice. Now the second test: AddingSameProductsShouldGroupThemTogether, and you’re realize that certain portion of the code could be reused, so you are refactoring that bit into a SetUp method.

While we’re here, there’s 1 thing that you don’t normally observe from TDD examples that only give you the end-result. Here you notice that refactoring into SetUp method is a “reactive” activity. We create SetUp method after couple of testcases. SetUp method is meaningless, and its purpose is merely to eliminate repetition by consolidating the least-common-denominator of all testcases.

Further down in this post, you will see that this is in contrast with BDD, where SetUp method has its own place as a first-class citizen. In BDD, SetUp method (aka Context) is created proactively before you start writing your first testcase.

Anyway, so here’s our first 2 testcases after refactoring:

[TestFixture]
public class AddingProductToShoppingCartTest
{
	// [ADDED] --------------------------- {
    [SetUp] public void BeforeEachTest()
    {
        cart = new ShoppingCart(
            this.customer = new Customer(),
            this.inventoryService = mocks.Mock<InventoryService>());
        nokia = new MobileHandset(42342);
        customer.OrderHistory.Add(createPreviousOrder());

        inventoryService.Expect(x=>x.AcquireProduct(nokia, Args.AnyInt)).ThenReturn(true);
    }
	// } -------------------------- [/ADDED]

    [Test] public void ShouldContainAddedProductWithCorrectQuantity()
    {
		// [REMOVED /]
        cart.Add(nokia, 12);
        cart.GetLineFor(nokia).ShouldNotBeNull();
        cart.GetLineFor(nokia).Quantity.ShouldBe(12);
    }

	// [ADDED] --------------------------- {
    [Test] public void AddingSameProductsShouldGroupThemTogether()
    {
        cart.Add(nokia, 10);
        cart.Add(nokia, 20);
        cart.GetLineFor(nokia).ShouldNotBeNull();
        cart.GetLineFor(nokia).Quantity.ShouldBe(30);
    }
	// } -------------------------- [/ADDED]
}

So far so good. You go ahead and write the next 8-9 testcases, before you stumble across our next testcase: CustomerFirstTimeOrder_ShouldAllowOnly1Product. Bugger! Our SetUp method no longer fits to cover this scenario. You will need to take line 10 from SetUp method back to each of the testcases.

[TestFixture]
public class AddingProductToShoppingCartTest
{
    [SetUp] public void BeforeEachTest()
    {
        cart = new ShoppingCart(
            this.customer = new Customer(),
            this.inventoryService = mocks.Mock<InventoryService>());
        nokia = new MobileHandset(42342);

		// [REMOVED /]

        inventoryService.Expect(x=>x.AcquireProduct(nokia, Args.AnyInt)).ThenReturn(true);
    }

    [Test] public void ShouldContainAddedProductWithCorrectQuantity()
    {
        customer.OrderHistory.Add(createPreviousOrder()); // <- ADDED

        cart.Add(nokia, 12);
        cart.GetLineFor(nokia).ShouldNotBeNull();
        cart.GetLineFor(nokia).Quantity.ShouldBe(12);
    }

	[Test] public void AddingSameProductsShouldGroupThemTogether()
    {
        customer.OrderHistory.Add(createPreviousOrder()); // <- ADDED

        cart.Add(nokia, 10);
        cart.Add(nokia, 20);
        cart.GetLineFor(nokia).ShouldNotBeNull();
        cart.GetLineFor(nokia).Quantity.ShouldBe(30);
    }

    // [ADDED] --------------------------- {
    [Test] public void CustomerFirstTimeOrder_ShouldAllowNotAllowMoreThan1Product()
    {
        Assert.Throws<IllegalRequest>( ()=> cart.Add(nokia, 2));
        cart.GetLineFor(nokia).ShouldBeNull();
    }

    [Test] public void CustomerFirstTimeOrder_ShouldAllow1Product()
    {
        cart.Add(nokia, 1);
        cart.GetLineFor(nokia).ShouldNotBeNull();
        cart.GetLineFor(nokia).Quantity.ShouldBe(1);
    }
	// } -------------------------- [/ADDED]
}

OK let me say it again. SetUp method consolidates the least-common-denominator of all testcases. Well, after you cover various scenarios for your test, you will no longer have too much common-denominator. Consequently, you will find yourself keep changing large number of testcases and refactor your test-code every time you add a new test-case that covers slightly different scenario.

For completeness sake, let’s add 1 more test-case: WhenInventoryRunsOutOfStock_ShouldRejectProduct(). Alas, once again we need to refactor the SetUp method. So this concludes the final result of our TDD process:

[TestFixture]
public class AddingProductToShoppingCartTest
{
    [SetUp] public void BeforeEachTest()
    {
        cart = new ShoppingCart(
            this.customer = new Customer(),
            this.inventoryService = mocks.Mock<InventoryService>());
        nokia = new MobileHandset(42342);
    }

    [Test] public void ShouldContainAddedProductWithCorrectQuantity()
    {
        customer.OrderHistory.Add(createPreviousOrder());
        inventoryService.Expect(x=>x.AcquireProduct(nokia, 12)).ThenReturn(true);

        cart.Add(nokia, 12);
        cart.GetLineFor(nokia).ShouldNotBeNull();
        cart.GetLineFor(nokia).Quantity.ShouldBe(12);
    }

    [Test] public void AddingSameProductsShouldGroupThemTogether()
    {
        customer.OrderHistory.Add(createPreviousOrder());
        inventoryService.Expect(x=>x.AcquireProduct(nokia, 10)).ThenReturn(true);
        inventoryService.Expect(x=>x.AcquireProduct(nokia, 20)).ThenReturn(true);

        cart.Add(nokia, 10);
        cart.Add(nokia, 20);
        cart.GetLineFor(nokia).ShouldNotBeNull();
        cart.GetLineFor(nokia).Quantity.ShouldBe(30);
    }

    [Test] public void CustomerFirstTimeOrder_ShouldAllowOnly1Product()
    {
        inventoryService.Expect(x=>x.AcquireProduct(nokia, 2)).ThenReturn(true);

        Assert.Throws<IllegalRequest>( ()=> cart.Add(nokia, 2));
        cart.GetLineFor(nokia).ShouldBeNull();
    }

    [Test] public void WhenInventoryRunsOutOfStock_ShouldRejectProduct()
    {
        customer.OrderHistory.Add(createPreviousOrder());
        inventoryService.Expect(x=>x.AcquireProduct(nokia, 20)).ThenReturn(false);

        Assert.Throws<IllegalRequest>(()=> cart.Add(nokia, 20));
        cart.GetLineFor(nokia).ShouldBeNull();
    }
}

Everytime we cover a new test-case, we change the equilibrium of our common-denominator, and you will spend most of your time refactoring. After 99 testcases, you have virtually nothing in common. Your SetUp method is practically empty. Each of your testcases are a giant repetitive set of arrage-expect-action-assert, all in every single method. Effort to maintain this test is out of control. The test is no longer readable. Try reading again the test we just wrote. Still readable? Try again tomorrow morning.

Behavior Driven Development

Let’s try converting the exact same testcases we just wrote into SpecUnit grammar. I’ll also switch my mock-framework from record-replay mode into AAA-syntax.

public class when_adding_a_product:
    Behaves_like_an_empty_shopping_cart_and_a_repeat_customer
{
    public override void Because()
    {
        cart.Add(nokia, 12);
    }

    [Observation]
    public void should_acquire_product_from_inventory()
    {
        inventoryService.AssertWasCalled(x=>x.AcquireProduct(nokia, 12));
    }

    [Observation]
    public void cart_should_only_contain_1_line()
    {
        cart.Lines.ShouldHaveSize(1);
    }

    [Observation]
    public void cart_should_contain_line_for_added_product()
    {
        cart.Lines.First().Product.ShouldBe(nokia);
    }

    [Observation]
    public void cart_item_should_have_added_quantity()
    {
        cart.Lines.First().Quantity.ShouldBe(12);
    }
}

public class when_adding_same_product_twice:
    Behaves_like_an_empty_shopping_cart_and_a_repeat_customer
{
    public override void Because()
    {
        cart.Add(nokia, 10);
        cart.Add(nokia, 20);
    }

    [Observation]
    public void should_acquire_both_requested_products_from_inventory()
    {
        inventoryService.AssertWasCalled(x=>x.AcquireProduct(nokia, 10));
        inventoryService.AssertWasCalled(x=>x.AcquireProduct(nokia, 20));
    }

    [Observation]
    public void cart_should_only_contain_1_line()
    {
        cart.Lines.ShouldHaveSize(1);
    }

    [Observation]
    public void cart_should_contain_line_for_added_product()
    {
        cart.Lines.First().Product.ShouldBe(nokia);
    }

    [Observation]
    public void cart_item_quantity_should_be_sum_of_added_quantities()
    {
        cart.Lines.First().Quantity.ShouldBe(10 + 20);
    }
}

public class when_inventory_runs_out_of_stock:
    Behaves_like_an_empty_shopping_cart_and_a_repeat_customer
{
    Exception exception;
    public override void Because()
    {
        inventoryService.When(x=>x.AcquireProduct(nokia, 2)).ThenReturn(false);

        thrownException = ((MethodThatThrows)()=>
            cart.Add(nokia, 2))
        .GetException();
    }

    [Observation]
    public void should_reject_the_request()
    {
        thrownException.ShouldNotBeNull();
    }

    [Observation]
    public void should_not_acquire_product_from_inventory()
    {
        inventoryService.AssertWasNotCalled(x=>x.AcquireProduct(nokia, Arg.AnyInt));
    }

    [Observation]
    public void should_not_add_any_product()
    {
        cart.Lines.ShouldBeEmpty();
    }
}

public class when_a_first_time_customer_add_more_than_1_product_quantity:
    Behaves_like_an_empty_shopping_cart
{
    Exception exception;
    public override void Because()
    {
        thrownException = ((MethodThatThrows)()=>
            cart.Add(nokia, 2))
        .GetException();
    }

    [Observation]
    public void should_reject_the_request()
    {
        thrownException.ShouldNotBeNull();
    }

    [Observation]
    public void should_not_acquire_product_from_inventory()
    {
        inventoryService.AssertWasNotCalled(x=>x.AcquireProduct(nokia, Arg.AnyInt));
    }

    [Observation]
    public void should_not_add_any_product()
    {
        cart.Lines.ShouldBeEmpty();
    }
}

public class when_a_first_time_customer_add_1_product_quantity:
    Behaves_like_an_empty_shopping_cart
{
    Exception exception;
    public override void Because()
    {
        cart.Add(nokia, 1);
    }

    [Observation]
    public void should_acquire_product_from_inventory()
    {
        inventoryService.AssertWasCalled(x=>x.AcquireProduct(nokia, 1));
    }

    [Observation]
    public void cart_item_should_contain_the_product()
    {
        cart.GetLineFor(nokia).Quantity.ShouldBe(1);
    }
}

[Concern("Shopping Cart")]
public class Behaves_like_an_empty_shopping_cart: ContextSpecification
{
    public override void Context()
    {
        cart = new ShoppingCart(
            this.customer = new Customer(),
            this.inventoryService = mocks.Mock<InventoryService>());
        nokia = new MobileHandset(42342);

        inventoryService.When(x=>x.AcquireProduct(Arg.Any<Product>(), Args.AnyInt)).ThenReturn(true);
    }
}

public class Behaves_like_an_empty_shopping_cart_and_a_repeat_customer:
    Behaves_like_an_empty_shopping_cart
{
    public override void Context()
    {
        base.Context();
        customer.OrderHistory.Add(createPreviousOrder());
    }
}

So this time, every user-story is no longer written in 1 (or several) methods. Each user-story is represented as 1 class which a clear contet. And each test-method only contains one single line of Assert statement. As a rule-of-thumb, 1 assert turns to 1 test-method. As such, we can quickly lookup the definition of each expected behavior. E.g., we can lookup that the definition of cart_item_quantity_should_be_sum_of_added_quantities is cart.Lines.First().Quantity.ShouldBe(10 + 20).

Same goes for the classes that represent each user-story. They are defined in a clear business vocabulary, comes with each definition. Programmers can lookup the definition of each business context instantly. E.g. the definition of when_inventory_runs_out_of_stock is inventoryService.When(x=>x.AcquireProduct(nokia, 2)).ThenReturn(false).
Our test becomes a handy place for programmers to lookup business glossary just by following Context() and Because() methods. So the definition of “inventory running out of stock” in the context of a shopping cart is when AcquireProduct() returns false.

Usually I stick all these classes together into a single file (e.g. AddingProductToShoppingCartSpecification.cs). A file where we can lookup a whole aspect of a little sub-functionality in our system in a neat grammar of Behaves_like, When, Should, Should, Should… where every one of them is immediately followed by its definition.

Now everytime we need to write a new scenario, we’re no longer worried if this will affect the other testcases. If the new scenario doesn’t fit with our current contexts, we simply create a new context. Just create a new when_xxx class… without needing to refactor any existing test-case.
This helps us to avoid the temptation to apply lazy-workaround. For instance, in our previous TDD example about “first-time-customer”, developers could actually just create a new method in Customer class to clear all OrderHistory to turn a repeat-customer back into a fresh first-time-customer. This makes the SUT fits with existing SetUp method scenario and test-cases dependency hell… saving them from refactoring effort. But this leads to smelly design, and probably doesn’t fit with the domain they’re representing. (In domain requirement, they probably will never have a customer clearing his order history).

When you execute this “testcases”, SpecUnit provides a tool that converts this into a pretty HTML document with nice given-when-then grammar, structured in an organized fashion. This document is readily consumable by non-technical people. After the business people see this document quite several times, they will start to understand how you work. They are able to see how we (developers) translate the requirement they convey into an executable specification in given-when-then manner. Soon enough, they will get excited and start to learn to speak in the same grammar.

They already have a general feeling of the way we talk, so they will start expressing their requirement in the same given-when-then language, which will immediately appear as our BDD specification. Closing the room for ambiquity. Bridges the gap of communication.

Big Is Beautiful..

.. NOT!
I found this from Pattrick Smacchia’s blog, a single method inside .Net framework class, System.Windows.Forms.DataGridView.GetClipboardContent().
I think it’s hilarious.

Yes, a single method. Flag1, flag2, flag3,… obj1, obj2, obj3…, Obfuscator is the last thing they ever need.

NDepend for Continuous Quality

Full Disclosure: Patrick Smacchia (lead dev of NDepend) offered me an NDepend Professional license for free so that I could use it and say some words on this blog. I gladly received it. Now I’m going to tell you how much it rocks, but not because I have to. I had been having interest in NDepend for quite some time. It’s a great tool in promoting good application code and architecture, something I’m quite fond of. This is a tool that I would have been using anyway if the project I’m working on was in .net environment.

What is NDepend?

In case you haven’t been within the reach of geeky hype signal recently and never heard about NDepend, it is a tool that reads into your source code, and statically analyses if the code meets certain design guidelines and quality. By “statically”, it means that it examines your code without actually running it. NDepend is often used in code-review process during development to easily identify spots in the code that could be improved.

If that sounds like FxCop to you, yes they share the same goal in principle, but they are different in 2 ways:

1. FxCop works on fine-grain details. It inspects each line of instruction or coding structure, and how they comply with framework design guideline. NDepend works more from high-level application architecture viewpoint. It visualizes the code in terms of dependencies, complexities, and other design metrics in quantifiable numbers, giving you a big picture idea of the state of the code and spot smelly areas. You can then define some thresholds to define an acceptable level of code quality. Further down this post, this difference will become crystal clear.
2. FxCop gives you a set of rules you can use, but and there is no easy way to express your own rule. Adding new rule involves writing new add-in to be plugged into FxCop. The primary strength of NDepend is on its CQL (Code Query Language), a DSL that allows you to express your own custom rule and evaluate it on the fly as you type.

Using NDepend

NDepend makes me wish my current project at work was written on C#. It is in fact written in Java. Incidentally, developed using big-ball-of-mud approach, the code has degraded to one of the worst source-codes I have worked on. That should have been absolutely the perfect place to use NDepend and refactor the big mess out loud. Alas NDepend doesn’t deal with Java. The good news is, NDepend just announced to release NDepend for Java, XDepend. But that’s for another post.

For our purpose, I pick a codebase from one of open-source .net projects I am quite familiar with, CodeCampServer, a web application developed using a Domain-Driven-Design and Test-Driven-Development on MVC platform. That perfectly mimics my idea of good application architecture.

UI appearance of Visual NDepend very much feels like Visual Studio environment.

 

Getting NDepend to analyse assemblies is incredibly easy! Literally couple of intuitive clicks, and before I know it, I am presented with fantastic details of information about the project.

Dependency Graph

Dependency graph is the best way to understand the architecture of the code.

You can navigate further deeper to get more intimate understanding about the code. Since CodeCampServer is built using Domain-Driven-Design, the domain-model layer is more than likely to be our primary interest in understanding the application, particularly when it’s developed by other people.

What we are looking at is a direct x-ray scan of CodeCampServer.Model.Domain namespace, and it says 1 thing: the code is impressive! Doesn’t take much to notice that CodeCampServer is developed using a solid Domain-Driven-Design: the code almost looks like business document. None of all classes within this namespace has any reference to some geeky technical details or infrastructure components. Indeed, the dependency graph of this code directly reflects the business models and their associations, which is readily consumable not only for developers, but also for business people. Big A+ for CodeCampServer.

Dependency graph can also be presented in the form of matrix.

3 rules to read this matrix:

• Green: number of methods in namespace on the left header that directly use members of namespace on the top header
• Blue: number of methods in namespace on the left header that is directly used by members of namespace on the top header
• Black: number of circular direct dependency between namespaces on the left and on the top headers

As such, the following graph generates English description: 42 methods of the assembly CodeCampServer.Website are using 61 methods of the assembly CodeCampServer.Model

 

So back to the previous complete matrix, it shows that CodeCampServer.Model.Domain namespace does not have any reference to any other namespace (all horizontal boxes are blue). To put in other word, domain-model is the core of this application (see Onion Architecture). All DDD-ish so far.

What I find interesting is that data-access namespace (CodeCampServer.DataAccess) makes *zero* direct communication with CodeCampServer.Model.Domain. It means that none of the repositories makes a direct communication with domain entities at all. This is impressive! It might seem counter-intuitive at first, how can those repositories create instances of domain-entities from DB, as well as save the state of domain-entities to DB without making any communication with any method/property/constructor of domain entity?

CodeCampServer uses NHibernate as ORM which completely decouples data-access concerns from the application code. Repositories in data-access layer simply delegate all CRUD operations of domain entities to NHibernate that will handle the creation and persistence of domain-entities. The repository itself makes totally zero direct communication with domain-entity.

Overall, CodeCampServer has a very good dependency matrix, but you will be surprised of how many unexpected smells that you can find in your application. You might find many things are not written in the way how you architect it. If one of your developers take a shortcut and, say, call security-checking from domain-entity, you will notice an unexpected reference from the dependency matrix (Domain->Security). You can even setup automatic alert when that happens, e.g. when domain layer has any reference to other namespace. More on that later.

Circular Dependency

This is what we normally need to be alerted with. CodeCampServer.Website.Impl makes a circular reference with CodeCampServer.Website. A reference from CodeCampServer.Website into a member in more specific namespace like CodeCampServer.Website.Impl signals some kind of smell.

You can navigate through the detail of the reference by clicking on the box. Turns out that CodeCampServer.Website.Globals make a reference to CodeCampServer.Website.Impl.RouteConfigurator.

The reference is apparently caused by Windsor IoC configuration in Globals.RegisterMvcTypes() methods.

Since Global is only referencing RouteConfigurator by its class, and not communicating with it in any way, NDepend indicates 0 on the reference. These 2 components are entangled for a conscious intention,so this analysis concluded no design problem. However this is a good example of using NDepend feedback to discover potential design smell in many parts of applications. For example look at the following dependency matrix for Spring.Net code:

The black areas indicates that those namespaces are tightly entangled together. Tight coupling means that all those 6 namespaces can only be understood together, which adds complexity in understanding and maintaining the code. Of course tight-coupling also means that they all can only be used together. I.e. to use dependency-injection capability from Spring.Core, you will need the presence of Spring.Validation in the baggage.

Metric

NDepend presents overview information about our code in various metric. For example, here is “Line of Code” metric for CodeCampServer.

CodeCampServer is written with TDD, so unsurprisingly, Unit-Tests takes up majority of the code lines. Followed by integration-tests. The biggest portion of application-code is on Website layer. And here is my favorite part: just like a street map, you can zoom deeply into this map by scrolling your map, from top assembly level down to classes and individual methods! In fact, you can use your mouse scroll to zoom almost anything in NDepend user interface.

Now let’s see the map for Cyclomatic Complexity metric. The more if/else/switch statements in your code, the higher Cyclomatic Complexity it gets.

CodeCampServer is developed using Domain-Driven-Design approach, so when it comes to complexity and decision-making, Model layer takes up most of the space. Unit-test takes just about the same size.

Code Query Language

I can’t repeat this enough: CQL is brilliant! It gives you the power to gather just about any information you need about your source code. CQL is very similar to SQL, but instead of querying the database, you are querying into your source-code. You can use CQL to quickly search for particular areas in your code that can be of your interest in term of code quality.

The information that you can query with CQL is comprehensive. NDepends provides a huge 82 code metrics that provides access to large range of information you need.

• Which methods have been refactored since last release and is not thoroughly covered by unit-tests?
  SELECT METHODS WHERE CodeWasChanged AND PercentageCoverage < 100
• What public methods could be declared as private?
  SELECT METHODS WHERE IsPublic AND CouldBePrivate
• What are the 10 most complex methods?
  SELECT TOP 10 METHODS ORDER BY CyclomaticComplexity

Code Inspection

So far we have been talking about all various statistics and analysis, but we haven’t seen the part that matters the most in a static analysis tool: code inspection. You can set up custom coding guidelines using CQS, and let NDepend regularly inspect your codebase to meet the project standard, and issue warnings when somebody violates the rule.

Let’s say, Architectural Guidelines:

• DDD Value Object has to be immutable
  WARN IF Count > 0 IN SELECT TYPES WHERE !IsImmutable AND HasAttribute "ValueObject"
• Web layer should not have direct dependency to Data layer
  WARN IF Count > 0 IN SELECT METHODS FROM NAMESPACES "Web" WHERE IsDirectlyUsing "Data"
• Repository should not flush NHibernate session. It violates unit-of-work pattern.
  WARN IF Count > 0 IN SELECT METHODS FROM NAMESPACES "Data" WHERE IsUsing "NHibernate.ISession.Flush()"
• All public methods of Services has to be surrounded by database transaction
  WARN IF Count > 0 IN SELECT METHODS FROM ASSEMBLIES "Services" WHERE IsPublic AND !HasAttribute "Transactional"

Quality Standards:

• Methods that are way too long, and obviously unreadable
  WARN IF Count > 0 IN SELECT METHODS WHERE NbILInstructions > 200 ORDER BY NbILInstructions DESC
• Methods that have too many parameters are smelly.
  WARN IF Count > 0 IN SELECT METHODS WHERE NbParameters > 5 ORDER BY NbParameters DESC
• Classes that have too many methods. Chances are they violate Single Responsibility Principle
  WARN IF Count > 0 IN SELECT TYPES WHERE NbMethods > 20 ORDER BY LCOM DESC
• Classes that overuse inheritance. You should consider favour_composition_over_inheritance principle 
  WARN IF Count > 0 IN SELECT TYPES WHERE DepthOfInheritance > 6 ORDER BY DepthOfInheritance DESC

Continuous Quality

Continuous quality is the ultimate goal of code-analysis tools. You can hook NDepend to your project’s continuous integration servers.

Hooking NDepend to TeamCity allows it to continuously monitor the quality of the project based on design rules you specifies, and ring the alarm when somebody checks-in some malicious code that violates design guidelines of the house. This gives an immediate feedback to the developers to fix the code before it degrades the codebase. Yes, we “Fix” code NOT only when something is broken, but also when the code does not meet certain quality threshold! It’s often difficult to get that across to some people.

This ensures not only that all codes being checked-in can actually be built into an actual application that satisfies all the requirements (unit-tests), but also ensures that the codebase continuously meet acceptable level of quality.

TeamCity is a really nice development dashboard that provides you a great detail of information about the current state of the application and its historical progressions, and NDepends supplies quantitative numbers for various quality metrics that you can track back. You can even use NDepend to analyse differences between 2 versions of source code and the implication to the metrics. You can observe how the metrics are evolving over time. In general, this will gives you a trend looking more or less like this:

At the start of the project, the code is looking good. Methods are short, classes are awesomely designed, loose coupling is maintained. Over time as the complexity grows, methods are getting stuffed up with if/else and non-sense code, classes are entangled together… The code starts to decay. When the code gets so bad and the heat goes beyond acceptable level or violates certain coding guidelines, NDepend starts complaining screaming for refactoring.

The developers starts cleaning up their mess and refactor the code, while the code is still reasonably manageable. The heat cools down a bit, and quality is getting better. As the development progressing, the heat keeps jumping up and each time NDepend gives you immediate feedback, allowing you to refactor the code before it gets too difficult. The overall code quality is constant over time.

Without continuous inspection to code quality, the diagram would have been:

The code decays steadily as it grows. It’s becoming less and less manageable with the intention to refactor it when you feel critical. But before you know it, the code is already too difficult to be refactored. Making a single change is totally painful, and people end up throwing more mud into it everyday until the code turns into a big ball of mud, that the only thing in your mind everyday is to scrap the code and move on to the next project. And I know that feeling all too well

Report

When it comes to code analysis report, NDepend is top notch. There are ridiculously massive details of information about your code. From trivial lines-of-code, test-coverage, and comments, to dependencies and warnings. You can even compare 2 versions of the source-code. And all of them can be kept track in historical order within TeamCity. All these details can be overwhelming, luckily it’s presented with a great high level summary at the top, followed by increasingly detailed information as you walk down. Here are big-picture summary of CodeCampServer’s codebase:

Stable or abstract do not mean anything bad or good about the code. That is just a fact about the code.

• Stable package means that a lot of components are using it. I.e., it has a lot of reasons NOT to change
• Abstractness indicates the ratio between interfaces, abstract classes, or sealed classes within the package. Abstract package gives more flexibility in changing the code.

Good quality code is a balance between these 2 properties. It’s about finding equilibrium between 2 extreme characteristics:

• Painful: the package used by a lot of people, but lack of abstractness, hence painful to change
• Useless: the package is used by no one, but it has a lot of abstractness, hence over-engineering.

The website and data-access of CodeCampServer are infrastructure implementations, and it is structured that there is nothing in the application that depends on them. Thus they’re located at the edge of instability in the diagram. The core of the application, Model layer are by nature stable. They cannot be easily changed without affecting the other part of the application. Luckily it is highly decoupled (using service-interface and repository-interface), so it has higher abstraction, although not enough to make it into green area.

Summary

Weakness

There are however some area that NDepend still falls short.

1. CQL is not as powerful as you would expect. The most significant one being the absence of any notion of subquery, or join, or any other mean that allows you to traverse relationship between components. What seem to be pretty simple queries are actually impossible to be expressed in current version of CQL:
   • Select all methods in classes that implement IDisposable
   • Any method that constructs any class from namespace "Services". (Rule: services should be injected by IoC)
   • Any member in namespace "Services" that has instance field of types from "Domain.Model" namespace. (Rule: DDD services should be stateless)
   • All namespaces that refer to another namespace that make reference to opposite direction (i.e. circular reference)
2. Continous Integration support is a bit weak. In fact, there is no direct support to TeamCity. Constraint violations can’t be fed into TeamCity alert system. And while TeamCity can store NDepend historical reports as artefacts, there is no easy way to follow the progression summary of the stats. E.g. growth of code size or dependencies (other than collecting all the reports ourselves), like what TeamCity does with code-coverage graphs. But maybe I’m just too demanding…
3. 82 metrics are impressive, added with CQL it makes NDepend far superior than other code-analysis tools in term of the easiness of customisation. However, I haven’t found any mean to actually extend the metrics (or the constraints). Even though FxCop’s customisability is very limited, it’s fairly easy to write your own plugin and extend the tool yourself.

Having said that, NDepend is a really impressive tool. Lives for architects and application designers have never been easier. They no longer need to deal only with whiteboards and CASE tools. NDepend allows them to deal straight with the actual code realtime. Looking into the code from architectural viewpoint. NDepend provides great deal of flexibility with its brilliant CQL. NDepend is very intuitive, and it took me almost no time to get going and do interesting things.

Are You a Mockaholic?

Mockaholic: (noun) – a person who compulsively takes mocking frameworks as the answer to every test-isolation.

In my current project, I use Specification pattern for my data-access query, and while it is testable, they were anything but fun.

Here’s some SUT of my MVC Customer controller:

public ActionResult Search(string name, DateTime? dob, bool? isTerminated)
{
ISpecification spec = null;
if(!name.IsNullOrEmpty())
spec &= whereCustomer.NameContains(name);
if(dob != null)
spec &= whereCustomer.WasBornInDay(dob.Value);
if(isTerminated != null)
spec &= isTerminated? whereCustomer.IsTerminated(): !whereCustomer.IsTerminated();

searchResult = repository.Query(spec);
return View(“search”, searchResult);
}

I stripped out all PTO related stuffs for clarity. Using Rhino-Mock, here is the unit-test:
(PS: &, |, and ! operators call And(), Or(), and Not() instance methods respectively behind the scene)

Test-Case:

IWhereCustomer whereCustomer = MockRepository.CreateStub();

ISpecification nameSpec = StubSpec();
ISpecification dobSpec = StubSpec();
ISpecification notTerminatedSpec = StubSpec();
ISpecification finalSpec = StubSpec();

whereCustomer.Expect(x => whereCustomer.NameContains(“Hendry”)).Return(nameSpec);
whereCustomer.Expect(x => whereCustomer.WasBornInDay(dob)).Return(dobSpec);
whereCustomer.Expect(x=> whereCustomer.IsTerminated().Not()).Return(notTerminatedSpec); // Recursive Mock

nameSpec.Expect(x=> x.And(dobSpec).And(notTerminatedSpec)).Return(finalSpec); // Recursive Mock

repository.Expect(x=> x.Query(finalSpec)).Return(stubCustomers);

controller.Search(“Hendry”, dob, false).ShouldRenderView(“search”).WithModel(stubCustomers);

Although the code bloat has been reduced significantly by the new recursive mock feature in Rhino-Mock 3.5, the test is still far than neat. It’s tedious to write, difficult to read. From reader’s perspective, I can’t immediately figure out what this unit-test is trying to express. If this is how I should write unit-test for anything dealing with ISpecification, I definitely need to find something better.

So I scrap the whole Rhino-Mock stuffs from my unit-test, and write my own hand-coded stub for ISpecification class myself, and I’m able to refactor my test-case into:

IWhereCustomer whereCustomer = StubSpecificationFactory();

repository.Expect(x=> x.Query(
whereCustomer.NameContains(“Hendry”) & whereCustomer.WasBornInDay(dob) & !whereCustomer.IsTerminated()
)).Return(stubCustomers);

controller.Search(“Hendry”, dob, false).ShouldRenderView(“search”).WithModel(stubCustomers);

We managed to reduce the test code by more than a half! And even better, you can use AAA style test that wasn’t quite possible previously.

controller.Search(“Hendry”, dob, false).ShouldRenderView(“search”).WithModel(stubCustomers);

repository.AssertWasCalled(x=> x.Query(
whereCustomer.NameContains(“Hendry”) & whereCustomer.WasBornInDay(dob) & !whereCustomer.IsTerminated());

How was that possible? Here’s the stubbed ISpecification that I wrote.

public class SpecificationMock: ISpecification
{
private MethodInfo methodInfo;
private object[] args;

public override bool Equals(object other)
{
SpecificationMock spec = other as SpecificationMock;
if(spec== null)
return false;
return methodInfo.Equals(spec.methodInfo) && args.SequenceEquals(spec.args);
}
}

So this stub is only responsible in comparing equality between expectated and actual ISpecification.
Mocked IWhereCustomer will forward all MethodInfos (NameContains() and WasBornInDay() in this case) and its arguments down to the our SpeficiationMock. I use Castle DynamicProxy here to automate that (hence StubSpecificationFactory utility).

By unleashing yourself from full blown mocking-framework and instead going back to basic using handcrafted mock objects, you can actually produce a much simpler test design, using a carefully established “test-oriented vocabulary” that suits your specific situation.

Extensible Query with Specification Pattern

Writing finder methods for data repositories is problematic. The following is an example of pretty simplistic customer-search screen:
Email Address: [________]
Name: [_______]
Age: from [__] to [__] year old
Is terminated: [ ]Yes [ ]No
{Search}

Now how are we going to implement data query behind this search? I’ll go through several alternatives. (Jump to solution if you can’t care less). Let’s start from the simplest approach.

1. Repository Finder Methods

IList<Customer> result = customerRepository.SearchByBlaBlaBloodyBla(email, name, ageFrom, ageTo, isTerminated);

Let me give more context about what I’m working on. I’m developing an application that is intended to be an extensible vertical CRM framework. The client will provide their own new screens/functionalities by writing specific plugin implementation in their own separate assembly, without requiring modification on the framework (Open-Closed Principle).
The approach you just saw above tightly couples the repository API with specific UI design. Any change with the design of the search screen will require the repository API to be reworked. And should we create one repository method for each search screen? Furthermore, when a new search screen or report functionality is plugged into the system by adding new plugin, we need to somehow extend the data repository API to cover each of those specific screen scenarios. This is not an easily extensible architecture.
The upside, this approach is simple and very easy to mock for unit-test. When flexibility is not an issue, I would go for this approach.

2. Lambda-Expression Repository

IList<Customer> result = repository.FindAll<Customer>(
	x => x.EmailAddress == emailAddress && x.IsTerminated == isTerminated);  // and so on

This code uses repository API from FluentNHibernate. I like this API because we only have one single general-purpose Repository. It decouples the repository completely from specific UI design. However I’m not comfortably about leaking naked Linq outside of repository. Exposing Linq to other layer will scatter the database concern all across application. Let’s consider what happens if we decides to refactor IsTerminated property that is currently implemented as a column in DB into C# code, say:

public bool IsTerminated {get {return this.TerminationDate != null; }};

The earlier Linq statement (possibly scattered all over the place) will start to fail since Linq is unable to map IsTerminated property into a correct SQL where clause.

3. Pipe and Filter Pattern

IQueryable<Customer> result = repository.All<Customer>().WithEmail(emailAddress).AgeAbove(ageFrom).AgeBelow(ageTo);
if(isTerminated != null)
	result = (isTerminated)? result.IsTerminated(): result.IsNotTerminated();

Or in this case, that should be wrapped into:

IList<Customer> result = repository.All<Customer>().WithBlaBloodyBla(emailAddress, name, ageFrom, ageTo, isTerminated).List();

This approach leverages fluent IQueryable and extenssion-methods. It still exposes IQueryable which leaks database concern outside repository, but it’s much better since the query is properly encapsulated behind easy-to-read and maintainable extenssion methods.
In the above example with isTerminated check, it’s obvious that this approach is doing pretty well in handling dynamic query that is very difficult to express using previous lambda-expression approach. But the flexibility is pretty limited, or to be specific, you can only chain multiple filters in an AND relationship.
Another problem of this approach, which is actually the main reason I steer away from approaches #2 #3 #4, is on unit-testing. Yes it is very easy to unit-test each of the filter independently, but it is extremely difficult to mock out those filters to unit-test services that depends on it. I’ll describe the problem in the next approach.

4. Specification pattern

IList<Customer> result = repository.FindAll<Customer>(new WithinAgeCustomerSpecification(ageFrom, ageTo));

My prefered solution is largely derived from specification pattern, I’ll give extra highlight to this approach later. Long story short, IMO this approach is best since it doesn’t leak any data-concern and linq to outside repository. It also separates the responsiblities between loading/saving domain entity (repository) and querying (specification). I’ll start with the problem.
As mentioned, it’s very easy to unit-test each of the specification using the infamous in-memory/sqlite repository testing. But it’s incredibly difficult to unit-test the UI controller and application layer that uses the specification.
Just to give a concrete illustration, this is how I write unit-test had I used the approach #1. (Simplified to search age only)

customerRepository.Expect(x => x.SearchByAgeBetween(20, 30)).Return(stubCustomers);
ViewResult view = customerSearchController.Search(20, 30);
Assert.That(view.Model, Is.EqualTo(stubCustomers);

But anyone has suggestion how I could test the following controller (simplified)?

public class CustomerController: Controller
{
	IRepository repository; //Injected

	public ActionResult Search(int? ageFrom, int? ageTo)
	{
		var customers = repository.Query(new WithinAgeCustomerSpecification(ageFrom, ageTo));
		return View("search", customers);
	}
}

That little call to “new WithinAgeCustomerSpecification(..)” makes it virtually impossible to mock the specification and take it out from the test concern. Linq and Extension method in approach #2 and #3 certainly don’t help.
Why do we care to mock the specification? Because, mind you again, testing queries _IS_ painful! It’s tedious to setup stub-data and verify query result. Each of the specification has had this kind of unit-test themselves, and we certainly _DO_NOT_ want to repeat the test in the controller. For sake of discussion, this is how the unit-test for the controller would look like using unmocked Specification.

ShouldLoadSearchView();
CanLoadCustomerByEmailAddressToViewData();
CanLoadCustomerByNameToViewData();
CanLoadCustomerByAgeToViewData();
CanLoadCustomerByNameAndEmailAddressToViewData();
// etc etc

Each test-case deals with tedious in-memory/sqlite stub data. I don’t even understand why I need to care about data and Sqlite to unit-test UI/Application layer. It just doesn’t make sense.
And guess how the unit-test for the specification looks like.

CanSearchByEmailAddress();
CanSearchByName();
CanSearchByAge();
CanSearchByNameAndEmailAddress();
// etc etc

That’s right, duplication. Not to mention tediously data-driven. Generally, you want to avoid testing that involves data and query. For comparisson, this is how the test for the controller would look like with mocked specification.

ShouldLoadSearchView();
ShouldSearchCustomerUsingCorrectSpecification();
ShouldLoadSearchResultToViewData();

Yes, that’s all we care: “controller should use correct specification to search the customer”. We don’t care if the specification actually does what it claims it does. That’s for other developers to care.

Solution

By wrapping Specifications into a factory, we decouple the controller from Specification implementation.

public class CustomerController: Controller
{
	IRepository repository; //Injected
	ICustomerSpecFactory whereCustomer; //Injected

	public ActionResult Search(int? ageFrom, int? ageTo)
	{
		var customers = repository.Query(whereCustomer.HasAgeBetween(ageFrom, ageTo));
		return View("search", customers);
	}
}

Unit-test is a breeze.

whereCustomer.Expect(x => x.HasAgeBetween(20, 30))
	.Return(stubSpec = MockRepository.Generate<Specification<Customer>>);
customerRepository.Expect(x => x.Query(stubSpec)).Return(stubCustomers);

var view = customerController.Search(20, 30); // EXECUTE CONTROLLER ACTION
Assert.That(view.Model, Is.EqualTo(stubCustomer);

EDIT: I posted a better way to write unit-test for this

I actually like it a lot! The specification is also amazingly flexible to mix and play. E.g.:

repository.Query((whereCustomer.HasAgeBetween(20, 30) || whereCustomer.LivesIn("Berlin")) && !whereCustomer.IsVIP());

And they’re still testable and mock-friendly. Now that we know I like this approach, let’s take a look on the implementation of specificatin pattern.

public class CustomerQuery: ICustomerQuery
{
	public ISpecification<Customer> MathesUserSearchFilters(string email, string name, int? ageFrom, int? ageTo, bool isTerminated)
	{
		var result = Specification<Customer>.TRUE;
		if(email != null)
			result &= new Specification<Customer>(x => x.email.ToLower() == email.ToLower());
		if(name != null)
			result &= new Specification<Customer>(x => x.Name.ToLower().Contains(name.ToLower());
		if(ageFrom != null)
			result &= IsOlderThan(ageFrom.Value);
		if(ageTo != null)
			result &= !IsOlderThan(ageTo.Value);
		if(isTerminated != null)
			result &= (isTerminated)?IsTerminated():!IsTerminated();
		return result;
	}
	public ISpecification<Customer> IsOlderThan(int yearOld) {/*..*/}
	public ISpecification<Customer> IsTerminated() {/*..*/}
}

Unlike Linq criteria, Specification plays incredibly well with building dynamic query! And that’s not the best part yet. These Specifications are not mere DB queries. Write once, use it everywhere. It can be used for object-filtering or validation.

var terminatedCustomers = customerList.FindAll(whereCustomer.IsTerminated());

Or:

Validate(customer, !whereCustomer.IsTerminated())
	.Message("The customer had been terminated. Please enter an active customer");

Code for sample Specification API can be found in Ritesh Rao’s post.

Where are we?

Oh yes, our initial objective: plugging in new screen/functionality in Open-Closed Principle fashion. Not a problem. The query can live in separate assembly, and it’s easy for the client to introduce their own set of Specifications that meets their querying needs for their plugins.
This approach also gives us the liberty to override the specification implementation, e.g. to comply with specific persistence technology, or database-structure. Say, if Customer.Name is implemented as FIRST_NAME and LAST_NAME columns. Overriding Specification implementation is not possible with “new” keyword or extension method approach, since the application is tightly coupled to specific Specification implementation.
This allows clients to extend the domain entity with their business-specific properties and persistence-structure.

Partial Mock is Your Friend

UPDATE: Mockito now supports real partial-mock since 1.8.0

Rhino-Mock has one dangerous ability that I find useful from time to time, known as partial-mock.
Right now, I’m working on a Java project using Mockito, and I find it “deliberately” lacking this dangerous yet delicious ability.
Mockito has spy method which is quite similar as partial mock, except that it is not. Here’s what I’m talking about

public void canGetAWarmWelcome()
{
	SUT sut = spy(new SUT());
	when(mock.getName()).thenReturn("Darling");

	assertEqual("Darling", sut.getName());
	assertEqual("Hello, Darling", sut.getWarmWelcome());
}

public class SUT
{
	public String getWarmWelcome()
	{
		return "Hello, " + this.getName();
	}

	public String getName()
	{
		return "Hendry";
	}
}

Using partial mock, both assertions in this test-class would pass. Using mockito spy, however, the second assertion will fail. I.e., you can’t invoke a spy method from the real SUT method from same object.

This is mostly by-design decision on Mockity side. The author argues that partial mocking smells fishy, and that it’s better to refactor the code and eliminates the need for partial-mocking.
I don’t completely agree. There are many occassions where partial-mocking can be useful particualrly when you are dealing with third party API.
I am trying to write a unit-test over a web-service client code using axis. Part of the code are autogenerated from WSDL, and I can’t refactor it to suit my unit-test.
As a brief picture, here’s the autogenerated the class I’m trying to mock (partially).

public AccountServiceBindingStub extends Stub implements AccountService_PortType
{
	public AccountDetailResponse getAccountDetail(AccountQueryRequest accountQueryRequest)
	{
		Call call = createCall();
		call.setOperationName(new QName("", "getAccountDetail"));
		/*
		call.setxxxx(...);
		call.setxxx(...);
		*/

		call.invoke(param);
	}

	protected Call createCall()
	{
		Call call = new Call();
		/*
		etc etc
		*/
	}
}

That little naughty call to “new Call()” closes the door for dependency injection. It makes it impossible for me to sneak in a fake Call instance that returns a stubbed constant string of SOAP response, substituting for making a real HTTP connection to the actual back-end.
Before you ask, I know you might be wondering why I want to involve autogenerated classes into the equation of my unit-test. Shouldnt I be mocking this whole class altogether and stub the AccountDetailResponse object directly (rather than stubbing the SOAP message)? Afterall, I’m not trying to test classes that is autogenerated by axis tool. Or am I?
The reasoning behind this is down to idiosyncrasies we have been facing in transforming SOAP message to domain model, which is exactly what we want to cover in this unit-test. For instance, cents/dollar units, negative/possitive balance sign, with/without currency symbols, PREPAID vs prepaid, and other more subtle issues. So we want to capture several actual SOAP response samples the server provides, and write some test-cases against them.
And yes, I could just write subclass of the class above, and override createCall(). But I have lots of those classes, and I don’t want to pollute my test-cases with obscure technical detail of how I fake the Call.
So I wanted to encapsulate this hacky faking-logic to a test helper that generates the runtime subclasses for me… which is precisely a partial-mock by definition (or reinventing thereof)