STEPHEN CHU . com

Speed up your MySQL data load

2008-12-01T18:22:00.004-06:00

Perhaps you have some large datasets you would like to load into MySQL, such as a big text file generated from mysqldump that contains many sql INSERT statements. You can obviously populate your empty database by redirecting the content of this file into the mysql command line like such:


  $ mysql -u root -p your_database < /tmp/table.dump

Another way to load data into your database or table is to use the sql syntax LOAD DATA INFILE. It requires you to provide a file that is in some delimiter format such as csv or tab-delimited values. One can use the mysqldump command line switches --fields-*-by and --tab to dump the content of the database to such format.

After you started either of these data load, you may want to kick off your shoes and take a nap, cos this will take a while if your MySQL server is not tuned and your dump file is pretty big (in this example, an InnoDB table of 790,000 rows [~500 MB dump] took 45 mins on my MBP using a Macports [mysql5 @5.0.67_1] installation with default settings). If you have to reload your database/tables often, this is unbearably slow.

Fortunately, there are a few MySQL tuning settings that you can specify at mysql server start-up time to tremendously speed up your data load time. Keep in mind, it is advisable to flush your MySQL logs (like the following) prior to tinkering with any log file size settings to avoid any log file corruptions:

(Before you proceed, I recommend you backup your data first)


  $ sudo mysqladmin flush-logs
  $ ... then shutdown your MySQL server ...
  $ sudo rm /path/to/mysql/datadir/ib_logfile*
  $ ... then start your MySQL server ...

So, now it's time to put in our magic sauce. On my Macports MySQL installation, the settings were meagerly defaulted to:


  +---------------------------------+---------+
  | Variable_name                   | Value   |
  +---------------------------------+---------+
  | innodb_additional_mem_pool_size | 1048576 |
  | innodb_buffer_pool_size         | 8388608 |
  | innodb_log_buffer_size          | 1048576 |
  | innodb_log_file_size            | 5242880 |
  | bulk_insert_buffer_size         | 8388608 |
  +---------------------------------+---------+

Add the following settings in your appropriate my.cnf file. Your settings may vary with the kind of machine you have (mine is a MBP 2GB RAM):


  [mysqld]
  innodb_additional_mem_pool_size=256M
  innodb_buffer_pool_size=512M
  innodb_log_buffer_size=256M
  innodb_log_file_size=512M
  bulk_insert_buffer_size=256M

Our settings will now look like:


  +---------------------------------+-----------+
  | Variable_name                   | Value     |
  +---------------------------------+-----------+
  | innodb_additional_mem_pool_size | 268435456 |
  | innodb_buffer_pool_size         | 536870912 |
  | innodb_log_buffer_size          | 268435456 |
  | innodb_log_file_size            | 536870912 |
  | bulk_insert_buffer_size         | 268435456 |
  +---------------------------------+-----------+

Then, as I reload my table, it only takes just 4' 55", a whopping 900% performance boost.

Bonus Tip:

You might notice that you cannot use SELECT COUNT(*) to track your data load progress while LOAD DATA INFILE is being executed (it always returns a count of zero). Instead, you can use the SHOW INNODB STATUS command. Find the line that says "undo log entries" under section TRANSACTIONS, and that number is the number of rows inserted so far. You can also look at how many inserts were performed per second by finding the line "inserts/s" under the ROW OPERATIONS section.

Go to Ruby ! and ? method definition with TextMate CTags

2008-09-26T12:18:00.003-05:00

If you are not yet using the TextMate CTags bundle in your Ruby/Rails development, you are missing out big time. This bundle allows you to go to a method definition as easy as pointing your cursor at the method and hit a keyboard shortcut. It has served me well and I highly recommend it.

But soon you might realize that this "go to definition" shortcut only works most of the time. Specifically, whenever you try to go to the definition of a Ruby method that ends with a bang (!) or question-mark (?), CTags will fail open to the line that defines that method for you.

Here's a fix for you. The perl script that got executed underneath this little utility actually uses a TextMate variable TM_CURRENT_WORD. This variable points at the "word" your cursor is currently laying on top of in TextMate. But since by default, TextMate does not consider ! or ? to be part of a word, your CTags fails on you. All you need to do, is go to Preferences / Text Editing, and change Word Characters to include ! and ?.

Object-Oriented Programming with Rails ActiveRecord

2008-06-22T23:02:00.005-05:00

Have you seen this before?

@posts = Post.find :all, :conditions => { :user_id => @user.id }

This happens to new Rails developers quite a bit. This code retrieves correctly an array of @posts and send them to the view for rendering. But what about it?

The key thing here is that this can also be done in a better way:

@posts = @user.posts

When you use the former method to retrieve @posts, you are actually thinking in database terms along the lines of "in the database table 'posts' and therefore ActiveRecord model 'Post', retrieve the rows whose foreign key column 'user_id' has the value @user.id." ActiveRecord is a pattern for accessing your data from objects, but you also have to combine it with the power of the object-oriented-ness of Ruby to create the chimes of beautiful code. Using Rails and Ruby does not make anyone a better programmer automagically. One can still write code in Java/C# in the same procedural style as if you were writing C. It is how to leverage the best of all worlds makes you a better problem solver.

Therefore, think like an object bigot. Think in objects, and not in database foreign key column values. Whenever you see any Rails code in the pattern of this:

:model_id => @model.id

Just stop. "Objects on Rails" sounds a lot better than "Ruby on Foreign Keys."

Learning encapsulation should come before learning Rails

2008-06-18T01:39:00.006-05:00

Do you see anything wrong in this one line of code?

puts 'Drink up your milk!' if @milk.expiration_date < 2.days.from_now

I think there is. In particular, the programmer can do better. It is about a programming concept that we have all heard and should be familiar with: encapsulation. If you are into testing, which you ought to be, you can probably identify such encapsulation violations by how smelly your tests are, as I believe in Test Driven Design. Going back to the line of code. What's wrong with it?

The pattern of this code is, you have code somewhere that yanks states (expiration_state) out of an object (@milk) and then interrogate against those states (< 2.days.from_now). By no means this is a rule, as exceptions do exist. But, when this happens, your programming siren in your head should go off as if the CI build is broken: can this be a method on the object @milk itself?

Ruby is a powerful programming language. It allows you to do all sorts of fancy programming: dynamically altering classes, add methods to only the selected few object instances, while being duck-typed and object-oriented all at the same time. There are examples galore in Rails itself. But to enable all such magic, all of your ActiveRecord domain models have getters and setters on all of their states (i.e. data). While that is convenient to access their states, sometimes you have to be careful. With great power comes great responsibility, and that responsibility comes down to you.

You might not think this applies to you. But have you ever written code like this in Ruby?

MyMailer.deliver_product_returned_notification if @product.state == State::Returned

total = @line_items.sum { |line_item| line_item.price * line_item.quantity }

if [:admin, :superuser].include?(@user.role)
  ...
end

Now, let's look at these rewritten examples:

puts 'Drink up your milk!' if @milk.expiring?

MyMailer.deliver_product_returned_notification if @product.returned?

total = @line_items.sum(&:subtotal)

if @user.superuser?
  ...
end

Forget about the loops, the blocks, the hype, and everything about Ruby for a sec. Code is code. Not only does the code become more readable, when you try to enrich your domain by naming things correctly, you could also very well be opening up new business concepts that wasn't previously clear or accurate. If you have a domain model, you are modeling it against domain concepts so that your app can interact with. You'd better be sure it is right, or else lots of time will go wasted on code that solves only half of the business problems all the time.

Launching apps in command line (Mac)

2008-06-09T16:03:00.003-05:00

In most cases, to launch an application on a Mac is just a matter of using the awesome QuickSilver keyboard shortcut and type in what you want to open. But, sometimes it is useful to be able to launch/open an application in command line or terminal in a scripting context.

So, instead of going through all the typing $ /Application/MyFavouriteApp/Contents/MacOS/MyFavouriteApp in your terminal, you can do:

$ open -a MyFavouriteApp

This will open any applications in your /Applications folder by name.

Rails composed_of validation

2008-05-21T03:10:00.005-05:00

ActiveRecord allows you to declaratively write validations (e.g. validates_presence_of) in any ActiveRecord::Base models. In addition, the errors will be stored in the @model#errors collection and be used by various view helpers and methods like FormBuilder, error_messages_for, and error_message_on, wrapping a nice error CSS around the offending html form elements. But when you have a normal domain model object that is non-ActiveRecord::Base such as a composed_of type Value Object, you will not have access to these declarative validation magic methods. Today let me try to elaborate on how to use ActiveRecord::Validation methods in your composed_of objects in Rails 2.0.

(The following entry revolves around the assumption that you are using f.fields_for to create and assign value object domain models onto your ActiveRecord::Base models. Using this approach eliminates most value object creation code in your controllers, achieving "Skinny Controllers". To learn about, visit, check out the Rails composed_of &block conversion blog entry.)

Using the same example in the previous entry, let's include our ActiveRecord::Validation module into our Money class like such:

class Money
  include ActiveRecord:Validations
  
  attr_reader :balance, :currency
  
  validates_presence_of  :currency
  validates_inclusion_of :currency, :in => ['USD', 'EUR'], :if => :currency?
  validates_presence_of     :balance
  validates_numericality_of :balance, :if => :balance?
  
  def initialize(balance, currency)
    @balance, @currency = balance, currency
    @errors = ActiveRecord::Errors.new self
  end
  
  def new_record?
    true
  end
  
  def currency?
    !@currency.blank?
  end
  
  def balance?
    !@balance.blank?
  end
  
  def self.human_attribute_name(attr)
    attr.humanize
  end
  
  def balance_before_type_cast
    @balance
  end  
end

You will notice a couple things. One, I have to define the #new_record? method. This method is defined on all ActiveRecord::Base objects, but since our PORO object is not a record per se, we just stub it out. Also, we need to store a collection @errors of type ActiveRecord::Errors.

Depending on what validation routine you will end up using in Money, you may have to stub out different methods. For example, I am showing error messages with error_messages_for (more on this later), and it requires stubbing out self.human_attribute_name (as of Rails 2.0, but no longer needed in future Rails). Using validates_numericality_of requires me to stub out balance_before_type_cast. Also, the validation :if conditions requires me to add the question-mark methods balance? and currency?. Remember, this approach does not give you all the validation magic. For example, validates_uniqueness_of will not work because it assumes too much about your object being a normal AR model and needs access to a database. But in practice, your Value Objects should not need such validations, and in most cases they contain only simple one-off validations, and provide simple functionalities such as formatting like this and this.

After all these, let's see our ActiveRecord Book class.

class Book < ActiveRecord::Bases
  composed_of :balance, :class_name => 'Money', :mapping => [%w(balance balance), %w(currency currency)] do |params|
     Money.new params[:balance], params[:currency]
  end
  validates_presence_of :name
  validates_associated  :balance
end

The composed_of conversion block remains. You will notice the validates_associated :balance line as well. This tells your book instances that they should not be persisted should there be any balance validations failing, just like any normal validations you would write. By default, any failing balance validation will add an error message 'Balance is invalid' in your @book#errors collection. If you want to suppress that message from showing up, you can pass in option :message => nil.

So, to put it all together, here is the view and the controller code:

<h1>New book</h1>

<%= error_messages_for :object => [@book, @book.balance] %>

<% form_for(@book) do |f| %>
  <p>Name: <%= f.text_field :name %></p>

  <% f.fields_for :balance, @book.balance do |ff| %>
  <p>Balance: <%= ff.text_field :balance %></p>
  <p>Currency: <%= ff.text_field :currency %></p>
  <% end %>

  <p>
    <%= f.submit "Create" %>
  </p>
<% end %>

<%= link_to 'Back', books_path %>

To show error messages from multiple objects on the view, I am using the view helper method error_messages_for(*args). The :object option actually allows you to pass an array of objects (c'mon, you should know this trick about ActiveRecord by now. If not, check it out here and here).

def create
  @book = Book.new params[:book]

  if @book.save
    flash[:notice] = 'Book was successfully created.'
    redirect_to @book
  else
    render :action => "new"
  end
end

Again, a skinny, thin, sexy-looking controller action.

A normal Rails un-RESTful page as a resource... or can it?

2008-05-06T14:53:00.006-05:00

An index page comes with Rails scaffolding. It is used to show a list of the "thing" you are CRUD-ing on. However, all too often we are tasked to show some pages that we aren't doing any CRUD operations on. Just a few examples:

Dashboard page after user logs in

The "Forgot your password?" page

Tabs or subtabs of multiple lists

List page showing multiple lists of different entities

A normal dashboard page isn't something you would normally CRUD on. It is more like a place holder page that a user sees the first thing after s/he logs in, showing many items of all sorts valuable to a user. In many Rails app, depending on what those items are, the dashboard page will end up being rendered by one of the items' index action. Worse yet, next time when you actually need that index page by that model, you have to call it something else, maybe 'list', and then play with the routes to get it wired up correctly.

class CustomersController < ApplicationController

  # The dashboard page
  def index
    @customers = Customer.find :all
    @products = Product.find :all
    @tasks_of_the_day = Task.find :all
  end

  # A list of customers
  def list
    @customers = Customer.find :all
  end
  
end

ActionController::Routing::Routes.draw do |map|

  map.resources :customers, :products, :tasks
  map.with_options :controller => 'customer' do |r|
    r.dashboard '/customers/list', :action => 'list'
  end

end

Here's a suggestion: how about put it in views/dashboards/show.html.erb, and, while you are at it, give it a DashboardsController? Then, put it in your routes.rb as a (singular) map.resource :dashboard, like such:

class DashboardsController < ApplicationController

  def show
    @customers = Customer.find :all
    @products = Product.find :all
    @tasks_of_the_day = Task.find :all
  end
  
end

ActionController::Routing::Routes.draw do |map|

  map.resources :customers, :products, tasks  
  map.resource :dashboard

end

By rendering the dashboard page in a completely different controller, you now have a very readable GET dashboard_path named route (GET: http://localhost:3000/dashboards), and you will not contaminate the index action of your other models' controllers with instance variables of all kinds. You also have a more readable routes.rb file.

One of the examples above for non-RESTful pages is the "Forgot My Password" page. Can you think of a good way to do it the Rails REST-ful way? Please go to Seeing Rails Resources Clearly to share some of your thoughts.

Pragmatic Rake tasks organization

2008-04-05T13:43:00.007-05:00

Do you have a lot of rake tasks on your Rails projects? I am sure you do. Rake tasks are handy, and developing your next-big-idea Rails app is a lot less attractive without it. Most of the time you drop a new foo.rake file in your lib/tasks folder, and you write code in it and get out. Over the course of development, that folder is likely to look like your favorite Halloween party house (ahem, Neal's...), except on an every day basis.

If you want your rake tasks to be more manageable, here are a few tips:

1) Namespace your rake tasks.

Namespaces in rake is nice. They make sure all tasks in your namespace are cohesive as a area of work. Still, your job is to keep your rake tasks not too crowded in a single namespace. For example, the db namespace that Rails provides very easily become crowded with tasks of all sorts, because, well, on a Rails application what you do almost always is related to your database: drop/create/migrate your database, importing reference/application data, database integration points with other applications, db related background tasks, backups... Phew! How about create a new namespace for import, integration, background, and backup for each of them?

2) Name your rake files by their namespaces.

Even if you namespace all your rake tasks, still, you are likely to end up with a bunch of .rake files in lib/tasks. But, now that you have your tasks grouped by their meaningful and not-so-crowded namespaces, you should name your files by your namespace. For example, db.rake, data.rake, import.rake, integration.rake, and backup.rake. Can you guess where that rake db:awesome is implemented? You certainly don't know what it does, but you got that right, in lib/tasks/db.rake.

3) Try not to nest more than 2 namespaces.

This tip is not a rule, the point being don't over do it. Sometimes, that third-level namespace really makes sense, For example, db:create:all from Rails creates databases for all environments. It isn't for everything, though. For example, I prefer rake data:load better than rake db:data:load. Not only is it shorter, but also the namespace db is simply implied. Short nested-ness forces you to name your namespaces more descriptively, and thus easier to navigate to their implementation.

4) Move your rake file and its external ruby files into a sub-folder, named by that namespace.

When your rake tasks are complicated enough, you obviously will use objects to do the job for you. When that happens, instead of having lib/tasks/integration.rake, you will instead want to have a sub-folder lib/tasks/integration/, and put files like integration.rake and integration_awesome_klass.rb in it. There really is no need to contaminate your lib folder with rake specific files.

Shameless Plug: Still using NAnt? Check out how to organize your NAnt scripts as well, on this blog post.

params[:fu] #5 ) Update multiple models in update action atomically.

2008-03-27T22:25:00.003-05:00

Updating multiple models is hard? It sounds complicated, but with Rails it actually isn't, if you know how to take advantage of it. Knowing what you know about Rails params, let's take a look at today's topic: the update action.

<% form_for @reader do |f| %>
  <%= f.text_field :name %>
  
  <% for @subscription in @subscriptions %>
    <p>
    <% fields_for @subscription do |ff| %>
      <%= ff.collection_select :magazine_id, Magazine.find(:all), :id, :name, {}, :index => @subscription.id %>
    <% end %>
    </p>
  <% end %>

  <%= f.submit 'Save' %>
<% end %>

generates:

<p>
  <select id="subscription_4_magazine_id" name="subscription[4][magazine_id]">
    <option value="101">PC Magazine</option> 
    <option value="102">IT Pro</option> 
    <option value="103" selected="selected">WIRED</option>
  </select>
</p>

<p>     
  <select id="subscription_5_magazine_id" name="subscription[5][magazine_id]">
    <option value="101">PC Magazine</option> 
    <option value="102">IT Pro</option> 
    <option value="103" selected="selected">WIRED</option>
  </select>
</p> 
  
... (and more)

Processing ReadersController#update (for 127.0.0.1 at 2008-01-14 21:12:56) [PUT]
Parameters: { "commit"             => "Update",
              "reader"             => { "name" => "stephen chu" },
              "subscriptions"      => { "4" => { "magazine_id" => "101" },
                                        "6" => { "magazine_id" => "102" }, 
                                        "7" => { "magazine_id" => "103" } },
              "authenticity_token" => "238ba79b8282882ba01d840352616c2cc79280f0",
              "action"             => "create",
              "controller"         => "readers" }

See the pattern? The POST-ed parameters are of the same structure as in one of our last example, hash of hashes. The sub-hashes are keying off of the subscription id, because this time around we are updating existing subscriptions. So last time we used params[:subscriptions].values, what would it look like this time? Let take a look.

def update
  @reader = Reader.find params[:id]
  @reader.attributes = params[:reader]
  @reader.subscriptions.update params[:subscription].keys, params[:subscription].values
  
  if @reader.save
    flash[:notice] = 'Reader was successfully updated.'
    redirect_to @reader
  else
    flash[:notice] = 'Failed.'
  end
end

Again, another ActiveRecord model method utilizes the array of hashes pattern! The update method source code on RDoc looks like it is just updating one at a time. But a peek at the source code says otherwise:

def update(id, attributes)
  if id.is_a?(Array)
    idx = -1
    id.collect { |id| idx += 1; update(id, attributes[idx]) }
  else
    object = find(id)
    object.update_attributes(attributes)
    object
  end
end

Again, the update method recognizes array! So, where to get the arrays that we will use in our controller action? They come from .keys and .values of course:

params[:subscriptions].keys    # => [ "4", "6", "7" ]
params[:subscriptions].values  # => [ { "magazine_id" => "101" }, { "magazine_id" => "102" }, ... ]

So in essence, our controller code is free from all those ugly params-munging activities. Remember, controller actions should not shuffle around their params, or otherwise it fails to abide the "Skinny Controller, Fat Model" principle, and they will stink.

Now, if you are thinking about by using update, we run the risk of not atomically saving all of our models should any of our models fail validation, you are correct. This is where rescue_from in controller saves the day. Just transact our update action using AR transaction, and re-render the edit page should it catches ActiveRecord::RecordInvalid error, you should be able to make your update action atomic. Given how lean our controller action looks like, having a transaction block that wraps around our code is not so much a nuisance anymore.

This also wraps up our params[:fu] series. Remember, how you assemble your views form elements have a lot to do with how thin and skinny your controllers look like. Thanks for reading!

params[:fu] #4 ) Use the magical _ids=([...array of ids]) association proxy method.

2008-03-26T22:53:00.003-05:00

In the last blog entry, we looked at an example of having to build a new parent model and build its join models all in two lines of code. But if you are trying to just assign a list of has_many or has_and_belongs_to_many models, you don't have to use the fields_for and index trick. You can just use an association proxy method. Let's check it out:

class Reader < ActiveRecord::Base
  has_and_belongs_to_many :blogs
end

<% form_for @reader do |f| %>
  <%= f.text_field :name %>

  <%= f.collection_select :blog_ids, Blog.find(:all), :id, :name, {}, :multiple => true %>

  <%= f.submit 'Create' %>
<% end %>

generates:

<select id="reader_blog_ids" multiple="multiple" name="reader[blog_ids][]">
  <option value="1">Martin Fowler</option>
  <option value="2">Stephen Chu</option>
</select>

Processing ReadersController#create (for 127.0.0.1 at 2008-01-14 21:12:56) [POST]
Parameters: { "commit"             => "Create",
              "reader"             => { "name" => "stephen chu"
                                        "blog_ids" => ["1", "2"] },
              "authenticity_token" => "238ba79b8282882ba01d840352616c2cc79280f0",
              "action"             => "create",
              "controller"         => "readers" }

And now, in my controller action, the creation of the reader only takes one line of code:

def create
  @reader = Reader.new(params[:reader])

... thanks to the following has_many (or habtm) association assignment method:

@reader.blog_ids = [...many ids...]

So, stop doing any more params-munging code in your controller action like these:

def create
  ...
  params[:blog_ids].each do |id|
    @reader.blogs << Blog.find(id)
  end
end

Get on the Rails bandwagon!

params[:fu] #3 ) Using fields_for and the index option to create a new parent model with multiple child models.

2008-03-25T21:19:00.003-05:00

Alright, the last couple days were easy. Today, let's take a look at a slightly more complicated example, but one that occurs on almost every single Rails project out there: saving multiple models on one POST. Let's say your new form allows you to create a new reader and attach 3 subscriptions to it. Here's the code:

class Reader < ActiveRecord::Base
  has_many :subscriptions
  has_many :magazines, :through => :subscriptions
  validates_associated :subscriptions
end

<% form_for @reader do |f| %>
  <%= f.text_field :name %>
  
  <% (1..3).each do |index| %>
    <% fields_for :subscriptions do |ff| %>
      <p><%= ff.collection_select :magazine_id, Magazine.find(:all), :id, :name, {}, :index => index %></p>
    <% end %>
  <% end %>

  <%= f.submit 'Create' %>
<% end %>

generates:

<select id="subscriptions_1_magazine_id" name="subscriptions[1][magazine_id]">
  <option value="101">PC Magazine</option>
  <option value="102">IT Pro</option>
  <option value="103">WIRED</option>
</select>  

<select id="subscriptions_2_magazine_id" name="subscriptions[2][magazine_id]">
  <option value="101">PC Magazine</option>
  <option value="102">IT Pro</option>
  <option value="103">WIRED</option>
</select>

... (and more)

Processing ReadersController#create (for 127.0.0.1 at 2008-01-14 21:12:56) [POST]
Parameters: { "commit"             => "Create",
              "reader"             => { "name" => "stephen chu" },
              "subscriptions"      => { "1" => { "magazine_id" => "101" },
                                        "2" => { "magazine_id" => "102" }, 
                                        "3" => { "magazine_id" => "103" } },
              "authenticity_token" => "238ba79b8282882ba01d840352616c2cc79280f0",
              "action"             => "create",
              "controller"         => "readers" }

By using the :index html option in our form builder generated fields, we are essentially inserting a unique index key to the posted value of that field. Why should we care? Well, here is how the controller code would look like:

def create
  @reader = Reader.new params[:reader]
  @subscriptions = @reader.subscriptions.build params[:subscriptions].values
  if @reader.save
    flash[:success] = 'Good.'
  else
    flash[:error] = 'Bad.'
  end
end

I just persisted a reader with multiple subscriptions, and the notable differences I added in the create action, was these characters: params[:subscriptions].values. Nothing much changed from the last has_one :computer example besides association related differences. There is no looping, map/collect-ing, gsub-ing, etc., in my action while creating and attaching these multiple subscriptions onto the new reader. The controller action is just doing its same-old routine: receives posted parameters, and shove them into the corresponding models. So how is this done?

Looking at the source of the #build method on the association proxy classes (e.g. HasManyAssociation), you will notice something interesting:

def build(attributes = {})
  if attributes.is_a?(Array)
    attributes.collect { |attr| build(attr) }
  else
    build_record(attributes) { |record| set_belongs_to_association_for(record) }
  end
end

The method recognizes array! If you pass in an array of hashes, it will process them one by one! So how do we get an array of hashes? In our case the way to get array of hashes is by calling Hash#values, because they are organized in sub-hashes. Calling .values will yield us the following array:

params[:subscriptions].values  # => [ { "magazine_id" => "101" }, { "magazine_id" => "102" }, ... ]

By organizing your params on your view in ways that they can be directly consumed by your models, you end up with a lot less code to write. No more bastardizing your action code!

(back to the TOC of the params[:fu] series)

params[:fu] #2 ) Put attributes into a different params key using fields_for if they belong to different models.

2008-03-24T23:03:00.002-05:00

fields_for is very useful for separating out your params:

<% form_for @reader do |f| %>
  <%= f.text_field :name %>
  
  <% fields_for :favourite_computer, @computer do |ff| %>
    <%= ff.text_field :name %>
  <% end %>
  <%= f.submit 'Create' %>
<% end %>

Processing ReadersController#create (for 127.0.0.1 at 2008-01-14 21:12:56) [POST]
Parameters: { "commit"             => "Create",
              "reader"             => { "name" => "stephen chu" },
              "favourite_computer" => { "name" => "macbook pro" },
              "authenticity_token" => "238ba79b8282882ba01d840352616c2cc79280f0",
              "action"             => "create",
              "controller"         => "readers" }

def create
  @reader = Reader.new params[:reader]
  @computer = @reader.build_computer params[:favourite_computer]
  if @reader.save
    flash[:success] = 'Good.'
  else
    flash[:error] = 'Bad.'
  end
end

class Reader < ActiveRecord::Base
  has_one :computer
  validates_associated :computer
end

Here, I am on a page that creates the reader and its favorite computer. Two models sounds hard? Not really, if you key your params to a different key for each of the model, your controller is still nice and thin. You do need to add one line to the action to create the computer though, but that maps nicely to the fact that you are operating on two models from the page post. I would not want to add more than one line of code to create one additional model, so forget those params-munging/looping.

Also, you can see that I am using validates_associated here. The point being I do not want to call save or valid? once for each model in my action, making my controller code fat.

Also, fields_for is not only just a helper method, but also a method available on your FormBuilder as well (you know, the f in your form_for is an instance of a FormBuilder. To see an example of how to call it off of a form builder, check out this blog entry.

(back to the TOC of the params[:fu] series)

params[:fu] #1 ) Wrap all relevant attributes in a single params key.

2008-03-23T22:47:00.009-05:00

This is the simplest form of using form_for and params:

<% form_for @reader do |f| %>
  <%= f.text_field :name %>
  <%= f.text_field :birth_place %>

  <%= f.submit 'Create' %>
<% end %>

Processing ReadersController#create (for 127.0.0.1 at 2008-01-14 21:12:56) [POST]
Parameters: { "commit"             => "Create",
              "reader"             => { "name" => "stephen chu", "birth_place" => "hong kong" },
              "authenticity_token" => "238ba79b8282882ba01d840352616c2cc79280f0",
              "action"             => "create",
              "controller"         => "readers"}

def create
  @reader = Reader.create! params[:reader]
end

By doing this, you eliminate the need to explicitly spell out each parameters posted, like the ugly Reader.new :name => params[:name], :birth_place => params[:birth_place].

Notice, Active Record is extremely hash-happy. Let's take a few examples and see how it is so hash-hungry (or rather, hash-happy).

@reader = Reader.new params[:reader]  # => { 'first_name' => 'stephen', 'last_name' => 'chu' }

@readers = Reader.find(:all, :conditions => params[:search])  # => { 'name' => 'chu' }

@reader.attributes = params[:reader]  # => { 'first_name' => 'martin', 'last_name' => 'fowler' }

@reader.update_attributes(params[:reader])  # => { 'birth_place' => 'hong kong' }

Notice the find with condition one. No one says I cannot have a hash or form_for that must map to a database table!

All of the above will work in just one line of code, and I am sure you can find more examples as well. Remember, hashes are one of the most underrated Rails toolset to reduce code you need to write. Stay tuned on how we can optimize the use of these AR one-liners.

(back to the TOC of the params[:fu] series)

Boost your Rails Controller params[:fu]

2008-03-23T22:46:00.010-05:00

In many ways, in a Rails controller POST action, what gets POST-ed in params are often overlooked. In many straight-forward tutorials, we slap a forms_for @customer in our view, and then the params will come out to be directly consumable by our ActiveRecord model. Unfortunately, many new Rails developers cannot see beyond the usefulness of hashing the posted parameters accordingly. In the next few blog posts, I intend to show you that if you take the time to assemble your params the way Rails likes it, you can dramatically reduce the amount of code you have to write in your controller actions and models, simply because Rails takes care of it for you. The end result is getting it done in less code, fewer tests, less trouble.

Here it goes, the params[:fu] series:

params[:fu] # => "1" Wrap all relevant attributes in a single params key.

params[:fu] # => "2" Put attributes into a different params key using fields_for if they belong to different models.

params[:fu] # => "3" Using fields_for and the index option to create a new parent model with multiple child models.

params[:fu] # => "4" Use the magical <association_name>_ids=([...array of ids]) association proxy method.

params[:fu] # => "5" Use ids to update multiple models in update action atomically.

update:

These tips have been chosen as the 8th runner-up in the Railscasts 100th episode contest. And for that, let me add another parmas[:fu] bonus to this list to all readers:

params[:fu] # => "bonus" Passing multiple params in an observe_field, now that you know your parmas[:fu].

Enjoy!

Rails composed_of &block conversion

2008-03-12T21:14:00.010-05:00

If you had read Domain Driven Design, you will remember the author, Eric Evans, talks about various types of model objects in your domain model. Two of which are Entity Objects and Value Objects.

Entity objects are those that we care about their identity. For example, on Amazon.com, two users obviously have different identities, despite the possibility that they have the same name. Value objects, on the other hand, are those that we only care about their internal states' equality. Two money objects each has amount 100 and currency 'US' are treated as equal in the entire application. These two types of objects are similar in that they can all be defined as a class, but their usage is quite different.

Usually, a big bulk of your domain objects will be Entity objects. Almost all of your ActiveRecord::Base subclasses belong to that category, since each object has a different 'id' to uniquely identify themselves.

Rails provides composed_of as a way for your Value Objects to be referenced by an ActiveRecord Entity object. Rails 2.0 even added some syntactic sugar to composed_of, allowing you to pass a block for value object conversion. For Rails 1.x users, you can achieve the same with the plugin called composed_of_conversion. Let's see how to beautify your Rails code.

class Account < ActiveRecord::Base
  composed_of :balance, :class_name => 'Money', :mapping => [%w(balance amount), %w(currency currency)] do |params|
    Money.new params[:balance], params[:currency]
  end
end

class Money
  attr_reader :amount, :currency
  
  def initialize(amount, currency)
    @amount, @currency = amount, currency
  end
end

class AccountsController < ApplicationController
  def create
    @account = Account.new params[:account]

    if @account.save
      flash[:notice] = 'New account created...'
      redirect_to @author
    else
      render :action => "new"
    end
  end
end

Notice how my @account creation does not require any additional Controller code for value object creation and assignment, keeping my controller away from snacks and candy (i.e. thin). But since I have redefined Account#balance and Account#balance= through the use of composed_of, what should my form_for fields in my view look like? The answer lies with a use of fields_for.

<% form_for(@account) do |f| %>

  <% f.fields_for :balance, @account.balance do |ff| %>
  <p>
    Balance:  <%= ff.text_field :balance %>
  </p>  
  <p>
    Currency: <%= ff.text_field :currency %>
  </p>
  <% end %>

  <p><%= f.submit "Create" %></p>
  
<% end %>


POST Parameters:
{ 
  "commit" => "Create",
  "account"   => { "balance" => { "balance"=>"400", "currency"=>"us" } }
}

By calling f.fields_for on the form builder (and not just fields_for), you are creating a new hash scope 'balance' under params[:account]. Since your composed_of is called balance, only the sub-hash 'balance' will be used in your composed_of block conversion. Then, your model will use the sub-hash to convert it into the Money object you want.

At this point, your Money value object need not be confined with a constructor that takes parameters by ordinal position anymore. You can have it take a hash just like any ActiveRecord::Base classes, so that any future addition of parameters to Money does not require changes to your conversion block. Just drop an html element using your fields_for form builder and it will just work.

Why do this? Now, you have a different class to handle money related operations, such as addition, subtraction, validations, money comparison, and even formatting your money amounts everywhere. Should you have another model that needs money, you know where to share code from.

Also, do not get into thinking that a value object must abstract at least 2 columns on a table. It doesn't have to be. A value object can abstract just one column but contains its own validation and formatting logic and just be fine.

Value objects are very important in any application, even though a lot of times they are simple. Make good use of them and your code quality will improve. Now it's time to roll up your sleeves and clean up those value object creation code inside your controllers!

A little #to_s can do wonder in DRY-ing up your views

2008-03-10T23:03:00.001-05:00

Do you do this every day?

<h1>Show - Blog <%= @blog.name %></h1>
<dl>
  <dt>Author:</dt>
  <dd><%= @blog.author.full_name %></dd>
</dl>

<ul>
  <% for comment in @blog.comments do %>
    <li>Posted by <%= comment.created_by.full_name %>: <%= comment.description %></li>
  <% end %>
</ul>

I am lazy. Having to spell out in my views what field to render for every object when it is that field 90% of the time in my application gives me too much typing. When I render a @comment, you bet I want its description field most of the time; similarly, when I render a @user, you bet I want his/her full_name.

Don't forget all these erb tags at the end of day only render #to_s anyways, so you can just define your default field there to get rid of some typing as well:

class Blog < ActiveRecord::Base
  def to_s
    name
  end
end

class User < ActiveRecord::Base
  def to_s
    first_name + ' ' + last_name
  end
end

class Comment < ActiveRecord::Base
  def to_s
    description
  end
end

Now, let's see how our view looks, 90% of the time:

<h1>Show - Blog <%= @blog %></h1>
<dl>
  <dt>Author:</dt>
  <dd><%= @blog.author %></dd>
</dl>

<ul>
  <% for comment in @blog.comments do %>
    <li>Posted by <%= comment.created_by %>: <%= comment %></li>
  <% end %>
</ul>

See, it's a lot DRY-er and less verbose.

You will also notice that by doing this I eliminated a need to use a possible Object#try(:method) case:

<dd><%= @blog.author.try(:full_name) %></dd>

is no longer necessary, because a nil @blog.author will automatically give an empty string on the view. The end result is we all get to be lazier every day :-)

Format your data using Rails composed_of Value Objects

2008-01-11T03:58:00.000-06:00

A requirement comes in as usual says, "I want prices to be formatted like '$2,000.00' on all rhtml's."

Your programming brain then associates the word "formatted" with "presentation". Afterall, the price column is simply stored in a decimal column in the database. And off you go, wrapping number_to_currency(@book.price) on all of your views, just like any tutorial says.

It's all good until the same formatting is required for all emails being sent, all web services requests being issued, all Excel/CSV reports being generated, all everything. Since you took advantage of formatting the prices with an ActionView helper method number_to_currency, you cannot do the same in places where ActionView helper methods are not available to you. You start creating wrapper classes around your Book class, maybe a presenter, to keep this "presentation" logic outside of your pristine business domain model.

class Book < ActiveRecord::Base
end

class BookPresenter < SimpleDelegator
  include ActionView::Helpers::NumberHelper
  
  def price
    nummber_to_currency "#{__getobj__.price}"
  end
  
  def isbn
    ... ...
  end
end

class BooksController < ApplicationController
  def show
    @book = BookPresenter.new Book.find(param[:id])
  end
  
<p>
  ISBN: <%= @book.isbn %>
  Price: <%= @book.price %>
</p>

To present formatted data is extremely common on any applications, be it a price, date/time, name, address, SSN, ISBN, or even a baseball game score. A lot of these information are so simple that they can be stored in a single database column and be represented by a primitive type like String, DateTime, or Fixnum in Ruby. Since they are so simple in nature, programmers think that all it takes to format them is wrap a helper method around it on the view. But this approach may backfire like the example above.

Alternatively, one can create a value object that has a #to_s method that does all the formatting for you. That way, everywhere that needs formatting can ask a book object for it, and not have to wrap a presenter around it for something as formatting.

class Book < ActiveRecord::Base
  composed_of :isbn, :class_name => 'ISBN'
  composed_of :price, :class_name => 'Money'
end

class ISBN
  def to_s
    ... ...
  end
end

class Money
  include ActionView::Helpers::NumberHelper

  def to_s
    nummber_to_currency @price
  end
end
  
class BooksController < ApplicationController
  def show
    @book = Book.find param[:id]
  end
  
<p>
  ISBN: <%= @book.isbn %>
  Price: <%= @book.price %>
</p>

So what's the difference? The primarily difference is, every Book object now knows how to format its information correctly, and now when a book needs to be used by various sources, be it on an rhtml page, a web service request, or an Excel/CSV report, those sources can interrogate a book object, but not "first fetch a book object, then wrap it with a presenter".

Some people will yell now, "but aren't you putting presentation logic into your domain model?" Yes, kinda. One of the toughest decisions that a developer always has to make is to distinguish between domain logic and non-domain logic. Now if I may ask, if my formatting is happening in multiple places, is this domain logic? I tend to say it is. Also, from experience, while it is extremely rare for me to put such presentation/formatting logic in an Entity Object domain model class (e.g. Book) because such classes are almost always complicated enough in just domain behaviors, having formatting logic in these usually less complicated Value Objects (e.g. Money) makes a lot of sense to me. And in Rails, composed_of provides me that convenience to me almost effortlessly.

The way Rails has these handy ActionView::Helpers is super useful, but every once a while we must step back and ask ourselves the question, "how can I use them beyond the tutorial way on my really complicated application?"

Tip #1: You can also have your #to_s methods to take an additional parameter to customize the formatting behavior. Take a look at how DATE_FORMATS are used in Time#to_s.

Tip #2: You may not need a new class for the sake of using composed_of every time you have a new formatting requirement either. Sometimes you may be able to get away with just extending the Ruby primitive types with a custom method/format, for example Price: <%= @book.price.to_s :money %>.

Go to mysql command shell using rake

2007-12-26T00:01:00.001-06:00

Tired of typing mysql -u foo -pbar -h db01.firewalled.com baz_development yet? Add to that when you have database environments galore like test, QA, selenium, staging, etc.

Here's a rake task for you. Just point it to a Rails environment and execute and you are in the corresponding mysql command prompt.

e.g.

$ rake mysql RAILS_ENV=qa

Loading mysql in RAILS_ENV=qa...
    Executing: mysql -u foo -pbar -h localhost baz_qa_test

Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.0.51 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

Code:

desc "Loads mysql command prompt"
task :mysql do
  puts "\n\nLoading mysql in RAILS_ENV=#{RAILS_ENV}...\n"

  database_yml_path = "#{RAILS_ROOT}/config/database.yml"
  database_yml = YAML.load(ERB.new(File.read(database_yml_path)).result)

  raise "Could not find environment #{RAILS_ENV} in database.yml" unless database_yml[RAILS_ENV]

  config = database_yml[RAILS_ENV]
  username = config['username']
  password = config['password']
  database_name = config['database']
  host = config['host'] || 'localhost'
  port = config['port'] || 3306
  socket = config['socket'] || '/tmp/mysql.sock'

  raise "Failed. Setup requires a user and database for environment #{environment} in '#{database_yml_path}'.\n\n#{database_yml.to_yaml}" unless username and database_name

  args = "-u #{username}"
  args << " -p#{password}" unless password.to_s.strip.empty?
  args << " -h #{host}" unless host.to_s.strip.empty?
  args << " #{database_name}"

  command = []
  command << "mysql #{args}"

  puts <<-EOS
    Executing: #{command.join(' ')}\n
  EOS
  system command.join(' ')
end

Rid of those TextMate "ctags timed out!" error

2007-12-17T02:04:00.000-06:00

If you use TextMate for your Ruby on Rails development, you must use the ctags bundle CTags.tmbundle. It gives you a keyboard shortcut CTRL-] to jump to files that defines the class/method your cursor is on. Similar to "go to file and line" in some IDE. Super handy.

Trouble is when your project gets large, it gets longer and longer to update the tmtags index file to a point where it exceeds the default 15 seconds timout and gives a "ctags timed out!" error.

What you can do is edit the file ~/Library/Application\ Support/TextMate/Pristine\ Copy/Bundles/CTags.tmbundle/Support/bin/tmctags and increase the timeout $CtagsTimeout value to something more reasonable. Problem solved.

p.s. CTags suffers from another shortcoming, and that is it cannot index ruby methods named ending with a ? or !. Anyone knows of a fix/workaround? Please comment if you do!

Why create database migrations, when you don't need to?

2007-12-14T00:15:00.000-06:00

Rails out of the box supports database migration. It allows Rails programmers to be more Agile, do less database BDUF and instead change the database schema as the requirements change and the business wants it. Writing a migration is also as easy as running a command line script/generate migration and then executing it with rake db:migrate.

Database migration is useful for two reasons. Number one is to allow developers to change the database schema. This can be as simple as a single ALTER TABLE statement to add a new column. Number two, more importantly, is to migrate the all-important data exist in the database. But if your application is still under development for initial release, then migration may not be buying you too much good, because chances are no one cares how your table structure got to where it is now, and you may not have a whole lot of data.

We all use migrations feverishly probably because most of the Rails books/references/tutorials begins with: "Let's start by creating a database migration, in it we insert some data into the newly created tables, and learn how to do XYZ." Thus, your db/migrate folder is stuffed with migrations that does everything: create new tables; alter existing tables; create data for those tables; update data created from previous migrations; and perhaps all of the above, while without justifiable reasons on what those migrations are trying to migrate. In practice, it is quite time-consuming for a developer to run 200+ migrations every time he blows away a database, which is not uncommon. Not only that, sometimes you have multiple migrations that basically cancel out each other's changes as our customers change their minds back and forth. As a result, you could be creating migrations left and right when there may not be any real beneficiaries: a database loaded with data.

Sometimes your QA team may have their own data set that tests your app, and thus their loaded database is a beneficiary. While that is true, I prefer their data to be scripted and be freshly generated by ActiveRecord models (using create!) every time instead of keep migrating them, because as my application domain model expands, I want not just QA data but all datasets to be cleansed and validated by my model validations. There is no guarentee that after running a drop_column migration the QA dataset does not violate any business logic. Keeping all these data valid while database migrations are being rapidly created is very hard.

To take advantage of the fact that your development environment has nothing to lose until you have an initial release, while maximizing the benefits of keeping all your data valid (for all enviroments) the whole time while your app is under development, it's a simple steps 1-2-3:

Step 1:

Have one migration file, 001_release_one_schema.rb, that captures all database object creations. For example, all your create_table, create_index, views, triggers (*yikes*), etc. After this migration, your database should contain all database objects for your Rails app but in a "blank", data-less state.

$ cat db/migrate/001_release_one_schema.rb


  class ReleaseOneSchema < ActiveRecord::Migration
    def self.up
      create_table "foos" do |t|
      end

      (... and many others ...)
    end

    def self.down
      (... ... ...)
    end
  end

Step 2:

Create a rake task to populate all reference data that your application requires to run with. Reference data meaning all data that your application cannot change through its screens, but are essential for your app to run. For example, all currencies that are used to populate a drop-down on your app. A lot of drop-down lists data are reference data.

$ cat lib/tasks/data.rake


  namespace :data do
    desc "Loads a default dataset of both reference and user data into database."
    task :load => [ :environment,
                    :configuration,
                    :reference_data,
                    :user_data ]

    private

    task :configuration do
      ENV['DATASET'] ||= 'slim'
    end

    task :reference_data do
      require "#{RAILS_ROOT}/db/data/reference_data/#{ENV['DATASET']}"
    end
  end

$ cat db/data/reference_data/slim.rb


  @us_dollar  = Currency.create! :name => "US Dollar"
  @yen        = Currency.create! :name => "Yen"
  @euro       = Currency.create! :name => "EURO"
  ... ... ...

Step 3:

Create a rake task to populate all user data that your app requires to run with. User data are data that a user can create/update within your application. They are also required for the Rails app to function properly the first day when it launches. For example, for your flashy Paypal application, the fees structure on how it charges its users. An application administrator is allowed to raise or drop fees in your app.


  namespace :data do

    private

    task :user_data do
      require "#{RAILS_ROOT}/db/data/user_data/#{ENV['DATASET']}"
    end
  end

$ cat db/data/user_data/slim.rb


  Fee.create! :amount => 1_000, :currency => @us_dollar
  Fee.create! :amount => 1_500, :currency => @yen
  Fee.create! :amount => 2_500, :currency => @euro
  ... ... ...

There are several benefits of managing your database schema and data this way:

It is easier and faster to re-populate your entire database to the latest schema from scratch with data, since there are no extraneous migrations.

Faster to locate and update data needed for application. They are always in your dataset generation scripts.

No need to worry about outdated/removed ActiveRecord classes and declare them inside the migration file itself. They is no "legacy" ActiveRecord models.

All data are valid all the time because they are created through Model.create! sanitized by your model validations.

Easy to specify datasets to load by preference for DEV (slim, loaded), BA (story sign-off), QA (scenario-based), or demo (full) environments. e.g. rake data:load DATASET=slim RAILS_ENV=qa

No worries about broken/incomplete migrations. Fewer code, fewer trouble.

Now, after your Rails app goes to a 1.0 production release, you should switch this back to the normal Rails database migration style. I suspect your application users won't be too happy if you blow away their data every time you roll out a minor update or a major release... (or not?)

The last "D" in TDD means more than just "Development"

2007-12-04T21:50:00.000-06:00

When asked "Do you write tests?", a lot of developers these days will say "of course" as their answers. However, not everyone can admit to doing TDD (Test Driven Development) correctly. Test Driven Development says, a developer will write a test that fails first, then write code to make the test pass, and refactor when possible, and repeat. This is what most people's TDD rhythm is. For the most part this is fairly easy to do. But to reach the next level, one has to understand TDD as a tool: TDD means more than just test your own code. Here is a couple tips on what the last "D" means:

Discipline

It takes a great deal of discipline to even write a failing test before writing the actual code. Sometimes, we write a little seudo-code here, or move a method definition there, or changing code else where trying to see if more new code needs to be written after it, and sooner than you think you are writing the actual implementation of the methods you wanted to test (Test Afterwards Development anyone?). Other times you write the test, but you are too anxious to even run it and see it fails. And other times you want to jump into the actual code immediately when you see your new test fails, but failing for the unexpected reasons.

Don't fall into these traps. If anything is true, testing is hard, but it is at the same time rewarding and fun. What's also true is, it will pay off. Write the failing test, draw up your list of tests you will need to write, and satisfy them one by one. Having discipline is the cornerstone of becoming a better programmer.

Design

It takes too long to write a test? Tests are running too slowly? Are your tests difficult to read? Are they too brittle and fail all the time? Hang in there! You ever had the feeling you saw code in the codebase that irks the living hell out of your mind written by someone else on your team? Well, it is time for you to get some of these feedback about your own code. Yay, your code sucks! Your tests are telling you that! Let's address each of these one by one.

Slow running tests? You shouldn't be hitting a database or web service in your unit tests, because you can mock/stub them out. Difficult to mock/stub it out? There probably is a better way to design your classes your tests are hitting. Ever heard of Inversion of Control (or Dependency Injection)? Master them. True TDD masters use them extensively.

Unreadable tests? Is it because of too many mocks/stubs? Or is it the code is 500 lines long and doing high octane 720-double-backflip logic? Either way, you have to learn to like small objects. Check this blog post of mine out.

Hard to test something? Tests too brittle? Perhaps you have encapsulation issues in your class design. If your classes are referencing 15 other neighbors, of course they are hard to mock/stub. Chances are, you have to spend time to debug your tests to find out what's wrong! Heard of Law of Demeter? Even if you have, take a look at this highly entertaining yet informative post. It might change your perspective a little.

The bottom line is, TDD is a way to guide you to writing good code, but only if you know how to use it as a tool. Now that you know, hopefully you will have a new perspective next time you write a test.

Passing multiple parameters in observe_field

2007-12-02T09:35:00.000-06:00

Sometimes you just want to pass multiple parameters on your ajax observe_field web request for processing. If you Google around, you will find out there are multiple ways of doing it.

And here is yet another way:

    observe_field :target_dom_id,
      :url => some_named_route_url,
      :method => :get,
      :with => "Form.serializeElements($('text_field_one', 'text_field_two', ...))"

where 'text_field_one' and 'text_field_two' are html form element ids.

I like this because if my text fields are rendered through form_for or fields_for (e.g. form_for :product), and my action does not want to explicitly specify what the posted parameters are upon processing (e.g. Product.create params[:product]), then by using Form.serializeElements I can hide all the params inside the key params[:product] still.

Parsing time in custom format in Ruby

2007-11-27T23:50:00.000-06:00

I am sure you have used Time#strftime (or Date, DateTime) to format your date/time into your own custom format for pretty time display. This is fine and great. But how do you parse a date or time in your own specific format in Ruby?

There is a little method called DateTime#strptime. Its second parameter takes a string, containing basically all the options available to you in Time#strftime (eg. %m-%d-%Y) and return you a DateTime object. This method is strangely not available in Time class. Fortunately from there you can use the Rails methods #to_time to convert to the Time object you want.

Usage:

  DateTime.strptime("12/25/2007 01:00 AM EST", "%m/%d/%Y %I:%M %p %Z")

Stop tangling those iPod earbuds!

2007-11-21T01:35:00.000-06:00

You have an iPod? If you ever use their white trademarked earbuds, chances are every time you pull them out of your pockets you spend 40 seconds of your life untangling those stupid wire. It adds up.

Ever notice there is a little sliding thing at the Y-split of your earbuds on the wire? I assume it is used to reduce the split between the two ear pieces when you are wearing them. If they are for that purpose I have never used them. But I did find a better use of it...

Next time when you are done using them, slide that thing all the way to the ear pieces. This will make your Y-splitted earbuds into essentially one straight wire, and thus reduce a lot of unnecessary tangling. Hopefully this saves you some annoying faces in untangling the thing (maybe that's why those Apple billboards have no face on them...)

Numbers don't lie, but your test coverage numbers might

2007-11-18T15:30:00.000-06:00

We have learned all though our education that our decisions should base off of numbers. Can you support your family given however much you are making? What is the velocity of gravity? How many roses do you give your girlfriend on Valentine's Day (no satisfying answers ever...)

The same happens in software. On all software projects, various flavors of metrics are gathered and read by various types of people on the team. Code coverage being one of them that is most commonly mentioned.

But there is a misconception about code coverage: When the percentage is high, it is good; otherwise, it is not good.

Then people try to find definition of "high", some say 80% is a good number, others say 90%. Some even strive for 100%.

But let's decode this message a little more thoroughly. If your number is low, this means your code is not very well tested. Clearly this is not desirable in a code base that you have to go in every day and make changes here and there - good luck in not breaking stuff. So, a low coverage number is bad.

Now, a high coverage number means your code is well tested. But this number does not tell you a few things:

1) The quality of your code. It does not tell you whether your objects are coupled like spaghetti; it does not tell you whether your code is doing crazily unreadable nested iterations mixing with multiple levels of recursions; it does not tell you whether you are violating encapsulation and separate of concerns; it does not tell you whether you are copy & pasting code everywhere. If your code exhibits all of these symptoms, a 99.9% code coverage still means future changes to the code base is going to be a nightmare.

2) The quality of your tests. Tests are also code that needs to be maintained. If it takes someone 30 seconds to make a single line of code change, but it breaks hundreds of tests and takes that person an additional hour to duplicate that fix in all failed tests, then the burden of maintaining those tests outweights the feedback of whether the system still works or not. Further, if tests are hard to read as they are long and mocks/stubs everywhere, and the test method names do not reflect what the test method body is doing, then having your tests unmaintained just adds test maintenance time.

So how should you read the coverage number? You should read this number in combination with other metrics to gauge the health of the code base. For example, do defects constantly come from a feature area of the code when changes are being made? Are story estimates higher in certain areas of the code base but not others? Are stories in certain functional areas always being under-estimated during Planning Meeting?

Code coverage is just a number. It does not tell you whether your code's flexibility to reflect the pace of your business requirements change. True productivity comes in with good code solving a particular problem well. Well tested code can still be bad.

Seeing Rails Resources Clearly

2007-11-15T20:02:00.000-06:00

REST-fulness is probably one of the hot topics you will face when you walk into the Rails world today. One of the terms you will notice when you dig deeper into Rails REST-ful routes is "resources". In Rails, after running the scaffold resource generator, you will get a database table migration, a domain model class, a controller, and 4 CRUD views. This is the skeleton that Rails provides for you to build upon.

However, a lot of Rails developers don't know how to bend this structure when it comes to non-CRUD operations (eg. search for something, reset password, login, various ajax actions etc.)

I have a solution for you: Routes Driven Development - Let your named routes guide you. Here's how.

As an example, suppose you have a Rails app that an admin can create/remove users that access your application. You have a database table named Users, which has a password column.

In your routes.rb, with a single line of:

map.resources :users

You automatically get these academically REST-ful urls:

  GET    /users          # => users_url
  POST   /users
  GET    /users/new      # => new_user_url
  GET    /users/:id/edit # => edit_user_url
  GET    /users/:id      # => user_url
  PUT    /users/:id      
  DELETE /users/:id

Then, there comes a requirement that says "From the login page, a user who forgot his password can go to a page to see directions on how to retrieve his forgotten password." We are all familiar with this concept, right?

The next thing you know, the developer jumps into the code:

  class UsersController < ActionController::Base
    def index...
    def show...
    def new...
    def edit...
    def create...
    def update...
    def destroy...
    
    def forgot_password
      # new code here
    end

And in your routes.rb:

  map.resouces :users
  map.forgot_password_url    '/users/forgot_password', :controller => 'users', :action => 'forgot_password', :method => :get

Oops. And you just made a wrong move in building upon the Rails skeleton. You put the action in the wrong controller.

Most developers make the assumption that a Rails "resource" is synonymous to a database table. This observation is made because this is how most people would start their Rails application, as in our case our Users concept. With this in mind, since password is a column rather than a table in the database, it should not have its own controller, and thus no model, nor view, etc. Still, we got to jam the code somewhere, and therefore the Users controller, the table on which it belongs. But this observation of one-table, one-controller is not always true.

Instead, I would keep my users controller free from password related operations and do this:

  class PasswordsController < ActionController::Base
    def forgot
      # new code here
    end

  map.resource :password, :member => { :forgot => :get }

This gives me an url of:

  GET: /passwords/forgot    # => Which gives you forgot_password_url

My named route of forgot_password_url is telling me that I am doing the right thing, because it is extremely readable. Had I not been creating a new passwords controller and put the action in my users controller, using the same technique I used to define my named route, I would end up with a not only incorrect but also nasty forgot_password_user_url(:id) by the naming convention (<action>_<controller>_url). Definitely not what I want to pass to any link_to() or button_to().

If you realize, you just created a controller (and its view) with no database table backing it what-so-ever. It doesn't even have a model class. But then again, how much more complicated can it be when the customer is asking for a page that shows directions?

By putting actions in the wrong controller, your controller actions will lack cohesiveness (i.e. actions don't operate on the right thing) which inhibits sharing, you will also run into problems in sharing views (pages, partials, rjs, helpers) like "why am I rendering a partial that is in an obscure folder?", etc. In short, a lot of bad code.

Using this routes driven technique to drive your actions and controllers, can you identify which action/controller combination is correct for a 'Login' button?

1) create_session_url  # => :controller => 'sessions', :action => 'create', :method => :post
2) login_user_url      # => :controller => 'users',    :action => 'login',  :method => :post
3) login_url           # => map.login '/login', :controller => 'whatever', :action => 'anything', :method => :post

The true answer is, it doesn't really matter, as long as your controller is responsible for the correct "resource", and the named route is screamingly readable to anyone using it.. No matter which controller you put your actions in, you can always fall back to creating a named route for it using map.connect like option (3) above. The bottom line is, Rails "resources" might not be what you and others think it is.

As homework, go read more about Rails resources. Not only can you specify :member, but learn how to use other options like :new, :collection, and nested routes. Also, there is also a singular "resource" method and a plural "resources" method. Get familiar with them. This rdoc page is IMHO pretty poorly documented, but hopefully this article gives you a reason to explore them.

Extreme Rails programming

2007-03-20T20:27:00.000-05:00

What is considered as extreme Rails programming? How do you know you are coding in a zone where no man has gone before you?

That's the feeling of you standing on top of Mount Everest, with the world underneath your feet. That's the feeling when history is in the making before your eyes.

So how do you get there?

Got dumped by your girlfriend because you love Rails more than her? Uh-uh.

Solving a problem no one has thought of before in Ruby on Rails? Negative.

Writing Rails code on the moon? Nah.

Today I looked over a programming pair's shoulders, and I immediately realized that they were in a programming zone where no one ever has gone before them. So I took a picture as proof.

And it is...

Writing a Rails app in vim, with a Dvorak keyboard.

This is real, and it's priceless.

So do you know what key you press to yank a line?

(Thanks to Muness Alrubaie and David Vollbracht for this shot)

Rails migration pitfalls

2007-03-06T23:27:00.000-06:00

When we create a migration, we are essentially creating a "delta" SQL script, whose intent is to change the state of the database at a given time, including its schema and its data, from point A to point B. Rails migration scripts even allow you to rollback from B to A. A lot of developers got the delta of the schema part right, because it is the most intriguing part, but they fail to recognize the data part in their migrations.

Take a look at the following migration:

class MoveColumnFooFromTableAToTableB < ActiveRecord::Migration
  def self.up
    remove_column :table_a, :foo
    add_column :table_b, :foo, :string
  end

  def self.down
    add_column :table_a, :foo, :string
    remove_column :table_b, :foo
  end
end

This migration drops column "foo" from Table A, and adds column "foo" to Table B. From the schema point of view, the migration achieves what it's supposed to do - moving a column. But this delta script is flawed from the data point of view.

Imaging your application is at 1.0, users start inserting 20MM rows into your Table A. This migration script is part of your release 1.5 upgrade. You run this script against your table A. Poof! The column is moved, but you just lost *all* of your data in the column "foo" forever. Table B now has a "foo" column with no data in it. Things went wrong, so you rollback, right? Try it. Still, your Table A now contains a "foo" column with all NULL values in it. The data are gone.

Worse yet, your boss is standing behind you, giving you 15 minutes to fix the whole mess. "Database migration sucks...", you mumbled.

This migration fails to migrate the data from point A to point B. So, what should developers have done differently? Well, here's one way to do it:

  def self.up
    add_column :table_b, :foo, :string
    execute("UPDATE table_b INNER JOIN table_a ON table_b.id = table_a.id SET table_b.foo = table_a.foo")
    remove_column :table_a, :foo

  end

  def self.down
    add_column :table_a, :foo, :string
    execute("UPDATE table_a INNER JOIN table_b ON table_a.id = table_b.id SET table_a.foo = table_b.foo")
    remove_column :table_b, :foo
  end

Now with this delta script, which accounts for data migration as well, will do what it is intended to migrate: schema and data.

Here is another gotcha situation:

  def self.up
    add_column :table_a, :foo, :null => true
  end

When you have a table that contains rows in it, like our 20MM row table, after this migration, the 20MM rows will contain NULL in column "foo". Your database will complain null column constraint violated after this migration.

So be careful when you are performing these migrations. My advice is, do a sanity check on all your migrations by running them against a database with tables populated with data. It actually may not be a bad idea to run a CI build on any migration check-ins, to tick off all migrations, against a database full of data, if your migrations are meant to migrate data.

Enhance Array#collect to become magical

2007-03-06T22:46:00.000-06:00

Do you do this often?

customers.collect { |customer| customer.name }
customers.collect { |customer| [customer.name, customer.id] }

Array#collect is indeed very powerful. But I still find myself to repeatedly declare a variable to keep a reference of the elements I am iterating. I could care less if I call it |customer| or |c|.

What if I enhance the Array class to do the following:

customers.collect_name
customers.collect_name_and_id

DRY-ness... Inspiration came from ActiveRecord's magic #find method.

class Array

  def method_missing(method_sym, *args)
    if collect_by_method?(method_sym)
      attributes = fetch_collect_attributes(method_sym)
      if attributes.size == 1
        block = lambda { |element| element.send attributes.first }
      else
        block = lambda { |element| attributes.collect { |attribute| element.send :"#{attribute}" } }
      end
      self.collect(&block)
    else
      super
    end
  end

  private

  def collect_by_method?(method_sym)
    method_sym.to_s =~ /^collect_/
  end

  def fetch_collect_attributes(method_sym)
    attributes = method_sym.to_s.gsub(/collect_/, '').split(/_and_/)
    raise ArgumentError, "Array#collect_* requires at least one method name after it. eg. #collect_id" if attributes.empty?
    attributes
  end

end

And of course, code is no good without tests:

class ArrayTest < Test::Unit::TestCase

  def test_collect_raises_exception_with_no_parameter
    assert_raise ArgumentError do
      [].collect_
    end
  end

  def test_collect_with_one_parameter
    array = []
    array << TestStruct.new(:id => 1, :foo => 'Foo 1')
    array << TestStruct.new(:id => 2, :foo => 'Foo 2')
    assert_equal ['Foo 1', 'Foo 2'], array.collect_foo
  end

  def test_collect_with_multiple_parameter
    array = []
    array << TestStruct.new(:id => 1, :foo => 'Foo 1', :bar => 'Bar 1')
    array << TestStruct.new(:id => 2, :foo => 'Foo 2', :bar => 'Bar 2')
    assert_equal [ ['Foo 1', 'Bar 1'], ['Foo 2', 'Bar 2'] ], array.collect_foo_and_bar
  end

  def test_collect_does_not_interfere_default_method_missing
    assert_raise NoMethodError do
    [].foo
    end
  end

end

Introduction video to ActiveResources RESTful Rails controller actions

2007-01-13T21:28:00.000-06:00

The world is talking about REST. Just when you are reading voraciously about Ruby and Rails, what would be nicer than having DHH to tell you what the next version of Rails' vision is regarding the whole REST idea?

Consider how you would write 4 named routes for the following ActiveController actions today:

POST /people/create
 GET /people/show/1
POST /people/update/1
POST /people/destroy/1

ActionController::Routing::Routes.draw do |map|
  map.with_options(:controller => 'people') do |people|
    people.connect 'people/create', :action => 'create'
    people.connect 'people/show/:id', :action => 'show'
    people.connect 'people/update/:id', :action => 'update'
    people.connect 'people/destroy/:id', :action => 'destroy'
  end
end

How many times have you done CRUD on a model? With ActiveResources, all your need is 1 line of route setting:

  POST /people
   GET /people/1
   PUT /people/1
DELETE /people/1

ActionController::Routing::Routes.draw do |map|
  map.resources :person
end

Convention over configuration can buy you a lot of coding. Rails does just that here (and more). Enjoy.

Presentation (pdf)
There is also a ActiveResources cheatsheet

Migrate Thunderbird mailboxes from Windows to Mac

2007-01-05T18:44:00.000-06:00

When I moved from my Windows machine to a Mac I had a bunch of mail in my Thunderbird that I wanted to migrate over. These are emails going back to the days of the war in online email services. As it turns out it isn't as easy as copying the mailboxes over from Windows to my shiny Mac. MozBackup didn't work for me because it is only for Windows-to-Windows. I found a solution after some soul searching from Google... Turns out it's almost as easy as copying over your mailboxes...

1) Install the T-Bird extension MBoxInput
This is the beauty in open standard. Mailboxes in T-Bird are stored in MBox standard, therefore they are portable to whichever email client you wish to use. Install this onto your Mac T-Bird.

2) Copy your mailboxes from your Windows T-Bird profile folder to your Mac
They are in your %APPDATA%\Thunderbird\Profiles folder. I used my good-ole USB drive...

3) On your Mac, right click on your Folders menu, there will be a "Import/Export" item. Select it, and then choose the mailbox in your Profile folder/sub-folder you want to import (eg. Inbox)
If you have multiple email accounts set up, you can right click on the account before selecting Import/Export. The mail will then be imported into the appropriate account as a subfolder.

Canadians love iPod...

2007-01-03T01:06:00.000-06:00

I mean, you gotta see this.

Read the story "the iPod-wearing native American".

Use your test classes to identify smelly code

2007-01-01T23:20:00.000-06:00

How can you tell if you have smelly code? Your test classes actually can tell you a lot about it. Here are a few unit test anti-patterns:

Big test class
Hopefully you have one unit test class per class/module/whatever-that-contains-behaviors you write code for. When your test class starts becoming unwieldily difficult to read and understand, your Class Under Test* is doing too much (i.e. God Object). So what is difficult to read? It is different for everyone. For me, when your unit test class stretches more than 4-5 page scrolls, it is starting to become too big. Think small, think manageable.

*By Class Under Test, I mean the class (e.g. class Laptop) that is to be tested by the test class (e.g. class LaptopTest < Test::Unit::TestCase).

Prefix test method names for categorization purposes
When you have to categorize your test methods in your test class by prefixing some text for grouping to improve readability, it is an indication your Class Under Test is doing too much:

class EBayChristmasShopperTest < Test::Unit::TestCase
  def test_browse_fetch_pricing_details_for_item...
  def test_browse_froogle_compare_item_prices...
  def test_browse_amazon_compare_item_prices...
  def test_auction_overbid_item...
  def test_auction_underbid_item...
  def test_payment_paypal_purchase_item...
  def test_payment_paypal_buy_it_now...
  def test_payment_credit_card_purchase_item...
  def test_payment_credit_card_one_click_checkout...

Instead of one EBayChristmasShopper object doing all of these, how about have Shopper to delegate some of these functionalities (browsing, auction, etc.) to other smaller objects (like ItemBrowser, Auctioneer)? This way you can write more well-intended unit tests for ItemBrowser and Auctioneer, and significantly simplify our EBayChristmasShopper class.

To me, one good way to categorize your unit test methods is to group them by the names of your public methods:

class IPodTest < Test::Unit::TestCase
  def test_prev_skips_to_previous_song...
  def test_prev_and_hold_rewinds_during_playback...
  def test_menu_displays_song_list_during_playback...
  def test_play_starts_playback...
  def test_play_pauses_playback_when_playing...
  def test_play_resumes_playback_when_paused...
  def test_next_skips_to_next_song...
  def test_next_and_hold_fast_forwards_during_playback...


class IPod
  def prev...
  def menu...
  def play...
  def next...
end

This way, each of your public methods can have a group of unit tests responsible for its behavior. In addition, it's easy to find out which method is missing tests (psssst....). But most importantly, you can easily see and logically group functionalities together, and refactor them into new classes should your Class Under Test becomes hideous to look at.

Mocks/Stubs frenzy
All too often, developers try to write code to make things work the way they want, at the expense of the complicating the fundamentals of object-oriented programming: object-to-object communication. Sure, the code works, but the tests tell you a different picture about how you get it to work. Take a look at the following example:

class DriverTest < Test::Unit::TestCase
  def test_drive_vehicle_forward_moves_appropriate_car_parts
    car = Car.new
    car.radio.expects(:off)
    car.clutch.expects(:shift_to_drive)
    car.acceleration_paddle.expects(:push)
    car.steering_wheel.expects(:turn).with(:straight)
    car.ignition.expects(:engine_on)
    car.timing_belt.expects(:run)
    car.muffler.expects(:stinks)
    car.battery.expects(:charging)


    driver = Driver.new(car)
    driver.driving_directions = 'FORWARD'
    driver.drive
  end
end

Did I lose you at the mock call for my car radio? Driving forward, in my mind, only needs half of those moving parts (ignition, clutch, steering wheel, acceleration). Others? I don't care (at least not when I only want to move the car forward as a driver). Sure the code works, but an overdosage of Mocks and Stubs is a good indicator that you are breaking encapsulation and/or coupling your class with too many other objects.

Long test methods
This indicates your Class Under Test requires a lot of set up in order to be useful, or your object is very coupled with the rest of the world (remember someone once said decoupling is good?), or your Class Under Test is very long and procedural. Remember the test_drive_vehicle_forward above?

Reduce the size of your test methods by re-thinking your object-object communication path. Maybe introducing an additional object to do all the dirty set up work would help decoupling the Class Under Test from the rest of the world immensely. Maybe this is a common object-coupling design problem that someone already has a design pattern to solve it succinctly, and all you need is google for it. There are always ways around it.

Specifying ruby 1.8.4 to install using DarwinPorts

2006-12-13T16:27:00.000-06:00

Recently I had to install ruby 1.8.4 onto my Mac. I have always been using DarwinPorts (v1.320 ) and it has served me well. But this time it is giving a little trouble. On the port tree (port list ruby) DarwinPorts gives me ruby 1.8.5, which as we all know breaks script/breakpointer for now. It does not give me the option to specify a previous ruby version to do port install, as if 1.8.5 is the only ruby that it recognizes.

I suppose you could compile ruby from source by hand, but we developers are just hard heads and refuse to give in...

Fortunately, there is a solution. (Thanks to Marc Selig)

1) Find out the svn revision number of the Portfile that has 1.8.4 by looking at:
http://trac.macosforge.org/projects/macports/log/trunk/dports/lang/ruby/Portfile
In my case it is 16709.

2) Set up a local port repository. In the file /opt/local/etc/ports/sources.conf, add the line:
file:///Users/Shared/dports

3) Install the port into your local repository.
cd /Users/Shared/dports
svn co --revision 16709 http://svn.macports.org/repository/macports/trunk/dports/lang/ruby/ lang/ruby/
portindex /Users/Shared/dports

4) Now you should be able to see ruby @1.8.4 in addition to @1.8.5 by running "port list". Run "port install ruby @1.8.4" and you are up and running.

Enhance your irb experience on Windows

2006-09-24T23:42:00.000-05:00

You can extend the functionality of your irb on Windows by creating a file .irbrc in your %userprofile% directory. Put some Ruby code in there, and they will get loaded when you launch your irb, as well as your script/console. I have put in some code snippets in there to aid my irb experience.

(You may have issues trying to create a file with its name beginning with a period on Windows Explorer. Try to run the command "notepad .irbrc" in your cmd.exe)

Tab Completion

require 'irb/completion'
ARGV.concat [ "--readline", "--prompt-mode", "simple" ]

Do you miss intellisense? If you do this, then in irb, type in like "text".s[tab][tab] will give you a list of methods in String that starts with s.

Ruby Interactive (ri) reference in irb

def ri(*names)
system(%{echo | ri #{names.collect { |name| name.to_s }.join(" ")}})
end

Then you can type ri 'String.strip' to see the Ruby index. The argument has to be a string though.

Remember, on a Windows machine, you have to put "echo | " in front of your ri line or else your irb will just return 'false'. I know it's counterintuitive that it isn't at the end, but it works.

Clear screen

def cls
system('cls')
end

I always wanted clear screen...

Console messed up with long irb lines
If you have a line of Ruby code that is way too long, and you want to go back and make changes, on Windows console it is not very friendly as it will almost guarentee to mess up your edits and disorient your typings. The only way I Googled on how to fix this is to take away the "--readline" switch from the Tab Completion tip above, and replace it with "--noreadline". But then you lose Tab Completion of course. I haven't found a workaround for it yet, but in the mean time I will happily just use Cygwin bash shell =)

Ruby constructor dependency injection... or not?

2006-09-13T21:25:00.000-05:00

Dependency injection has proven to be something a black belt unit tester must know about if you are serious about unit testing. If you have written some sort of unit tests, would you be jealous if I tell you for some of us, running all these unit tests takes sub-1 second? In C# and Java, actively practising Dependency Injection makes mocking and stubbing out dependencies much easier, and thus tests become easy to write, and run fast because they do not need to make time consuming calls. In fact, constructor injection is one of my favourite design techniques:

    public class Laptop
    {
        private IBattery battery;
 
        public Laptop(IBattery battery)
        {
            this.battery = battery;
        }
 
        public void PowerOn()
        {
            if (battery.Meter >= 10)
            {
                // Booting Vista...
            }
        }
    }

Then to unit test Laptop, you could use NMock like such:

        [Test]
        public void PowerOnFailsWhenBatteryIsTooLow()
        {
            mocks = new Mockery();
            mockBattery = mocks.NewMock<IBattery>();
            Expect.Once.On(mockBattery).Method("Meter").Will(Return.Value(9));
            Laptop laptop = new Laptop(mockBattery);
 
            laptop.PowerOn();
 
            Assert.AreEqual("Off", laptop.PowerStatus);
            mocks.VerifyAllExpectationsHaveBeenMet();
        }

It may not be worth it to mock out Battery, but think about a lengthy web service class.

That's all true in C# and Java. Now in Ruby, I don't even need to constructor inject my Battery instance to unit test my Laptop class. I can just as easily unit test my Laptop class without having to inject my mock Battery:

class Laptop
  def initialize
    @battery = Battery.new
  end
  def power_on
    if @battery.meter >= 10
      # Booting Max OS X
    end
  end
  
  def power_status
  end
end

class LaptopTest < Test::Unit::TestCase
  def test_power_on_fails_when_battery_too_low
    Battery.any_instance.expects(:meter).returns(9)
    laptop = Laptop.new
    laptop.power_on
    assert_equal :off, laptop.power_status
  end
end

Basically I am mocking using Ruby Stubba/Mocha, but I don't even need to write an extra constructor to inject the Class Under Test's dependencies! With no interface IBattery, nothing! This is some cool trickery of programming in a dynamic language like Ruby, and I am discovering these things every day with my colleagues!

I know you are going to say "well, I can use reflection to do the same thing...", and I will tell you, sure, you try to do it in a readable manner and with one line of code. I didn't say you can't do it with C# or Java, I am just saying, this is how I can do it with one line of highly readable Ruby. Happy programming.

The most revolutionary usernames and passwords, ever

2006-09-05T22:10:00.000-05:00

Haven't you written the authentication piece for your app? Isn't it annoying after it is implemented in your app, every time you try to access your app's functionality during development, you have to start logging in every time by typing in some dev-only user id with a fake password? Having a good set of tests helps alot in terms of not having to go through the app starting from login while new features are still undergoing development, but there are still many cases where you have to debug something starting from the first login page of your application.

Over time, I have developed one way to mitigate this nuisance, by creating a few dev usernames and passwords of the following pattern:

Users:
+ dave
+ fred
+ cat
+ red

Passwords:
+ create
+ database
+ case
+ ware
+ vase
+ card
+ garbage
+ beta

Now the question is, what is the pattern? Answer me in 10 seconds...

That's right, I can finish typing all of them with my left hand, while my right hand still holding my mouse and not have to switch from my mouse to my keyboard to type the password, then my mouse to hit the 'Login' button.

Did you get the pattern in 10 seconds?

Size does matter

2006-08-24T20:12:00.000-05:00

When it comes to managing an Agile project team, one question usually arises for planners and managers is how big should a story really be. For the most people a story should be sized at the minimum as something that can be completed within an iteration by someone (or a programming pair). Each story is tagged with a difficulty level, whether you measure using 1-2-3 ideal development days or gummy bears, for best-guess estimates on how long it will take to implement the solution for the story.

I think while it looks trivial, story sizes is very important to an Agile team to manage business wishes of the application and development delivery of business values.

On some projects, a story could be one small thing on one screen (eg. enter customer address). Others, it spans multiple screens (eg. enter, edit, and delete customer info). For me, I would much prefer they being as small as they can be. For something like entering, editing, and customer info, strive to break things out to something meaningfully small, like:
- enter customer name
- enter customer address
- enter customer billing info
- edit customer name
- edit customer address
- edit customer billing info
- delete a customer

Small story has a few advantages:

More tailored to business needs
Suppose you have a specific need for a desktop computer, like you are a professional graphics designer or gamer. Would you be more satisfied for your purchase if you spend your budget at an online store for one of their 20 brand computers, or custom build it with hardware parts you get to play with one by one to address your specific needs? Is 1GB RAM gonna be enough? How about 2GB? Or 1GB plus more video card memory is good? If you purchase your computer parts by parts, chances are you will be buying something much more tailored to what you need now, at the same time pay for something that can accomodate tomorrow's change better (eg. a motherboard with more RAM slots than normal). The smaller your application's features are, the more flexibility you have. But don't go overboard and purchase your parts at microscopic level (buying transistors) which will cause you nightmare.

You can need it later, or YAGNI at all
You are building your dream vehicle from scratch. You have a tight budget, and a tight deadline. Do these constraints sound familiar? We are all working in this competitive business environment every day. So if you are building this car you need, can it satisfy your need with only 2 wheels like a motorbike? Does it still address your needs if it has three wheels? Do you need this car to have a 5th backup tire? Of course all these depends on what you use your car for. You might only need 2 wheels and as light weight as possible for a marathon or endurance race, or you might absolutely need a 5th wheel if you use it in the Africa dessert where there is no concrete. But the tight budget and deadline does not change, what changes is the delivery and whether the users' needs are satisfied. So, between one story that says "build 5 wheels for the vehicle" and five stories that says "add 1 additional wheel to the vehicle", what would you pick? Prioritization is key.

More consistent and better measured velocity
As previously mentioned, stories should be sized as something that can be completed in one iteration. Based on last iteration's performance, if you are planning for 3 stories to play the following iteration, constituting a 1-point, 2-point, and 3-point story. If the team were 99% complete with it but was unable to complete that 3-point story, your team's velocity dropped from a 6 down to a 3. This is a pretty big change in velocity. Now imagine you redo your iteration with 8 stories, same scope, a total of 20 points based on last iteration velocity. At the end of the iteration if your team cannot deliver one 2-point story and one 3-point story (note: much smaller scale), you team's velocity is 15. Your tracking of your development team's progress can be more accurately measured simply by splitting your stories up.
But with smaller stories it does come at a cost. There are some disadvantages.

Code is better, analysis is easier
Smaller stories encourage team members to complete something faster, meaning they get a chance to go back and refactor code easier and more often. It also allows programming pair to switch pairing partner more often to get more understanding of the code base. Smaller stories can also be defined in much more granularly fashion. Things that business do not understand yet can be split out to another story in order to get things going. Better tested and better understanding the code and stories, obviously makes your application more robust and easier to change.

With smaller stories, one must also understand there are drawbacks too. Here are a few I can think of:

Story explosion
Most people use an Excel spreadsheet to manage their day-to-day story progress. Having a spreadsheet of 60 stories is alright, now consider representing the same story list with 180 stories. If you are organizationally challenged, you will quickly run into issues with your tracking. How to categorize such information? Well, in today's blogsphere world there is something called tagging. You can tag a story with multiple tags, and later click on a tag topic (eg. UI) and a list of things you previously tagged will show up. This allows someone to quickly sort and filter the obviously larger story list. I don't know if there is a product out there that supports such information sorting and filtering yet, but I would start looking there if I were to solve this issue. Del.icio.us is a good place to explore tagging.

Story selections
When there are more stories to choose from, playing the right stories becomes more tricky. It is common to have groups of stories that are tied to an area that is under heavy business analysis and are subject to change completely, and groups of stories that are fairly stable and not prone to change. But only picking stories that are stable and easy to complete is not the correct way to play stories. Whichever stories to play should depend on the complexity of the technical solution
and business value that can be recouped from that solution. The application is meaningless to the business if out of the 180 stories, 90 are complete, 60 are still under intense business analysis and are subject to change, and 30 are ready to go, while the business value generated by those 90 completed stories are trivial. Get down to the dirty aspect of the problem the application is trying to solve, and start flushing them out. Without a defined problem to solve, any application is way to expensive to build.

So should you go with diet size story or super-sized story? This question is like asking how diverse your investment portfolio should be. It depends. Now that you know why size does matter, apply your thinker's head to see whether it applies to the project you are on!

The "Not-Enough-Objects" anti-pattern

2006-08-18T00:19:00.000-05:00

Definition:
This is a pattern that every single programmer in the world had used at one point or another in their programming careers. Classes become unnecessarily bloated and multi-talented in their ability to single-handedly accomplish almost anything and everything an application requires.

Root Causes:
Being lazy to create new classes for new but not aligned behaviors into existing objects; being ignorant on good objects commuication and interactions; lack of concern of each objects' goal of existence; failure to realize the importance of programming to interfaces; putting off code refactoring forever (the "kitchen sink" syndrome ¹); embraces code babies² as if they are the next American Idols; failure to effectively organize, categorize, and classify application functionalities.

UML:
eg.

Customer

+ OrderSoda() ~20 LOC
+ Sleep() ~100 LOC
+ RPCCallTolMom() ~250 LOC
+ PlayTVGames() ~500 LOC
+ WebServiceCallToWalkMyDog() ~3,000 LOC
+ RepeatRoutineEveryDay() ~275 LOC
.
.
.
(many more methods)

Symtoms:
When it appears in your code, they make your code become demoralizingly hard to read or comprehend; extremely difficult to touch code without introducing new errors; code become very brittle; code starts to look like POOP (Procedural Object-Oriented Programming); exponentially increase your programmers' after-hour caffeine consumption to life-threatening level against others (e.g. "I will kill the xxx if I knew who wrote this").

How to Avoid:
Avoid 1,000 line classes and/or 200 line methods; avoid class behaviors that are mutually exclusive; avoid class states that are mutually exclusive; avoid classes to have multiple responsibilities in favor of delegating to new objects that the classes use; avoid too many if-else statements in favor of object hierarchy (see Strategy); effectively categorize or organize your classes/namespace/file/folder structures into meaningful and humanitarian fashion; encourage collective code-ownership; frequently communicate with team members the intent of each object; feverishly unit test the heck out of each class and object using mocks and stubs; use TDD as test-driven design to think how other objects will use the object you are unit testing; pair-program with someone who reminds you of code quality is of utmost importance.

Short List of Skills Required to Avoid:
1. Programming to abstraction and not implementation
2. Object encapsulation
3. "Tell, Don't Ask" and the Law (or guideline) of Demeter (link), go back to #2 if you do not understand
4. Short methods/classes with well-intended names in favor of plethora of comments
5. Unit testing
6. All your unit tests run in sub-10 secs. because you understand Mock vs. Stub (link)
7. Dependency Injection (link)
8. Decoupling and cohesiveness (related to object-to-object communications)
9. Knowledge of design patterns of all types
10. Knowledge of anti-patterns

After-Thought:
Have we understood why good programmers are rare?

(1) kitchen sink syndrome: If it is clean, it stays clean. Once dirty dishes goes in and no one bothers cleaning up, dishes will start piling up and they never get cleaned.
(2) code babies: A snippet or chunk of code, or application functionality that is given birth by its genetically related programmer that no one else in the planet understands or dares to understand.

Time is money... Stop using your Quick Launch or Start menu

2006-08-13T10:45:00.000-05:00

Do you have a list of 30+ icons of applications you use daily in you Quick Launch taskbar and you turned on its auto-hide feature? Do you try to actively manage your Start/Programs folders to make it easier for you to find and launch applications? Are you running out of keyboard shortcuts to launch your favourite applications? If you say yes to all these questions, chances are you are a power keyboard-driven user of your computer. It's simply a breeze without having to move your hands away from the keyboard and do everything on your computer.

To add more ammunition to you keyboard freaks, download and try Colibri. This is an application that lets you launch your apps by simply typing in a pattern that matches your apps' names! I mapped it to the F12 key on my laptop. When I want to launch Firefox, I press F12, then type in "fox", and hit enter and it is launched; When I want to launch MSN Messenger, I press F12, type in "mess", hit enter and it is launched...

It's pretty much similar to Google Desktop, but in GD after typing I have to use the up and down arrows multiple times to nav away from other doc types to get to app's exe. Colibri is strictly for app launching. If I type in "mess" for MSN Messenger, it will remember that you have used it to launch MSN Messenger, so next time rather than suggesting you to launch Y! Messenger it will default to MSN Messenger, while letting you to arrow down to select Y! Messenger should you want to.

I rarely use my Quick Launch or my Start menu anymore...

Bonus Tip 1: Did you know shift-F10 opens up your mouse right-click menu?
Bonus Tip 2: Turn on your "Begin finding when you begin typing" feature in your Firefox. I use this to "type-and-hit-enter" to surf the web =P

Applying PicoContainer.NET with Presentation Patterns - Part II

2006-07-19T01:05:00.000-05:00

(Part I)

Let's think about what Pico is good at in practical sense. I think Pico is great at wiring dependencies together. That means, it is good at wiring dependencies starting from your Presentation Model/Presenter layer and back, because these are all classes that you have complete control over. For your Views, at least if you develop in Visual Studio .NET, the restriction on their default constructors conflicts with the way Pico works. There are ways around it (setter injection as said in Part I), but they are ugly.

But what if I don't register my Views into my Pico container like other classes? Then the question becomes: how can my View get a reference to its dependent Presentation Model/Presenter?

Before I show you my solution, I have to say one more thing. The method InitializeComponent() synonymously refer controls as components for a reason. Every control in .NET is a component. What is a component? It is something you put in a container. Every container has many components. So yes, I am telling you that these controls/widgets are all being put in a container (not Pico) in the .NET control hierarchy framework. So what does each control being a component in this .NET container hierarchy enable me to do? It allows you to have access to the functionalities from other components in the same container. Let me rephrase: each View, being a component, can have access to other classes like a Presentation Model/Presenter, so long as they are all in the same .NET container hierarchy framework. Didn't I just say that this container framework is not Pico? Yes, I did, but it doesn't mean we have to use only one over the other. We can use them both together to each of their strengths to achieve the best of both worlds. Let me explain.

We have already said using Pico container to host everything starting from the Presentation Model layer is a good thing. So here is some example code on how to do this:

        private static IMutablePicoContainer createMutablePicoContainer()
        {
            IMutablePicoContainer pico = new DefaultPicoContainer();
            pico.RegisterComponentImplementation(typeof(IPageOnePM), typeof(PageOnePM));
            pico.RegisterComponentImplementation(typeof(IPageTwoPM), typeof(PageTwoPM));
            pico.RegisterComponentImplementation(typeof(IWebService), typeof(WebService));
            return pico;
        }

In order to allow the Views to have access to other components (Presentation Model/Presenter), we have to create a .NET container. It is just a class that subclass from System.ComponentModel.Container. I am going to call this an ApplicationContainer to avoid being confused with our Pico container.

        internal class ApplicationContainer : System.ComponentModel.Container
        {
            // ...
        }

To put our Views into this ApplicationContainer, you instantiate an instance of it, and put the Form object that you will start your application with into it like this:

        [STAThread]
        static void Main() 
        {
            ApplicationContainer applicationContainer = new ApplicationContainer();
 
            MainForm form = new MainForm();
            applicationContainer.Add(form);
 
            Application.Run(form);
        }

From our Views, the method that they will use to get access to other components is called GetService(Type serviceType). When this method is called from within our Views, if our Views are put in a System.ComponentModel.Container, by default the method will ask for its containing container to traverse all registered components and see who can provide this "service" to it. If it finds such component, the container will return it, and now the requesting component gets a reference to that object. How does the container traverse its registered components and decide to give the component requesting a service the service that it asks for? Well, interestingly a System.ComponentModel.Container also has its own GetService() method to do just that. Now, since we have our own subclass ApplicationContainer, what if we override its GetService(), and while our ApplicationContainer object receives a request for service from any of its components, we can instruct it also look to see if Pico has the stuff that the requesting component has. More concretely, when a View uses GetService(typeof(MyPresentationModel) to get its Presentation Model/Presenter dependency, ApplicationContainer will ask for a Pico container that has already been fully registered to return an instance of that class if it is found, like such:

        internal class ApplicationContainer : System.ComponentModel.Container
        {
             private IMutablePicoContainer _pico = createMutablePicoContainer();
 
            protected override object GetService(Type service)
            {
                object instance = _pico.GetComponentInstanceOfType(service);
 
                if (instance != null)
                {
                    return instance;
                }
 
                return base.GetService (service);
            }
        }

To sum it all up, you need to do the following to get things to work:
1. Create an ApplicationContainer class subclassing System.ComponentModel.Container.
2. Add your starting Form into the ApplicationContainer instance, prior to starting it.
3. Set up a Pico container within the ApplicationContainer instance.
4. Register all dependencies that your Presentation Model/Presenter classes will need in your Pico container. You do not need to register your Views.
5. Override the ApplicationContainer's GetService() method, make it to look further into its already setup Pico container for anything that it should return.

Now from your Views, you can gain access to its dependency, in this case a Presentation Model/Presenter, by calling:

        private void PageOneView_Load(object sender, System.EventArgs e)
        {
            // Add this View to ApplicationContainer. Otherwise we have to instantiate
            // each and every view in public static void Main() and do the adding there.
            base.FindForm().Container.Add(this);
 
            // This GetService() call will now find what we need in ApplicationContainer.
            IPageOnePM service = (IPageOnePM)base.GetService(typeof(IPageOnePM));
        }

And modify your PresentationModel's constructor to include an additional parameter to take in the web service class. Then your class can start using the web service functionality, while in unit testing you can mock/stub it out!

    public interface IWebService
    {
    }
 
    public class WebService : IWebService
    {
    }
 
    public class PageOnePM : PresentationModel, IPageOnePM
    {
        private IWebService webservice;
 
        public PageOnePM(IWebService webservice)
        {
            this.webservice = webservice;
        }
    }
 
    // ApplicationContainer class
    private static IMutablePicoContainer createMutablePicoContainer()
    {
        IMutablePicoContainer pico = new DefaultPicoContainer();
        // ...
        pico.RegisterComponentImplementation(typeof(IWebService), typeof(WebService));
        // ...
        return pico;
    }

So now, we have eliminated the child user control not knowing how to get a Presentation Model/Presenter problem, because a user control is also, well, a component in the same ApplicationContainer. We have also solved the ugly setter injecting child user controls' dependencies problem. You also did not modify a single View default constructor! We can now happily use our Visual Studio .NET IDE to do WYSIWYG design and manage good OO design, plus Inversion of Control-ing our dependencies.

After-thought: Though the title of these two posts say Presentation Patterns, in my mind they are more geared towards just for Presentation Model, due to in my opinion the difference in directional references between Presentation Model and Model-View-Presenter. I mentioned briefly about this in my earlier post here.

By the way, note that now each page can have access to multiple Presentation Model objects, instead of the what-it-used-to-be 1-to-1 relationship between a page and its Presentation Model, so one can do something similar to Ruby on Rails where each View can make calls to multiple Controllers which operates on various Models to complete the desired action! This makes code-sharing between Presentation Models much easier, and also each Presentation Model can be named not after their page but by application functionality!

Applying PicoContainer.NET with Presentation Patterns - Part I

2006-07-19T01:02:00.000-05:00

There is a couple favourite presentation-layer design patterns that I have been consistently using to build .NET applications. In particular, they are Presentation Model and Model-View-Presenter. Both are extremely handy when it comes to making the code-behind of your View more testable by delegating its responsibilities to another layer of code. From that layer, you can start chipping in various flavors of dependency injections and stubs and mocks to start going all-out unit testing assault to your code.

In many cases, on a per View basis (like a page or user control), there is a Presentation Model class, or a Presenter class, sitting behind it ready to receive a call from its corresponding View, then execute the behavior called upon.

PicoContainer.NET (Pico from here on) encourages more decoupled object-oriented class design by helping you to manage your classes' dependencies. As a quick example, suppose you have a class iPod (you know what it is, right?) and a class Battery. Obviously iPod depends on a battery. However, the iPod class at construction time creates its own Battery instance to use and cannot take any other battery types (to the dismay of iPod users). Now your iPod stopped working and you have to figure out why. If you need to test your iPod, you have to be extremely creative to figure out where the problem is related to its battery or not somehow. How do you know if the problem lies in the iPod, or its battery? This is why good code minimize dependencies. Suppose, however, your iPod receives a Battery instance through its constructor. Then in the unit tests of your iPod class, you can then mock/stub the battery out and start exclusively interrogating and verifying the behaviors of your iPod class. iPod now can be injected with a mock Battery instance, and hence your iPod class is more testable, and its tests are more "unit" and not touching other parts of your code. You can then start writing more automated unit tests that run in milliseconds easily.

The question now is who then is responsible for passing a Battery instance into iPod. That's where Pico shines. If, as mentioned, your iPod has such a constructor that takes in the Battery dependency, using Pico as a container, you register your iPod and Battery class into the container, when you ask the container for an iPod instance, the Battery instance (since it is also registered) will be instantiated automagically and you can now start using your iPod object, without having to hard-code your iPod class to instantiate its own Battery.

Because Pico is such a great tool, I have used it in various .NET projects of mine. Since Pico uses a registration scheme, whatever you put into a Pico container you get its dependency-wiring functionality for free, many things tend to be put into it. Afterall, Pico encourages good design, right? As a result, Views, Presentation Models, Presenters, etc., are all registered into Pico container. The Pico container is set up at public static void Main() time, and when you run Application.Start(form) you pass it into your form object. From there all of your Views can programmatically ask for its dependencies. Life is good.

Why are Views being put into the container? Because every View needs a delegating Presentation Model or Presenter to handle its behavior, meaning, every View "depends" on a Presentation Model or Presenter. As a result, they all get registered into Pico in order for the form to get navigate the user to the correct View.

Life is good - until stuff happens. Since constructor injection is the preferred way to inject dependencies, if your View "depends" on something, then your View would need a constructor passing in its dependencies before Pico can start wiring dependencies on your behalf. Every .NET programmer knows that, every View (regardless you use a Form or User Control) has a default parameter-less constructor that has a default line InitializeComponent(). This is because the Visual Studio .NET IDE uses this constructor to support WYSIWYG at design time. Tampering with or getting rid of this constructor and/or the InitializeComponent() method call will get your IDE trouble in supporting editing your View at design time.

A second problem that would arise is that, consider I have a User Control. It is also a View, albeit being used and instantiated by its parent View (maybe another User Control). Since this child user control instance is created by the auto-generated code from the parent Views InitializeComponent() method call, you cannot modify this child user control's default constructor to take in its "dependency" Presentation Model/Presenter, and then hope that its parent Form can somehow inject the child control's dependencies inside its InitializeComponent() method. Even if you can modify these auto-generated code, your IDE design-time support is now in jeapardy. Because of this problem, some Pico programmers would buck the constructor injection coding consistency, pass in the child user control's dependencies into the parent View's constructor, and start using setter injection to inject the child user control's dependencies, after the parent Form's InitializeComponents() method call. That way at least you can leave the child user control's default constructor alone and continue to use VS's strong IDE support, while managing your child control's dependencies.

What to do? Now comes Part II - My attempt to get the best sides of all worlds: Use Pico to manage dependencies; and have VS.NET IDE WYSIWYG support.

Ruby sugar you can't find in C#

2006-07-12T16:04:00.000-05:00

Coming as a C# programmer to learn Ruby, as I journey from a compiling language to a dynamically typed language, of course there are some concepts in Ruby that are easier (or harder) for me to grasp than others. I want to write out loud some of these interesting bits and pieces for others who are also interested in trying out Ruby.

String vs. Symbol
In Ruby a string can be single-quoted or double-quoted (eg. "foo"). A symbol is a variable name prefixed with a colon (eg. :foo). Symbols somtimes are used pretty much synonymously with strings. The reason they exist is that by using symbols, in memory they only exist as a single copy per symbol no matter how many times you refer to them in your code. String however will create a new copy of the string in memory every time you refer to them, even though they are exactly the same. This in C# is equivalent to string interning, and is the difference between using between a StringBuilder and regular string concatenation. So don't be baffled when you see them next time.

Methods vs. Messages
In Ruby, everything is an object, including a number and a string. Ruby programmers like to think that when you are calling a method on an object, you are sending a message to that object in hope of it doing something for you. One of the cool features in Ruby is that one can call methods that are not defined at programming time but define and call them at runtime. For example:

class Foo
  def programming_time_method
    puts "programming time method"
  end

  def define_more_methods
    self.class.class_eval do
      define_method(:new_born_method) { puts "i was born!" }
    end
  end
end

Now if you do:

f = Foo.new
f.programming_time_method
puts f.respond_to?(:new_born_method) #Does f have method by that name?
f.define_more_methods
puts f.respond_to?(:new_born_method) #Does f have method by that name, now?
f.new_born_method

then you get results:


>ruby test.rb
programming time method
false
true
i was born!

One can do much more fancier stuff with Ruby too. Read on.

Class Methods vs. Static Methods
In C# most developers I worked with frowned upon static methods. They are flat-out evil. You cannot effectively mock method calls of them, and that they encourage non-OO style procedural programming. Pretty much all of the arguments that they are good for something are trumped by the hard-to-test reason. Introducing class methods in Ruby:

class Customer
  def self.class_method
    puts "class method getting called"
  end
end

Customer.class_method
>ruby test.rb
class method getting called

At first glance they act just like static methods, you do not need an instance but just the class type and you can start making calls on them. However, they are no longer un-mockable. The unit testing framework that comes with Ruby also contains a mock framework. It can mock class methods. All of a sudden class methods, or should I say methods that associate to a class, deserves another look. In fact, Ruby on Rails uses class methods extensively for some of its magical operations. For example, if my Customer class inherits from Model, and has an id and a name in the database, I automatically get these magic methods for free - yes, free!

Customer.find_all                # returns the list of all Customer instances from database
Customer.find_by_id(2)           # returns the Customer instance that contains id=2 in database
Customer.find_by_name('STEPHEN') # returns the list of all Customer instances whose name is STEPHEN

When you add a column to your table in the database, you get a free find method without coding. That's Rails.

Method-missing
Any message (aka method) you send (aka call) to an object, if that object does not yet have that method defined, it will go into its method_missing() method, which you can override. From that point, one can be fairly creative given the dynamic nature of what Ruby allows a programmer to do. In fact, method_missing() plays a pretty important role in constructing a DSL (Domain Specific Language).

class Customer
  def method_missing(method_symbol, *args)
    puts method_symbol.to_s
    return self
  end
end

customer = Customer.new
customer.pays.me.five.dollars

>ruby test.rb
pays
me
five
dollars

More Ruby sugar will follow. Stay tuned.

How to use GMail SMTP server to send emails in Rails ActionMailer

2006-06-22T21:30:00.000-05:00

Recently I had to use Ruby to send emails. I don't want to go through the hassle of setting up a mail server on my build box, and since I use GMail's hosted webmail service I am thinking I might be able to use their smtp mail server to do it.

Rail's ActionMailer was simply the automatic choice, since I am building a rails app.

Turns out GMail supports only SSL SMTP mailing service, meaning if you cannot create a SSL connection to its SMTP server, you cannot send email through them. My Rails and Ruby (1.84) version do not yet support creating a SSL SMTP connection through Net::SMTP. DHH writes about how to do so through installing msmtp here, but we developers just obviously love options =)

The dynamic nature of Ruby allows me to enhance the functionality of that class. I found the following code from a couple Japanese posts here and here on the web (fairly low Google ranking). Pasting them in my /vendor/plugins as follow:


$ cat vendor/plugins/action_mailer_tls/init.rb
require_dependency 'smtp_tls'


$ cat vendor/plugins/action_mailer_tls/lib/smtp_tls.rb
require "openssl"
require "net/smtp"

Net::SMTP.class_eval do
  private
  def do_start(helodomain, user, secret, authtype)
    raise IOError, 'SMTP session already started' if @started
    check_auth_args user, secret, authtype if user or secret

    sock = timeout(@open_timeout) { TCPSocket.open(@address, @port) }
    @socket = Net::InternetMessageIO.new(sock)
    @socket.read_timeout = 60 #@read_timeout
    @socket.debug_output = STDERR #@debug_output

    check_response(critical { recv_response() })
    do_helo(helodomain)

    raise 'openssl library not installed' unless defined?(OpenSSL)
    starttls
    ssl = OpenSSL::SSL::SSLSocket.new(sock)
    ssl.sync_close = true
    ssl.connect
    @socket = Net::InternetMessageIO.new(ssl)
    @socket.read_timeout = 60 #@read_timeout
    @socket.debug_output = STDERR #@debug_output
    do_helo(helodomain)

    authenticate user, secret, authtype if user
    @started = true
  ensure
    unless @started
      # authentication failed, cancel connection.
        @socket.close if not @started and @socket and not @socket.closed?
      @socket = nil
    end
  end

  def do_helo(helodomain)
     begin
      if @esmtp
        ehlo helodomain
      else
        helo helodomain
      end
    rescue Net::ProtocolError
      if @esmtp
        @esmtp = false
        @error_occured = false
        retry
      end
      raise
    end
  end

  def starttls
    getok('STARTTLS')
  end

  def quit
    begin
      getok('QUIT')
    rescue EOFError
    end
  end
end

So now, in your ActionMailer::Base's server settings if you have:


ActionMailer::Base.server_settings = {
  :address => "smtp.gmail.com",
  :port => 587,
  :domain => "mycompany.com",
  :authentication => :plain,
  :user_name => "username",
  :password => "password"
}

You call your ActionMailer::Base's deliver method (perhaps from a custom subclass), it will send an email through GMail. Mission accomplished.

Some unspoken tips on managing an Agile project

2006-05-25T11:41:00.000-05:00

Want to have an easier time to manage an Agile project and avoid common pitfalls while have some fun? Read on...

Food
Believe it or not, food is not a luxury item on an Agile project. Food is *essential*. Agile practices encourages as much open-air communication as quickly as possible, be it computer-to-human (through tests) or human-to-human (short iterative cycle). Yet sometimes it is hard to force down onto people to communicate more effectively from a Project Manager. By having good food/snacks, people who are not co-located are going to come to the room much more often than usual, and a casual "how are you guys doing" while shuffling popcorn into his/her mouth is usually a conversation starter on many things the development team needs. And food, of course, improves morale inmeasurably =)

Team co-location
This is key to being able to communicate more effectively, which is one of the tenets of a successful Agile project. By more effectively, I mean the project can execute in less time because people can make more informed decision based on an abundant amount of open-air information. For whatever reasons, people are generally good at filtering useless information but not good at carrying information to the appropriate people who need them. By bringing the team together in a relatively open area with maximum face-to-face communication, you are encouraging team members to make more informed decisions before they waste time to work on the wrong thing based on false assumptions on continually ongoing changes within the development team and the business.

CI build sound
If you use CruiseControl.NET, try putting a sound clip that reminds of some embarrassing moments of some development team members as the build success/failure cctray settings. This can get quite amusing some times. Another morale booster.

Quote sheet
During the course of development, there is bound to be someone who inadvertently said something hilarious about someone or something. Keep a list of them on a wiki on the build server where the team members have access to. This is a must as part of the team-building process. At ThoughtWorks's annual national gathering day, we collect great and funny quotes from ThoughtWorkers from projects everywhere, print them on T-shirts, then auction them out. The money collected goes to donation. Funny and meaningful.

My funny quote of my current project? Nah. You have to ask me personally for that =P

Second story wall for issues after 1.0 release
In a XP story wall, which represents the progress of stories planned for the current iteration, usually contains columns ("DEV Ready", "DEV In Progress", "QA Ready", "QA Complete") that mark the stages of a story from ready for development to QA complete. Now how does the wall change when your application goes 1.0 release?

Generally, after a 1.0 release, the team will create a new branch of the source code for urgent production defect fixing purpose. The problem is that when you have a large team of developers split across urgent production defect fixes and release 1.1 development, or there are many small and frequent 1.x releases coming after your initial release, or for some uncommon reason you have more than one branches of source code (after you have determined it is ultimately unavoidable). Then managing the merging of code from your release branch to your active development branch becomes tricky, because if developers forget to merge, your active development branch now has bugs.

How about at 1.0 release, create a new story/issue wall next to the original story wall, but it only contains columns "Issue Verified", "Issue In Progress", "Issue Resolved", and have the developers merge their source code before they can move any issue cards from "Issue Resolved" column of that wall to the other wall's "QA Ready"? This physical movement of an issue card from one wall to another always remind developers to merge their changes. If that doesn't help the busy and forgetful developers, have them move issue cards to a new column called "Issue Merged".

Stand-up token
A good stand-up meeting is not an avenue to identify "solutions" of problems participants face. It should be a quick meeting to identify individuals who need to hold face-to-face conversations after the stand-up and then move on. But people being people, sometimes conversations go out of hand during a stand-up. Have a stand-up token, like a football, in stand-up so that only the person who has the token can talk. When you see people exchange the token too many times in a conversation, perhaps it's time to ask them to offline it till after the stand-up.

Stacking "using" blocks together

2006-04-27T21:02:00.000-05:00

I had always been thinking of using the using-block to instantiate multiple disposable objects so I don't have to remember explicitly calling Close() or Dispose() on them:

            using (IDisposable one = new DisposableOne(),

                   IDisposable two = new DisposableTwo())

                // Compile time error

But unfortunately C# does not support that. So as a work around, one would do these nested using statements to do the trick:

            using (IDisposable one = new DisposableOne())

                using (IDisposable two = new DisposableTwo())

                    // Ops. Code is now indented

But now, you have messed up your pretty-looking indentation of your code. Remember back in the acadamia your programming class instructor told you that all your if-else/for/while statements will work without curly brackets? Well, surprise, same case for using blocks:

            using (IDisposable one = new DisposableOne())

            using (IDisposable two = new DisposableTwo())

                // This works!

Time to show off to your pairing friend you can write prettier code than him/her =D

Multi-threading applications tips

2006-04-21T11:54:00.000-05:00

Recently I have been involved with a multi-threading application, and throughout development I have been acquiring tips and gotchas here and there. One of the fun things about such applications is that you get to play with some sort of almighty powerful server box. In my case, a cool 8-processor dual core box with 8 GB RAM... How much is it? $70,000. =)

Test your application in a comparable multi-processor testing server. Better yet, test it on your production box
Yes, you heard me. Usually multi-threading applications are performance critical. Otherwise how else can you justify a pricy machine and more complicated code? If you are not able to perform testing on the actual production box, chances are the users will ultimately find the problems. Which means developers, QA, and project managers all have to work OT hours to fix the problem and push those fixes to production, plus the number of phone calls and help desk tickets you have to make. When you multiply these hours and stress, the justification of having another pricy testing server before going into production all of a sudden is not too pricy anymore.

Server GC vs. Workstation GC
.NET garbage collection behaves differently when it is installed on a single processor machine versus on a multi-processor machine. If you install it on a single processor box, it is in Workstation GC mode, meaning there is at most one thread doing garbage collection. When you install .NET onto a multi-processor box, it is installed as Server GC mode, meaning there is one GC thread per CPU. I strongly advise that the environment you deploy to before your production environment to be a multi-processor machine and have Server GC mode turned on. That way you are testing more closely to the real environment. In performance-intensive applications, garbage collection, albeit automatic, is usually worth monitoring as well.

Raise an event thread-safe
In C#, many of us have raised an event this way. In fact, this is the way most books/lectures/tutorials demonstrate it.

    1 public class Battery

    2 {

    3         public const int LOW_BATTERY_LEVEL = 20;

    5         public event EventHandler LowBattery;

    6         public event EventHandler Depleted;

    8         private int remainingBattery = 100;

   10         private void OnLowBattery()

   11         {

   12                 if (LowBattery != null)

   13                 {

   14                         LowBattery(this, EventArgs.Empty);

   15                 }

   16         }

   18         private void OnDepleted()

   19         {

   20                 if (Depleted != null)

   21                 {

   22                         Depleted(this, EventArgs.Empty);

   23                 }

   24         }

   25 }

Did you know that this is not thread-safe? Thread A could be executing this code, does a null check at line 12, when it is just about to raise the event, Thread B grabs the subscriber and unregisters/unwires the event. When Thread A resumes execution, it tries to raise the event when no one is subscribing to it. Null reference exception.

In order to make the code thread-safe, we can take advantage of delegates are value-types, in other words, you can only create"copies" of them, but not by reference.

   10 private void OnLowBattery()

   11 {

   12         EventHandler handler = LowBattery;

   13         if (handler != null)

   14         {

   15                 handler(this, EventArgs.Empty);

   16         }

   17 }

Use of lock(this) and lock(typeof(Foo))
Every C# programmer is aware of the lock keyword. It prevents multiple threads from referencing the resource while the resource is being modified. When you want to modify a member variable thread-safe, you can go and lock the object that contains it:

public Foo()

            lock(this)

                    _memberVariable = "SOMETHING";

But what about static variables? Well, interestingly the lock statement allows you to lock a type object as well:

public class Foo

            public static string StaticVariable = null;

            public void SetUpStatic()

                    lock (typeof(Foo))

                            StaticVariable = "SOMETHING";

Give you threads names
Yes. Threads in the old days have only a thread id, which gets generated every time when a new thread is genereated, and that makes debugging very painful. Now in .NET you can programmatically give each thread your application creates a name with its t.Name property. Use them, then in your VS.NET Debug/Windows/Threads (Ctrl-Alt-H while debugging) you will see each thread with their name. Now they look much more friendly. Better yet, name your threads after each project manager you have worked with =)

Use .NET synchronization classes
Don't tell people you know how to write multi-threading applications knowing only how to use lock(). If you don't know how Monitor.Wait() and Monitor.Pulse() and the shopping bag of the synchronization classes that .NET provides, you are missing out alot. As a starter, try this excellent article here. It's a must read for anyone programs C# and multi-threading.

Checking duplicate Sql statements in Excel

2006-04-19T19:14:00.000-05:00

In my current project we are doing database Continuous Integration (CI). This means that every developer will have their own database instance for development, and at every check-in the build server kicks off the CI cycle, which starts getting the latest source code, compile, rebuild the database from scratch through running SQL scripts, re-insert data (reference data) these tables should have, compile, test, etc. When done right, this allows the much-required freedom for developers to develop against the database without interfering other developers' databases, and significantly reduces the time it takes to ask the DBA's to modify the database table for you.

When you acquire from the customer/business analysts a new set of reference data that they would like you to insert into these database tables, I have seen in many cases where these new data conflicts with the data that already exists due to duplication (mostly human error). Since most of these reference data are scripted in text files and there are usually hundreds if not thousands of them, sometimes finding that one duplicating line is like trying to find a needle in a sea.

Fortunately in Excel there is a function to rescue: =COUNTIF(set, to_find).

Usage: Increment count if "to_find" exists in "set".

So to quickly find out which is the offending PK constraint INSERT line, you do this:
=COUNTIF(E3:E11, E3)

And as you see you will quickly find out where that awful INSERT statement is... Time to bug the customer =)

Pragmatic NAnt scripting

2006-03-16T23:10:00.000-06:00

Why am I talking about NAnt when the rest of the world is in love with Ruby and Rake? Well, I still think NAnt has its value. Rake is still pretty raw (it doesn't even have an NUnit task yet), and until it has matured there is still a huge group of developers out there not wanting to spend a huge deal of time learning Ruby/Rake. Plus, most companies out there who have already adopted NAnt is stuck with their NAnt code. This article provides some insights on how to provide such companies business value by making NAnt scripts more maintainable.

Based on my experiences with NAnt, I am concluding the followings are true:

NAnt scripts are very hard to maintain, due to reasons such as:
- Target dependencies are all intermingled into a big unmanageable web.
- No consistencies across NAnt scripts. There is no coding convention in NAnt scripts.
The scripts have to handle a lot if you think about it. It has to handle...
- Pre-Build (like build artifact directory creations)
- The CI (Continuous Integration) cycle like GetLatest, Compile, UnitTests, OtherTests.
- Post-Build (like package, stage, distribute, deploy, blah...)
Build scripts are nasty and no one wants to inherit, and that's why names like "build monkey" (and even more provocative ones) are created and stick to people who initially start the script and tend to stick to those people until they leave the project. It is a thankless job.
Build scripts are, nevertheless, crucial to any projects. If you have a good set of build scripts, they can help developers writing higher quality code, BAs to verify story functionalities quicker, QA to mitigate environment nightmare, and streamlines deployment process by deployment team.

And here are some tips that I have found to be useful to extend the maintainability and lifespan of your build scripts.

Have multiple build script files
I would break down what I see in many projects one monolithic projectname.build file into many smaller build files named after their build function. For example, the file below is called Sesame.build. It has one and only one purpose: to build a project, and test it. I am putting these two build functions together into one build file because the targets involving testing the build output is usually small, and usually go hand-in-hand together. Conveniently co-locating them into one file I think is better than creating a small and isolated test.build file. But other build disciplines, such as package & deploy, by itself could be a big deal, so if complexity warrants I will have a separate file for them. The only encapsulation we can apply to build scripts is really separating them into files.

<?xml version="1.0" ?>
<project name="cruise" default="all">

    <target name="all" depends="get, build, tag" description="Target executed by CCNet." />

    <target name="get" depends="get.get_latest" description="Gets the latest source code from Subversion." />
    <target name="build" depends="build.build" description="Compile and build the source code." />
    <target name="tag" depends="tag.tag" description="Tag the successful build by the naming convention tags\\CRUISE-B999" />

    <target name="get.get_latest">

    </target>

    <target name="build.build">
        <nant buildfile="Sesame.build" target="all" />
    </target>

    <target name="tag.tag">

    </target>

</project>

<?xml version="1.0" ?>
<project name="Sesame" default="all">

    <target name="all" depends="build, test" description="Compile, build, and test the Sesame project." />

    <target name="build" depends="build.compile, build.database" description="Compiles the .NET source code and setup local database instance." />
    <target name="test" depends="test.unit_test, test.other_test" description="Runs unit tests and functional tests." />

    <target name="build.compile">
            ...
    </target>

    <target name="build.database">
            ...
    </target>

    <target name="test.unit_test">
            ...
    </target>

    <target name="test.other_test">
            ...
    </target>

</project>

NAnt target categorizations and naming convention
The ultimate sin of unmaintainable build scripts more or less attribute to the over-profileration of target dependencies. When targets start having a complex web of dependencies, build scripts will take an enormous amount of courage and time to repair.

By breaking down all targets in a single NAnt build file into three categories, and through coding consistency and naming conventions, build scripts can last for a very long time. I find the following categorizations of all NAnt targets in your build scripts useful.

<?xml version="1.0" ?>
<project name="Sesame" default="all">

    
    <target name="all" depends="build, test" description="Compile, build, and test the Sesame project." />

    
    <target name="build" depends="build.compile, build.database" description="Compiles the .NET source code and setup local database instance." />
    <target name="test" depends="test.unit_test, test.other_test" description="Runs unit tests and functional tests." />

    
    <target name="build.compile">
        // Compile using the build.solution_configuration property value...
    </target>

    <target name="build.database">
        // Rebuild database using the database_server property value...
    </target>

    <target name="test.unit_test">

    </target>

    <target name="test.other_test">

    </target>

</project>

Points of interests:

Level 1: These are targets that orchestrates the order of executions of various Level 2 targets and not Level 3 targets. They will only contain depends, but never have a target body. They are the common entry points into the build function the script file represents (eg. target "all" in cruise.build will be called by CruiseControl.NET to kick off the CI build process). They must have descriptions. I prefer names to be underscore-delimited.

Level 2: These are targets that group Level 3 targets together to make a cohesive unit of work of function. For example, a "clean" target might do altogether a few things to clean a build: build artifacts directories, build results directories, VS.NET bin/obj/VSWebCache, etc. They again will never have a target body. These targets will also have descriptions. I again prefer their names to be underscored.

Level 3: These targets will *never* contain depends. They will *only* do one very refined detail piece of work. In addition, their names should be namespaced by a Level 2 target name using a period, and then their function, so that they are easily distinguishable from other targets. This helps newcomers to recognize them and start treating them differently. They never have descriptions (think of these as private methods in a class that does one very specific function for you).

The namespacing naming convention can also be expanded to properties as well.

Use properties to consolidate paths
Have a file (eg. common_paths.build) that extensively use NAnt properties to consolidate your source's folder structure. This will save a lot of code duplication in your build scripts in the longer run if you intend to keep your build scripts for a while.

<?xml version="1.0" ?>
<project name="common_paths.nant" default="all">
    <property name="trunk.dir" value="." />

    <property name="build_output.dir" value="${trunk.dir}\build_output" />
    <property name="build_results.dir" value="${trunk.dir}\build_results" />
    <property name="source.dir" value="${trunk.dir}\source" />
    <property name="tools.dir" value="${trunk.dir}\tools" />

    <property name="dotnet.dir" value="${source.dir}\dotnet" />
    <property name="dts.dir" value="${source.dir}\dts" />
    <property name="sql.dir" value="${source.dir}\sql" />

    <property name="nunit.dir" value="${tools.dir}\nunit" />
    <property name="nant.dir" value="${tools.dir}\nant" />
    <property name="nantcontrib.dir" value="${tools.dir}\nantcontrib" />

</project>

Use the target description attribute
NAnt provides a -projecthelp command line switch to list all of the targets in a given build file. When you give targets a description these targets gets first-class recognition in the listing as "Main Targets":

Combining this tip with the naming convention of the Level 3 targets can be a very powerful technique in improving the readability of your build scripts. As a bonus tip, consider also implementing a target named "help" to display all this -projecthelp target lists.

Use depends or call, but not both
If you start using both, you will be impairing the readability of your build scripts. They more or less do the same thing anyways. I would use depends over call because newcomers to NAnt would be much more likely to learn about depends ahead of call.

Know your property inheritance
NAnt's property inheritance convenience is commonly overlooked and under-valued. Many people chose using <call> task to preserve locally declared property values, rather than experimenting property inheritance of various kinds. I am publishing my findings here.

<?xml version="1.0" ?>
<project name="example" default="example">

    <include buildfile="common_paths.nant" unless="${property::exists('trunk.dir')}" />

    <target name="example" depends="calling_script.properties">
        <nant buildfile="example_two.build" inheritall="true">
            <properties>
                <property name="solution.configuration" value="DEBUG" />
                <property name="build_output.dir" value="c:\modified_build_output" />
            </properties>
        </nant>
    </target>

    <target name="calling_script.properties">
        <property name="calling_script.depends_inherited" value="Inherited from depends clause." />
    </target>

</project>

<?xml version="1.0" ?>
<project name="example_two">

    <include buildfile="common_paths.nant" unless="${property::exists('trunk.dir')}" />

    <echo message="1) Explicitly inheriting property from nant task tag: solution.configuration='${solution.configuration}'" />
    <echo message="2) Explicitly overriding property in calling script's nant task tag: build_output.dir='${build_output.dir}'" />
    <echo message="3) Implicitly inheriting property from calling depends clause: calling_script.depends_inherited='${calling_script.depends_inherited}'" />

    <echo message="Is (1) readonly? ${property::is-readonly('solution.configuration')}" />
    <echo message="Is (2) readonly? ${property::is-readonly('build_output.dir')}" />
    <echo message="Is (3) readonly? ${property::is-readonly('calling_script.depends_inherited')}" />

</project>

1 & 2) Inherit properties through <nant> task
You can call the <nant> task and include a <properties> set within the tags, while setting the inheritall attribute to true to pass those properties to the build script that is getting invoked. In addition, as shown in (2), you can also override the loaded property "build_output.dir" in the properties section prior to passing it into the callee build script.

If you use this style of properties passing, it is a very good idea to pass them as readonly=true to avoid your customization of script behaviors to be reset again by the callee script.

3) It's important to know that properties declared earlier in the depends clause can be used further downstream in the later depends targets, even into another callee build script. This technique is useful when you do not want to load all properties for all targets up front (with the side-effect that the last target in your execution will have access to all properties declared in every single previously executed target, but I am okay with it because I rarely run into properties nightmare compare to build scripts nightmare). Therefore, the following readable script can be achieved:

<?xml version="1.0" ?>
<project name="Sesame" default="all">

    <target name="all" depends="build, test" description="Compile, build, and test the Sesame project." />

    <target name="build" depends="build.properties, build.compile, build.database" description="Compiles the .NET source code and setup local database instance." />

    <target name="build.properties">
        <property name="build.solution_configuration" value="DEBUG" unless="${property::exists('build.solution_configuration'}" />
        <property name="build.database_server" value="localhost" unless="${property::exists('build.database_server'}" />
    </target>

    <target name="build.compile">
        // Compile using the build.solution_configuration property value...
    </target>

    <target name="build.database">
        // Rebuild database using the database_server property value...
    </target>

I am sure there are lots of other tips as well, but that's it for this post. Comments, feedback welcome.

A good way to create a custom Value-Type object in C#?

2006-02-02T22:30:00.000-06:00

Have you ever written code like this?

State state = new State("AZ");

Money money = new Money(299.95);

State state = State.Parse("AZ");

Money money = Money.Parse(299.95);

How many domain-specific Value Type classes (you care about equality but not identities) have you been creating every project? The question of should I use the "new" keyword or a static .Parse() method to create them is always a coding consistency problem.

What if I tell you that you can do this in C#:

State state = "AZ";

Money money = 299.95;

Simple and clean. Here's how:

public class State

    private readonly string _state;

    private State(string state)

        _state = state;

    public static implicit operator State(string state)

        return new State(state);

Few findings about Presentation Model

2006-01-24T22:14:00.000-06:00

Almost every project that I have been on strives for code testability. By increasing code testability one can better the design of the solution as well as continue to change code without worries. However when it comes to increase testability of the view in the classic MVC, it is always hard to do. Microsoft proclaims that they have achieved MVC (or Model-ASPX-CodeBehind), but it fails to address the testability issue of the code in V and C: CodeBehind logic are extremely hard to test against. Both the Presentation Model (PM) and the Model-View-Presenter (MVP) pattern address this issue.

Having used both patterns for my past few projects, I think I am seeing some pros/cons. Today let's look at the Presentation Model first:

Presentation Model:
I like using Presentation Model (PM) when I am developing a web application. In web apps, there are a couple major headaches when it comes to increase code testability, the first of which is carrying state across requests. Secondly, in ASP.NET, the Page class gets created and disposed of every request, making using an intermediary class (such as a Presenter) to "push" content from the model to the view very difficult, and hence a directional referencing problem.

PM employs a "pull" model, meaning the consumer class (aka the code-behind) will ask for data that the PM has prepared for it. Because of this model, at each Page cycle step (Page_Load, Page_Init, etc.) the view will be in the driving seat as far as controlling when to reload the controls and what to load them with. The PM simply sits there and wait for someone to tap his shoulder to ask him to produce something. Notice the directional reference: the view has a reference to the PMs. The PMs do not know about the view, and do not push content out to force the view's state to be refreshed.

Usually you will have one PM per ASPX page, which I call a Page PM. When the page gets busy, your Page PM is likely to be a gynormous 1,000 line class, containing many properties for the use of the many dynamic view controls' use. Therefore, use sub-PM classes to better organize your code and avoid code duplication. The levels of sub-PM classes from my experience probably won't go deeper than 3 inheritance hierarchy.

Most of your Page PMs usually will have reference to the session object, in which most probably the aggregate root of your Domain Model will be stored. For a more complex Domain Model, the PM's task of flattening out this 3-dimensional beast starting with this aggregate root will make your view look extraordinarily simple.

Because PMs usually flatten out your Domain Model into some easily readable grid or string format, your Domain Model objects may not be the ultimate data holder objects that your view 's controls bind to. For example, you will obviously need a new object for a grid showing an insurance policy's type, coverages, each and every insured entity, and each coverage's premiums. (Hopefully they live in separate objects in your Domain Model). Create some what I call "Data Item" objects to facilitate data binding to your grids or lists. Data Item objects are just for holding your diced and sliced Domain Model data and have no behaviors. Think of them as classes that represent a row of data in every single grid on your page. In the above example, your PolicyInfoDataItem class will have public properties of [PolicyType :string, CoverageName :string, InsuredEntityName :string, CoverageAmount :double]. This inspiration comes from DataGridItem, and various ListItem variants in the .NET Framework.

Unit testing your Page PMs in NUnit is always not easy because it uses the Session object, which is not available in NUnit testing. I have seen people use reflection to create a fake HttpContext object that stores Session data on the current thread, which then during [SetUp] set the context object (yes it does have a setter). Then your PM unit test classes have access to HttpContext.Session.

Speaking of unit testing PMs, I find it to be much more work to use mocking to unit tests your PMs. Especially if you love constructor injection, having your PM's ctor to take all those IEverything classes and during unit tests have them all mocked out is a lot of work because PMs is much more object state dependent than behavior loaded. As a result, I would prefer state-based testing (Stubs or the Object Mother pattern) to facilitate unit testing PMs. One must also manage the creation and customization of your domain model's state carefully, because these tend to duplicate code across your test classes easily if you go with state-based testing.

Your view also should not directly instantiate or reference any Domain Model objects (as a guideline, not rule). Whenever possible, pass primitive type user input information gathered off the Page and pass them directly to the Page PM for actions. This can consolidate exception handling routines (instead of scattering them all over your views), and also decouple your view from your Model, which you might later put them into separate assemblies.

Your Page PM can also handle behaviors that your view requests. For example, whenever a button is clicked, its code-behind event handler will make a call to the Page PM's "Save" method, passing the necessary information gathered off the controls, and then the Page PM will delegate the responsibility of the actual save to the Domain Model layer.

Agile story tips

2006-01-14T22:13:00.000-06:00

Stories in an Agile project tend to be less talked about than Test Driven Development, Continuous Integration, and Food :-). But in fact, when it's done well it can profoundly affect the project team's productivity and end users' satisfaction towards the software. The following are a few findings from my experience related to stories management that will drive better software. Their importance are rated on a a scale of 1 to 5 asterisks:

Stories should be a thin-thread from the frontend all the way to the backend. (*****)

This helps the features being developed each iteration to be completed more consistently, and consistency drives predictability, and thus increases visibility and helps business to prioritize and plan given the available time and resources.
Business analysts (or the sponsor/end user) should own the stories. During IPM (Iteration Planning Meeting), they should be signed-off, meaning no changes should be made when developers start developing them. (*****)

By freezing the requirements, developers can be much more productive. Compare a 100m race between two runners, one can go all the way from start to finish without stopping, while the other has to stop-and-go every 10m because he has to worry about the next 10m track he runs will be changed. Also, by someone owning the stories, after the stories were developed the owner of the story will have the responsibility to verify the correctness of the solution. Thus this requires better-written story specifications (tying back to freezing requirements), and in the end the developed story becomes exactly what business wants.
Stories should be measured in terms of difficulties or story points, instead of ideal development days (IDD). (****)

There are two camps of people when it comes to this bullet. One camp uses IDD to estimate the difficulty of a story, then uses Load Factor to measure how off they were in their original estimates. The other camp uses Level of Difficulty such as small/medium/large, or Story Points such as 1-5, so they can measure purely based on yesterday's weather, and not a number that someone uncomfortably being forced to make up.

IMHO, the first camp's glossaries were created at "post-mortem," literally. Consider the following conversation after a project failure:
Business: Why did you guys fail to deliver? You only delivered half of what I want.
Developers: Because we were not productive.
Business: Why?
Developers: Too many damn meetings.
Business: Hm... so in "ideal time" if you had no meetings you would be able to deliver?
Developers: Oh yea...
Business: So in hindsight, the original estimates you guys told me was off by a "load factor" of 1/2. Next time should I want a new software I better take that into my budget account...

The problem is that all these numbers are only meaningful within the context of that one project. But people being people, they like to carry these numbers with them to any projects they walk into wherever they go... because apparently they have been burned before. Next time when business asks for budget for a brand new project, guess what. They will bump up the budget by half, because of that "load factor."

By using the terms of the second camp, one is implicitly forced to think of these measurements in the context of the current project. When I say this story's difficulty is a Large, one has to ask: Relative to what? Of course, the answer is relative to the other stories of this project. When I say, Story A is a 1 and Story B is a 3, again you are forced to think in the context of the current project.

You might ask, now if you don't bump up the budget by half, then doesn't the business not get all of what they ask for? The answer is yes. But that's the beauty of short, iterative releases. If we can do that, then in the end the business without bumping up the original budget, yes they will exactly get what they have asked for according to their business priority, perhaps the 6 out of their 10 features, but since we have delivered at least some of the features in early iterations, not only the business has saved money due to those features being rolled out, but also the business is in better shape when it comes to repositioning itself to face more real world challenges, and thus will pump more budget to continue develop the software to give them what they want.
If stories are small enough, then there is no need to task them out during an IPM (Iteration Planning Meeting). If stories are to be tasked, they should be estimated. As each of them are completed, actuals should be measured.

Admit to it. Small estimates are much more accurate than big estimates. Therefore, if each story is tasked out into small chunks of time estimates, and after story completion we have its corresponding actual time spent measurement, we then can find out how much work really is to complete say Story A, a medium-difficulty story or one that has Story Points of 3.

So what are these estimates and actuals for? They are actual prove (or tracked history) of how we complete our Stories. Let's say in Iteration One we have a story "Public user login" that has a Story Points of 3. In that iteration (two-weeks), at the IPM the development team estimates that there are a total of 10 tasks to be done to complete that story, and they busted their ass to complete that and only that story. Then, in Iteration Seven, a similar story "Restricted user login" shows up. Relatively speaking, it also has a Story Points of 3, since they are about equally difficult. However, since most of the one-time tasks to do that has been completed, the actual number of tasks to complete this story in Iteration Seven might be just 3 tasks. Now the team can use the rest of the time to build other stories, and thereby achieve more Story Points. If it turns out the team achieved total of 7 points in the end, then we say the team is kicking some ass and is more productive than they were in Iteration One. You would notice the total time spent on all tasks between the two iterations will be somewhat the same (assuming no resources change), but completed Story Points increased. From the point of the business, it rocks, because they are seeing more stuff being churned out by the team, in exactly the manner they want it to be.

This brings up another very important point, if you notice...
For story difficulty measurements, whatever you use (IDD, Small/Medium/Large, Story Points), you should always estimate it using the same scale as you estimate the entire story list. (*****)

This is the only way to measure whether the team is improving over the course of the development.

Using the example in the last bullet, if in Iteration One the team thinks that "User login story" is a 3 Story Points story, and in Iteration Seven the team completes a similar also 3 Story Points story "Restricted login story", then if in Iteration Ten business comes back and say they want a brand new but similarly difficult "CEO only login story", now despite the fact that in Iteration Ten, after doing those two login stories, this new story requires very very little work to complete, we must again make this story have 3 Story points.

This way, the measurements will tell the business the following:
In Iteration One, the development completed 3 Story Points.
In Iteration Seven, the development completed 7 Story Points (because the tasks required to do the "Restricted login story" has reduced.
In Iteration Ten, the development completed 14 Story Points (because even few tasks is required to complete the new 3-Point "CEO only login" story).
In terms of the business people, Story Points = functionalities = business value. They know what they are getting in a consistent basis.

Should there be a case somewhere in Iteration Eight the number of Story Points dropped, then one has to figure out why. Here's the task actuals can come into handy. In Iteration Seven, 7 Story Points and total of say 100 actual hours of time were needed to complete all tasks. In Iteration Eight, only 6.5 Story Points were completed. But if we look at the hours of the actual, only 90 hours were recorded from all tasks. Now we know the time the team spent on the actual tasks for all stories are about the same. Probably because people having vacations or public holidays that contributes to the drop in productivity.

Do you check in your OSS source into trunk?

2005-11-29T11:47:00.000-06:00

How to use Open-Source Software? What I mean is, should you see OSS source code as part of your project's source code, keep their source code around and maintain/update them, or should you just use them in your project and wait for it to be maintained by their authors? This is a question for a lot of development teams, because by now almost anyone would have known the pros and cons of OSS. The bottomline is, you will use them at one point or another. The bigger question is, how to use them to your project's benefits, without carrying too much overhead.

For me, I want to maintain as few lines of code as possible. So my answer is, don't ever give me the OSS's source code. Give me your project's source code plus the OSS assemblies that are in use. I should be able to checkout your project's trunk and go. Don't assume I have stuff installed after I downloaded your project's source code. If at some point in the future the code does not run because of an OSS bug, fix the bug then submit it back to the community. If this happens more than few times, use something else.

This solves the problem of trying to maintain a chunk of code that a team has no idea about. At the minimum, it solves any new dev's problem of not having to download a 100MB trunk with 80MB of it is OSS source code (admittedly it's more a nuisance than a problem).

One of the good things about OSS is that there usually is an abundant amount of alternatives out there. Take functional testing as an example, Selenium and WATIR. There are things Selenium is good at (cross-browser testing), and there are things WATIR is good at (more powerful script coding). Mock objects anyone? NMock, EasyMock.NET, Rhino.Mocks, etc. Code coverage? There are even TWO NCovers out there that has the same name...

I think the problem of merging a tweaked, home-brew version of an OSS back into using a latest version down the road is much more painful than using multiple OSS in your code base. For the latter at least the breaking changes are documented.

If it hurts to use something in solving your problem. Don't use it. Problem solved. The problem you are getting paid to solve is delivering business value.

The best Visual Studio.NET blogging companion

2005-11-19T12:30:00.000-06:00

Check this out, all I did was in VS 2003, right-click, select "Copy as HTML...", click OK, and CTRL-V in blogger. All of a sudden you get this stylish code colorization in a blog:

    public class Bootstrap : IDisposable

        private IMutablePicoContainer picoContainer;

        [STAThread]

        public static void Main()

try

                using (Bootstrap bootstrap = new Bootstrap())

                    IMainForm mainForm = bootstrap.BuildMainForm();

                    Application.Run((Form) mainForm);

            catch (Exception e)

                MessageBox.Show(e.ToString());

Awesome VS.NET Add-in! CopyAsHTMLSource

Web 2.0... the what?

2005-11-19T12:07:00.000-06:00

Recently I am hearing more and more about Web 2.0. So what the hell is it? It seems to me no one really knows exactly what it is, but among all opinions they all point to the same direction: enhanced web application's user experience. I guess the industry is looking for the next buzz word after AJAX.

In my company's forum, a couple web sites have been mentioned, and they really impressed me with what they mean by "user experience":
script.aculo.us
Rico

Man I am falling in love with the head-shaking textbox in script.aculo.us!

How to mock out event calls in NMock?

2005-10-10T16:50:00.000-05:00

When it comes to .NET eventing, a lot of developers barf at and not knowing how to test them. one is that the event handler methods are correctly wired to the corresponding events, and two the event handler code does what it's supposed to. The first is hard to test, because the wiring of a method to the event is internal to the containing class only. The second is easier, because you could stub out the event handler method and making sure that it is getting called, but pure mockists would dislike this approach.

Consider the following example:


public interface IBattery {
    event EventHandler Low;
    event EventHandler Depleted;
}

public class Battery : IBattery {
    public event EventHandler Low;
    public event EventHandler Depleted;
    
    public Battery() {
        Low += new EventHandler(OnLowBattery);
        Depleted += new EventHandler(OnDepleted);
    }
    
    public void SomeMethodThatConsumesBattery() {
        .
        .
        .
        if (IsLowBatter("10%")) {
            OnLowBattery(this, EventArgs.Empty);
        }
    }

    protected virtual void OnLowBattery(object sender, EventArgs e) {
        if (Low!= null) {
            Low(sender, e);
        }
    }
}

public interface ILaptop { }

public class Laptop : ILaptop {
    private IBattery battery;

    public Laptop(IBattery battery) {
        this.battery = battery;
        
        batter.Low += new EventHandler(OnLowBattery);        
    }
    
    protected virtual void OnLowBattery(object sender, EventArgs e) {
        // Count down 15 mins before shutdown!
    }
}

We want to unit test the events are wired for our Laptop class, but we want to avoid changing anything any of the classes for the purpose of unit testing, since I am a big advocate of "never change your production code just for testing." You may argue that I have a "protected virtual void" event handler method there to leave room for myself for stubbing, but as far as event handling method goes, I think it is actually a good idea to allow my subclasses to override and extend my default implementation. This is the standard and encouraged way of writing and declaring event handling methods too in writing custom ASP.NET server controls. Check out Nikhil Kothari's book.

So, solution one, create a stub class for our Battery class in testing Laptop, and add extra methods to manually raise the events:


public class BatteryStub : Battery {
    privately bool lowBatteryExecuted = false;

    public void RaiseLowBattery() {
        Low(this, EventArgs.Empty);
    }
    
    protected override void OnLowBattery(object sender, EventArgs e) {
        lowBatteryExecuted = true;
    }
    
    public void Verify()
    {
        if (!lowBatteryExecuted) throw new ApplicationException();
    }
}

[TestFixture]
public class LaptopTests {
    
    [Test]
    public void StartCountdownWhenLowBattery() {
        IBattery batteryStub = new BatteryStub();
        ILaptop laptop = new Laptop(batteryStub);

        batteryStub.RaiseLowBattery();

        // Assert laptop countdown started.

        battery.Verify();
    }
}

That's a lot of code to just unit test a single event is wired correctly. Also this stub is really doing a lot of a mock's work. Look at the Verify() and the simplistic stubbed event handler method. The fact that they are there and are simply is because in testing we are not interested in what they do, but that they are getting called. Now, duplicate this kind of test stub class for every object you have events, and you will quickly lose appetite on how many you have to write for all your domain objects.

Fortuntely there is a second solution, if you use NMock 1.1:

(credit to my co-worker and talented friend Levi Khatskevitch)


public class DynamicEventMock : DynamicMock
{
    private const string ADD_PREFIX = "add_";
    private const string REMOVE_PREFIX = "remove_";
    
    private EventHandlerList handlers = new EventHandlerList();
    
    public override object Invoke(string methodName, params object[] args)
    {
        if (methodName.StartsWith(ADD_PREFIX))
        {
            handlers.AddHandler(GetKey(methodName, ADD_PREFIX), (Delegate) args[0]);
            return null;
        }
        if (methodName.StartsWith(REMOVE_PREFIX))
        {
            handlers.RemoveHandler(GetKey(methodName, REMOVE_PREFIX), (Delegate) args[0]);
            return null;
        }
        return base.Invoke(methodName, args);
    }
    
    public void RaiseEvent(string eventName, params object[] args)
    {
        Delegate handler = handlers[eventName];

        if (handler == null)
        {
            if (mockedType.GetEvent(eventName) == null)
            {
                throw new MissingMemberException("Event " + eventName + " is not defined");
            }
            else if (Strict)
            {
                throw new ApplicationException("Event " + eventName + " is not handled");
            }
        }
        
        handler.DynamicInvoke(args);
    }
    
    private static string GetKey(string methodName, string prefix)
    {
        return string.Intern(methodName.Substring(prefix.Length));
    }
}

Now, in your test class:


[TestFixture]
public class LaptopTests {
    
    [Test]
    public void BatteryLowIsRaised()
    {
        DynamicEventMock mockBattery = new DynamicEventMock(typeof(IBattery));
        
        ILaptop laptop = new Laptop((IBattery)mockBattery.MockInstance);
        
        mockBattery.RaiseEvent("PlayClick", EventArgs.Empty);
        
        // Assert the laptop instance's 15 mins count down started
        
        mockBattery.Verify();
    }
}

Should the Low event wasn't wired, this test fails. You have explicitly told the mock object to raise its event, and assuming your events are wired correctly to the correct event handler, you have control over when to fire them.

There are drawbacks, of course. Notice the event name is represented by a string, making event renaming a pain to do, as it is for NMock for method expectations. My only advise is try out EasyMock.NET if that is a real pain for you.

Ruby on Rails demo

2005-10-10T11:37:00.000-05:00

A lot of buzz has been generated by Ruby, and given that Ruby on Rails pushes that hype to another level it sure is worth a look. This video from rubyonrails.com is a great tutorial on what is Ruby on Rails and how it works.

http://www.rubyonrails.com.nyud.net:8090/media/video/rails_take2_with_sound.mov

Container framework in .NET - yes - 1.1

2005-09-23T11:12:00.000-05:00

Haven't blogged for a good while... =P

I have been looking at the System.ComponentModel namespace. It is obviously an area where I have constantly overlooked. What is in there? There is a ton in there that the Visual Studio .NET IDE extensively uses, so it means it is useless to us as developers, right? Maybe not.

Recently I have found myself interested in digging deeper into PicoContainer.NET. It allows you to design better OO business objects by decoupling your business objects from one another (through IoC). For those who aren't familiar with IoC (Inversion of Control), basically it means in this context instead of having object A to instantiate and use object B in it, object A will ask for an instance of object B in its ctor at its construction time. How it helps OO design is that now object A no longer has a irreplaceable link to object B, thus they are decoupled, and it allows us to test object A easily by either stubbing or mocking out object B.

In the above example, object A and object B are "Components". They will be put in a "Container" and Pico will automagically instantiate object B when you request for an instance of object A.

Back to the System.ComponentModel namespace, it also contains interfaces IContainer, IComponent. Now things get interesting, each IComponent is called "sited" after being added into a IContainer (like the glue between the Container and the Component). A component can now call base.Site.GetService(typeof(anotherComponent)) to access another component's functionality. For our example, in object A's ctor, it can call base.Site.GetService(typeof(ObjectB)) to retrieve an instance of object B and create itself, without having to know how to "hard create" an instance of object B.

How does it help testing? For testing object A, one can stub or mock out its Site property and provide a testing implementation when its GetService() is getting called. This pattern (or good practice) is very important to design decoupled OO systems.

So when to use what? My take is that one should use Pico in your bootstrap class (public static void Main(args[])), and since it is considered harmful to use Pico in anywhere deeper than your bootstrap class, I will consider using the System.ComponentModel stuff in everywhere else I think using a Container-Component pattern will help decoupling my design.

Some excellent reads on the topic:
PicoContainer
[Urban Potato] Geek: System.ComponentModel
[Daniel Cazzulino] Lightweight Containers and Plugin Architectures: Dependency Injection and Dynamic Service Locators in .NET

Want to whip up some quick UI Tree?

2005-06-10T14:10:00.000-05:00

I had to produce for one of my clients some form of UI Tree to help them to focus on how various pieces of user interactions with the system is going to take place from a UI perspective. It went from paper bubbles to Visio-lizing it and client still don't like it. Finally, I stumbled across this product that saves the day.

It is basically a map of thoughts organized in a click-me-and-drill-down-as-you-go application. The point is, it proides a very easy drag-and-drop tree creating inetrface to do the job very quickly.

At the end of the day this did wow the people who saw it, and it is extremely easy to use. It got the job done. I am however interested in the many ways of people who create the so-called UI Tree how they do it.

Connecting a Pocket PC emulator to host internet in Whidbey

2005-05-24T16:45:00.000-05:00

This tip is for developing environment which has Whidbey Beta 2. I do not know if it will work (probably won't) on a Virtual PC environment.

If you are trying to connect your Pocket PC emulator to the internet because you are developing/deploying perhaps a web service on your host system, and you want your emulator to be able to consume it in your debugging, here is a few things in sequence I have tried to get it to work:

- Restart your computer. Yes.
- Open your .sln by double-clicking it in your Explorer, not by opening your Whidbey and then selecting it from your File/"Recent Projects". Apparently there is a funky difference.
- Run F5 to bring up the emulator.
- It will probably tell you "Cannot connect to device", but that's okay.
- In Whidbey, go to Tools/"Device Emulator Manager", set your emulating device to Cradle.
- Reset the state of your emulator.
- Close down the emulator.
- Run F5 again to bring it back up.
- In your emulator Connections/Advanced/"Select Networks", select "My Work Network" in the first droplist for programs that need access to the internet.
- Go to its Edit/"Proxy Settings", uncheck the box "This network connects to the internet". Notice this box will automatically be checked again by the emulator once it can connect to the internet.
- And finally, verify that you have turned your firewall software off.

This would work in the emulator IE whether you type "http://localhost/MyService.asmx" or "http://localhost/MyService.asmx". If I were you, I would see if I could even connect to Google first in your emulator IE before testing the web service.

Dynamic deployment config settings

2005-05-17T14:24:00.000-05:00

Using Visual Studio .NET 2003, there are different options to dynamically use app/web.config settings depending on what environment the build is deployed on. One of which is in your code, use preprocessor directives like #if, #elif, and #endif. I don't like this approach, period. This is really ugly. What if you have 5 deployment environments? You will end up with values with keys "dbConnStringDev", "dbConnStringTest", "dbConnStringQA", "dbConnStringProd", etc.

Another option is have inside your app/web.config file, just create one key-value pair for each setting regardless of how many deploying environments you might have (eg. ), and then after each deployment, manually (or automate this by some scripts) go and edit the settings for all these. Well... this is just as bad as a single mistype would deem a deployment failed.

A third option would be, if you have separate a set of physical machines per deployment environment, one could use each machine's machine.config as the definitive setting storage, and then override them with local app/web.config in for example your developer's .NET projects. Question arises if for example you have five different deployment environments (dev, test, QA, stage, and prod) but you only have two boxes for your web services server across all of these environments: how do you direct traffic for dev, test, and QA to box A while stage and prod to box B. It's the same problem all over again.

The solution (well, not really) I have found so far is this. In the .csproj that could house a app/web.config file, you can find the following lines for each solution configuration:

<Config
Name = "Release"
AllowUnsafeBlocks = "false"
BaseAddress = "285212672"
CheckForOverflowUnderflow = "false"
ConfigurationOverrideFile = "app.release.config"
DefineConstants = "TRACE"
DocumentationFile = ""
DebugSymbols = "false"
FileAlignment = "4096"
IncrementalBuild = "false"
NoStdLib = "false"
NoWarn = ""
Optimize = "true"
OutputPath = "bin\Release\"
RegisterForComInterop = "false"
RemoveIntegerChecks = "false"
TreatWarningsAsErrors = "false"
WarningLevel = "4"
...

if you create a new .config file for each deployment environment that you have, then create a new solution configuration for each deployment envionment (eg. Debug, Release, QA, Stage, Production), then change that bolded line for each deployment into the appropriate .config file, then create a Setup project in your solution and build it with the solution configuration you want for your deployment, the output .msi will include *a* app.config file, with it being replaced with the file you specified in the above bolded text. So in the example above, app.release.config will be used and renamed to app.config when the output .msi is deployed onto the environment.

This is so far the cleanest solution for me. But it does tie you into using Setup project for your deployment configuration. And from what I have heard, .msi has not been a good deployment strategy for complex application deployment (yes, the fact that there is a relational database built into a .msi does not make it suit for enterprise level if it is complex to use).

Still looking into alternatives... on a side note, the fact that Microsoft is switching from "no-touch deployment" into "one-click deployment" means to me that the marketeers at MS is going one step to far in their marketing effort =P