Posts
58
Comments
103
Trackbacks
10
May 2007 Entries
SOA, Change, Agility and Reuse

After listening to Udi's most recent podcast, a few things stuck out at me.  First, he mentions potential business drivers for SOA adoption: agility (ie..time to market), cost-savings, and reusability.  While many people have jumped on the bandwagon of reusable services and BPEL, what really struck me in his podcast were his comments on reusability.  After listening to them, several things "clicked."

On every major paradigm shift, from object-oriented programming, component-oriented programing, service-oriented programming, etc, reusability and composability have been driving factors.  Yet, the often touted killer concept of stringing the reusable things (components, objects, services) together in an ad-hoc fashion has never materialized.  Building the "one Customer object to rule them all" is a myth.  In turn, if you are relying on reusability to enable composability and are looking to composability for cost savings, that's quite a large gamble, especially given technology's track record on these concepts.

Agility, on the other hand, can be grasped with a proper SOA.  I'll dig deep here.

What is at the heart of agility?  Being able to successfully manage change.  The change in developer mindset is from one of fear of change (or should I say fear of bugs and missed deadlines) to one of embracing change.  This is a very hard thing to accomplish as a coder.

One of our greatest tools for managing change in our application is the tool of isoloation.  We isolate different parts of our system to make change easier to deal with.  Layering is a simple example of this.  We isolate persistance concerns to the data access layer of our system so that when we need to change anything related to that single concern, we know exactly where to go and how it affects the rest of our application.  We can do this because the Data Access layer of our application has a single responsibility and single purpose.  No other part of the system should have any care how the persistance works.  Objects work best with the same principle.  This is why we have such things as the Open/Closed Principle and the Single Responsibility Principle.  It's also why we measure cohesion as part of our code quality.  Single purpose "things" have the greatest chance for reuse and allow us to better manage change when it occurs, and it is this single-purpose concept which we use to draw lines of isolation (boundaries for loose coupling), which in turn enables agility.

Here are the relevant principles:

  • Single purpose items (objects, components, services) yield the greatest chance for reuse.
  • Single purpose items plus loose coupling (boundaries) yield isolation of change.
  • Isolation of change enables agility.

If we want to build service-oriented architectures, we must not forget the lessons and techniques of the past.  What often makes sense in the little things (inside an application) often make sense in the bigger things (multi-service interaction).  We must be wary of the naive promise of ad-hoc composability until if and when it becomes a proven concept.

And when building those services, don't forget the eight fallacies of distributed computing.

Technorati tags: , ,
posted @ Sunday, May 20, 2007 11:02 PM | Feedback (0)
Two More for the Bookshelf

I just picked up a couple more books from Amazon.  I'm really looking forward to reading them.

Pattern-Oriented Software Architecture, Volume 1: A System of Patterns

Pattern-Oriented Software Architecture, Volume 2, Patterns for Concurrent and Networked Objects

Here's a good summary of the 5 books in this series.  If you reach much software architecture literature, you will probably see these books referred to as the POSA books.  They get quoted often.

I'm mostly interested in the Event Handling patterns in POSA2, but POSA1 is quoted so often in architecture literature that I feel like I really need to read it.  I see quite a bit of overlap between POSA1 and some of my other readings, but I want to make sure I have a strong foundation.  I may learn more just reading another perspective on the same topics.

On a related note, there was a decent video presentation from Ian Cartwright (ThoughtWorks/Martin Fowler) on Eventing Patterns up on InfoQ.

 

posted @ Sunday, May 20, 2007 9:32 PM | Feedback (0)
The Consumer Owns the Contract

In software, we have many forms of contracts.  In xml-based interactions, we have DTD and XSD contracts.  In class interactions, we have abstract classes and interfaces.  In libraries and frameworks, we have APIs.  I'll keep this simple.  The client code owns these contracts.

When we are talking about XSD schemas for webservices (ie..in the form of WSDL documents), the client owns the contract.  You as a service provider are saying to the consumer, "this is the interface my implementation conforms to".  After publishing the contract, you are free to change the implementation as long as the contract remains the same.  If you break the contract, all bets are off.

The same applies for interfaces inside an application.  As long as your class implementation complies with the interface, you are free to change the implementation as desired.  Once you break the contract, again, all bets are off.  The client code must be updated.

It is this simple concept which drives many other parts of software development.  Test-driven development is nothing more than a client-code driven methodology (with tests to *prove* the implementation meets the contracts).  This is why test-driven development is typically used in conjunction with interface-based programming.  They just go hand in hand.

Another simple example comes to use from Bertrand Meyer in his famous concept of Design by Contract.  We explicitly declare preconditions, postconditions, and class invariants in our code to protect the contract that a method has with its other interacting pieces.  A precondition is the contract the method has with it's calling code.  It specifies that a certain parameter contains a valid value or that it may or may not be null.  Postconditions are contracts that a function gives to its calling code.  It specifies that it *will* return a specific range of values.  This may include a specific set of values or simply a non-null value, for example.  A class invariant is contract that a method has with it's owning class.  It specifies that it will not leave the class in an inconsistent state (ie..by setting a specific field to null, for example).

What else can we learn from this simple yet powerful concept? It can be useful when packaging our code.  Udi Dahan, one of my favorite bloggers, has a post regarding the packaging of interfaces and classes.  As his first rule of design, he states that interfaces (contracts) should be packaged separately from the implementation.  I agree entirely, as this allows the implementation to be swapped without a recompile of the client code (assuming you haven't broken the abstraction by coupling the calling code to the implementation).

I would add one exception to this rule however.  I think there are times when the contract (interface or abstract class) can be packaged with the client code.  A classic example of this is the .NET framework itself.  If we look at the framework, we see interfaces such as IConvertable<T> and IComparer<T>.  Given the usage of these interfaces (among many others), we can see that in the case of these abstractions (which are really just implementations of the Strategy pattern), the .NET framework is the consuming code (you can go look at their usage if you like).  Therefore, the contract (these interfaces) belong to the client (the .NET framework).  Packaging the interface separately would also work, but it's not necessarily required.  Pretty much all strategies and callbacks fall within this niche (especially where you are already tied to a particular implementation, such as the .NET framework).  In an individual application, this could mean packaging the data access interfaces in the same physical distribution unit (the same assembly) as the entities.  In a single application scenario, it will not break the consuming contract to completely replace the data access logic.  This is because the entities are the consumers of the data access layer.  If the data access layer has to break the contract both will have to be recompiled anyway.  What this will break, however, is large-scale reuse.  If you have a nice data access API,  packaging its contracts (interfaces) separately from the implementation and consumer will allow for greatest reuse.  Neither situation breaks a design principle in my opinion, however, splitting the two yields nice benefits in the reusability department.

This simple concept, that the client owns the contract, is simple yet amazingly powerful.  It's one of the architectural principles that can drive decisions both large and small.  The corollary to this principle is to package your contract either with the client code or as a standalone artifact, depending on which yields the greatest benefits in your particular situation (maintainability vs reusability).

I may expand on the reusability tradeoff on another post in the future.  For now, I'm still working through some areas (especially when it comes to physically packaging large applications using a modular packaging approach--ie..not the usual assembly per layer approach).

posted @ Tuesday, May 15, 2007 12:04 AM | Feedback (1)
Just a Reminder

Let's keep our eye on the ball..

We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more.

Manifesto for Agile Software Development

posted @ Monday, May 14, 2007 12:25 AM | Feedback (0)
Useful SQL Server Links

Straight from my bookmark list:

I had planned to try out Quest's Benchmark Factory for Databases, but between the SQL Server 2005 install on my laptop and the 2005 SP2 install, I'm calling it a night.  I'll let you know how the trial goes, and where you can get a copy for super cheap.

 

posted @ Sunday, May 13, 2007 10:58 PM | Feedback (31)
Orthogonal Platform Upgrades

Our team is currently looking to migrate to SQL Server 2005 from 2000.  An interesting thought occured to me as I was researching the changes and upgrade procedures.  Our team is very Agile-friendly (we are continually trying to become more and more Agile), and as such, we have kept as much as is possible out of the database.  One side effect of this is that the database upgrade comes as an orthogonal upgrade.  If you aren't familiar with Orthogonality, I highly recommend you to read this post over at Jeremy Miller's blog.

In short, the teams who choose to embed much of the application inside the database (ie..the database is the application mindset) are forced to cope with some pretty massive changes.  I can't imagine trying to upgrade the database and multiple third party database packages at the same time.  That would force either a HUGE testing effort or risk a meltdown during deployment.  Our upgrade, in contrast, involves only a handful of minor changes.

On another note, our current test coverage is really going to be an ROI multiplier when Orcas and Windows Server Longhorn come out.

 

Technorati tags: , , ,
posted @ Saturday, May 12, 2007 6:12 PM | Feedback (0)
Implementing the Mapper and Dao on the Query and DbGateway

Now we will look at leveraging the queries and dbgateway from my last post into a usable framework.  Our goal is to be able to take our query objects and use them to get real objects back (instead of just DataSets and DataRows).  Let's look at the following new interfaces:

    public interface IMapper<T>
    {
        T Map(DataRow row);
    }
    public interface IDao<T>
    {
        T Get(IQuery<T> query);
        IEnumerable<T> Find(IQuery<T> query);
        void Execute(IQuery<T> query);
    }

We will let the mapper specialize in converting our data into objects, and we will use the Dao (Data Access Object) to whip the whole thing together.  For this stage, we also added a 2nd query interface.  Here they both are.

    public interface IQuery
    {
        DbCommand DbCommand { get; }
    }

    public interface IQuery<T> : IQuery
    {
    }

You may wonder why we used a generic type argument for the 2nd interface.  In short, this let's our query objects expose the type of data they return.  If the query object returns data for a customer class, we can now enforce that at compile time (so we don't pass an Order Query and try and map the results onto a Customer object).

You will also probably notice I changed the IQuery interface to use a DbCommand instead of a string.  I did this so I could leverage the .NET Enterprise Library inside the query object.  I wish the EntLib designers would have used IDbCommand, but instead they used DbCommand.  So I am stuck with it.  It's really not that bad though.  DbCommand is still an abstract class, so we are still safe from change.  You could stick to the string implementation if you prefer.

Here's our gateway (using EntLib3):

    public interface IDbGateway
    {
        DataRow Get(IQuery query);
        DataTable Select(IQuery query);
        void Execute(IQuery query);
    }

 

    public class DbGateway : IDbGateway
    {
        public DbGateway()
        {
            _db = DatabaseFactory.CreateDatabase(); // entlib
        }

        private Database _db;

        public DataRow Get(IQuery query)
        {
            if (query == null)
            {
                throw new ArgumentNullException("query","query cannot be null");
            }
            DataTable dt = Select(query);
            if (dt == null || dt.Rows.Count == 0)
            {
                return null;
            }
            else
            {
                return dt.Rows[0];
            }
        }

        public DataTable Select(IQuery query)
        {
            if (query == null)
            {
                throw new ArgumentNullException("query", "query cannot be null");
            }
            DataSet ds = _db.ExecuteDataSet(query.DbCommand);
            if (ds.Tables.Count == 0)
            {
                return null;
            }
            else
            {
                return ds.Tables[0];
            }
        }

        public void Execute(IQuery query)
        {
            if (query == null)
            {
                throw new ArgumentNullException("query", "query cannot be null");
            }
            _db.ExecuteNonQuery(query.DbCommand);
        }
    }

The Mapper is trivial to implement for each type.  It simply takes the DataRow and creates an object from it.

And finally, we can bring everything together.  Here's the implmentation for the Dao.

 

    public interface IDao<T>
    {
        T Get(IQuery<T> query);
        IEnumerable<T> Find(IQuery<T> query);
        void Execute(IQuery<T> query);
    }
    public class Dao<T> : IDao<T>
    {
        public Dao(IMapper<T> mapper, IDbGateway gateway)
        {
            _mapper = mapper;
            _gateway = gateway;
        }

        private IMapper<T> _mapper;
        private IDbGateway _gateway;

        public T Get(IQuery<T> query)
        {
            DataRow row = _gateway.Get(query);
            return _mapper.Map(row);
        }

        public IEnumerable<T> Find(IQuery<T> query)
        {
            DataTable dt = _gateway.Select(query);
            foreach (DataRow row in dt.Rows)
            {
                yield return _mapper.Map(row); // iterator
            }
        }

        public void Execute(IQuery<T> query)
        {
            _gateway.Execute(query);
        }
    }

You might notice that we return an IEnumerable<T> instead of an IList<T> or some other type.  This allows us to use an iterator for the data mapper while letting the client code do foreach's across the objects.  If you'd rather have a List<T> from the Dao, you should notice that List<T> takes an IEnumerable<T> in one of its constructors.  This is the best of both worlds.

posted @ Tuesday, May 08, 2007 10:08 PM | Feedback (0)
Learn C# 4.0

Want to get a head start on the rest of the .NET developers out there?  Start learning how to use C# 4.0 today.  How, you ask, can I learn to use C# 4.0 when 3.0 3.5 isn't even out yet?

Invest your time and effort into learning how to properly leverage programming languages in general.  That's right.  It works the same for application architecture.  How do you prevent the next release of ADO.NET from forcing you to rewrite your application?  You isolate it and build using an ADO agnostic API.  You chain it to the data layer where it belongs and don't let it see the business logic it is enabling.

It works the same for us as developers, and while you will need to learn the new language improvements for C# 4.0 when it comes around, investing in general purpose techniques is the best thing you can do for yourself today.  To put it in business terms, investing in technology agnostic knowledge yields a better ROI than memorizing the latest API out of Redmond.  That should be painfully clear right now, given the amount of new technology they have released in the past 6 months.  Did you know that they are pushing the next version of SQL server into testing in June?

You may be wondering how to start.  Enter design patterns.  Hate design patterns?  Try starting with functional programming, test-driven development, responsibility-driven design, domain driven design, messaging or any number of other topics out there available to you.

Ask yourself what you currently know that will be useful 5-10 years down the road.  If all you can answer is c# or t-sql, you are going to be continually reinventing yourself down the road.  Ouch.

[Updated: It helps to proof your posts when you aren't really tired...lol]

posted @ Tuesday, May 08, 2007 9:10 PM | Feedback (6)
Not Invented Here Syndrome

Yes, this is a bit of a rant.  You have been warned. 

As developers (and human beings), we have a tendency to reject things which were Not Invented Here.  This tends to include processes, tools, techniques, and a host of other useful things.  In a lot of cases, this is Bad (with a capital B).

When it comes to components and tools, not only are you robbing yourself of a productivity boost by not having to develop every little thing in your app, but you miss the quality benefits that a specialized vendor can bring to the table.  If you look around at the component vendors and open-source projects that exist on the web, more than likely you can find one that will scratch your itch and save you money.

Look at it this way, take your hourly rate and multiply times the number of hours it will take to build that little utility or component.  Now remember that you always want to rewrite that tool as soon as you finish it.  Multiply by two again.  Assuming you aren't being paid $6 an hour, the $300 that component costs is a drop in the bucket, and we aren't even factoring in debugging time.  Also realize that the off the self version, assuming you pick a decent vendor, will be much more tested than anything you grow at home and will probably have a lot more features too. 

If you are currently writing off open-source projects, you are missing some of the best stuff out there.  Not only do you get it for a $0 cost, but the upgrades are free too.  Some of the best developers in the world contribute to open-source projects.  Who knows, maybe if/when you have to modify their stuff, some of their talent will rub off. ;-)

While I definitely wouldn't want to use an off-the-self solution for a core business component, leave the other oddities for someone else to write.  Focus on your core business strengths, and let them focus on theirs.  Remember this the next time you shun that cool open-source tool or decide to write your own component instead of spending a couple hundred bucks to get a professional one.  Don't fall victim to Not Invented Here Syndrome.

[Update: Just ran into a similar posting on my RSS reader over at Dru's blog.]

posted @ Monday, May 07, 2007 10:53 PM | Feedback (0)
Separating the Query from the Gateway

In short, this is an easy way to simplify certain aspects of data access.

Let's look at the following interfaces:

    public interface IQuery
    {
        string Text { get; }
    }
    public interface IDbGateway
    {
        DataRow Get(IQuery query);
        DataTable Find(IQuery query);
        void Execute(IQuery query);
    }


In this case, we wrap each of our data access commands (sql, hql, etc) inside a simple object, implementing our IQuery interface.

    public class AllCustomersQuery : IQuery
    {
        public string Text
        {
            get { return "exec up_Customers_GetAll"; }
        }
    }
 
    public class GetCustomerQuery : IQuery
    {
        public GetCustomerQuery(int id)
        {
            _id = id;
        }

        private int _id;

        public string Text
        {
            get { return string.Format("exec up_Customers_GetById @Id = {0}", _id); }
        }
    }


These objects are great candidates for code generation (especially if you are already generating stored procedures).  Unlike in this simple example, make sure you are filtering your sql queries properly (ie..for sql injection attacks, etc). 

We can group our queries in a utlity class for easy access.

    public static class CustomerQueries
    {
        public static IQuery GetAll()
        {
            return new AllCustomersQuery();            
        }

        public static IQuery GetById(int id)
        {
            return new GetCustomerQuery(id);
        }
    }

The gateway class itself is simple to implement, as it only has four methods.  I'll leave that as an exercise for the reader, but here's a preview of what your code could look like after you've put it all together.

 

            int customerId = 1;
            DataRow customerInfo = gateway.Get(CustomerQueries.GetById(customerId));

By separating our queries from our gateway (which actually executes them against the database server), we have simplified a lot of other concerns.  Looking at database caching, for example, we can see that now, writing a caching layer for our application has become simpler.

 

    public class CachingDbGateway : IDbGateway
    {
        public CachingDbGateway(IDbGateway gateway)
        {
            _gateway = gateway;            
            _tableCache = new Dictionary<string, DataTable>();
            _rowCache = new Dictionary<string, DataRow>();
        }

        private IDictionary<string, DataTable> _tableCache;
        private IDictionary<string, DataRow> _rowCache;
        private IDbGateway _gateway;

        public DataRow Get(IQuery query)
        {
            string queryText = query.Text;
            if (_rowCache.ContainsKey(queryText))
            {
                //cache hit
                return _rowCache[queryText];
            }
            else
            {
                //cache miss
                DataRow row = _gateway.Get(query);
                _rowCache.Add(queryText,row);
                return row;
            }
        }

        public DataTable Find(IQuery query)
        {
            string queryText = query.Text;
            if (_tableCache.ContainsKey(queryText))
            {
                //cache hit
                return _tableCache[queryText];
            }
            else
            {
                //cache miss
                DataTable table = _gateway.Find(query);
                _tableCache.Add(queryText,table);
                return table;
            }
        }

        public void Execute(IQuery query)
        {
            _gateway.Execute(query);
        }
    }


In this example, we built a simple object which implements our gateway interface.  It takes the real gateway in as an argument to the constructor and adds a layer of functionality on top.  In fact, there's no reason why this caching gateway needs to take the real gateway either.  If you have other concerns you want implemented between your data layer and your business layer, you are free to stack these decorators on top of each other (ie.. a LoggingDbGateway).  Note that you will want this particular caching implementation to run as a singleton (as the dictionaries get initialized in the constructor). 

Since we probably don't want to cache all the queries, we might also extend our IQuery interface to support this.

    public interface IQuery
    {
        string Text { get; }
        bool Cacheable { get; }
    }


We can modify our CachingDbGateway to respect each query's caching expectations.

In summary, while not a production-ready example, this gives us a good base to build a data access layer on top of.  By splitting the query from the object which executes the queries, we can simplify our code.  Separating the responsibility of executing the query from the query itself also gives us the ability to chain different responsibilities on top of the actual execution mechanism.

posted @ Sunday, May 06, 2007 3:48 PM | Feedback (2)