Thursday, February 14, 2008

Right-sizing the architecture

I work at a medium-sized technology company as an e-commerce architect. The company started out in the dot-com days supporting a single web site (primarily content driven). When I joined about two and a half years ago, the company decided to expand out into new sites hosting different content for different partners. As technologists, we decided break apart what was a monolithic application into separate services that could be reused amongst the sundry sites we would now be hosting. With SOA being a more accepted idiom at the time, we decided to build out a "platform" to handle our needs.

As for my role specifically in e-commerce, we have two aspects of the business: digital subscriptions and merchandise. The digital subscription e-commerce is handled in-house via home-brewed systems. The merchandise e-commerce engine and site was hosted and managed by third company. Early on in my requirements gathering for the e-commerce part of the new platform, there was talk of ditching off the external company and handling the merchandise ourselves. As the talk "seemed" rather serious, and the thought of running an e-commerce engine that handled both both merch and digital subscriptions sounded really fun, I decided to incorporate many of the merchandising aspects into my design up front. This problem number one.

As for the platform itself, it is a collection of synchronous web services and asynchronous message publishers/listeners. The web services are most SOAP-style, but there are a few REST services, as well. (I don't want to discuss the advantages of one web service technology over the other here, just the costs/implications involved with choosing one of them.) On the web services, we have a fully defined WSDL/XSD contract. For the messaging components, it's more-or-less a collection of name/value pairs (expressed as XML) going across the JMS, although some of the value elements are full XML documents conforming to a defined XSD. We found that specifying specifying the service interface and data entities via WSDL and XSD was helpful for defining the XML documents that would be exchanged between client and service, but that it's application was limited outside of that scope. This is problem number two, and the reason why it is a problem is largely dependent on problems number one.

The assumptions (and aspirations!) I started with unfortunately did not bear as much fruit as hoped. Let's start with problem number one. In my desire to design and build out a bad-ass e-commerce system, I jumped in full force. Given the requirements I had, I created what I call the "Big Vision Up Front". It's sort of a mix of Big Design Up Front (a/k/a/ the waterfall model) and the agilists' notion of YAGNI. Basically, I aggregated all the requirements and derived a vision of the future platform. This did not include hardcore UML diagrams or extensive documentation, but lots of collected thoughts about the various models and interactions amongst them. Then, as a given set of features that we knew were needed for the next project, we would build them out.

OK, this all seems cool so far, build components we need when we need them based on the original vision. Unfortunately, in the initial build out (call it phase 1), I built in more functionality, more of that big vision, than was necessary at the time. In other words, we overbuilt expecting that the unused components would be "needed soon". For some components that has turned out to be true (it may have taken a year or two, but they are finally being used). For others, it feels like random appendages hanging off in semi-sensibile directions. Of course much of this was due to my enthusiasm to create something big and great, but what I failed to recognize and understand was the nature of the business side of our, and really any, company. The business folks are, on the whole, nice people and are trying to do their jobs well. Unfortuantely, I didn't apply enough discretion to understand the business vision and goals in both the short and long terms and how I could apply my technical skills to best serve that. Hence, we overbuilt some things. Nothing awful or completely atrocious, but some design and code are growing a funky odor...

So, the lesson for problem one is to understand your business. Figure out what is really being asked for, both in the short and long term, and learn how to marry the two. Of course, this kind of knowledge only comes with experience, so I think I've begun down that path of enlightenment.

As for problem two, the purely technical matter: as part of taking over the merchandising site, we would have needed to integrate in with about 75 external partners (manufacturers, shippers, and such) and several other entities for user management, e-commerce, and miscellaneous services. With this in mind, we created full-blown WSDLs and XSDs (by hand - it's not that hard) as our service interfaces. Well, we built it, but nobody came. In fact, our only service consumers are internal. Now, having services even just for our internal clients (web-apps we host and so on) has turned out to be a great boon to seperation of concerns and being able to upgrade some of our sites without a big-bang release, there has been a development cost. We need to create the WSDL (basically define a few operations - we follow the SOAP Messaging style, not SOAP XML-RPC disaster), define the data structure in XSD, then create XML to object mapping files (we use JiBX) so everything gets populated into our domain models objects. This style isn't so bad once you've learned WSDL, XSD, JiBX, proper domain modeling, ...., but, there is a cost whenever we want to add a new operation to a service or create a new service. Everytime this happens, we keep looking at each other saying, "Something's just nor right here...". I feel that once again, we didn't understand our business fully and ended up satisfying their initial requirements but perhaps added in more infrastructure costs than what we really needed to incur.

So, the lesson for problem number two is to understand your business and know how to apply how the most cost-effective technology you can. By cost-effective, I don't necessarily mean cheapest in terms of licensing and deployment costs (although these factors cannot be totally discounted), I'm really talking about knowing your development team - the poor engineers who will have to implement and live with the architectural decisions. Ruby on Rails might (might?) be the best solution for a project in isolation, but if all your developers are Java cats with no Ruby (let alone Rails) experience, the cost of the project just went up. Similarly, the WS-* stack contains many inherent costs just to get something up and running.

To sum up, architects needs to keep eye on the business folks and the engineering resources who will be responsible for the building and ownership of the software we help them manifest. Otherwise it's a gonna be a long time in refactoring land....

No comments: