31 July 2009

Lost in Reflection

The time for a new project had come. I was, as usual, excited about the idea of sowing the seed of a new technological creature and bear it for the initial gestation period, until it was mature enough for someone else to take over the incubation and seeing it through to begin its autonomous digital life.

This time it was about web services. We had already decided to write them in Java, and we had a rough idea of the interfaces we were going to expose, but the rest was "carte blanche". This was particularly exciting because, even though I had worked with web services on other projects, this was the first time I was actually going to design web services from the start, so there I was, rolling up my sleeves and reading the Axis2 reference pages.

The other exciting part was that someone else was going to develop an application that would consume my web services, and due to certain timing requirements in their development lifecycle they would have to do their work in parallel with mine. This introduced an interesting challenge: the interfaces would have to be pretty much rock-solid long before any actual web services code was written.

No problem, I thought. We already knew more or less what interfaces we were going to expose, so I can literally code them in Java, build the WSDL from the compiled interfaces and ship the WSDL to the developers on the "consumer project". This way, we can then work out the web services implementation and the consumer implementation independently.

First things first: the contract is not just about the information that a specific method takes as arguments and the information it will spit out as a response. There is a bit more to it. For example, we knew that in our implementation of the web services every method should have a set of "standard" arguments in its request, like the id of the application that originated the request, or the id of the specific language requested by the consumer (e.g. EN, DE, FR, etc). We also knew every method response must include some sort of status code and status message. Finally, we wanted to also include support for digitally signing both the request and the response.

Easy, I thought, we would obviously stick all of these in the relevant super-classes in our implementation, so I created the related super-interfaces in the package that I would use to create the WSDL.

Just for clarity, it worked out to be something like this:

// Base interface for all requests and responses
public interface WebServiceMessage {
    public String getSignature();
    public void setSignature(String signature);
}

// Base interface for all requests
public interface WebServiceRequest extends WebServiceMessage {
    public String getLanguageId();
    public void setLanguageId(String languageId);
}

// Base interface for all responses
public interface WebServiceResponse extends WebServiceMessage {
    public int getStatusCode();
    public String getStatusMsg();
}

// The request for the method doSomething() in the web service
public interface SomeRequest extends WebServiceRequest {
    public int getSomeArg();
    public void setSomeArg(int someArg);
}

// The response for the method doSomething() in the web service
public interface SomeResponse extends WebServiceResponse {
    public int getSomeResult();
    public void setSomeResult(int someResult);
}

// This is the actual web service
public interface MyWebService {
    public SomeResponse doSomething(SomeRequest request);
}

Just to prove the concept, I carried on and mocked up a basic implementation for a couple more methods, then packaged and published the mock service and proceeded to give it a simple test through SoapUI. I gave SoapUI the address of my service endpoint and it quickly digested the web service's WSDL, presenting all the right methods, which was good... until I asked SoapUI to generate a sample request for one of the methods.

The request was incomplete. There were some elements completely missing. So I checked the WSDL and... shock, horror... the schema for SomeRequest only showed some members but not others!
<complexType name="SomeRequest">
        <complexContent>
          <extension base="impl:SomeRequest">
            <sequence>
              <element name="someArg" nillable="true" type="xsd:int"/>
            </sequence>
          </extension>
        </complexContent>
      </complexType>

Where were the inherited members, like the language id? Lost. Gone. No trace.

Did I use the wrong arguments in java2WSDL? Nope. It turned out that java2WSDL did not support interface inheritance. Digging around online forums and googling up "java2WSDL support inheritance" unfortunately confirmed that. Apparently this is a limitation of the reflection engine they use in Axis2, to be rectified in a future release.

This exercise has now resulted in a couple of learning points for me.

  1. even when you think you have identified your assumptions, think again: there are a lot of implicit assumptions (these are the assumptions that are not actually spelled out) in a piece of architecture, and "support for inheritance" is one of those
  2. after you have identified your assumptions, check them out: it turned out that java2WSDL supported class inheritance, but not interface inheritance
  3. hold on writing your mocks until you've been through the above: I could have saved a whole day of refactoring if I had waited mocking up the web service until I had checked the WSDL generation

Mea culpa.
M.

01 July 2009

Cloud Sweet Cloud

What's this cloud thing? Where is it? What does it do? Why should I bother? Why should I care?
I care because I'm a geek, and any new-tech stuff simply gets my unconditional attention, but aside from that I have recently found more reasons to care about it.
I am on this project with a fairly complex project whose main characteristics are:
  • zero client entry points (all web-based)
  • multi-tier architecture
  • inter-layer messaging based on HTTP or HTTPS
  • scalability is achieved by simply adding more modules
  • pretty much any kind of load balancing solution can be implemented at the client entry point
When we implement it, we figure out how many concurrent users we might expect, and size the implementation accordingly, in terms of number of processors needed, amount of memory needed, dedicated bandwidth, etc.

The key to any implementation is, indeed the number of concurrent users, but guess what... in many cases getting the project number of concurrent users is like choosing the numbers to play at the lottery: a wild guess. I've seen the same happening many through my career. It's just the way it goes.

You can call the country or regional manager, ask about the market projections for the first year, then talk with the marketing guys, see if there have been pre-registrations etc, and finally they all agree with a ballpark estimate of 'N' concurrent users. Perfect! What's the confidence level on this? Hmmm... they say pretty good, with a 20% floor and a 50% ceiling. Excellent!

So what do you do? You size for N x 2 concurrent users and proceed with the implementation. Then, of course demand will be way over the projections and your business unit will have to rush new equipment to take care of it, but until it does it will effectively lose business, and inevitably people will start pointing fingers...
Maybe sizing for N x 2 is not enough, so what do you do? Size for N x 3? N x 5? And what if the actual demand end up sticking to the projections or be less? You then have a lot of spare capacity that pulls down the return on investment. So what's the magic multiplier for sizing a projection of N concurrent users?

Of course, there is no such thing. But wait a minute...

Enter the cloud.

There have been a lot of discussion about 'what' actually is 'the cloud'. To me, there is no such thing as 'the cloud'. There are service providers. Some of the solutions are geared towards running web applications, like Google App Engine or Windows Azure. Other solutions are geared towards storage and backup, like Nirvanix. Some other solutions are geared towards data centre resources outsourcing, like Amazon EC2.
Amazon EC2 is the stuff I've been playing with and it's brilliant. The cost structure is somewhat exotic: you pay for CPU/hour, TB/month transfer rates, GB/month storage rates, etc. However, once you get to grips with this model I find it a lot more deterministic than traditional costing. A systems architect will have a good idea about these figures for system usage, so budgeting should not be a serious problem. But budgeting and costing is not why I'm a big fan of this type of cloud solution.

The reason why I like it is that in the cloud there are no such things as spare capacity, forecast errors, starved resources, supplier delays, or last-minute server re-allocation traumatic stress disorder.
Mind you, I'm not saying this s the silver bullet of all scalable implementations. It has its pros and cons, and things to consider are the loss of control over parts of your data centre, the (un)reliability of internet, remote administration costs, staff training, securing the data channels with the cloud, encription, compression, and so on.

Yes, nobody said it doesn't need any homework, and each project is different, but for me and some of my projects this is a godsend. Need more capacity for the Christmas rush online sales? No problem, let me open my ElasticFox plugin and I'll double your capacity in less than an hour, already configured, already load balanced, and fully operational... oh, and with no downtime of course. Then what? Need to downsize after the Christmas rush? No problem, let me open my ElasticFox plugin and remove some spare capacity.

Hold on, that's too simple. Let's try this... I am going to release an updated version of the sytem and planning for a good load test and a stress test to see how far I can push it, so I want a single stack system for the stress test, and a 3-stack load balanced system for the load test, then I want the same again but pointing at a different back-end (one has standard test data, the other has localised data in different languages). No problem, let me open my ElasticFox plugin and... well... you know the story.

M.