Monday, 9 June 2014

Winging it and the Peter Principle

This story from the Guardian came at me from multiple people who +1's it or tweeted it or whatever. It's good, feels true etc. Read it now, I'll wait.

Like I said, it feels so true and at one level it certainly is. I was sitting at my laptop struggling with writing some tricky code, trying to figure out why whatever ought to work wasn't and thinking 'yeah, I am totally winging it here' which was true except for the 'totally'.

We work on the edge of our expertise a lot of the time. For many of us that's where the fun is, the challenge. It makes us grow, or it tells us our limits or something. We don't even think about the things we do that are second nature.

This is related to the 'Peter Principle', the notion that we are all promoted to one level beyond our expertise, which is just another way of saying we work on the edge. The Peter Principle is about people being promoted until we find they are in a position they cannot perform in. They've been good up until now, but the latest rise up the corporate ladder was one too many. They stay there because the people who put them there are too gutless (or incompetent) to push them back down again.

Mostly this is rubbish. It assumes there is just one point where people switch from competence to incompetence and that that this cannot be changed. I suggest that every new promotion has its challenges and each one takes time to work through, but what is really true is that we work on the edge of our expertise all the time. And it is true even for those of us who don't work in a corporate environment where 'promotion' is a foreign concept. We take on challenges because they are fun.

But are we totally winging it? Of course not.

Take some basics. I code using a laptop, mouse, second monitor etc. Do I have the slightest difficultly operating the mouse, getting it to point where I want and so on? No. It's not a trivial skill. Try teaching it to someone over 80 who hasn't used a mouse before. The development environment I prefer is Eclipse and I use it to write Java and C++ code. Navigating the file structures in that environment is something I do without hardly a thought. And, of course, those two languages have syntax which is as natural to me as speaking English. This isn't blowing a trumpet. Anyone who programs in this environment works this way.

A little while ago I reorganised my projects to build with maven instead of ant/ivy. I didn't know maven and I hate reading manuals. I generally read a little and try it, fall over, do a little more reading (just enough, I'm lazy) and get past that problem and so on. I was winging it the whole time. Now I use maven every day, and I don't think about it.

So it is true that we wing it all the time, but that is because we like working on the edge, but we're only on that edge because we have a rump of expertise to be on the edge of. So winging it, sure, but not totally winging it.

Sunday, 1 June 2014

Madura Demos

I've re-deployed my demos. This is more interesting than it sounds, well I hope it is anyway. I've already posted about each of them. This is about the pain of getting them up there and running in the cloud.

Long ago I had them running on CloudFoundry and they worked fine, then the CloudFoundry business model changed and their free option went away. Since I make no money from this stuff the demo site has to be free, so I went to AppFrog who were running the same CloudFoundry software and they lived there for a while...until the same thing happened.

Recently, after completing the Workflow project, I had some free time so I thought I'd try and get them going on Google Application Engine. But I gave up on that yesterday (yes, it was only yesterday) and loaded them onto OpenShift (Red Hat's cloud offering). They worked first time. It was so easy. I was surprised.

The reason I was surprised was that it was so very hard to do Google Application Engine. There were two main issues.

First, GAE doesn't like Logback, which is a commonly used logging mechanism and I use it everywhere (because commonly used means it should work everywhere, right?) GAE only likes JUL and no one I know uses JUL. JUL is the native Java logging mechanism and it is the reason for products like Logback, ie people hate JUL and want something else.

But switching to JUL was not a big deal. I wouldn't want to have to work with it, but these demos are already debugged so that's mostly okay.

Next GAE needs lots of classes to be serializable so it can flick sessions between servers. I see the sense of this and I went off and made all my classes serializable. It didn't take very long, actually. Of course I tried it out using GAE's Eclipse plugin to make sure it was working and it was.

But when I deployed it for real it complained of more classes that needed to be serializable. However these aren't my classes, they're libraries from other people. Spring etc. I can't change them (well, not unless I want to maintain them ever after, and I don't). I suspect there is a solution to that particular problem but I'd noticed that the only way to find out any of this stuff is with the remote deploy. The local test told me nothing. This was true of the Logback issue too. So I realised this might go on forever and I bailed.

Openshift, in contrast, was quite happy with Logback and didn't care about Serialization. That may cost them something in efficiency, but it sure got up and running fast.

There seem to be two general approaches with Openshift: using Git and using scp. The Git option assumes you store your project source on their Git repository and they build and deploy from there when you tell them. Sounds fine, but I already have a public Git repository at GitHub, and my maven build works just fine. I don't feel a pressing need to learn their build syntax etc, though it is likely easy enough.

The scp option is basically 'I have this war file, upload it and deploy it'. Great! That's what I did. Worked first time. I suspect the running applications are not as fast as they might be but for a free service I can live with that. I'll even listen to arguments that GAE's serialization requirements really help Google deliver a faster application. But working trumps speed and I have it working.

Madura Pizza Order Demo

I'd meant to post an entry about putting my demos on line so people can play with them and not have to build the software. Not everyone, it seems, is actually a software engineer :-). Then I realised I had never actually posted anything about the basic pizza order demo. So here it is.

The purpose is to show off the Madura software I've been working on some time now. All of it can be found on my Github repositories, all open source and, with one exception, all of it free. The idea with Madura is that it makes use of rich business objects (Orders, Pizzas, Employees etc) to remove code from the application. These objects are very rich, in that they can include business rules that monitor what is going on. Most people do this with application logic and UI logic which is harder to write and maintain than what we have here.

The demo needs to be run from Firefox. It works on IE6 (I don't run Windows normally but I had an old version lying around). Don't use Chrome because that will show you the mobile UI, which is a different demo. I guess I should add don't run it on your mobile device either. It works, but it isn't the demo I'm about to describe.

We are supposedly ordering a pizza (but don't worry, no pizza will be delivered). There are several pizza related products but the most interesting is the Configured Pizza, which dynamically adjusts the choices to be compatible with what has already been picked. The premise is that only some combinations of topping, base and size of pizza are allowed and the Madura-driven UI only ever
presents valid choices, dynamically altering the fields as necessary.

Click on the URL and log in as admin/admin. You should see something like this:
We'll configure a pizza in a moment, but first click on the Customer button.

Notice the four buttons at the bottom and that one of them, Dynamic,
is disabled. Enter the name 'fred' (without the quotes) in the Name field and tab off it. The Dynamic button is now active. Change 'fred' to something else. The button is disabled again.

What is going on here? Do we have some custom javascript or custom java driving this? Actually no. That's what other people do.

There is a rule defined using Madura Rules which controls whether the button is disabled or not. This is the rule:

rule: Customer "dynamic" {
    if (name == "fred") {
        dynamic = true;


 Normally this kind of thing gets done in something like Java, and you'd write code to test if the name field was 'fred', and you'd also write code to test if it was not 'fred' and then set dynamic to false. One of the nice things about rules is that their converse is implied without having to write it, so this is all we need.

The next demo step is to enter something into the email, eg 'abc'. Now you see an orange '!' beside email. This is the usual way Vaadin indicates an error. Roll over it with the mouse and you will see

Failed email: label=email not a valid email address, attempted=abc 

which is the default error message for this kind of error. The email field
requires an '@' sign so this is a simple validation error. There actually isn't a proper rule for this because it is too simple. We just add an Email annotation on the field and the rest happens. Naturally you can customise that error message and you can add I18n alternatives (which means it can show up in French etc if you need it to).

Also notice that the Save button is now disabled. This happens automatically because there is an error. You don't even have to write a rule. When the error is cleared the Save button will enable.

Change the email field to something valid and click the Save button to get back to the home page. You have a list of products on the left hand side. Expand the Sides heading and click on Boneless Chicken.
There isn't any configuration for this product, just a price. But note
that the message about the shopping cart now says it contains one item. This message is controlled by the following rules:

rule: Order "shoppingcartsize" {
    if (count(orderItems) > 0) {
        shoppingCartStatus = format("shopping.cart.status",count(orderItems));

rule: Order "shoppingcartsize" {
    if (count(orderItems) == 0) {
        shoppingCartStatus = format("shopping.cart.status.empty",0);

We could make this simpler if we were happy with saying 'Shopping cart contains 0 items', then we would only need one rule. But 'Shopping cart is Empty' looks nicer. The 'shopping.cart.status...' references are to a properties file that holds the actual messages we need, including the error messages mentioned above. The French messages are in a similar file, so if your user was French it would say "Le panier est vide" when the shopping cart is empty. To be clear, the language thing is just ordinary Java stuff, though it is rarely applied this well.

Now press the Save to Order button to keep the product in the shopping cart. The Cancel button would remove it so don't press that. The next step is to expand Pizzas and pick Configured Pizza.
This is a more complex form. There is a date field, which like all the other fields, is a standard Vaadin widget. The amount at the bottom is zero, until we pick enough options for the rules to work out a price.

But the first thing to note, which you cannot see very well on the form, is that this form is generated using exactly the same Java as the Boneless Chicken one. The only difference is that object it maps to is different. This is part of the 'rich objects' thing. The objects themselves have enough information in them for the Java code to build us a form. So defining a new product doesn't involve writing more Java (necessarily). It just needs the right object and the right rules.

Now we start configuring our pizza. Change the default Base to 'Puff' and pick Size='Medium'.
There is a new field called testing now and it has a red asterisk beside it, which is standard Vaadin for indicating a required field. The Save to Order button is disabled because we now have a required field that is not filled in. Finally you have an amount calculated. This is the relevant rule:

rule: Pizza "p3" {
    if (size == "Medium") {


amount = 15;

activate(testing) makes the field visible and require(testing) makes it required. There are no rules to disable the Save to Order, that is handled in the same way as the error in the Email field, ie automatically.

But there is more. Check the options for Topping. They should be Italian and Spanish. Now clear the Size (and notice the testing field vanishes and the button re-enables). Check the options for Topping again. There are three others. The lists of valid values are being dynamically constrained depending on what else is picked. This is independent of the order they are picked, so if you pick Topping first then Size will be constrained and if you pick Size first then Topping will be constrained.

The rules behind this aren't convenient to express in the if/then format we've seen so far, and they are awful in an ordinary language like Java. Instead we use decision tables. The format of the table in this case is XML but it could be a database table or, if you're prepared to write some code, something else. I won't bore you with the XML here. Conceptually the table looks like this:

topping          size
Seafood            Small
Italian               Medium
Spanish             Medium
Hawaiian          Large
Greek                Large

Pick some valid combination, press Save to Order then press Checkout.
This just lists the resulting order items and their prices with a total. The total is
calculated using a rule:

formula: Order "sum" {
    amount = sum(orderItems.amount);

This is fired whenever we add or remove an item from the orderItems list, or if we change the amount of something already in there.

Part of the richness of the objects is that they include permissions on the fields. For example the admin user you are logged in as has all permissions. Another user operator/operator has slightly less capability. For example he doesn't have access to the address field of the customer so when he goes to that form it doesn't show up. There is no code to write to handle that, it is just part of the object definition. Similarly if you type 'fred' into the name field the Dynamic button doesn't enable. Operator doesn't have permission to use the Dynamic button, so regardless of the value of the name it won't enable even if the rule tries to enable it. Declared permissions trump rule permissions.

All of this, apart from making the application a whole lot simpler is taking us to an interesting question. Since the business objects contain all the logic how come the UI can know what is going on?

It does it by knowing only that the objects are rich, that it can query them for labels and currently valid picks (the pizza sizes and toppings) and if they are required or not etc. It does not ever know anything about the rules, including the simple validation things like the Email validation. And this actually makes it brilliant when we want to implement another user interface.

Usually when a business decides to implement a new interface, such as something that runs on a mobile platform, people discover they've implemented a lot of business logic in their UI and they have to rewrite it for the new system. Then they have to keep the two in sync, doubling the maintenance effort. But we don't have to do any of that. We just write the new UI and leave the business rules where they are.

To prove the concept I went ahead and wrote another Vaadin UI for the pizza order. If you browse to the same URL with your iPad or Android phone or a Chrome browser you'll see it. The login is a bit fiddly, you may need to press the user and password fields to activate them, depending on your device. They are over on the right.

This mobile demo is a bit simpler, it just does the configured pizza so there's no need for a blow-by-blow walk through. You know enough now to try picking different options and see what happens. It ought to be much the same as before.

We said it before but it is worth saying again. Although the UI is different all the rules and objects are the same. The only thing we had to change was the UI code. But we do mean the bare UI. We did not have to write code to restrict the choices based on earlier selections, nor did we have to write any code to track the total order or make the submit button (actually the Log Out button here) disabled until all the required fields are complete. All of those things are managed behind the UI with the same code as before. This shows the power of keeping the business logic clearly separated from the UI, and Madura is clearly an effective way to do that.

Wednesday, 21 May 2014

Deploying an Eclipse Update Site

I have been working on an Eclipse Plugin to support people who use my rules
engine and workflow engine. These both require you to specify stuff in a text file, rules in one case and process definitions in another. There is syntax to get right and the plugin provides a helpful editor that tells you about syntax errors, suggests things you might need to enter, colours keywords etc. It isn't nearly as smart as the Java editor in Eclipse but it should help.

Some people might be surprised I've done it this way rather than a pointy-clicky user interface that non-programmers can use to define their workflows and rules. But I have been around this stuff a while now and

I have yet to see a non-programmer actually write any rules or any workflow. Instead I have seen pointy-clicky user interfaces that are too dumb to actually do the job because some things are just too hard to make that easy.

Also, even if non-programmers did get directly involved, they would have to learn how to handle issues like version control and rollout which would mean they would turn into... programmers.

So these products accept reality and make it as easy as possible for programmers, especially Java programmers, to work with them. Hence the need for an Eclipse plugin that helps with the editing. I use Eclipse all the time, have done for years, partly because of the way people extend it with plugins. There are alternatives to Eclipse but one plugin is enough for now.

Actually writing the plugin is pretty simple. It took less than a week to throw it together. The writing and testing part just needs one Eclipse plugin project and you write the code and tell it to run. That launches a second copy of Eclipse with the plugin installed and you try it out, step through with the debugger etc. Easy stuff.

Deploying it however, that was much more complicated.

The way the people who designed the plugin architecture would have me do it is to create several different projects which are interrelated. I need one for the plugin, of course, another for the 'feature' which contains the plugin, another for the 'update site' which builds the repository other people can use to download from. If I use Tycho, which is what I'm supposed to do if I want to build with Maven, then I need another couple of projects as well. That seems too complicated, at least for a single plugin project. If I had several plugins I wanted to roll into one site then maybe, there would never be more than 3 'overhead projects' but...not this time.

Instead I cooked up an Ant script which builds everything in one project. That should make it very simple for anyone who wants to pull my source code from Github and try building it. One complication is that the build.xml file (which is what Ant looks for by default) kept getting deleted by the Eclipse wizards so I changed its name to build1.xml. The file is a bit messy still but it does run okay and it delivers an update site in the 'site' subdirectory.

That means I can commit the site directory and use the Github project as its own public update site. So people who don't care about the source code and just want to install the plugin binaries into their copy of Eclipse can point to there and get it. It took me ages to work that out. I was expecting to deploy it into Sonatype, which is where I deploy all my other binaries, but they are Maven binaries and Eclipse update sites have a different structure. Fair enough, but there were hints around that it was possible which made me keep looking too long. Anyway, this approach is easy and seems to work fine.

Thursday, 1 May 2014

Madura Workflow

Here's what I have been working on recently: Madura Workflow.

At this point your eyes glaze over, of course, but let me assure you that this stuff is really neat, even though you almost certainly don't need it.

Over the decades I've come across quite a few Workflow products, sometimes they're called Business Process Engines and such like. There may be a problem with specific definitions I've never got to the bottom of, anyway I was never happy with them. So here's my effort.

You use workflow when you're building the kind of computer application that needs multiple tasks done by different users, some of them external users, at different times, possibly over weeks or months. Maybe you need to handle a complex insurance application which needs several actuaries to look at different aspects of it, you might want to send parts of it out to external consultants and get a response back, some of those might be automated. The application's route through your system might need decisions made which influence its path. For example if some amount is larger than a cutoff point the application may have to go to a supervisor who is authorised to handle it rather than the lowly minion it would go to by default. It might need to send a credit check request off to the credit reference agency who have an automated response system, and they charge for each request so you want to avoid sending to them if you don't need to. Choices again.

The process definition works through the steps, but under that there are tightly integrated rules provided by my Madura Rules Engine. This means that you don't have to write code to make it all work. The rules fire automatically (tightly integrated, see?) and stuff just happens. This ensures the process definition is not polluted by a load of non-relevant logic. In other products like this they often are polluted, you see, and it means business rules are spread all over the place in the system, making it hard to find what you have to change. There are validation rules, process rules, UI handling rules and so on. In this they are all in Madura Rules.

We can send Web Service messages out to external services and process the responses. Again this is done with no code. We use Spring Integration to handle this at the moment, though the connection point to that is very thin and it could be replaced by something similar. Anyway, it means that, unlike many workflow systems, the process definition is not polluted by details of the messaging.

When you have a task that needs user interaction you can present a form generated from the data structure (again no code required) for the user to complete. Naturally the form is connected to an object that is monitored by Madura Rules. So you get all the validation and error handling etc you could want. You can, if you want something more specific, write some Java+Vaadin code for a specific form.

You'll gather by now that implementing a process is pretty simple because it is detached from the generally nasty business of defining application logic, UI code and messaging. That's the idea but there's another trick here too. In practice these kinds of coordinated processes can take weeks or months to pass through a business system. They might even get passed back and forth between two or more people several times. What happens during that time if you want to change the process definition, or the rules, or the form or the messages?

Obviously I wouldn't raise the issue unless I had good news. Madura Bundle allows us to package all of that stuff into a separate bundle file that is hot-swappable without having to restart the server(s). Existing process instances remember what bundle they started with and they keep using that. Newly launched processes use the latest bundle. Easy. So, for example, if that cut-off point mentioned above needed to change you could just regenerate the bundle with a new value.

If you've read this far you might want to take a look at All this stuff is open source and, except for commercial implementations of the rules engine, entirely free.

There's a UI which is fairly complete based on Vaadin, but the underlying workflow library is agnostic to the UI technology, so if you prefer something else go for it. I already said the messaging platform, Spring Integration, can be switched for something else and actually even Madura Rules could be replaced because it is implemented as a plugin to Madura Objects and you could replace it with another plugin.

Monday, 7 April 2014

Strange Java issue with reflection and statics

Yesterday I found myself struggling with an odd Java problem. I suspect it is a Java bug, though I have deeply, deeply ingrained resistance to blaming anything on my environment. I can hear university lecturers and early employers pointing out 'it is always you' when I naively suggested that possibility years ago. But this just may be the exception.

I'm using reflection to discover the data type of the first parameter on this method:

public void init(Email annotation, PropertyMetadata propertyMetadata)

in this case Email is actually an annotation rather than an ordinary class. It has a full type of

But when I locate the init method using reflection and get the type of the first parameter like this:

Class p = (Class)method.getParameterTypes()[0];

the type I get is


rather than

and this causes my validation engine to ignore validating email fields. I have other kinds of validation (eg Length) which work perfectly well, the type for Length is returned correctly no problem. So I went over the code to see what the difference was. The good news is I eventually found it, the bad news is it doesn't really make a lot of sense.

The init method shown above lives in a class called EmailValidator. Again, I have a list of these such as LengthValidator, RegexValidator and so on. They all look the same, except that EmailValidator misbehaves. The one difference is that EmailValidator has some static Strings defined. It looks like this:

public class EmailValidator implements FieldValidator
    private static String ATOM = "[^\\x00-\\x1F^\\(^\\)^\\<^\\>^\\@^\\,^\\;^\\:^\\\\^\\\"^\\.^\\[^\\]^\\s]";
    private static String DOMAIN = "(" + ATOM + "+(\\." + ATOM + "+)*";
    private static String IP_DOMAIN = "\\[[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\]";

See? Just ordinary  static Strings. They don't seem to have any relationship to the init method. But they do. If I make them non-static (ie remove the 'static' from the definition) then... it all comes right. The datatype comes as
as it should. There is no downside to making the statics into ordinary fields because these classes are singletons, instantiated only once. And, of course the validation now works. 

The result, when I edit '@' out of Amy's email address and tab off the field shows the error signal and disables the 'save' button, which is what is supposed to happen if there is an error. When I roll the mouse over the error signal it pops up with more detail. The totally cool thing about this is that all I need to do to add validation to the field is annotate it with @Email, everything else just happens (as long as those static fields are changed).

Friday, 4 April 2014

The End of Windows XP

Microsoft have announced they will stop supporting Windows XP around now. Apparently there is still a lot of it about. It was a pretty good operating system in that it was stable and didn't use too much resource. So people hung onto it (including Mrs).
It won't suddenly stop working, of course, but there won't be any new patches to plug security holes that inevitably get found and exploited. That's really what 'support' means. And I'm listening to a discussion on the radio about your options if you are still running it.
The obvious thing to do is pay a couple of hundred dollars to MS and get a copy of Windows 8, install that (it will be a clean install, not an upgrade) and learn to use a whole new interface. Except that Windows 8 probably won't run on your old computer because W8 needs more resources (memory, disk space, CPU). Your old 3rd party software will run okay though, probably, if there is enough resource.
What they didn't mention on the radio is another option: Linux.
Here's why you should consider Linux, specifically if you are running on an older machine you want to look at Lbuntu. Here is why:
  • It will run on your existing machine, you don't need to upgrade your hardware. I'm running it just fine on a machine I bought in 2005.
  • It looks enough like XP to make you feel comfortable enough, probably more like it that Windows 8 does.
  • It will probably run your existing 3rd party software. No guarantees, but Word and Excel run just fine under a compatibility product called Wine. Setup is simple.
  • It is free. This point would be unimportant if it did not work, but it does work, so I'll say it again FREE (as in beer). You just download it. You don't need your credit card.
  • Support is not going away and there is a vibrant community of helpful people. I've never yet had to ask anything, a quick Google always finds someone else with my question, and the answer.
Yes, you will have to do a complete re-install to get there, but you were up for that anyway. Linux is seriously simple to install these days, at least as easy as Windows and it happily auto-detects your hardware and sets it up without any fuss.
Lbuntu is one of many 'packaging' options for Linux and you can find lots of others, but Lbuntu was specifically designed to look more like XP to the user and to be lighter weight, ie to run on old machines.
You can even give it a quick try out. Download the CD image from here and burn it to a CD, then boot from the CD. The CD gives you the option of running stand-alone (without touching your disk drive, it is all in memory) or installing a system on your disk. Use option one to give it a try.
When you decide you like what you see you can install it alongside your Windows system but you are better off putting in a new disk and installing onto that. The reason I recommend this is so that you absolutely know that you aren't going to press the wrong button somewhere and overwrite you existing system.
You can then put your old drive into a USB enclosure and copy your data across to the new drive.
All this is no more hassle than installing Windows 8, far less if it saved you a hardware upgrade.