Celebrated winning the project commission: dinner for three at Rules, London's oldest restaurant. Classic English food perfectly done. We warmed up with Stilton & celeriac soup, fried whitebait and walnut & Stilton tart, moved onto venison and duck and finished with heart-stopping puddings. Thankfully a decent walk back to the car, so heads could clear.

Yesterday a planning session. We can cannibalise the shell of a similar system, which gives us system infrastructure. We'll use the RTF printing layer I wrote last year, which makes it easy to produce editable, well-formatted documents. Today I'll complete analysis of the processing rules for the first of the 8 groups of procedures, which means we'll have tested code for them. Paul will adapt the other system's GUI to provide review and editing of the data sucked from the mainframe. All that's missing is the live link to the mainframe. We announced that we would have Release 1 in place, without the mainframe link, but with everything else working for Group 1 procedures, on 24 March.

Managing change and embracing change: visited one of the customer's managers to follow up on our request for an internal programmer to help us link to the mainframe. His informal request didn't make it past the change-management processes that protect the IT resources. A formal request will be considered by the Change Board, which meets once a month. That's great we said, we just want you to know we're delivering Release 1 in just over two weeks -- without the mainframe link. Perhaps that provides a useful context for the request?

Chief output is letters and documents. How to automate acceptance testing? We could compare RTF code streams, but this is too fine-grained. Trivial changes in appearance would break the tests. We invented a new data structure, the diktat to represent output document contents. A diktat consists of a list of APL expressions, a dictionary (list of name-value pairs) and a list of text-table references. A text-table reference is either a pointer to a text fragment in a library of such fragments, or a table definition in the same library. A text fragment is just a character string with escape characters used to embed the names of values to be mapped from the dictionary. A table definition is, I think, just the name of an array from the dictionary. Executing the APL expressions in sequence (so maybe the list is a fn) gets you an RTF document object. But before you do that, ie ignoring the formatting expressions, you can compare the dictionary and text-table refs with another diktat to see if, at this granularity, the document contents match.

Notice we can edit text fragments in the library without breaking tests. Also that we can tinker with presentation without breaking tests! This a breakthrough for us.

The client has divided into 8 groups the processes we are automating. We have nearly finished the acceptance test cases for the first group. This is a cracking pace: I'm not sure we have been allocated a project code yet; I don't yet have a desk or a machine on the client's network, nor even a pass to the building.

I'm hoping that by the time I've coded the procedures in the second group I'll have enough confidence in the identification of mainframe data used to have someone start work on the interface.

Switched today to my second trainer. I'm now being trained on the procedures by 2 clerks, alternating so I can work a longer day than they do. Although they are training me in different procedures, there is enough overlap that I encounter with one rules that I have coded from the training of the other. Today we hit that for the first non-trivial time.

Revised the calculation of Single Premium penalties so that it now returns a pair of results where before it had returned one. That got today's new case to work, but when I ran the test suite we got a scatter of red lights where earlier test cases broke. Revised the programs to use the result pair and we got green lights. Took maybe 10 minutes, during which the clerk went back to signing paperwork.

I'd been concerned about evolving the code like this, having to return to fix earlier cases. In principle, the problem could grow geometrically with the test suite; in practice probably much less.

In all we had two such breaks today. The earlier one was while I coded the split -- another ten minute break. Otherwise the coding takes about as long as it would take me to make a note in English of what to do, and we just have a pause in the conversation.

The point -- in terms of XP distinctions -- is not that it takes less time to code. The point is that it takes so little time that it can be done in conversation with the user. It's all about facilitating communication. So a conversation which otherwise would produce paper notes, followed by a gap for coding, then follow-up questions, followed by reviews and acceptance tests and analysis of where my code varies from whatthey think they told me to do -- all that gets collapsed into a single conversation. Not just less time, but more freedom and far fewer errors. Or so I say.

I've looked up the London XP community website and will go to their Extreme Tuesday Club in two weeks. Will I be the only non-Java programmer there?

Paul Mansour has linked to this still experimental blog. I guess we're not in Kansas any more, Toto.

Writing code that shadows the structure of the clerks' checklists and procedures makes it easy to match what I'm doing with what they're doing. And the code cries out for refactoring! But right now isomorphism with the paper procedures is too valuable, so refactoring will wait until the procedure code has stabilised.

Some paper procedures refer to other procedures, so there is a degree of modularity already.

But many of the questions recur from procedure to procedure, and refer almost entirely to facts about a policy, so should become methods of a Policy class. AgeAt is an obvious one.

The test rig is a delight. RunSuite a mere handful of code lines; ShowSuite a couple of 1-line D fns. The clerks can read the displayed test cases without help.

In following the application code, Kim is starting to recognise and remark on primitive APL expressions, and wonders whether she could use J to support her daughter's school math. She and Sarah are intrigued by how I generate and handle collections of data in immediate-execution mode. No wonder actuaries were among the first to adopt APL.

This is exciting. Had difficulty analysing the pension valuation procedures. The clerks can perform them, but get confused trying to talk or reason about them. The documentation they use themselves turns out to be full of ambiguities resolved in practice by conversation with or among the senior clerks. Given the flakiness of their own documentation, what hope of communicating via any formal specification?

XP urges me to resolve the problem verbally. But I can't get enough certainty about a procedure to go and code it with any confidence that it will do what a clerk does. Nor can I break their procedures up into chunks small enough to tackle, because reasoning about the structure of the procedures drives them nuts. They can show me what they do, but they can't discuss it much!

Solution. I've used Dyalog namespaces to encode a hierarchical structure for policy data, in which the nodes are named after mainframe screens and the leaves after field labels. We should be able to fill structures like this either by screen scraping (yecch) or some inquiry process.

I've built a tiny but effective test rig -- the simplest thing that could possibly work. A case is represented by a namespace containing (at least) a variable lx. A RunTest fn executes the string in lx and reports passed, failed or broken. Name references in lx get resolved first in the case namespace, so into the case as variables go any values needed by the test expression. A tiny recursive D fn suffices to display the contents of the case. An APL component file holds all the cases (a test suite) and a simple fn RunSuite returns a string of 1s, 0s and ¯1s corresponding to passed, failed or broke.

So far I've accumulated a score or so test cases, all readable by the clerks, where we can see the test data and the asserted result.

Here's how I now work. Forget analyse, design, code. This is code, test, refactor. Sitting with the user, I record the test case in immediate-execution mode. Then the clerk tells me what she does, and I code the rules as she tells them to me. She generally spends more time stating the rule than I spend coding it. For example:

Go to IIMG screen 17, find all the records where field PH is zero and sum all the positive values of AMOUNT.

pol.IIMGS17.((PH=0)+.×0˜AMOUNT)

Decide whether policy payments stopped at any time. Do any of the AMOUNTs in phase 0 total the sum of all earlier AMOUNTs in phase 0?

0Ÿ.=+\pol.IIMGS17.(AMOUNT/þPH=0)

In January this year I was with Adrian & Gill Smith, working on Vector, the British APL Association's journal. Adrian & Gill live in a small village in North Yorkshire, handy for tramping snow-covered moors. The kitchen is the place to work, with laptops in comfortable range of the solid-fuel Aga stove. There I saw shoved into a pile of papers a copy of Kent Beck's Extreme Programming Explained: Embrace Change.

Ray Cannon had been drip-feeding me pearls of wisdom about Extreme Programming (XP) for a year. "Extreme Programmers test each line as soon as they've written it," he would announce. Madness, I'd think. "Extreme Programmers program in pairs." Right. I wondered whether there was such a thing as XP, or whether Ray was simply making this stuff up.

Now here was a book. Ray was definitely not making this stuff up. In any case: testing each line of code as you wrote it. Who would make up stuff like that? It turned out Ray hadn't read Beck's book, the original source book on XP. He'd read another book, one that compared XP with other processes, but didn't say a lot about what XP is. Ray had inferred what he thought he knew about about XP from the comparisons. Some of the time, he was right.

Did it matter? Apparently it did. The days I'd spent with Gill & Adrian, Scott Adams had been lampooning XP in the his Dilbert comic strip. If Dilbert was making fun of it, it must be getting around. While waiting for the morning porridge to emerge from the Aga, I dug in.

In Zen Buddhism there is a tradition of public question-and-answer sessions with a master. The monks ask questions to provoke the master to make present something of the dharma. A good question also makes the questioner look pretty good, but if you don't have a really confronting, original question, there are some proven standards to ask. One is "Why did Bodhidharma come from the West?"; that is: what is the point of all this? Another is, "What, in Zen, is worth keeping?"

A version of this had been puzzling me for some time, and Adrian and I had been chewing it over. Decades back, APL had looked like the future of computing. Now fashion had moved on. What, I wondered, in APL is worth keeping? Put another way, what is it I love so much about making software this way that I'm fighting for it rather than investing in learning C++ or Java?

One answer is the aesthetic satisfaction of extreme elegance. APL solutions can pare a problem down to its bones, and then solve it with a simple rearrangement and a fast rattle. Pure solution, no marbling. Verbose languages don't even come close.

Another is a fast, informal style of development. I'd moved away from this in the 1980s, caught up in the certainties promised by Software Engineering. Like many others, I had instead the experience of working on projects that produced more diagrams and specification documents than usable code. In retrospect I can read my own history as having been discouraged into quitting software development for over a decade. How are the flighty maulen.

Beck's book was a breath of fresh air. Or, in Adrian's kitchen, an icy draught. In his view, producing systems quickly with small teams and minimal process, as we'd done in the early days, was right. But he brought to it rigorous practices I'd never seen before, such as Test-First Design and Pair Programming. He and others were reporting the kind of astonishing results successful APL projects had been famous for, and doing it with verbose languages like Java. It seemed as if APL's lightweight development had been reincarnated. That which was worth preserving had emerged once more, or at least part of it had.

With the good people at Optima Systems, I'd been positioning APL as a vehicle for tactical systems solutions. This should have been a productive line to take. The customer is a major UK life office. Its recent mergers have left it with hundreds of systems and interfaces, while a rapidly changing regulatory environment and plunging stock market have intensified the pressure for results.

In January this year, we got the break for which we had been lobbying. Last week we got approval for what is to be an exemplary piece of software development. We are working with one of the customer's operating divisions. We have to get a useful system into production within 3 months, and it has to repay in this calendar year, out of savings from budgeted operational costs, all its development and first-year maintenance costs. Our kinda game.

It doesn't meet all the selection criteria for using XP. We're a 2-man team, and our software process isn't "broken". But I can see the potential in practices like Test-First Design for lifting our already impressive game.

We're off.