Extreme Programming in APL

2003-07-06T20:09:00.000Z

Posted an article The Experience of Being Understood. Requirement specification looks a lot like language acquisition. Reflecting on what Wittgenstein had to say about language acquisition suggests an unusual development practice.

2003-06-27T01:42:00.000Z

Tonight I want to say something about the effect that communicating through the code is having on the project.

When Sarah & I started working together we quickly ran up against the limits of our abilities to reason verbally about the processes we are automating. There are very many variables and cases. The calculations she described were not complicated, but the conditions for applying them were. Very. I suspected that this reflected optimisation long ago to minimise calculation, after which it just got difficult to reason across the cases. An actuary would be better trained to abstract a general framework within which to reason, but Sarah isn't an actuary.

So we tackled the case examples one by one, using control structures (IF, THEN, ELSE, &c.) in the code to relate the rules to the conditions for applying them. Case by case the control structures proliferated.

At first I refactored the code only inside the control structure elements. Had I refactored the control structures as well I should have destroyed the framework that anchored the work we were doing together. Sarah & I were already using the code as a visual representation of the conditions, to ensure we could see while we talked.

I came to see that I could refactor the control structures only where Sarah could follow the refactoring. Sometimes I have been able to teach her this. She sees how multiplying a number by a condition is the same as saying it's zero unless the condition holds. I would eventually be able to refactor to my heart's content only when we had finished and no longer needed to refer to it!

But that severely limited the benefits we could get from refactoring. Pity, because they included finding simpler abstractions from among the detail.

In retrospect it seems I did two things right at this point by luck. We both find the pair programming work as intensive as XP practitioners report it. So we break for a few minutes each hour to stretch. (At the moment, when we've achieved a milestone we've been aiming for, I pick and read aloud a poem from A.A Milne's When We Were Very Young.) I also vary the rhythm a bit by spending a few minutes every now and again deconstructing some recent expression in immediate-execution mode. So besides the variables and functions she's named herself, Sarah also has some insight into what the primitives are doing. For example, I've showed her how a Boolean expansion maps a list of numbers into a longer list, and she has used her knowledge to spot and challenge a line where this was not done as we had agreed it should be. So the first thing I think I did right was to keep discussing and explaining the code as I wrote it.

The second thing was to build upon this learning to support aggressive refactoring. Sarah likes the XP prescription for “merciless refactoring”, and enjoys seeing simplicity emerge from complexity.

When early in the project we coded the rules for calculating Market Value Reductions, we wandered in a depressingly long sequence of nested control structures. When we came recently to the rules for calculating annuities and policy fees, we reached for an array structure that would allow us to represent all the cases. This entailed analysing and recreating rules embedded in an Excel workbook, rules that Sarah doesn't know — she just uses the workbook.

We started by refactoring the Excel spreadsheets in the workbook, displaying and naming new intermediate results until we could see what varies from case to case and what is the same. This allows us to rewrite the rules in APL.

In my last post I compared the ephemeral views Excel's auditing tools provide of the information flow with the stable view from the APL code. Even after refactoring, the visual metaphor of the spreadsheet limits how much can be displayed on a monitor. Sarah made the same comparison today and said she finds the APL code considerably easier to follow than the Excel workbook.

I've posted a small Dyalog APL workspace with a working copy of the function we've been editing today. I think many programmers will find it remarkable that a non-programmer should prefer this description to an Excel workbook.

When I began this post about communicating through the code I intended to write separately about the effects on communication quality and on the code quality. But it turns out that they are closely entangled. Weak communication produced loose code which was hard to refactor. Tight communication produced dense code and lots of refactoring opportunities.

In fact Sarah is now contemplating using the insights she's gained from our work to review and refactor the manual processes we're automating.

Next post: thinking about requirements specification as a Wittgensteinian language-game suggests some interesting practices.

2003-06-24T22:32:00.000Z

Pair programming with the customer

Earlier I described how I was cutting code as part of my communication with the trainers. They tell me what the system is to do, I code that and we verify the results. In this application domain, my description in APL of what's wanted takes similar time to the trainer describing it in English. So it's like a translation exercise, translating English to APL.

At first I described this as "pair programming with the customer". Then after reflection, I said it wasn't. The aim of pair programming is to raise code quality. The aim of what the trainers and I are doing is to raise communication quality.

We've been doing this for 2 months now and it's time to look at the results. I started working with 2 or 3 trainers, for the last month with Sarah alone.

Tonight, some key practices and communicating through the code with the customer, an unexpected result. Tomorrow, some unexpected effects on communication and code quality.

Programming practices

When the code produces wrong answers Sarah and I step through it with the interpreter, examining the intermediate results. So I write the code to produce the same intermediate results that the manual procedures do.

I don't want any other intermediate results, so ideally the only variable names in the code are names for Sarah's intermediate results.

Ideally, to avoid creating any other variables, I use a single line to represent how one intermediate variable is calculated. I tend to write long lines.

I also create anonymous D-fns 'in line' to avoid creating other variables, even to avoid referring to the same variable twice in one line. (Leading me, I think towards the verb trains in J.)

I want to support skim-reading the code. In skimming code, I need to see only the flow of information between variables. In skimming, dense chains of symbols between the curly braces around a D-fn register simply as do something here.

I also use D-fns merely to highlight the flow of information. For example, where a new value is calculated from a few other variables, I often compose the line as one or a few D-fns so that in skimming the code it is clear that the new value is derived as some function of certain others.

I cannot overemphasise the value of being able to read the code at two levels.

This is similar to the use of Excel's Auditing tools, which allow you to display the precedents or dependents of cells in a spreadsheet. The tools draw arrows over the spreadsheet to show where one number is used or obtained. (You have to read the cell formulas to see exactly how. This corresponds to reading the contents of my D-fns.) But it does not take many arrows to make a big mess, so you need to turn Excel's Auditing on and off for cell after cell to trace the flow of information. In contrast, everything in the APL code is available to a visual scan. The code can be skimmed to follow the flow of information, or the eye can stop to review the detail encapsulated in a D-fn.

A related practice is to name quite small fragments of code, either a D-fn or a derived function, and comment it, so that later it can be read in context by its semantics rather than its mechanics. For example, in code we were writing today, in two places we needed to convert 3-lists into 4-lists by duplicating the first element. That was the mechanics. The semantics was 'replicate the NPR to produce a Reduced'. So we declared a function

rnr „ 2 1 1°/¨ © replicate NPR as Reduced

and used rnr in lines where the reader needs only its semantics, not its mechanics. Unsurprisingly, this semantics for the derived function works only in the context of a single function, so rnr got defined in and localised to that one function.

Communicating through the code

I discuss the code with Sarah as if she were a novice programmer. I review the information flow, pointing to the variable and function names (which she chose) and saying in English what the primitive functions are doing. Occasionally I deconstruct a primitive expression in immediate-execution mode to illustrate or confirm a point, sometimes just to share a particularly elegant or powerful expression. Much of this is in the spirit of play: playing with a complex and fascinating toy.

Today Sarah challenged a line of code. Correctly. In a few related lines, I'd been using Boolean expansions to map calculated results into parts of a collection of arrays. Suddenly Sarah leaned forward and pointed at one of them. "Shouldn't they be for all the columns?" She was right, and had been able to see that the code didn't do it.

So Sarah skim-reads the code and through it we communicate about the rules we're automating.

This is fascinating. I had similar experiences in the 1970s with customers who actually were novice APL programmers. But Sarah doesn't write any APL.

It reminds me of my ability to read Italian. In limited contexts I'm a competent reader and hearer of Italian. Drawing on menu Italian and word roots shared with English, French and Latin I've acquired a vocabulary that meets my needs on holiday and allows me to skim-read newspaper articles.

It also reminds me of the language games Wittgenstein imagined in Philosophical Investigations. Sarah & I play a language game. It doesn't involve her speaking or writing any APL; it's not the same language game that programming or learning programming is. What are the uses of code fragments in this language game?

Similarly, we couldn't play the game with just any APL code. Certain properties of my code are essential to this working. I suspect that the way it supports skimming is crucial, also the use of Sarah's nomenclature in recognisable spellings.

It would be interesting and useful to identify what those properties are and consider how they can be replicated in other languages.

I would like to pay tribute here to what Ken Iverson taught me in 1977 about teaching APL; his emphasis on exploration, hypothesis testing, Socratic dialogue and working in pairs.

2003-06-05T18:10:00.000Z

Back from Italy and XP 2003.

How do you automate in three months a procedure it takes 6 months to train a new clerk to do?

No matter how smart you think you are, you've got an analysis challenge.

A pitfall to avoid is refactoring the manual procedures. These have been optimised to to minimise human work. In this project's case, some of the complexity comes from shortcuts taken to minimise calculation. With the machine, we would prefer to optimise to minimise the number of rules. Calculation is mostly free.

So spending time in analysis to reduce the number of rules to follow -- refactoring the original process -- is tempting. But it takes time we don't have. So Plan A is always to code it exactly as the clerks do it.

I can still refactor the code without consulting the trainers. I've already done some of this. The very concise APL source code allows patterns to show up at a high level. Since we now have a useful library of test cases, I can refactor these and test. In fact, doing this allowed me to catch what I believe were misunderstandings on my part, which had not been caught by the application test.

But refactoring this way has a drawback. The first version of the application code follows the trainer's instructions precisely. So it's easy to resolve any discrepancies between our results: we use the interpreter to step through the calculation, examining the results as we go.

I can't overemphasise how useful this has been. The communication process has been dramatically shortened. Effectively I'm now pair programming with the customer, who is reading the APL code off the screen, seeing her instructions reflected in control structures. (So APL is 'unreadable'? Match that, Java programmers!)

Refactoring the original process to simplify the rules is a whole new ball game. Slavishly following the existing rules, no matter how baroque they might be, has a clear advantage. You know that it will turn out, that you will produce the same result after some predictable amount of work. In refactoring the procedures, you don't have the same assurance. It's almost the same as redesigning the procedures, except you've got a test suite to get you home. Given that the trainers understand the reality represented by their procedures, it will probably all work out in reasonable time.

These reflections prompted by several days effort spent now on redesigning a table in which intermediate results are presented to the users. Using our system to display this provoked the users to suggest a more orderly presentation than the one they use themselves. We're still working out the consequences.

I felt a twinge of unease when I heard we were doing it. The lesson learned: on a sprint project like this, refactoring the users' processes can be a significant cost.

That said, we are getting it straight and it looks like the calculation rules will come out significantly simpler, with good consequences for later work.

2003-05-13T17:12:00.000Z

The winner of the Life-70 Interface Sweepstake, Roger Bunker, has asked that the winnings be donated to the 'Cancer Research' Society. Perhaps we should have a photo of Roger on the site too !!!!!!!!! Alot of IT interaction today, which gave me an insight as to how much work goes on behind the scenes, when an interface is being put in place.

2003-05-09T21:49:00.000Z

Kevin wanted his picture on this site! Here you go, Big Guy.

2003-05-04T10:30:00.000Z

Ted Codd died.

2003-05-02T17:58:00.000Z

We've mopped up most of the rules for the iteration's output documents and have coded the easy ones. We're way ahead of of our estimated progress, so it's time for our business user to add stories.

Since we came so far by dropping out complicated low-volume quotes, we could drop them back in again. Instead, as we've recommended, we're to push on to the next types of quotes.

This will reveal sooner more of the data we need for the mainframe link.

2003-05-01T19:04:00.000Z

Sidelined analysis of annuity calculations while we find out how much of this we already have coded in other SVQ applications.

Joe started coding up output documents with Sarah and me. We hadn't expected to get here so quickly, so Sarah will take tomorrow morning to assemble a complete set of the output and mark up the variable segments. After an afternoon coding up documents with me, Joe should be able to work up the rest over a couple of days without leaving Hampshire. The limits on co-location: we generally work faster together, but the travel is a burden worth avoiding when we can.

I hear IT wants fully documented requirements for the mainframe link before any build work starts. How to reconcile this with our process? Perhaps we can exploit the 'best seams first' strategy. Since we've tackled the the most valuable processes first, we can specify with some confidence what we need for the first phase and come to some 'best efforts' agreement for a second phase, with perhaps a third phase later to mop up spills.

2003-05-01T07:38:00.000Z

Back on business rules with Joe sitting in. He started off yesterday by coding non-trivial expressions so I could keep working with Sarah. Yesterday it became clear that his application knowledge is much better than mine, so we switched places.

I'm seeing that with his application knowledge he's able to elicit processing rules with confidence that he and Sarah have understood each other. That's more efficient and less rigorous than my working through test cases until Sarah can't think of any variations we haven't covered. Let's see how we've done when we've got some test cases on file for the new rules.

Set aside coding some of the more complicated rules. Unsurprisingly, they are used infrequently. They also require information we can't grab from the mainframe, thus need GUI objects to get from user. Chose to defer spending project budget on these: we can always come back later.

2003-04-30T06:25:00.000Z

Roger wins the mainframe interface sweepstake! His was the most optimistic date (30 April) and closest to the date work started (29 April). He gets to pick the charity the stakes get donated to.

2003-04-29T19:43:00.000Z

Met our mainframe interface programmer today, and we got aligned on a lightweight approach. Once we'd relaxed a bit, she saw that she could meet our request for a 'spike' interface simply by finding an existing link. We have that now and can work out how to use it.

Pair-programming with the client again. That same fear of "this is never going to come together". But it did before, and we're already halfway through the biz rules for this iteration.

2003-04-29T06:37:00.000Z

Planning meeting yesterday for our second iteration. Results largely as expected: more business rules, more documents. Plus driving a 'spike' for the mainframe interface to demonstrate that we can link all the way through.

We have our nominated mainframe programmer! And at the main customer site! We'll have our first conversation today.

Feedback from the users from our 'releases' is useful and keep us on track; but nothing like as useful as feedback from a production system. How soon after getting a mainframe link running can we get something into production?

Back yesterday afternoon to reviewing test cases with Sarah, one of the trainers. Good to be back in close communication again.

I notice the reported XP effect. Work is faster and more intense in close communication. Last week worked again one day at Joe's home in the Hampshire countryside, then the following 2 days co-ordinating by phone and email: hardder to maintain the intensity and momentum without co-location.

2003-04-29T06:27:00.000Z

What we did last week. Back after the Easter break, but without our On-Site Customers -- school holidays still going.

Our mainframe link project has shown up, and I look like winning the sweepstake we set up for the date work starts! (One for the optimists.)

Joe and I were able to use the time to rework the exchange of data between the mainframe link I've faked and the SVQ editor.

Refactoring has left the code organisation looking less like the worksheets. I regret this and am vaguely uneasy about it, but the logic for it seems compelling, driven by what values the users might edit. Might have a 'refactoring insight' about this later.

Still having to fight for YAGNI. The habit of coding what 'we're going to need later' is still strong. It's not our money to spend like this.

2003-04-06T14:55:00.000Z

Friday we delivered Release 2.

We'd estimated that we would complete stories equivalent to 6 ideal days. We didn't get it all done: we omitted complete editing of the 'policy' information as the implications of editing derived data turned out more complicated than we'd seen. So we completed 4 days work, plus some.

What's working/not working? Code quality for the OO code I've written for handling policy information is proving robust, and easy to work with. I'm identifying errors and making changes fast; once I know what behaviour I want, I can produce it quickly.

That's less true for the SVQ system we're adapting. When I pair with Joe, the SVQ code looks clunky to me, with data and derivatives held in globals. But I don't 'have the theory of the system' to use Naur's expression.

Impressed reading Peter Naur's article on programming as theory building. I doubt the philosophical references are as relevant as the author thinks, but the central notion of a 'theory of the system' is persuasive, and so too, the assertion that code quality is measured by consistency with that theory as well as by the external behaviour of the system that it produces. Programmers operating without the theory of a system will degrade the code quality.

Paul has more of this theory than Joe; we need his advice on how best to do what we gotta do.

Still, we now have a library of policies standing in for the mainframe link. (The users and we are running a sweepstake on the date that a mainframe programmer starts work on the link.) We can calculate derived data from them and produce the relevant documents. Rough edges everywhere, but an outstanding achievement for a month's work.

Business expert users are disappearing over the Easter holiday. Me too, I'm driving my mother around Ireland. We'll pick up again after Easter.

Happy trails.

2003-03-27T23:46:00.000Z

Delivered first release on Tuesday, as promised. Just enough system to (1) verify that we'd gone 'end to end' and (2) to give the users a starting point to steer us from.

Wednesday we played the Planning Game for the first time. For several reasons it suited us to make the next iteration short: it ends Friday week! I encouraged everybodyto think short-term, not Final Solution.

We had 2 x 6 = 12 calendar days. Planning Extreme Programming suggests an initial estimate of 1:3 for ideal:calendar time. But Joe & I are a small team, so we may get 1:2. Our estimates for the user stories were 6 days, so we accepted all of them. Which then prompted a few "Oh, and another thing..." so I dealt out blank cards to everyone to make notes for next time.

So far, only Kevin, the most senior person on the business side, is writing stories. But everyone's contributing, and there's an exhilarating sense of partnership.

The biggest dampener of spirits is the uncertainty about whether we will get a mainframe programmer to help us build an interface. Fortunately we've got screenscraping as a tested fallback.

From one of the stories two trainers took on the task of identifying the 'minimum set' of data to be recorded, so that we can drive completely from a keyboard until we can get a mainframe interface. Today they reported that so many data are needed that there would be no operational advantage in doing it. It would take as long as the present methods, but be less familiar.

Fortunately, I've got time in hand from coding up the output documents -- pair programming with one of the trainers. So I will hand bake more test cases into the library and we will run THOSE through. So no production use before the mainframe link, but we can test.

Time I implemented the automated testing for the output documents. No -- I keep catching myself -- that's a story for the next planning session.

2003-03-27T22:10:00.000Z

Nothing like a 'green fields' XP project: Friday Joe and I integrated my work with the SVQ system. We had hoped we might be done by lunchtime, installed by teatime.

Nah. We'd got close by midnight, when I called a halt. After 12 hours we had entered the 'negative productivity zone'. An hour in this zone costs more than an hour later fixing the mistakes we make. Joe slept over and we nailed it on Saturday morning.

2003-03-21T09:46:00.000Z

Too busy to report project status? That's an old chestnut!

Joe & I focused on assembling pieces for Release 1. Not quite a ziffer (Zero Feature Release) but just enough to let our users steer the next iteration.

To put it another way: rather than exploration or analysis or even collecting user stories, given that we're adapting a previous system (SVQUOTE, hereinafter SVQ), we're presenting a version of SVQ with a fraction of the required logic and output installed. Next week, when we plan the next iteration, the users can tell us how much effort to spend on changing the existing SVQ features, and how much to spend on new capabilities.

2003-03-20T12:23:00.000Z

Joe joined us yesterday with experience from the SVQUOTE system. That means he can figure out how to adapt the system to handle this project, while I focus on the specifics for this project. He manages the bone and muscle, and I, the squishy innards.

He was a bit appalled at all the new things to digest: object-oriented code organisation, data and code nested into namespaces passed as arguments and file components, but is now bearing down on adapting the shell's GUI data editors.

The XP practice of adapting scope to a fixed timetable is serving us well, and keeping us focused on what can be achieved in the next few days. The training I had as a Course Supervisor at Landmark Education is making a big difference too.

I've also had an insight that sharpens the sometimes blurry distinction between unit tests and acceptance tests. If I don't understand why an acceptance test fails, the customer can help me. But if it's a unit test, she can't.

2003-03-17T23:07:00.000Z

So I cheated. XP deprecates working overtime, but I spent some hours on Sunday working on the Diktat class. This was my first attempt to practise Test-First Design, and the results were encouraging.

At first, it was hard to think of the tests. But I got the hang of it after a while. The Dyalog Explorer is really coming into its own as I define a test case in a name space c, edit its comment and lx (test expression) values and edit or define any other fns or values it needs. Then Tests.RunTest c gives me a pass or fail.

XP talks about establishing a development rhythm: write a test, run it to see it fail, write code, see it pass or fail, revise code until it passes test, write another test...

The beauty of this rhythm is how focused it keeps me. For the Compare method of the Diktat class I wrote three tests: compare a diktat with itself, with a variant where the text fragments match but the dictionary values don't, and a variant where neither match. (I can think of other tests, but they seem unnecessary.) Defining the tests first had me focus on coding just to pass those tests without thinking about other issues. That shortened coding time. When the method passed all three tests, I reviewed the code for refactoring opportunities and I was done.

Reading Wake on Extreme Programming Explored, full of practical illustrations. Sadly, all the illustrations are in (tedious, verbose) Java, but it's still possible to follow the narrative. Leaves me strongly impressed with how much faster I can follow the same paths in APL; just so much less to write or refactor.

Miki's hanging on Monday for her first exhibition (see www.mikiy.com) and for technical support called me back from sunny Surrey and the walk I had planned for after the meeting of the Vector working group. But we did take a break in the new spring weather and played frisbee on Hampstead Heath.

2003-03-14T22:31:00.000Z

Back to clients today after working solo yesterday. Classes for printing are shaping up though slower going than I had estimated. I occasionally imagine myself pair-programming the new Diktat classes. In my imagination we seem to be working faster, but then in my imagination I was working faster than this solo.

While I was away the trainers devised new test cases. With the refactored code, setting up the tests is slightly easier, and reconciling differences with the model answers is quite a bit easier.

Joe joins me next week for 2-3 days to see if we can work out how to share development. It's a slight but manageable risk to having Release 1 ready for 25 March. (We deferred by one day so I can attend a wedding out of town.)

2003-03-12T22:10:00.000Z

A day spent refactoring. Expected to spend 2 hours on it this morning, spent the whole day.

First renamed some extracted data fields to fix mistakes in my understanding of which mainframe screens they come from. That broke most of the 32 cases in the test suite, so I edited it to reflect the new names, and wrote simple tools for that as I worked. That did take about 2 hours before we were passing the tests again.

The APL calculation functions started off closely following the clerks' checklists. As I've developed them, the structure of a Policy class started to clarify in my mind. That got clear enough today that I rewrote the code around a class structure, then revised the test suite to correspond. I also took out a number of algorithmic redundancies seen in passing. By 4.30 all the tests were passing again.

Conclusion: while I had a different agenda in mind, the refactoring was better done today. I'm now in good shape for tackling the output documents tomorrow. And I would never have been able to complete such radical refactoring in a single day without the automated tests. Never, ever.

2003-03-11T18:32:00.000Z

Pair programming with the customer. The clerks training me are reading over my shoulder now, checking rules on screen. The code to them is like a foreign language in which they recognise some words and phrases. I have laid out the high-level code to resemble their own procedure checklists, with similar comments and headings. So they can find their place and look for familiar terms. And I've been careful to use their terms as I build up my vocabulary of APL functions.

Today Kim returned from a conference with other trainers to find out whether we had included a rule they had been discussing. Do we, she asked, check whether the valuation date is earlier than the third policy anniversary before the original retirement date? She scanned the screen, found her place and saw

valdate>2 yrsBefore pol.StartDate aoob4 origNRD

They can't write code, but I've been translating it into English, pointing at the words, and they follow the thread. It's like my Italian. I've never studied Italian, but I can recognise enough to say what a newspaper article is about.

I'm ruining them for traditional software development.

Got hold of an output document and its embedded macros today. Tomorrow I'll build the text library, the scribe to produce an RTF document from a diktat, and convert the Word macros into a script. Yee ha.

2003-03-11T01:02:00.000Z

Another day away from the customers: doesn't feel right any more. Visited Paul to discuss the data fields to be retrieved from the mainframe. Do we know what they are yet? Yes and no. Yes, because the clerks have reviewed the mainframe screens with me and highlighted the fields they copy. No, because that process missed some fields, which we found when they taught me some of the procedures and I coded them. I haven't coded all the procedures ergo we don't know what all the fields are yet. So I upgraded the testing rig to index the data fields from the test suite.

Also created a Policy class and started converting existing functions into its methods, revising test cases to match.

Simon Garland suggests having the mainframe data exported as a Web Service, eerily echoing the BCS-OOPS evening seminar series Peter, Paul and I've been attending.

Tomorrow (today by the time I post this) the users and I will finish the procedures for Release 1 and review the output documents and the rules used for writing them.

2003-03-07T22:26:00.000Z

Celebrated winning the project commission: dinner for three at Rules, London's oldest restaurant. Classic English food perfectly done. We warmed up with Stilton & celeriac soup, fried whitebait and walnut & Stilton tart, moved onto venison and duck and finished with heart-stopping puddings. Thankfully a decent walk back to the car, so heads could clear.

Yesterday a planning session. We can cannibalise the shell of a similar system, which gives us system infrastructure. We'll use the RTF printing layer I wrote last year, which makes it easy to produce editable, well-formatted documents. Today I'll complete analysis of the processing rules for the first of the 8 groups of procedures, which means we'll have tested code for them. Paul will adapt the other system's GUI to provide review and editing of the data sucked from the mainframe. All that's missing is the live link to the mainframe. We announced that we would have Release 1 in place, without the mainframe link, but with everything else working for Group 1 procedures, on 24 March.

Managing change and embracing change: visited one of the customer's managers to follow up on our request for an internal programmer to help us link to the mainframe. His informal request didn't make it past the change-management processes that protect the IT resources. A formal request will be considered by the Change Board, which meets once a month. That's great we said, we just want you to know we're delivering Release 1 in just over two weeks -- without the mainframe link. Perhaps that provides a useful context for the request?

Chief output is letters and documents. How to automate acceptance testing? We could compare RTF code streams, but this is too fine-grained. Trivial changes in appearance would break the tests. We invented a new data structure, the diktat to represent output document contents. A diktat consists of a list of APL expressions, a dictionary (list of name-value pairs) and a list of text-table references. A text-table reference is either a pointer to a text fragment in a library of such fragments, or a table definition in the same library. A text fragment is just a character string with escape characters used to embed the names of values to be mapped from the dictionary. A table definition is, I think, just the name of an array from the dictionary. Executing the APL expressions in sequence (so maybe the list is a fn) gets you an RTF document object. But before you do that, ie ignoring the formatting expressions, you can compare the dictionary and text-table refs with another diktat to see if, at this granularity, the document contents match.

Notice we can edit text fragments in the library without breaking tests. Also that we can tinker with presentation without breaking tests! This a breakthrough for us.

2003-03-06T00:30:00.000Z

The client has divided into 8 groups the processes we are automating. We have nearly finished the acceptance test cases for the first group. This is a cracking pace: I'm not sure we have been allocated a project code yet; I don't yet have a desk or a machine on the client's network, nor even a pass to the building.

I'm hoping that by the time I've coded the procedures in the second group I'll have enough confidence in the identification of mainframe data used to have someone start work on the interface.

Switched today to my second trainer. I'm now being trained on the procedures by 2 clerks, alternating so I can work a longer day than they do. Although they are training me in different procedures, there is enough overlap that I encounter with one rules that I have coded from the training of the other. Today we hit that for the first non-trivial time.

Revised the calculation of Single Premium penalties so that it now returns a pair of results where before it had returned one. That got today's new case to work, but when I ran the test suite we got a scatter of red lights where earlier test cases broke. Revised the programs to use the result pair and we got green lights. Took maybe 10 minutes, during which the clerk went back to signing paperwork.

I'd been concerned about evolving the code like this, having to return to fix earlier cases. In principle, the problem could grow geometrically with the test suite; in practice probably much less.

In all we had two such breaks today. The earlier one was while I coded the split -- another ten minute break. Otherwise the coding takes about as long as it would take me to make a note in English of what to do, and we just have a pause in the conversation.

The point -- in terms of XP distinctions -- is not that it takes less time to code. The point is that it takes so little time that it can be done in conversation with the user. It's all about facilitating communication. So a conversation which otherwise would produce paper notes, followed by a gap for coding, then follow-up questions, followed by reviews and acceptance tests and analysis of where my code varies from whatthey think they told me to do -- all that gets collapsed into a single conversation. Not just less time, but more freedom and far fewer errors. Or so I say.

I've looked up the London XP community website and will go to their Extreme Tuesday Club in two weeks. Will I be the only non-Java programmer there?

2003-03-05T08:44:00.000Z

Paul Mansour has linked to this still experimental blog. I guess we're not in Kansas any more, Toto.

2003-03-05T08:38:00.000Z

Writing code that shadows the structure of the clerks' checklists and procedures makes it easy to match what I'm doing with what they're doing. And the code cries out for refactoring! But right now isomorphism with the paper procedures is too valuable, so refactoring will wait until the procedure code has stabilised.

Some paper procedures refer to other procedures, so there is a degree of modularity already.

But many of the questions recur from procedure to procedure, and refer almost entirely to facts about a policy, so should become methods of a Policy class. AgeAt is an obvious one.

The test rig is a delight. RunSuite a mere handful of code lines; ShowSuite a couple of 1-line D fns. The clerks can read the displayed test cases without help.

In following the application code, Kim is starting to recognise and remark on primitive APL expressions, and wonders whether she could use J to support her daughter's school math. She and Sarah are intrigued by how I generate and handle collections of data in immediate-execution mode. No wonder actuaries were among the first to adopt APL.

2003-03-03T22:40:00.000Z

This is exciting. Had difficulty analysing the pension valuation procedures. The clerks can perform them, but get confused trying to talk or reason about them. The documentation they use themselves turns out to be full of ambiguities resolved in practice by conversation with or among the senior clerks. Given the flakiness of their own documentation, what hope of communicating via any formal specification?

XP urges me to resolve the problem verbally. But I can't get enough certainty about a procedure to go and code it with any confidence that it will do what a clerk does. Nor can I break their procedures up into chunks small enough to tackle, because reasoning about the structure of the procedures drives them nuts. They can show me what they do, but they can't discuss it much!

Solution. I've used Dyalog namespaces to encode a hierarchical structure for policy data, in which the nodes are named after mainframe screens and the leaves after field labels. We should be able to fill structures like this either by screen scraping (yecch) or some inquiry process.

I've built a tiny but effective test rig -- the simplest thing that could possibly work. A case is represented by a namespace containing (at least) a variable lx. A RunTest fn executes the string in lx and reports passed, failed or broken. Name references in lx get resolved first in the case namespace, so into the case as variables go any values needed by the test expression. A tiny recursive D fn suffices to display the contents of the case. An APL component file holds all the cases (a test suite) and a simple fn RunSuite returns a string of 1s, 0s and ¯1s corresponding to passed, failed or broke.

So far I've accumulated a score or so test cases, all readable by the clerks, where we can see the test data and the asserted result.

Here's how I now work. Forget analyse, design, code. This is code, test, refactor. Sitting with the user, I record the test case in immediate-execution mode. Then the clerk tells me what she does, and I code the rules as she tells them to me. She generally spends more time stating the rule than I spend coding it. For example:

Go to IIMG screen 17, find all the records where field PH is zero and sum all the positive values of AMOUNT.

pol.IIMGS17.((PH=0)+.×0˜AMOUNT)

or:

Decide whether policy payments stopped at any time. Do any of the AMOUNTs in phase 0 total the sum of all earlier AMOUNTs in phase 0?

0Ÿ.=+\pol.IIMGS17.(AMOUNT/þPH=0)

Because the code is terse, I can write code instead of notes and just resume the conversation. The clerk can't read the APL expressions, but can see the field labels and screen names. She can see the control structures (often deeply nested) and my variables, which I take pains to name in her vocabulary. (Terseness, functional programming style and D fns allow me to avoid storing any values she doesn't relate to.) As we reach and execute code, I translate to English.

So we code up her rules. When we're done, we run the test case, which either passes, fails or breaks. If it breaks I can fix it without stopping our session. If the test fails, we can run the code a line at a time, tracing the rules and examining values. When the test passes, we append it to the test suite and rerun all the tests to see if our code changes broke anything.

We follow this path for one kind of policy, then return and modify the rules to accommodate the next. We never need to discuss the rules as framed for all cases, only follow specific cases and ensure we don't break earlier test cases.

No analysis phase. No specification document. Just a client-developer conversation which finishes with accepted code. This is like breathing pure oxygen. I think I'm going to get a nose bleed.

2003-03-03T02:17:00.000Z

In January this year I was with Adrian & Gill Smith, working on Vector, the British APL Association's journal. Adrian & Gill live in a small village in North Yorkshire, handy for tramping snow-covered moors. The kitchen is the place to work, with laptops in comfortable range of the solid-fuel Aga stove. There I saw shoved into a pile of papers a copy of Kent Beck's Extreme Programming Explained: Embrace Change.

Ray Cannon had been drip-feeding me pearls of wisdom about Extreme Programming (XP) for a year. "Extreme Programmers test each line as soon as they've written it," he would announce. Madness, I'd think. "Extreme Programmers program in pairs." Right. I wondered whether there was such a thing as XP, or whether Ray was simply making this stuff up.

Now here was a book. Ray was definitely not making this stuff up. In any case: testing each line of code as you wrote it. Who would make up stuff like that? It turned out Ray hadn't read Beck's book, the original source book on XP. He'd read another book, one that compared XP with other processes, but didn't say a lot about what XP is. Ray had inferred what he thought he knew about about XP from the comparisons. Some of the time, he was right.

Did it matter? Apparently it did. The days I'd spent with Gill & Adrian, Scott Adams had been lampooning XP in the his Dilbert comic strip. If Dilbert was making fun of it, it must be getting around. While waiting for the morning porridge to emerge from the Aga, I dug in.

In Zen Buddhism there is a tradition of public question-and-answer sessions with a master. The monks ask questions to provoke the master to make present something of the dharma. A good question also makes the questioner look pretty good, but if you don't have a really confronting, original question, there are some proven standards to ask. One is "Why did Bodhidharma come from the West?"; that is: what is the point of all this? Another is, "What, in Zen, is worth keeping?"

A version of this had been puzzling me for some time, and Adrian and I had been chewing it over. Decades back, APL had looked like the future of computing. Now fashion had moved on. What, I wondered, in APL is worth keeping? Put another way, what is it I love so much about making software this way that I'm fighting for it rather than investing in learning C++ or Java?

One answer is the aesthetic satisfaction of extreme elegance. APL solutions can pare a problem down to its bones, and then solve it with a simple rearrangement and a fast rattle. Pure solution, no marbling. Verbose languages don't even come close.

Another is a fast, informal style of development. I'd moved away from this in the 1980s, caught up in the certainties promised by Software Engineering. Like many others, I had instead the experience of working on projects that produced more diagrams and specification documents than usable code. In retrospect I can read my own history as having been discouraged into quitting software development for over a decade. How are the flighty maulen.

Beck's book was a breath of fresh air. Or, in Adrian's kitchen, an icy draught. In his view, producing systems quickly with small teams and minimal process, as we'd done in the early days, was right. But he brought to it rigorous practices I'd never seen before, such as Test-First Design and Pair Programming. He and others were reporting the kind of astonishing results successful APL projects had been famous for, and doing it with verbose languages like Java. It seemed as if APL's lightweight development had been reincarnated. That which was worth preserving had emerged once more, or at least part of it had.

With the good people at Optima Systems, I'd been positioning APL as a vehicle for tactical systems solutions. This should have been a productive line to take. The customer is a major UK life office. Its recent mergers have left it with hundreds of systems and interfaces, while a rapidly changing regulatory environment and plunging stock market have intensified the pressure for results.

In January this year, we got the break for which we had been lobbying. Last week we got approval for what is to be an exemplary piece of software development. We are working with one of the customer's operating divisions. We have to get a useful system into production within 3 months, and it has to repay in this calendar year, out of savings from budgeted operational costs, all its development and first-year maintenance costs. Our kinda game.

It doesn't meet all the selection criteria for using XP. We're a 2-man team, and our software process isn't "broken". But I can see the potential in practices like Test-First Design for lifting our already impressive game.

We're off.