The Digital Labyrinth

Chapter 1 - Trouble in the Cathedral

Why is software development so difficult? Why is it not more like other forms of engineering? What is intrinsically so different about it? Perhaps it is just a reflection of the relative immaturity of the art - much like the problems of the early cathedral builders compared to todays civil engineers. Perhaps then better methodologies and analysis tools can improve the quality of our designs. Or is there something fundamentally different about software that prevents tried and trusted reductive design methodologies from succeeding. To answer this we need to understand the design process at an abstract level.

Chapter 2 - What is Software Engineering?

To understand the underlying essence of software engineering and computer programming we introduce the Turing Machine - the fundamental model of all computation devised by the mathematician Alan Turning in 1936. The Turing Machine is an abstract description of a simple mechanical device able to compute a specific mathematical result . Turing shows how this concept can be extended to the design of a single Universal Turning Machine (UTM) able to simulate the working of any specific Turning Machine. This Universal Turning Machine is able to computer any conceivable mathematical problem capable of being suitably codified (with some clear limitations). Even though all modern computers operate with a radically different physical and logical architecture, they can be shown to reduce logically to implementations of the Universal Turing Machine. While the UTM is hopelessly slow and inefficient for any practical real world computing, its abstract model is an ideal vehicle to explore the concept of a computer program being ‘correct’ and hence the inherent difficulties of software engineering.

Chapter 3 - What is Design?

Design is essentially about searching a VAST space of possible configurations to find the varnishingly small set of viable solutions. Search spaces can be visualised as a 3-dimensional 'landscape' where the peaks represent the fittest solutions to the problem. But how large are these search spaces? We examine three sample spaces: the Library of Babel (the space of all possible 200-page books), the Library of Turing (the space of all possible Turing Machines), and the Library of Mendel (the space of all possible genomes). These search spaces are small in comparison to those for complex design problems, and yet are VAST - requiring billions of years to be searched by our fastest computers. How is it possible to home in on the fitness peaks of the search space without resorting to blind search? Are there clues to be found in the landscape that guide us to the best solutions?

Chapter 4 - Hill Climbing with Darwin

We examine strategies for searching design space using the clues in the underlying fitness landscape. Smooth - or correlated - landscapes can be searched efficiently by simple strategies such as hill climbing or annealing. These strategies can still become 'stuck' however on local peaks - missing much higher peaks separated by valleys of 'unfitness'. Sex is a highly effective algorithm which can search space far more effectively by taking longer leaps in design space. Once the landscape becomes randomly 'jagged' however, each point in the space must be searched to find the best solution. But what affects how smooth the search landscape is?

Chapter 5 - Tuning the Lie of the Land

The degree of 'fitness' of a possible solution depends upon the values of the state variables (analogous to 'genes' in biological systems) that are the dimensions of the design search being searched. But in many real-world systems, the 'genes' do not act in isolation but are affected by each other (epistatic coupling.) Real design projects are complex multi-variable optimisation problems. Simply optimising the value of one variable can reduce the fitness of others - so reducing the overall fitness of the solution. The NK model, devised by Stuart Kauffman, provides a simple way of exploring how the degree of coupling directly affects the randomness of the fitness landscape.

Chapter 6 - The Tangled Bank

So far we have considered a single 'organism' acting in isolation; trying to maximise its own fitness in a static world. What happens when we have several (or many) such organisms completing for scarce resources in the same design space? As one organism finds a niche where it has optimal fitness, so another adapts to take advantage of this situation - so reducing the fitness of the first organism. An evolutionary 'arms race' ensues, with organisms co-evolving together. The NK model can be adapted to model this situation where the peaks in the fitness landscape are constantly moving. The fittest individuals are those that can 'keep up' with the moving peaks and maintain their viability in a changing world. But what determines how easily an organism can adapt and how is this related to the coupling between 'genes'?

Chapter 7 - Living on the Edge

How is the average 'fitness' of a population of co-evolving systems affected by the coupling between them? We examine how behaviour and adaptability changes - from static orderly systems unable to adapt to random, chaotic systems unable to settle to any viable solution. The NK model is used again to show how the average fitness of a the overall system is greatest at the point of transition from order into chaos. Living organisms exist at the very 'edge of chaos'. But how do systems find this magical balance point between order and chaos? - evolution drives them there. We show how systems which have the ability of modify their own internal coupling have a natural tendency to move towards the edge of chaos i.e. to automatically tune themselves to the fitness landscape.

Chapter 8 - On Being the Right Size

Complex organisations, such as businesses, animal societies or even distributed computer networks, are composed of individual units that are coupled and so co-evolve. Each unit has its own set of 'genes' which determine its individual fitness and behaviour. But each unit is also affected by the degree of coupling with its neighbours. How does the 'average fitness' of units change as the size of each unit and the degree of coupling changes? Do we get the best results with a small number of large units or with a large number of much smaller units? In the extreme, we contrast the 'Stalinist Regime' of a single unit with the 'Banana Republic' of chaotically competing units. Once again, the overall best fitness occurs on the transition between order and chaos. This balance point corresponds to an optimal size of organisation unit. The key to building the best and most adaptive solution then would seem to be in picking the right size of unit. Although these competing units behave in a totally selfish manner, at the right level of coupling the average fitness of all units is maximised across the system as a whole . But how do we find the right size of unit? Are the any practical applications of this?

Chapter 9 - Looking for a Better Crane

The design process depends on finding the right tools to help us explore the vastness of design space more efficiently. Tools then are like cranes which help 'lift' us through design space. Unlike 'skyhooks' though, cranes are routed in the very design space they operate within. Cranes can themselves be used to build bigger and better cranes. Sex for example is a highly effective crane discovered by natural selection. Where can we find analogues to sex in other problem domains - from software design to business organisation and strategy? The problems with conventional software systems can be explained in terms of the NK model. Today's programs are 'Stalinist' in nature and are fatally brittle in the face of change. How might new developments such as object oriented programming, genetic algorithms, self-organisation and seed A.I. help us build more flexible and reliable systems? Similarly, how should we best manage large multi-national businesses? If a top-down management results in a Stalinist Regime unable to adapt to the changing market, how far should we delegate to small autonomous units? Will the corporation really be fitter overall if its organisation departments compete fiercely with each other and without having a global corporate goal in mind?

Chapter 10 – Taking the Moral Highground

What is morality? How is it based upon objective values that can be explored using the scientific method? Are moral truths imposed upon the world by virtue of revelation from the supernatural totalitarian monster we see in the Bible and the Koran? We argue that morality is grounded solely in the human condition and provides a true and objective basis for deciding whether actions are right or wrong without appeal to divine guidance. Morality is concerned with maximising the well being of sentient beings within a complex and interconnected biosphere. As such we can consider moral decisions as optimal solutions to searching a moral fitness landscape. Does this model provide insight into how we may build a better and more ‘moral’ society based upon reason and rationality rather than fear and superstition?