ONA: ANALYSE THIS

Paul Monk on fairytales and analysis in Canberra’s secret world

            Andrew Wilkie resigned from the Office of National Assessments (ONA), in March, in protest at the Australian government’s decision to join the war against Saddam Hussein. Since then, he has accused the Prime Minister and his staff of skewing the truth in the matter, of misusing and distorting intelligence, of accepting “fairytales” from Washington, in order to remain in step with President Bush.

            The Prime Minister has responded by saying, “I don’t know on what he bases those claims. If he has got evidence of that, let him produce it, otherwise stop slandering decent people. I am denying his allegations…ONA has indicated that he had virtually no access to the relevant intelligence.”

Mr Wilkie cannot produce his evidence, of course, even assuming he had such access, since to do so would open him to prosecution under the Crimes Act.  How, then, are we to judge the matter? Those immovably opposed to the decision to overthrow Saddam Hussein will be inclined to take Wilkie’s charges at face value. The Prime Minister’s political enemies will be inclined to do so, because of their animus against him and their belief that he has misled the country on other matters.

            If you belong in either of these categories, your mind may well already be made up. However, you may be mistaken. Why? Because you are almost certainly making unwarranted assumptions about what Wilkie saw, what intelligence ONA itself had access to and, above all, about how well analysis is generally done at ONA.

A few cautionary tales from the secret world of ONA analysis might throw some light on why these assumptions need to be examined closely. I should add that none involves disclosure of intelligence materials, only exposure of errant judgements made by senior ONA analysts to my certain knowledge. These judgements were all about substantial matters and were made by analysts who did have access to the relevant intelligence.

            It is worth noting, by the way, that the sort of problems to which I am about to draw attention are by no means confined to ONA or to Canberra. When Douglas McEachin took over as Deputy Director (Intelligence) at the CIA, in 1993, by his own account, he found that roughly a third of already-published CIA reports meant to assist policymakers had “no discernible argumentation to bolster the credibility of intelligence judgments and another third suffered from flawed argumentation.”

            However, it is ONA, in particular, that we are concerned with here. My first tale dates back to mid-1990, when I was working for the Defence Intelligence Organisation (DIO), as an East Asia analyst. A draft report on China’s international position a year after Tiananmen came across from ONA, written by the organisation’s senior China analyst. He shall, of course, remain anonymous for present purposes. Three DIO officers, all senior to me, had read it before it was shown to me and none had taken issue with it in any way.

I was stunned, when I read it. It was littered with errors of fact and reasoning which I found simply incredible. I wrote a blistering set of comments on it and took it in to my boss. “Have you seen this ONA draft?”, I asked. Yes, he said, adding that he thought it was basically sound. “Well”, I declared, “I think it’s an absolute crock.” I showed him why.

A comical game of bureaucratic politics ensued, before the ONA analyst was induced to come across to DIO to discuss our differences of opinion. There were many, but I single out one. He had written, based on no intelligence of any kind, that “When the United States withdrew from Indochina, in 1975, it ceded hegemony over South East Asia to China.”

            Given leave by my DIO boss, I challenged the ONA analyst on ten counts. Apropos of this particular claim, I asked him: “When you say that the US ceded hegemony over South East Asia to China in 1975, I take it that by ‘hegemony’ you mean political domination. Over what, the Philippines? Indonesia?” “Oh!” he replied. “What I really meant was continental South East Asia.” “Ah!” I came back. “Thailand, Malaysia, Singapore?” “Well, er, what I actually had in mind was Indochina”, he offered.

            “Indochina?” I queried. “Well, that’s not exactly co-extensive with South East Asia, is it? But, correct me if I’m mistaken, it was the Soviet Union, not China, that supplied the North Vietnamese with the weapons to overrun South Vietnam. It was the Soviet-backed Vietnamese who exerted control over Laos and then overthrew China’s ally, Pol Pot, and occupied Cambodia for a decade. Where does that leave Chinese hegemony?” “Well”, he sniffed, “I don’t see that this affects the substance of my argument.” “I don’t think you’ve got an argument”, I retorted.

            There was a lot more, but this snippet will serve to convey the nature of the analysis I had encountered at the highest levels of ONA. What is even more noteworthy is that the analyst in question, evidently unhindered by any process of review within ONA, proceeded to put out the report without changing a word, specifically including the passage about Chinese hegemony over South East Asia from 1975.

            I was so thunderstruck by this that I went to see a friend who was, at that time, head of intelligence coordination for the Department of Prime Minister and Cabinet (PM&C). Had he seen this ONA report? I inquired. It appears that he had, but only in passing. The flaws in it had been no more apparent to him than to my senior colleagues at DIO, or the senior analysts at ONA who had cleared it for publication.

            New to the game, I concluded that either of three things must follow from this disturbing episode. Either ONA reports were in general not taken seriously, so that it mattered little what sweeping claims they made, however ill founded; or they were taken seriously only when an urgent matter was in hand, which did not include this particular report; or they were routinely taken seriously and actually shaped the way policy was framed in PM&C and DFAT. The problem was, I had no idea which of these unsettling inferences was correct.

             Was I dealing with an unusually incompetent or eccentric ONA officer? I have no reason to believe so. In any case, the individual in question was promoted again and again (in DFAT) in later years. My colleagues and I used to joke that onanist was short-hand for ONA analyst. Yet equally sloppy and ill-informed analytic work was done, in my experience, by various government officials on different desks in both intelligence and policy agencies.  

            Sometimes, of course, it was not sloppy, merely mistaken. I remember well how ONA’s Japan desk was trying to predict, at one point, whether Prime Minister Kaifu would be sacked or not. They completed a report on a Friday afternoon, which went into circulation first thing Monday morning. It was titled ‘Japan: Keeping Kaifu’. The problem was that Kaifu had been forced to resign on the Saturday. Chastened, but nimble, ONA issued a report a couple of days later titled ‘Japan: Not Keeping Kaifu’. We all had a chuckle, but no one felt scandalized by ONA’s error.

            By 1995, I was head of DIO China analysis. I was sent a draft ONA report on the prospects for war between China and Taiwan. The draft was almost pure judgment, with very little intelligence content at all. Its pivotal claim was that, if things came to an impasse, China might feel that it “no effective alternative but to invade Taiwan”. There was no attempt in the draft to establish that this was itself ‘an effective alternative’ for China.

            I had my staff prepare a study of the military balance across the Taiwan Strait. Our conclusion was that China lacked the capability to invade Taiwan and risked a humiliating defeat if it tried.  We stipulated the capacities required and pointed out China’s weaknesses and Taiwan’s defensive strengths. The ONA team accepted our estimate and attached it to the next draft of the report – but left the wording about invasion as an effective alternative for China totally unchanged.

            Through several drafts and many critical annotations from my office, the report’s authors doggedly avoided any change to this wording. Yet at no point did they challenge the military assessment written by my staff or delete it from their report. I was baffled by this blatant logical contradiction and was never able to extract from ONA an explanation for it.

            The same report included, in the original draft, an estimate of China’s GDP based on exchange rates, which suggested that it was actually smaller than Australia’s GDP. I challenged this, based on the argument by Nicholas Lardy and other economists that a more realistic basis for estimating China’s GDP was the purchasing power parity (PPP) method, which yielded a Chinese GDP three to four times as large as did the exchange rate based method.

            ONA’s initial response, in the second draft, was to dismiss PPP estimates as woolly and unreliable. I therefore supplied chapter and verse from Lardy’s research papers. At this point, something remarkable happened. ONA produced a third draft in which they dropped the exchange rate based figure and substituted for it a PPP based estimate twice as large as Lardy’s. They did this without acknowledging my citations from Lardy or supplying any rationale whatsoever for their change of position. I was astonished and wrote back: “What does ONA actually believe, as a matter of analytic integrity?” I never received a reply.

            I could add further tales, but I trust that the ‘moral’ here is tolerably clear. That a senior ONA analyst should claim to have seen a lot of intelligence and to have drawn a conclusion from it (which happens to be at variance with that of the government of the day) is not sufficient reason to believe that the analyst is right in his judgement and the government wrong.

            There is, of course, the possibility that, dealing with complex intelligence data, under high pressure and amid passionate international controversy, both Wilkie and  Howard’s staff misread the matter in certain particulars. Intelligence is, after all, an arcane art at the best of times; and policy making involves taking responsibility for public affairs in the light of many considerations, which often do not allow for black and white certainty about the facts of the case.

            Error comes easily in complex domains. We have intuitive, picture-forming, pattern-hungry brains that have severely limited working memory capacity. We form erroneous impressions quickly and cling to them doggedly. We find it more adrenally gratifying to take a partisan position and to accuse others of immorality and stupidity than to carefully test hypotheses. So the world goes, belief clashing with belief, policy defended against all criticism, the disillusioned resigning on the basis of innocent illusions.

Is there, then, a larger lesson to be drawn from all this? I think there is. It is that making arguments explicit and, therefore, accessible both to understanding and correction, is the foundation of both sound intelligence analysis and responsible government. Achieving this requires specialist techniques, since it is not easy and does not come naturally.

Avoiding it is the default behaviour of the fearful, the confused and, of course, the dishonest. Even the well-intentioned flounder, especially when the subject is complex and the stakes are high. Yet, if we seek clarity in argument such explicitness is indispensable – both inside and outside the secret world.