The British believe they bade a graceful goodbye to empire. But Peter Marshall (left) discerns weakness behind the grace, while Patrick French reveals how Britain relied on espionage to prop up its rule in India

In March 1966, in one of the more impressive foreign policy achievements of his premiership, Harold Wilson announced plans to publish sets of official documents from important periods in Britain's foreign relations. This led to the arrival of the 12 magisterial volumes of Constitutional Relations Between Britain and India: The Transfer of Power 1942-47, the final one appearing in 1983. Trawling through literally millions of pieces of paper, a group of historians built up a remarkably detailed documentary collage of the last six years of British rule in India, showing the way in which policy was conceived, formulated and implemented.

As I worked my way through The Transfer of Power while researching a book on Indian independence and the creation of Pakistan, it became clear that one class of document had been excluded: intelligence. This was an important omission, since, like all unelected regimes, the British-controlled government of India was forced to rely heavily on surveillance and covertly obtained information in order to retain supremacy.

When I examined the archives of the India Office (the British government department that used to give the Viceroy of India his orders), I found that important categories of "public and judicial" documents had been judiciously withheld from the public. They appeared to relate in particular to a shadowy organisation called Indian Political Intelligence or IPI, which was charged with countering any kind of subversion of British rule in India, and was the only imperial intelligence agency permitted to operate out of London. I asked the declassification unit of the Foreign Office to review the status of this archive, and after more than year of prodding, it finally agreed to release it. This represents the first time that the complete working files of any British intelligence agency have been opened to public access.

É«ºÐÖ±²¥

It was not an easy task, and involved numerous telephone calls and letters to different government departments. Part of the problem, I later discovered, was that IPI also functioned as the India and Burma section of the Security Service (M15) on account of the empire being a "home" rather than a "foreign" concern, and that therefore after independence IPI was subsumed into the Security Service. Today, however, dealings with the Indian subcontinent are the responsibility of the Foreign Office and SIS (M16). This meant that all decisions which the Foreign Office took about the IPI files had to be cleared with different parts of the Whitehall bureaucracy.

I had two advantages in my efforts to get to see the papers. The first was that the IPI files belonged to a class of documents whose status was already under review, and so I intervened in a process which was already theoretically underway. The second was that I knew exactly what I was looking for; the great advantage that Whitehall has over researchers is that historians usually do not know what documents are being kept hidden from them, and so cannot make a specific request for access.

É«ºÐÖ±²¥

The release of the IPI papers does not, I think, represent a change in policy by the intelligence agencies, although there are indications that the Security Service may be about to release some of its own archive. The high administrative cost of vetting papers for declassification continues to tally with the desire of more traditional elements within the intelligence community who wish to keep files secret as a matter of principle, and as a result access is still restricted to a handful of chosen historians with links to the service. This is unlikely to change without legislation.

Although many of the released IPI files are riddled with the researcher's scourge - blank pages marked "original retained in the department under section 3/4 of the Public Records Act 1958" - my impression is that most of the archive has genuinely been released. Insulting material about certain prominent Indian nationalists has been retained, as have various documents regarding communist activity by Indians.

The main revelation in the released material is that Indian Political Intelligence was a far more significant operation that has previously been acknowledged. From around the time of the first world war until the transfer of power in 1947, the British authorities in India operated a surveillance and intelligence operation of great skill and range against the forces of nationalism. Serious political organisations such as Congress and the Muslim League were submitted to detailed research, while violent revolutionaries and smaller groups were trailed, watched and infiltrated.

It has been assumed until now that imperial intelligence operations were confined to the perpetrators of violence. According to the intelligence historian Richard Popplewell, "the intelligence agencies of the British Empire were not concerned with spying on non-violent nationalists".

É«ºÐÖ±²¥

Although this statement may have been true up until 1914, it is clear from the volume of material in the IPI collection that the policy did not last. As terrorism declined with the growth of Gandhian passive resistance, IPI's remit was widened, and by the 1930s it was pursuing all manner of people. For the last decades of British rule in India, intelligence-gathering was extended massively, since with the rapid rise of Congress it was seen as the only way of retaining the political advantage.

By the early 1940s, however, IPI had lost its grip. The files show that British rule in the subcontinent had been destabilised to a fatal degree. One of my most extraordinary discoveries was that the Congress leadership had even managed to take over substantial control of IPI's sister organisation, the Intelligence Bureau in New Delhi in 1946. By the time that Lord Mountbatten arrived as viceroy in March 1947, the intelligence machine was crippled, and because of a constitutional coup by Congress, Mountbatten was not even allowed access to the director of the Intelligence Bureau.

The IPI files are being recatalogued by the Oriental and India Office Collections of the British Library for public release later this year. I concentrated on documents relating to the 1940s, and only scratched the surface of the archive. There are substantial files on figures such as Krishna Menon, Subhas Chandra Bose and Jawaharlal Nehru which would repay many months of careful study. An ambitious PhD student could have a field day.

Patrick French is the author of Liberty or Death: India's Journey to Independence and Division, which will be reviewed in The É«ºÐÖ±²¥S on August 15.

É«ºÐÖ±²¥