On the first of January every year, history hits the headlines. That’s when the National Archives of Australia, or NAA, releases the most recent batch of cabinet records, documenting discussions at the highest level of government. This year, records from 1994 and 1995 were made available for the first time. Among other things, we learnt that prime minister Paul Keating’s government didn’t tackle climate change with great gusto.
But what about the files we’re not allowed to see? During 2017 at least 1140 government files were withheld from the public. Amid the celebrations each year, we should perhaps take time to investigate the limits of access.
Recent days have provided an example of accelerated release of cabinet documents via filing cabinet. The official process is more controlled. Under the Archives Act, files more than twenty years old (reduced from thirty years in 2010) are said to be in the “open period.” But that doesn’t mean we automatically get to see them. Before they’re released to the public, records go through a process known as “access examination.” Their contents are checked against a series of exemptions defined under section 33 of the Archives Act, which cover things like national security and individual privacy. Most records are opened without restriction. Some end up as “open with exception” and have pages removed or text redacted. A smaller number are withheld completely and are assigned the access status of “closed.”
Each January for the last three years, I’ve harvested the details of all “closed” files from RecordSearch, the NAA’s online database. RecordSearch only displays the current status of a file, so by taking annual snapshots I’m hoping to expose trends and anomalies, and build up a historical picture of the access examination process. But snapshots of an active system will always be missing data, so anomalies appear in the details presented below. As you’ll see, it’s more of an archaeological dig than a rigorous statistical analysis.
The important thing is that we can ask questions of the harvested data that we can’t ask through RecordSearch’s own search interface. For example, why are files closed? The 2016 harvest showed that the privacy exemption, section 33(1)(g) of the Archives Act, was invoked most frequently. No surprises there.
• “Closed period” was cited in 2565 cases, but a little analysis revealed that the contents of most of these files were, by 2016, in the open period. There’s no time limit or mandatory review for closed files — they stay closed until someone requests them to be re-examined. More mysteriously, there were 3410 closed files labelled “Pre access recorder.” It turned out these were files closed before the introduction of the Archives Act in 1983.
The NAA dealt with some of these oddities in 2016 by changing the access status of most “Pre access recorder” and “Closed period” files from “closed” to “not yet examined.” They’re no longer closed, but neither are they any more open. Before being released they’ll need to re-enter the access examination pipeline and may yet be closed again. Swings and roundabouts.
These changes meant that the total number of closed files in the 2017 harvest dropped dramatically by about 25 per cent. The year’s harvest didn’t show any major shifts, but the total number of closed files increased by 484 to a total of 11,235.
Number of closed files at each harvest date
1 January 2016 — 14,370
9 January 2017 — 10,751
1 January 2018 — 11,235
Comparing the reasons cited across the three annual harvests, it seems that the number of national security exemptions (33(1)(a) & (b) of the Archives Act) are trending down. Privacy (33(1)(g)) looks steady after a big jump in 2016 due to the processing of repatriation files from men and women who served in the first world war. What’s most worrying for researchers is the steady increase in closed files citing “Withheld pending advice.” These are records that have been referred to other government agencies for their advice. As we’ll see, this can cause significant delays.
• Enlarge this chart
So, what happened in 2017? As I mentioned, at least 1140 files were closed. But on the other side of the ledger, the access status of 623 files changed from “closed” to something else. Overall, there was a net increase of 517 closed files.
You might have noticed I keep fudging my findings. There were at least 1140 files closed in 2017, because it’s possible that some files were closed, then “un-closed” between my snapshots. And why un-closed? Couldn’t we just say “opened”? Unfortunately, it’s a bit more complicated. Most of the files that stopped being closed in 2017 are now “open with exception,” with parts of them still being withheld. Only 210 closed files were opened without restriction:
Number of files with new access status
Open — 210
Open with exception — 408
Not yet examined — 5
Just to add to the uncertainty, 189 files apparently went missing between harvests, while 173 mysteriously appeared carrying access decision dates prior to 2017. This is probably due to changes in the organisation and description of the records on RecordSearch, but I won’t know until I’ve examined them all in detail. Once again, it’s a case of trying to fill in the gaps between snapshots.
If we break down 2017’s “closed” and “un-closed” files by reason, we can see where the action was last year. Most of the changes were in the “Withheld pending advice,” or WPA, category. As mentioned, these files have been referred to other agencies for their input on the access examination process. They’re not finally closed — they’re in a sort of archival limbo, awaiting judgement on their access status. As a result, we’d expect a fair bit of coming and going — but the net increase of 377 files in this category indicates there might be a blockage in the pipeline.
• What sort of files are they? If we look at the 973 files closed in 2017 as WPA we see that the overwhelming majority come from just one series — A1838. This is the main correspondence series of the Department of Foreign Affairs and Trade, or DFAT, and is critical in understanding the evolution of Australia’s external relations. Of the 4215 closed files citing WPA on 1 January this year, 2121 (50.3 per cent) were waiting for DFAT.
• This will come as no surprise to anyone investigating the history of Australian foreign policy. Delays in gaining access to files from A1838 have become the stuff of horror stories for would-be researchers.
When I analysed the 2016 harvest I noted that there were 1467 files in the WPA bucket that had been waiting more than three years for a decision. If we look at the files that were “un-closed” in 2017, we can calculate the length of the access examination process. A total of 595 files emerged from the WPA backlog last year, taking on average three years and seventy-seven days to complete their journey. If we focus just on A1838, the average time taken was three years and 195 days. The quickest turnaround was just 163 days, but four files clocked up over seven years in limbo. A researcher wanting to see a file from 1967 entitled “Hong Kong — Communist Activities Within the Colony” could have finished two PhDs in the time it took for the file to be processed.
Up until last year, the NAA provided information on access examination outcomes in its annual report. But WPA files were excluded from the total number reported as “closed.” You can understand why — a final decision hadn’t been made on the fate of these records. But while they might not be finally closed, delays of three years or more make them effectively closed. This is the reality of access. A lack of transparency and accountability frustrates users and undermines confidence in the process.
In recent years the NAA has attempted to streamline access examination. But with the number of WPA files rising by about 400 per year, more needs to be done. The NAA’s 2016–17 annual report highlighted a review of the access examination process by Paul O’Sullivan, formerly a deputy secretary in DFAT and head of ASIO from 2005 to 2009. In response the NAA has, according to its director-general, “taken steps to strengthen the integrity of the process and to ensure that as much as possible is released in a timely way.” We can only wait and see.
None of this should distract us from the fact that government records more than twenty years old are expected to be open to public scrutiny. The rules in section 33 of the Archives Act are called “exemptions” for a reason. The files released on 1 January each year are not a gift to researchers. They are opened because we have the right to see them. Only by understanding the legislative and bureaucratic processes through which access is constructed can we protect these rights. •
