On Good Policy

A policy document states who owns it, who approved it, what version it is, and when it must next be looked at. None of those dates has passed.

This is the cheapest element to check and the one most often failed. It is also the one that makes every other element assessable, because a document with no owner has nobody to fix it and a document with no review date has no moment at which anyone is obliged to try.

The Australian National Audit Office found that approximately half of the Australian Federal Police's governance instruments were overdue, or possibly overdue, for review at the time of audit.¹ The Audit Office of New South Wales has reported policies past their scheduled review date in successive annual internal controls reports, together with findings that sixteen per cent of large agencies had not updated financial delegations and twenty six per cent held no delegation for writing off certain debts.² These are not exotic failures. They are the ordinary condition of an unmanaged library.

How fast does a document go out of date? The best quantified evidence comes from clinical guidelines, where the question has been studied directly. Shekelle et al.³ found that about half were outdated within 5.8 years, with a confidence interval of 5.0 to 6.6 years, and that only ninety per cent remained valid at 3.6 years. Working at the level of individual recommendations rather than whole documents, Martínez García et al.⁴ found that twenty five of one hundred and thirteen recommendations, or 22.1 per cent, required updating, with validity falling from ninety two per cent at one year to 77.8 per cent at four. A document that has not been examined in a decade is not merely old. It is, on the arithmetic, wrong in several places.

Standards give the answer to what a control block should contain. ISO 9001:2015 clause 7.5 requires identification, description, review and approval for documented information,⁵ and ISO 15489-1:2016 sets the equivalent requirements for records.⁶ The National Archives of Australia makes identifier, creator and date created mandatory metadata for Commonwealth records.⁷ NSW Police attaches a Document Control Sheet to its guides and procedures carrying title, subject, responsible command, authorisation level, security classification, publication date, version number, review date and record number, together with a modification history that references the decision authorising each operational change.⁸

Every requirement is either traceable to a cited provision that actually imposes it, or openly labelled as the agency's own choice.

This is a question of lawfulness, not of drafting taste, which is why it sits ahead of everything about style.

Australian administrative law is settled on the point. An agency's policy cannot displace a statute. Where the two conflict, the law applies and the policy does not. Policy must not prevent a decision maker from exercising a discretion that Parliament conferred, and it must not be applied so inflexibly that the individual case goes unconsidered.⁹ In Green v Daniels¹⁰ the departmental criteria applied to an unemployment benefit claim were held invalid because they substituted the department's own test for the one the legislation laid down. The applicant met the statutory conditions. She failed the policy. The policy was the thing that was wrong.

The Royal Commission into the Robodebt Scheme is the modern case. Its findings turned in significant part on internal guidance and public correspondence that asserted a legal position the agency could not sustain, applied by staff who had no practical way of telling which parts of what they had been given were law and which were departmental preference.¹¹

That distinction is the working heart of this element. A great deal of what sits in an operational manual is not legislative at all. Age thresholds, evidentiary standards, form requirements, escalation rules and timeframes are frequently agency decisions, and frequently reasonable ones. The defect is not that they exist. The defect is that they are written in the grammar of legal obligation, with no citation and no signal, so that neither the officer applying them nor the citizen subject to them can tell the difference. An officer who believes a policy rule is a statutory requirement will not exercise a discretion they in fact hold. A citizen who believes the same will not contest a decision they could have contested. The Commonwealth Ombudsman makes the same point in the context of automated decisions, where the distinction between what the law requires and what the agency has chosen to build becomes invisible to everyone downstream.¹²

Transcription deserves a note of its own. Copying legislative text into a policy document rather than citing it creates a copy that will not be updated when the provision is. Prescribed forms, statutory notices and warnings that must appear in a specified form are the exception, and they must be reproduced exactly. Everything else should point.

From a sentence alone, a reader can tell whether they must do it, how strongly it binds, and who exactly has to do it.

Requirement language is the most concentrated source of ambiguity in policy writing, and the easiest to fix. Two disciplines carry almost all of the value.

The first is a controlled vocabulary. RFC 2119 is the canonical statement that requirement levels must be explicit and used sparingly,¹³ and the Queensland Government Enterprise Architecture reproduces it almost verbatim as Australian government practice.¹⁴ GOV.UK publishes its own short set, distinguishing what a reader must do from what they need to do and what they can do.¹⁵ The choice of vocabulary matters less than its consistency. What breaks a document is using "must", "shall", "will", "is to", "is expected to" and "it is important that" interchangeably across a hundred pages, so that a reader learns to treat all of them as advisory.

The second is a named actor. Obligations written in the agentless passive voice have no owner. "Applications are to be assessed within twenty eight days" tells nobody that the twenty eight days are theirs. The Australian Government Style Manual requires active voice on the ground that it makes clear who is doing what,¹⁶ and agency policy frameworks that address the point require responsibilities to be attributed to a position, rank or business unit rather than to the organisation at large.¹⁷

There is some evidence that the strength of written requirements predicts what happens next. WellSAT, an instrument developed to score written organisational policy, codes each provision as absent, weakly stated, or strongly stated.¹⁸ Using it across 151 district policies, Schwartz et al.¹⁹ found that stronger and more comprehensive written policies were associated with greater implementation at school level. The association is not proof of cause, and districts that write firmer policies may simply be districts with more administrative capacity. It is nonetheless the closest thing the field has to evidence that requirement language changes behaviour.

A caution runs the other way. Converting every recommendation into a requirement is not an improvement. It creates fettering exposure where a discretion exists, and it creates unworkable obligations of the kind described in element seven. Over-binding is a defect in the same family as vagueness.

A competent officer can tell exactly when the rule applies, and exactly what to do, without ringing someone.

These two properties come from GLIA, the GuideLine Implementability Appraisal, an instrument built specifically to work out why a written recommendation fails to be acted on.²⁰ Decidability asks whether a reader can consistently determine that the conditions triggering a requirement have been met. Executability asks whether the required action is stated specifically enough that different people would do the same thing. They are the formal names for the two complaints every front line officer has about the manual.

The failure patterns are consistent. Conditions expressed in prose with two or more connectives, so that resolving a case requires reading the sentence three times. Enumerated conditions with no statement of what happens when none of them applies. Evaluative terms carrying decision weight with no elaboration anywhere in the document. Instructions built on abstract verbs, where the officer is told to ensure, manage, monitor or address something with no accompanying step, artefact or deadline. Specificity of recommendations is also one of the four AGREE II items that Hatakeyama et al.²¹ found, across 206 guidelines, together explained 72.1 per cent of the variance in overall quality judgements.

A distinction matters here, and getting it wrong produces bad advice. Some evaluative language in a regulatory scheme is deliberate and legally freighted. "Fit and proper" and "as soon as practicable" are statutory terms of art with decades of case law construing them. They are undefined in the legislation because their content comes from authority, not from a definitions clause, and a policy document that tried to define them exhaustively would be fettering the discretion they preserve. The right treatment is to identify where they sit and name the considerations that bear on them.

Agency-invented evaluative language is a different matter. "Adequate storage", "a satisfactory standard", "reasonable steps", used at a decision point with no stated considerations, no examples and no escalation path, is not preserved discretion. It is an unfinished sentence, and it is the reason two officers reach different outcomes on the same facts.

Presentation carries much of the load. Rules with several conditions and exceptions are far easier to apply from a table than from a paragraph. The Australian Government Style Manual treats sequential structure as the correct choice for step by step instruction and hierarchical structure for reference lookup, and warns that unconventional or inconsistent structures make people work harder to find and understand content.²² A procedure written as continuous narrative prose is a structural defect, not a stylistic preference.

A reader arriving with a question reaches the governing clause quickly, and that clause makes sense on its own.

Nobody reads a policy document from front to back. Readers arrive with a question, enter at a point of need, extract an answer and leave. Every structural decision should be made for that reader rather than for a hypothetical one who begins at page one.

The evidence on this is unambiguous and slightly bleak. Bakos, Marotta-Wurgler and Trossen²³, tracking 48,154 monthly visitors across ninety retail software sites, found that only one or two shoppers per thousand accessed the licence agreement at all, and most of those who did read a small fraction of it. Nielsen²⁴, reporting testing across forty two organisations, found average task success on intranets at seventy four per cent, essentially unchanged from seventy five per cent a decade earlier and below the roughly eighty per cent typical of public websites. A decade of investment in search and portal design moved nothing, because findability is a property of how content is written and organised rather than of the layer placed on top of it.

The counter-evidence is equally clear, and it comes from the rewriting studies collected by Kimble²⁵. When Federal Express restructured its operations manuals, the time taken to find information fell from five minutes to 3.6 minutes and, more importantly, the success rate rose from fifty three per cent to eighty per cent. Nearly half the searches that had previously failed began to succeed. When the Federal Communications Commission rewrote its marine radio rules, mean scores on a comprehension test rose from 10.66 out of twenty to 16.85, and the time taken to answer fell from 2.97 minutes to 1.62. When the United States Veterans Administration rewrote a standard letter, the proportion of recipients who failed to understand it fell from fifty six per cent to eleven per cent, and inbound calls on a related form letter fell from 1,128 per 750 letters sent to 192 per 710.

Those gains came from structure, ordering and heading design, not from vocabulary substitution. The Australian Government Style Manual sets out the rules concretely enough to check: exactly one top level heading, no more than three or four heading levels, no skipped levels, no orphan subheadings, no heading longer than seventy characters, and keywords in the first two or three words of every heading. It advises against question headings on the ground that they push the main idea to the end, and it treats deep hierarchies as a signal that a document is trying to do too much.²⁶

Two further properties matter and are less often discussed. The first is addressability. Every substantive provision needs a stable number that can be cited, linked to and referred to in training material, and cross references need to point at those numbers rather than at "the section above" or a page number that changes on reformatting. This is the principle behind the naming conventions used for legislation itself, where every provision carries a persistent identifier independent of how the document is rendered.²⁷ The second is self-containment. A reader who lands in the middle of a document has not read what precedes it, so a section that opens with "this requirement" or "the said officer" without saying which requirement or which officer has failed before it starts.

Sentences a reader can hold in working memory, one term for one concept, and no arithmetic required to determine an obligation.

This element is the one most often reduced to a number, and the number does not carry the meaning.

Mac et al.²⁸ gave 2,235 Australian adults the same health information written at four different reading grade levels in a four arm randomised trial. Mean knowledge scores were 9.0, 9.1, 8.9 and 9.1 out of fourteen. There were no statistically significant differences, and no differences in perceived ease, acceptability or trustworthiness either. Lowering the grade level, on its own, achieved nothing.

That result is easier to accept once you see what the formulas measure. Flesch-Kincaid, SMOG and their relatives count syllables, words and sentence lengths. As Schriver²⁹ observes in her account of the continuum from text-focused to reader-focused evaluation, a passage scores identically whether its sentences run forwards or backwards. Jarrett and Redish³⁰ set out the practical consequences at length. A formula cannot see whether a document answers the reader's question, whether the answer is findable, or whether the rule is stated before three paragraphs of history. Readability scores are useful as a triage signal for finding the worst material in a large library. They are not a verdict.

What does change comprehension has been identified precisely. Martínez, Mollica and Gibson³¹, analysing a corpus of roughly 225 million words, found that contract-style writing overuses four features: low frequency jargon, centre-embedded clauses, passive voice, and non-standard capitalisation used for emphasis. An accompanying experiment with 184 participants found that those features reduced recall and comprehension, with centre-embedding, the insertion of material between a subject and its verb, the worst offender. The follow up tested practising lawyers on the same material.³² They comprehended legalese worse as well, rated simplified contracts as equally enforceable, and preferred them. The mechanism is a limit on working memory rather than the difficulty of the underlying concepts, which is why professional expertise does not confer immunity.

Plain language is not the same thing as informality, and the two should not be confused. Linos et al.³³, across six studies with a combined sample of 67,632 including randomised field experiments in live government programmes, found that formal government letters outperformed informal ones on real compliance behaviour, with relative improvements of twenty five per cent on business self-certification, forty five per cent on emergency medical transport enrolment and twenty eight per cent on tax credit engagement. More than eighty four per cent of expert forecasters had predicted the opposite. The formal condition in those experiments varied several features together, so the finding bears on register rather than on sentence structure, but the direction is clear enough: an instrument that carries legal obligation should sound like one. The work is to simplify its structure, not to make it chatty.

The scale of the plain language task is also routinely underestimated. The International Plain Language Federation, which co-developed ISO 24495-1, reports that only about seven per cent of the standard's roughly 485 discrete ideas concern word choice.³⁴ The other ninety three per cent addresses identifying readers and their purpose, selecting content, structuring the document, designing headings, and evaluating the result with real readers. The standard's four governing principles are framed as reader outcomes rather than text properties: readers get what they need, can easily find what they need, can easily understand what they find, and can easily use the information.³⁵

Terminology discipline sits inside this element and is worth separate attention. One concept should have one term, used identically across the document and across the documents around it. Synonym drift, where a licence holder becomes a permit holder becomes the registered person, forces the reader to work out whether three things are being described or one. Every specialised term should be defined once, at or before first substantive use, and every acronym expanded on first use in each document, because readers enter documents individually rather than reading the suite in order.³⁶

The document describes work that can actually be performed, in the time and with the resources available.

This is the element that separates a document which is correct from a document which is used, and it is the one most consistently absent from written quality frameworks.

The evidence comes from process safety and from healthcare, where the consequences of a procedure that cannot be followed are visible enough to have been studied. Hale, Borys and Else³⁸, reviewing the literature on safety rules and procedures for the Institution of Occupational Safety and Health, collect the relevant surveys. In one study of four hundred chemical industry operators and managers, forty per cent said procedures were unworkable in practice, sixty two per cent said the job could not be done in the time available if procedures were followed to the letter, forty eight per cent found them too restrictive, and forty four per cent said there were simply too many. In a separate industrial survey, fifty seven per cent of workers were unaware that written procedures for their job existed at all. Dutch railway workers reported at ninety five per cent that keeping to the rules meant the work could never be finished on time, seventy nine per cent that there were too many rules, and seventy seven per cent that the rules were sometimes contradictory.

The consequences show up in incident data. A Dutch analysis of loss of containment incidents, also reported in that review, found that half were procedure related, and the breakdown is the interesting part: ten per cent involved no procedure or an unclear one, twelve per cent involved a procedure that was wrong, and twenty eight per cent involved the correct procedure not being followed. Roughly two fifths of procedure related incidents traced to a defect in the document rather than to indiscipline. The UK Health and Safety Executive reaches the same conclusion in its guidance on procedures, which treats an unusable procedure as a management failure rather than a compliance failure.³⁹

Volume is part of the same problem. Hale, Borys and Else⁴⁰ report that the sheer size of rule books in complex industries acts as a barrier to their use, and that increases in the number of procedures have been associated with worse rather than better safety performance. Adding a rule is the cheapest visible response to an incident and the one most likely to be taken, which is why document sets grow in one direction only. A suite whose size is never measured and never reduced is a suite getting steadily less usable. Degani and Wiener⁴¹, writing on flight deck procedure design, make the related point that procedures which do not match how the work is actually done invite improvisation, and that the feedback path from operator to procedure author is the control that keeps the two aligned.

Publication is not implementation. Runciman et al.⁴² found that Australian patients received care in line with guideline recommendations at fifty seven per cent of 35,573 eligible encounters, with a confidence interval of fifty four to sixty per cent, despite the guidelines being expert-endorsed and widely disseminated. Compliance ranged from thirteen per cent for alcohol dependence to ninety per cent for coronary artery disease. The variation between conditions, using the same guidelines and the same clinicians, points at the documents and the systems around them rather than at willingness. Cabana et al.⁴³ supply the framework that separates the reasons: a clinician may be unaware of a recommendation, unfamiliar with it, in disagreement with it, lacking confidence in performing it, doubtful that it will work, or blocked by circumstances outside their control. Only the first two are solved by publishing.

The authoritative version is structured content rather than an image of a document.

The way a document is encoded determines who and what can read it. Headings applied as bold body text are not headings. A table built with tab characters is not a table. A scanned page is not text at all.

The accessibility case is settled and, for Australian agencies, carries legal weight. WCAG 2.2 success criterion 1.3.1 requires that information, structure and relationships conveyed through presentation be programmatically determinable, including the relationship of each table cell to its row and column headers.⁴⁴ The obligation reaches government documents through the Disability Discrimination Act 1992 and the Commonwealth's digital service requirements.⁴⁵ A licensing eligibility table that a screen reader cannot parse is not a formatting preference. It is a barrier to a class of applicants.

The same structural work has a second payoff. Content that a screen reader can navigate is content that search, indexing and automated retrieval can navigate. Studies of document question answering find structure-aware chunking outperforming fixed-size chunking by a wide margin, and adding structured metadata to content lifts retrieval precision materially.⁴⁶ The Government Digital Service, whose estate has accumulated on the order of two hundred thousand PDFs, has published its own account of why the format is the wrong default: PDFs are harder to find and search, do not adapt to the device, behave unpredictably with assistive technology, and are updated far less often than the pages around them.⁴⁷ Its later work on serving that estate through an assistant found retrieval failing where pages were too long or poorly structured, which is the same defect surfacing in a different channel.⁴⁸

Format policy follows from this. The Australian Government Content Guide treats HTML as the default and requires a demonstrated user need before publishing as PDF.⁴⁹ Where a PDF must exist, it should be tagged, its text layer intact, its language set and its tables real. Automated extraction of tables from PDFs remains unreliable enough that decision content held only in that form should be treated as at risk.⁵⁰

The document is one declared kind of thing, it behaves like that kind of thing, and it should exist at all.

Readers cannot tell whether a document binds them unless it says what it is. A policy states a position and the mandatory minimum. A procedure states the sequence of steps. A standard states a measurable requirement. A guideline advises and may be departed from with reason. Agencies that publish this taxonomy, and attach a compliance consequence to each tier, give readers a way of calibrating everything they open. NSW Health treats policy directives as mandatory and guidelines as departures permitted only where sound reasons exist,⁵¹ and comparable frameworks operate in Queensland and in the university sector.⁵²

The taxonomy should be small. Published frameworks range from three types to fourteen, and there is no evidence that a richer taxonomy produces better governance. A set that staff cannot recite is worse than a coarse one they can. What matters is that every document declares its type, that the type comes from the published set, and that the content matches the claim. The information typing principle in technical communication makes the same argument from a different direction: mixing content types in a single unit loses the reader's focus and prevents reuse.⁵³

Content mismatch is common and diagnostic. A document labelled policy that consists mostly of numbered steps is a procedure with the wrong cover. A document labelled guideline that imposes mandatory requirements is binding readers who have been told they have discretion. A single document that combines organisational position, process detail, historical background and transcribed legislation is doing four jobs and doing none of them where a reader would look for it.

Instrument fitness also raises the question a policy suite rarely asks of itself, which is whether a written procedure is the right control at all. Some tasks are better handled by training, by a form that cannot be completed incorrectly, or by a system that will not permit the wrong action. Procedures written for tasks that did not need them are a substantial part of what makes a document set unusable.

Someone can act on it, someone can check that it was done, and someone has watched a real reader try.

This is empirically the weakest property of guidance documents, and knowing that is useful. Dijkers et al.⁵⁴, reviewing 544 rehabilitation clinical practice guidelines appraised with AGREE II, found that the Applicability domain, which covers practical tools, resource implications and monitoring criteria, averaged thirty four out of one hundred against seventy two for Scope and Purpose. Only thirty six per cent of the guidelines were recommended for use without modification, forty four per cent with modification, and twenty per cent not at all. Documents are good at saying what they are for and poor at saying how anyone would know they had been followed. Hatakeyama et al.⁵⁵ reach a compatible conclusion from the other side: the provision of advice and tools for implementation is one of the four AGREE II items that carry most of the weight in an overall quality judgement.

Three things close that gap.

The first is tools. A document that requires an action should supply or link what the reader needs to perform it: the form, the checklist, the template, the decision aid, the worked example.⁵⁶ A requirement whose associated form is described but not provided has offloaded the work of finding it onto every reader, every time.

The second is monitoring. The document should state an observable indicator of compliance, a role responsible for checking it, and a frequency or trigger. Most policy documents contain no monitoring clause at all, which means non-compliance can persist indefinitely without anyone being obliged to notice.

The third is reader testing, and it is the element that no amount of analysis substitutes for. ISO 24495-1 makes evaluation with readers a governing principle,⁵⁷ and the practice is well established in the one field that regulates it: medicines regulators require patient information leaflets to be user tested before approval, using a defined protocol and a stated pass mark.⁵⁸ The Therapeutic Goods Administration applied the same method to Australian consumer medicine information and found that the changes which moved documents to the pass benchmark were structural rather than lexical.⁵⁹ Almost no organisational policy document has ever been tested on the people who must use it. The minimum viable version costs an afternoon: write down the ten questions the document exists to answer, give three people who must use it one section each, and ask them to say in their own words what they must do. Any divergence is a finding. Record the test date in the register alongside the review date so that testing carries the same governance visibility as review.

A feedback channel is the fourth thing, and the single best diagnostic of whether a policy suite is alive. Does the document tell the reader how to raise a problem with it, and can the agency point to a change made in the last twelve months because someone did?

The corpus is usable by the reader it actually has, not the reader who has been travelling with it since the beginning.

Every set of guidance encodes an implicit model of its reader, and the model is rarely written down. When a framework develops over a decade, with foundational concepts held in one document and applied guidance spread across twenty more, that model quietly becomes someone who has followed the whole thing as it was built. That reader exists. They are the people who wrote it, plus a small number of specialists who work with the framework every week.

The reader who actually arrives is different. They did a one day induction three years ago. They meet the situation the framework governs a handful of times a year. They open a document at the moment they need it, under time pressure, often with a client in front of them. For that reader, a corpus organised as a body of knowledge to be acquired is close to unusable, and not because any individual document is badly written. It is because acting on chapter fourteen requires knowledge that was distributed across a separate foundation document, two earlier chapters, and a definitional shift that happened between one release and the next.

This is the reason the element sits here rather than earlier. Everything above assesses a document against a reader arriving at it. This one assesses the whole corpus against a reader's working life, and it is the point at which document quality stops being a property of documents.

The pattern has several recognisable symptoms.

Foundational material is held in its own document, which every applied document assumes and none restates. The foundation document is read once, at induction, and never again.

Terminology evolves across releases, so the same term carries slightly different content depending on the vintage of the chapter a reader happens to open. Nothing in either chapter says so.

Parallel document sets are maintained for different populations, with much of the guidance restated and the audience terms swapped. A practitioner who works across both must hold two nearly identical mental copies and know at every moment which one governs. Two chapters may share more than half their content and still differ in ways that matter.

Documents are organised by professional role or by framework structure rather than by the situation a reader is in, so someone with a problem must first work out which role they are occupying before they can find the guidance for it.

Nothing states the entry cost. No document says what a reader is assumed to know before opening it, or where to acquire it.

The supporting evidence is indirect but consistent. Cabana et al.⁶⁰ separate the reasons a practitioner does not follow guidance, and the first two, being unaware of a recommendation and being unfamiliar with it, are knowledge distribution failures rather than motivation failures. Hale, Borys and Else⁶¹ report an industrial survey in which fifty seven per cent of workers did not know that written procedures for their job existed at all. Carroll⁶², in the work that established minimalist instruction, found that people do not read documentation in order to learn a system; they read to act, they start in the middle, and they abandon material that requires them to build up context first. Bakos, Marotta-Wurgler and Trossen⁶³ show what happens to material that a reader is not compelled to open.

The fix is not to copy the foundation into every document, which recreates the duplication problem described in the next section. It is layering. Each applied document carries inline the minimum knowledge required to make the decision it governs, with a pointer to the fuller treatment for the reader who wants it. The test of the layer is whether an occasional user can act correctly without leaving the document, not whether the document is complete.