Happy summer and holidays

Dear HTCaN friends,

As I break for some summer holidays, I wanted to wish you a good period of rest and fun.

I hope to see you again in the blog in September or October. During academic year 2022/23, I will be mainly blogging about draft chapters of my forthcoming monograph on “Digital technologies and procurement governance. Gatekeeping and experimentation in digital public governance”, and related topics. I hope we will have interesting exchanges about the ideas for the book.

Until then, all best wishes for the rest of the summer,
Albert

© liebeslakritze / Flickr.

Digital technologies, hype, and public sector capability

© Martin Brandt / Flickr.

By Albert Sanchez-Graells (@How2CrackANut) and Michael Lewis (@OpsProf).*

The public sector’s reaction to digital technologies and the associated regulatory and governance challenges is difficult to map, but there are some general trends that seem worrisome. In this blog post, we reflect on the problematic compound effects of technology hype cycles and diminished public sector digital technology capability, paying particular attention to their impact on public procurement.

Digital technologies, smoke, and mirrors

There is a generalised over-optimism about the potential of digital technologies, as well as their likely impact on economic growth and international competitiveness. There is also a rush to ‘look digitally advanced’ eg through the formulation of ‘AI strategies’ that are unlikely to generate significant practical impacts (more on that below). However, there seems to be a big (and growing?) gap between what countries report (or pretend) to be doing (eg in reports to the OECD AI observatory, or in relation to any other AI readiness ranking) and what they are practically doing. A relatively recent analysis showed that European countries (including the UK) underperform particularly in relation to strategic aspects that require detailed work (see graph). In other words, there are very few countries ready to move past signalling a willingness to jump onto the digital tech bandwagon.

Source: PWC, The strategic use of public procurement for innovation in the digital economy (2021: 9).

Some of that over-optimism stems from limited public sector capability to understand the technologies themselves (as well as their implications), which leads to naïve or captured approaches to policymaking (on capture, see the eye-watering account emerging from the #Uberfiles). Given the closer alignment (or political meddling?) of policymakers with eg research funding programmes, including but not limited to academic institutions, naïve or captured approaches impact other areas of ‘support’ for the development of digital technologies. This also trickles down to procurement, as the ‘purchasing’ of digital technologies with public money is seen as a (not very subtle) way of subsidising their development (nb. there are many proponents of that approach, such as Mazzucato, as discussed here). However, this can also generate further space for capture, as the same lack of capability that affects high(er) level policymaking also affects funding organisations and ‘street level’ procurement teams. This results in a situation where procurement best practices such as market engagement result in the ‘art of the possible’ being determined by private industry. There is rarely co-creation of solutions, but too often a capture of procurement expenditure by entrepreneurs.

Limited capability, difficult assessments, and dependency risk

Perhaps the universalist techno-utopian framing (cost savings and efficiency and economic growth and better health and new service offerings, etc.) means it is increasingly hard to distinguish the specific merits of different digitalisation options – and the commercial interests that actively hype them. It is also increasingly difficult to carry out effective impact assessments where the (overstressed) benefits are relatively narrow and short-termist, while the downsides of technological adoption are diffuse and likely to only emerge after a significant time lag. Ironically, this limited ability to diagnose ‘relative’ risks and rewards is further exacerbated by the diminishing technical capability of the state: a negative mirror to Amazon’s flywheel model for amplifying capability. Indeed, as stressed by Bharosa (2022): “The perceptions of benefits and risks can be blurred by the information asymmetry between the public agencies and GovTech providers. In the case of GovTech solutions using new technologies like AI, Blockchain and IoT, the principal-agent problem can surface”.

As Colington (2021) points out, despite the “innumerable papers in organisation and management studies” on digitalisation, there is much less understanding of how interests of the digital economy might “reconfigure” public sector capacity. In studying Denmark’s policy of public sector digitalisation – which had the explicit intent of stimulating nascent digital technology industries – she observes the loss of the very capabilities necessary “for welfare states to develop competences for adapting and learning”. In the UK, where it might be argued there have been attempts, such as the Government Digital Services (GDS) and NHS Digital, to cultivate some digital skills ‘in-house’, the enduring legacy has been more limited in the face of endless demands for ‘cost saving’. Kattel and Takala (2021) for example studied GDS and noted that, despite early successes, they faced the challenge of continual (re)legitimization and squeezed investment; especially given the persistent cross-subsidised ‘land grab’ of platforms, like Amazon and Google, that offer ‘lower cost and higher quality’ services to governments. The early evidence emerging from the pilot algorithmic transparency standard seems to confirm this trend of (over)reliance on external providers, including Big Tech providers such as Microsoft (see here).

This is reflective of Milward and Provan’s (2003) ‘hollow state’ metaphor, used to describe "the nature of the devolution of power and decentralization of services from central government to subnational government and, by extension, to third parties – nonprofit agencies and private firms – who increasingly manage programs in the name of the state.” Two decades after its formulation, the metaphor is all the more applicable, as the hollowing out of the State is arguably a few orders of magnitude larger due the techno-centricity of reforms in the race towards a new model of digital public governance. It seems as if the role of the State is currently understood as being limited to that of enabler (and funder) of public governance reforms, not solely implemented, but driven by third parties—and primarily highly concentrated digital tech giants; so that “some GovTech providers can become the next Big Tech providers that could further exploit the limited technical knowledge available at public agencies [and] this dependency risk can become even more significant once modern GovTech solutions replace older government components” (Bharosa, 2022). This is a worrying trend, as once dominance is established, the expected anticompetitive effects of any market can be further multiplied and propagated in a setting of low public sector capability that fuels risk aversion, where the adage “Nobody ever gets fired for buying IBM” has been around since the 70s with limited variation (as to the tech platform it is ‘safe to engage’).

Ultimately, the more the State takes a back seat, the more its ability to steer developments fades away. The rise of a GovTech industry seeking to support governments in their digital transformation generates “concerns that GovTech solutions are a Trojan horse, exploiting the lack of technical knowledge at public agencies and shifting decision-making power from public agencies to market parties, thereby undermining digital sovereignty and public values” (Bharosa, 2022). Therefore, continuing to simply allow experimentation in the GovTech market without a clear strategy on how to reign the industry in—and, relatedly, how to build the public sector capacity needed to do so as a precondition—is a strategy with (exponentially) increasing reversal costs and an unclear tipping point past which meaningful change may simply not be possible.

Public sector and hype cycle

Being more pragmatic, the widely cited, if impressionistic, “hype cycle model” developed by Gartner Inc. provides additional insights. The model presents a generalized expectations path that new technologies follow over time, which suggests that new industrial technologies progress through different stages up to a peak that is followed by disappointment and, later, a recovery of expectations.

Although intended to describe aggregate technology level dynamics, it can be useful to consider the hype cycle for public digital technologies. In the early phases of the curve, vendors and potential users are actively looking for ways to create value from new technology and will claim endless potential use cases. If these are subsequently piloted or demonstrated – even if ‘free’ – they are exciting and visible, and vendors are keen to share use cases, they contribute to creating hype. Limited public sector capacity can also underpin excitement for use cases that are so far removed from their likely practical implementation, or so heavily curated, that they do not provide an accurate representation of how the technology would operate at production phase in the generally messy settings of public sector activity and public sector delivery. In phases such as the peak of inflated expectations, only organisations with sufficient digital technology and commercial capabilities can see through sophisticated marketing and sales efforts to separate the hype from the true potential of immature technologies. The emperor is likely to be naked, but who’s to say?

Moreover, as mentioned above, international organisations one step (upwards) removed from the State create additional fuel for the hype through mapping exercises and rankings, which generate a vicious circle of “public sector FOMO” as entrepreneurial bureaucrats and politicians are unlikely to want to be listed bottom of the table and can thus be particularly receptive to hyped pitches. This can leverage incentives to support *almost any* sort of tech pilots and implementations just to be seen to do something ‘innovative’, or to rush through high-risk implementations seeking to ‘cash in’ on the political and other rents they can (be spun to) generate.

However, as emerging evidence shows (AI Watch, 2022), there is a big attrition rate between announced and piloted adoptions, and those that are ultimately embedded in the functioning of the public sector in a value-adding manner (ie those that reach the plateau of productivity stage in the cycle). Crucially, the AI literacy and skills in the staff involved in the use of the technology post-pilot are one of the critical challenges to the AI implementation phase in the EU public sector (AI Watch, 2021). Thus, early moves in the hype curve are unlikely to translate into sustainable and expectations-matching deployments in the absence of a significant boost of public sector digital technology capabilities. Without committed long-term investment in that capability, piloting and experimentation will rarely translate into anything but expensive pet projects (and lucrative contracts).

Locking the hype in: IP, data, and acquisitions markets

Relatedly, the lack of public sector capacity is a foundation for eg policy recommendations seeking to avoid the public buyer acquiring (and having to manage) IP rights over the digital technologies it funds through procurement of innovation (see eg the European Commission’s policy approach: “There is also a need to improve the conditions for companies to protect and use IP in public procurement with a view to stimulating innovation and boosting the economy. Member States should consider leaving IP ownership to the contractors where appropriate, unless there are overriding public interests at stake or incompatible open licensing strategies in place” at 10).

This is clear as mud (eg what does overriding public interest mean here?) but fails to establish an adequate balance between public funding and public access to the technology, as well as generating (unavoidable?) risks of lock-in and exacerbating issues of lack of capacity in the medium and long-term. Not only in terms of re-procuring the technology (see related discussion here), but also in terms of the broader impact this can have if the technology is propagated to the private sector as a result of or in relation to public sector adoption.

Linking this recommendation to the hype curve, such an approach to relying on proprietary tech with all rights reserved to the third-party developer means that first mover advantages secured by private firms at the early stages of the emergence of a new technology are likely to be very profitable in the long term. This creates further incentives for hype and for investment in being the first to capture decision-makers, which results in an overexposure of policymakers and politicians to tech entrepreneurs pushing hard for (too early) adoption of technologies.

The exact same dynamic emerges in relation to access to data held by public sector entities without which GovTech (and other types of) innovation cannot take place. The value of data is still to be properly understood, as are the mechanisms that can ensure that the public sector obtains and retains the value that data uses can generate. Schemes to eg obtain value options through shares in companies seeking to monetise patient data are not bullet-proof, as some NHS Trusts recently found out (see here, and here paywalled). Contractual regulation of data access, data ownership and data retention rights and obligations pose a significant challenge to institutions with limited digital technology capabilities and can compound IP-related lock-in problems.

A final further complication is that the market for acquisitions of GovTech and other digital technologies start-ups and scale-ups is very active and unpredictable. Even with standard levels of due diligence, public sector institutions that had carefully sought to foster a diverse innovation ecosystem and to avoid contracting (solely) with big players may end up in their hands anyway, once their selected provider leverages their public sector success to deliver an ‘exit strategy’ for their founders and other (venture capital) investors. Change of control clauses clearly have a role to play, but the outside alternatives for public sector institutions engulfed in this process of market consolidation can be limited and difficult to assess, and particularly challenging for organisations with limited digital technology and associated commercial capabilities.

Procurement at the sharp end

Going back to the ongoing difficulty (and unwillingness?) in regulating some digital technologies, there is a (dominant) general narrative that imposes a ‘balanced’ approach between ensuring adequate safeguards and not stifling innovation (with some countries clearly erring much more on the side of caution, such as the UK, than others, such as the EU with the proposed EU AI Act, although the scope of application of its regulatory requirements is narrower than it may seem). This increasingly means that the tall order task of imposing regulatory constraints on the digital technologies and the private sector companies that develop (and own them) is passed on to procurement teams, as the procurement function is seen as a useful regulatory mechanism (see eg Select Committee on Public Standards, Ada Lovelace Institute, Coglianese and Lampmann (2021), Ben Dor and Coglianese (2022), etc but also the approach favoured by the European Commission through the standard clauses for the procurement of AI).

However, this approach completely ignores issues of (lack of) readiness and capability that indicate that the procurement function is being set up to fail in this gatekeeping role (in the absence of massive investment in upskilling). Not only because it lacks the (technical) ability to figure out the relevant checks and balances, and because the levels of required due diligence far exceed standard practices in more mature markets and lower risk procurements, but also because the procurement function can be at the sharp end of the hype cycle and (pragmatically) unable to stop the implementation of technological deployments that are either wasteful or problematic from a governance perspective, as public buyers are rarely in a position of independent decision-making that could enable them to do so. Institutional dynamics can be difficult to navigate even with good insights into problematic decisions, and can be intractable in a context of low capability to understand potential problems and push back against naïve or captured decisions to procure specific technologies and/or from specific providers.

Final thoughts

So, as a generalisation, lack of public sector capability seems to be skewing high level policy and limiting the development of effective plans to roll it out, filtering through to incentive systems that will have major repercussions on what technologies are developed and procured, with risks of lock-in and centralisation of power (away from the public sector), as well as generating a false comfort in the ability of the public procurement function to provide an effective route to tech regulation. The answer to these problems is both evident, simple, and politically intractable in view of the permeating hype around new technologies: more investment in capacity building across the public sector.

This regulatory answer is further complicated by the difficulty in implementing it in an employment market where the public sector, its reward schemes and social esteem are dwarfed by the high salaries, flexible work conditions and allure of the (Big) Tech sector and the GovTech start-up scene. Some strategies aimed at alleviating the generalised lack of public sector capability, e.g. through a GovTech platform at the EU level, can generate further risks of reduction of (in-house) public sector capability at State (and regional, local) level as well as bottlenecks in the access of tech to the public sector that could magnify issues of market dominance, lock-in and over-reliance on GovTech providers (as discussed in Hoekstra et al, 2022).

Ultimately, it is imperative to build more digital technology capability in the public sector, and to recognise that there are no quick (or cheap) fixes to do so. Otherwise, much like with climate change, despite the existence of clear interventions that can mitigate the problem, the hollowing out of the State and the increasing overdependency on Big Tech providers will be a self-fulfilling prophecy for which governments will have no one to blame but themselves.

 ___________________________________

* We are grateful to Rob Knott (@Procure4Health) for comments on an earlier draft. Any remaining errors and all opinions are solely ours.

Algorithmic transparency: some thoughts on UK's first four published disclosures and the standards' usability

© Fabrice Jazbinsek / Flickr.

The Algorithmic Transparency Standard (ATS) is one of the UK’s flagship initiatives for the regulation of public sector use of artificial intelligence (AI). The ATS encourages (but does not mandate) public sector entities to fill in a template to provide information about the algorithmic tools they use, and why they use them [see e.g. Kingsman et al (2022) for an accessible overview].

The ATS is currently being piloted, and has so far resulted in the publication of four disclosures relating to the use of algorithms in different parts of the UK’s public sector. In this post, I offer some thoughts based on these initial four disclosures, in particular from the perspective of the usability of the ATS in facilitating an enhanced understanding of AI use cases, and accountability for those.

The first four disclosed AI use cases

The ATS pilot has so far published information in two batches (on 1 June and 6 July 2022), comprising the following four AI use cases:

1. Within Cabinet Office, the GOV.UK Data Labs team piloted the ATS for their Related Links tool; a recommendation engine built to aid navigation of GOV.UK (the primary UK central government website) by providing relevant onward journeys from a content page, with the aim of helping users find useful information and content, aiding navigation.

2. In the Department for Health and Social Care and NHS Digital, the QCovid team piloted the ATS with a COVID-19 clinical tool used to predict how at risk individuals might be from COVID-19. The tool was developed for use by clinicians in support of conversations with patients about personal risk, and it uses algorithms to combine a number of factors such as age, sex, ethnicity, height and weight (to calculate BMI), and specific health conditions and treatments in order to estimate the combined risk of catching coronavirus and being hospitalised or catching coronavirus and dying. Importantly, “The original version of the QCovid algorithms were also used as part of the Population Risk Assessment to add patients to the Shielded Patient List in February 2021. These patients were advised to shield at that time were provided support for doing so, and were prioritised for COVID-19 vaccination.”

3. The Information Commissioner's Office has piloted the ATS with its Registration Inbox AI, which uses a machine learning algorithm to categorise emails sent to the Information Commissioner's Office’s registration inbox and to send out an auto-reply where the algorithm “detects … a request about changing a business address. In cases where it detects this kind of request, the algorithm sends out an autoreply that directs the customer to a new online service and points out further information required to process a change request. Only emails with an 80% certainty of a change of address request will be sent an email containing the link to the change of address form.”

4. The Food Standards Agency piloted the ATS with its Food Hygiene Rating Scheme (FHRS) – AI, which is an algorithmic tool to help local authorities to prioritise inspections of food businesses based on their predicted food hygiene rating by predicting which establishments might be at a higher risk of non-compliance with food hygiene regulations. Importantly, the tool is of voluntary use and “it is not intended to replace the current approach to generate a FHRS score. The final score will always be the result of an inspection undertaken by [a local authority] officer.”

Harmless (?) use cases

At first glance, and on the basis of the implications of the outcome of the algorithmic recommendation, it would seem that the four use cases are relatively harmless, i.e..

1. If GOV.UK recommends links to content that is not relevant or helpful, the user may simply ignore them.

2. The outcome of the QCovid tool simply informs the GPs’ (or other clinicians’) assessment of the risk of their patients, and the GPs’ expertise should mediate any incorrect (either over-inclusive, or under-inclusive) assessments by the AI.

3. If the ICO sends an automatic email with information on how to change their business address to somebody that had submitted a different query, the receiver can simply ignore that email.

4. Incorrect or imperfect prioritisation of food businesses for inspection could result in the early inspection of a low-risk restaurant, or the late(r) inspection of a higher-risk restaurant, but this is already a risk implicit in allowing restaurants to open pending inspection; AI does not add risk.

However, this approach could be too simplistic or optimistic. It can be helpful to think about what could really happen if the AI got it wrong ‘in a disaster scenario’ based on possible user reactions (a useful approach promoted by the Data Hazards project). It seems to me that, on ‘worse case scenario’ thinking (and without seeking to be exhaustive):

1. If GOV.UK recommends content that is not helpful but is confusing, the user can either engage in red tape they did not need to complete (wasting both their time and public resources) or, worse, feel overwhelmed, confused or misled and abandon the administrative interaction they were initially seeking to complete. This can lead to exclusion from public services, and be particularly problematic if these situations can have a differential impact on different user groups.

2. There could be over-reliance on the QCovid algorithm by (too busy) GPs. This could lead to advising ‘as a matter of routine’ the taking of excessive precautions with significant potential impacts on the day to day lives of those affected—as was arguably the case for some of the citizens included in shielding categories in the earlier incarnation of the algorithm. Conversely, GPs that identified problems in the early use of the algorithm could simply ignore it, thus potentially losing the benefits of the algorithm in other cases where it could have been helpful—potentially leading to under-precaution by individuals that could have otherwise been better safeguarded.

3. Similarly to 1, the provision of irrelevant and potentially confusing information can lead to waste of resource (e.g. users seeking to change their business registration address because they wrongly think it is a requirement to process their query or, at a lower end of the scale, users having to read and consider information about an administrative process they have no interest in). Beyond that, the classification algorithm could generate loss of queries if there was no human check to verify that the AI classification was correct. If this check takes place anyway, the advantages of automating the sending of the initial email seem rather marginal.

4. Similar to 2, the incorrect prediction of risk can lead to misuse of resources in the carrying out of inspections by local authorities, potentially pushing down the list of restaurants pending inspection some that are high-risk and that could thus be seen their inspection repeatedly delayed. This could have important public health implications, at least for those citizens using the to be inspected restaurants for longer than they would otherwise have. Conversely, inaccurate prioritisations that did not seem to catch more ‘risky’ restaurants could also lead to local authorities abandoning its use. There is also a risk of profiling of certain types of businesses (and their owners), which could lead to victimisation if the tool was improperly used, or used in relation to restaurants that have been active for a longer period (eg to trigger fresh (re)inspections).

No AI application is thus entirely harmless. Of course, this is just a matter of theoretical speculation—as could also be speculated whether reduced engagement with the AI would generate a second tier negative effect, eg if ‘learning’ algorithms could not be revised and improved on the basis of ‘real-life’ feedback on whether their predictions were or not accurate.

I think that this sort of speculation offers a useful yardstick to assess the extent to which the ATS can be helpful and usable. I would argue that the ATS will be helpful to the extent that (a) it provides information susceptible of clarifying whether the relevant risks have been taken into account and properly mitigated or, failing that (b) it provides information that can be used to challenge the insufficiency of any underlying risk assessments or mitigation strategies. Ultimately, AI transparency is not an end in itself, but simply a means of increasing accountability—at least in the context of public sector AI adoption. And it is clear that any degree of transparency generated by the ATS will be an improvement on the current situation, but is the ATS really usable?

Finding out more on the basis of the ATS disclosures

To try to answer that general question on whether the ATS is usable and serves to facilitate increased accountability, I have read the four disclosures in full. Here is my summary/extracts of the relevant bits for each of them.

GOV.UK Related Links

Since May 2019, the tool has been using an algorithm called node2vec (machine learning algorithm that learns network node embeddings) to train a model on the last three weeks of user movement data (web analytics data). The benefits are described as “the tool … predicts related links for a page. These related links are helpful to users. They help users find the content they are looking for. They also help a user find tangentially related content to the page they are on; it’s a bit like when you are looking for a book in the library, you might find books that are relevant to you on adjacent shelves.“

The way the tool works is described in some more detail: “The tool updates links every three weeks and thus tracks changes in user behaviour.” “Every three weeks, the machine learning algorithm is trained using the last three weeks of analytics data and trains a model that outputs related links that are published, overwriting the existing links with new ones.” “The average click through rate for related links is about 5% of visits to a content page. For context, GOV.UK supports an average of 6 million visits per day (Jan 2022). True volumes are likely higher owing to analytics consent tracking. We only track users who consent to analytics cookies …”.

The decision process is fully automated, but there is “a way for publishers to add/amend or remove a link from the component. On average this happens two or three times a month.” “Humans have the capability to recommend changes to related links on a page. There is a process for links to be amended manually and these changes can persist. These human expert generated links are preferred to those generated by the model and will persist.” Moreover, “GOV.UK has a feedback link, “report a problem with this page”, on every page which allows users to flag incorrect links or links they disagree with.” The tool was subjected to a Data Protection Impact Assessment (DPIA), but no other impact assessments (IAs) are listed.

When it comes to risk identification and mitigation, the disclosure indicates: “A recommendation engine can produce links that could be deemed wrong, useless or insensitive by users (e.g. links that point users towards pages that discuss air accidents).” and that, as mitigation: “We added pages to a deny list that might not be useful for a user (such as the homepage) or might be deemed insensitive (e.g. air accident reports). We also enabled publishers or anyone with access to the tagging system to add/amend or remove links. GOV.UK users can also report problems through the feedback mechanisms on GOV.UK.“

Overall, then, the risk I had identified is only superficially identified, in that the ATS disclosure does not show awareness of the potential differing implications of incorrect or useless recommendations across the spectrum. The narrative equating the recommendations to browsing the shelves of a library is quite suggestive in that regard, as is the fact that the quality controls are rather limited.

Indeed, it seems that the quality control mechanisms require a high level of effort by every publisher, as they need to check every three weeks whether the (new) related links appearing in each of the pages they publish are relevant and unproblematic. This seems to have reversed the functional balance of convenience. Before the implementation of the tool, only approximately 2,000 out of 600,000 pieces of content on GOV.UK had related links, as they had to be created manually (and thus, hopefully, were relevant, if not necessarily unproblematic). Now, almost all pages have up to five related content suggestions, but only two or three out of 600,000 pages see their links manually amended per month. A question arises whether this extremely low rate of manual intervention is reflective of the high quality of the system, or the reverse evidence of lack of resource to quality-assure websites that previously prevented 98% of pages from having this type of related information.

However, despite the queries as to the desirability of the AI implementation as described, the ATS disclosure is in itself useful because it allows the type of analysis above and, in case someone considers the situation unsatisfactory or would like to prove it further, there are is a clear gateway to (try to) engage the entity responsible for this AI deployment.

QCovid algorithm

The algorithm was developed at the onset of the Covid-19 pandemic to drive government decisions on which citizens to advise to shield, support during shielding, and prioritise for vaccination rollout. Since the end of the shielding period, the tool has been modified. “The clinical tool for clinicians is intended to support individual conversations with patients about risk. Originally, the goal was to help patients understand the reasons for being asked to shield and, where relevant, help them do so. Since the end of shielding requirements, it is hoped that better-informed conversations about risk will have supported patients to make appropriate decisions about personal risk, either protecting them from adverse health outcomes or to some extent alleviating concerns about re-engaging with society.”

“In essence, the tool creates a risk calculation based on scoring risk factors across a number of data fields pertaining to demographic, clinical and social patient information.“ “The factors incorporated in the model include age, ethnicity, level of deprivation, obesity, whether someone lived in residential care or was homeless, and a range of existing medical conditions, such as cardiovascular disease, diabetes, respiratory disease and cancer. For the latest clinical tool, separate versions of the QCOVID models were estimated for vaccinated and unvaccinated patients.”

It is difficult to assess how intensely the tool is (currently) used, although the ATS indicates that “In the period between 1st January 2022 and 31st March 2022, there were 2,180 completed assessments” and that “Assessment numbers often move with relative infection rate (e.g. higher infection rate leads to more usage of the tool).“ The ATS also stresses that “The use of the tool does not override any clinical decision making but is a supporting device in the decision making process.” “The tool promotes shared decision making with the patient and is an extra point of information to consider in the decision making process. The tool helps with risk/benefit analysis around decisions (e.g. recommendation to shield or take other precautionary measures).”

The impact assessment of this tool is driven by those mandated for medical devices. The description is thus rather technical and not very detailed, although the selected examples it includes do capture the possibility of somebody being misidentified “as meeting the threshold for higher risk”, as well as someone not having “an output generated from the COVID-19 Predictive Risk Model”. The ATS does stress that “As part of patient safety risk assessment, Hazardous scenarios are documented, yet haven’t occurred as suitable mitigation is introduced and implemented to alleviate the risk.” That mitigation largely seems to be that “The tool is designed for use by clinicians who are reminded to look through clinical guidance before using the tool.”

I think this case shows two things. First, that it is difficult to understand how different parts of the analysis fit together when a tool that has had two very different uses is the object of a single ATS disclosure. There seems to be a good argument for use case specific ATS disclosures, even if the underlying AI deployment is the same (or a closely related one), as the implications of different uses from a governance perspective also differ.

Second, that in the context of AI adoption for healthcare purposes, there is a dual barrier to accessing relevant (and understandable) information: the tech barrier and the medical barrier. While the ATS does something to reduce the former, the latter very much remains in place and perhaps turn the issue of trustworthiness of the AI to trustworthiness of the clinician, which is not necessarily entirely helpful (not only in this specific use case, but in many other one can imagine). In that regard, it seems that the usability of the ATS is partially limited, and more could be done to increase meaningful transparency through AI-specific IAs, perhaps as proposed by the Ada Lovelace Institute.

In this case, the ATS disclosure has also provided some valuable information, but arguably to a lesser extent than the previous case study.

ICO’s Registration Inbox AI

This is a tool that very much resembles other forms of email classification (e.g. spam filters), as “This algorithmic tool has been designed to inspect emails sent to the ICO’s registration inbox and send out autoreplies to requests made about changing addresses. The tool has not been designed to automatically change addresses on the requester’s behalf. The tool has not been designed to categorise other types of requests sent to the inbox.”

The disclosure indicates that “In a significant proportion of emails received, a simple redirection to an online service is all that is required. However, sifting these types of emails out would also require time if done by a human. The algorithm helps to sift out some of these types of emails that it can then automatically respond to. This enables greater capacity for [Data Protection] Fees Officers in the registration team, who can, consequently, spend more time on more complex requests.” “There is no manual intervention in the process - the links are provided to the customer in a fully automated manner.”

The tool has been in use since May 2021 and classifies approximately 23,000 emails a month.

When it comes to risk identification and mitigation, the ATS disclosure stresses that “The algorithmic tool does not make any decisions, but instead provides links in instances where it has calculated the customer has contacted the ICO about an address change, giving the customer the opportunity to self-serve.” Moreover, it indicates that there is “No need for review or appeal as no decision is being made. Incorrectly classified emails would receive the default response which is an acknowledgement.” It further stresses that “The classification scope is limited to a change of address and a generic response stating that we have received the customer’s request and that it will be processed within an estimated timeframe. Incorrectly classified emails would receive the default response which is an acknowledgement. This will not have an impact on personal data. Only emails with an 80% certainty of a change of address request will be sent an email containing the link to the change of address form.”

In my view, this disclosure does not entirely clarify the way the algorithm works (e.g. what happens to emails classified as having requested information on change of address? Are they ‘deleted’ from the backlog of emails requiring a (human) non-automated response?). However, it does provide sufficient information to further consolidate the questions arising from the general description. For example, it seems that the identification of risks is clearly partial in that there is not only a risk of someone asking for change of address information not automatically receiving it, but also a risk of those asking for other information receiving the wrong information. There is also no consideration of additional risks (as above), and the general description makes the claim of benefits doubtful if there has to be a manual check to verify adequate classification.

The ATS disclosure does not provide sufficient contact information for the owner of the AI (perhaps because they were contracted on limited after service terms…), although there is generic contact information for the ICO that could be used by someone that considered the situation unsatisfactory or would like to prove it further.

Food Hygiene Rating Scheme – AI

This tool is also based on machine learning to make predictions. “A machine learning framework called LightGBM was used to develop the FHRS AI model. This model was trained on data from three sources: internal Food Standards Agency (FSA) FHRS data, publicly available Census data from the 2011 census and open data from HERE API. Using this data, the model is trained to predict the food hygiene rating of an establishment awaiting its first inspection, as well as predicting whether the establishment is compliant or not.” “Utilising the service, the Environmental Health Officers (EHOs) are provided with the AI predictions, which are supplemented with their knowledge about the businesses in the area, to prioritise inspections and update their inspection plan.”

Regarding the justification for the development, the disclosure stresses that “the number of businesses classified as ‘Awaiting Inspection’ on the Food Hygiene Rating Scheme website has increased steadily since the beginning of the pandemic. This has been the key driver behind the development of the FHRS AI use case.” “The objective is to help local authorities become more efficient in managing the hygiene inspection workload in the post-pandemic environment of constrained resources and rapidly evolving business models.”

Interestingly, the disclosure states that the tool “has not been released to actual end users as yet and hence the maintenance schedule is something that cannot be determined at this point in time (June 2022). The Alpha pilot started at the beginning of April 2022, wherein the end users (the participating Local Authorities) have access to the FHRS AI service for use in their day-to-day workings. This section will be updated depending on the outcomes of the Alpha Pilot ...” It remains to be seen whether there will be future updates on the disclosure, but an error in copy-pasting in the ATS disclosure makes it contain the same paragraph but dated February 2022. This stresses the need to date and reference (eg v.1, v.2) the successive versions of the same disclosure, which does not seem to be a field of the current template, as well as to create a repository of earlier versions of the same disclosure.

The section on oversight stresses that “the system has been designed to provide decision support to Local Authorities. FSA has advised Local Authorities to never use this system in place of the current inspection regime or use it in isolation without further supporting information”. It also stresses that “Since there will be no change to the current inspection process by introducing the model, the existing appeal and review mechanisms will remain in place. Although the model is used for prioritisation purposes, it should not impact how the establishment is assessed during the inspection and therefore any challenges to a food hygiene rating would be made using the existing FHRS appeal mechanism.”

The disclosure also provides detailed information on IAs: “The different impact assessments conducted during the development of the use case were 1. Responsible AI Risk Assessment; 2. Stakeholder Impact Assessment; [and] 3. Privacy Impact Assessment.” Concerning the responsible AI risk assessment, in addition to a personal data issue that should belong in the DPIA, the disclosure reports three identified risks very much in line with the ones I had hinted at above: “2. Potential bias from the model (e.g. consistently scoring establishments of a certain type much lower, less accurate predictions); 3. Potential bias from inspectors seeing predicted food hygiene ratings and whether the system has classified the establishment as compliant or not. This may have an impact on how the organisation is perceived before receiving a full inspection; 4. With the use of AI/ML there is a chance of decision automation bias or automation distrust bias occurring. Essentially, this refers to a user being over or under reliant on the system leading to a degradation of human-reasoning.”

The disclosure presents related mitigation strategies as follows: “2. Integration of explainability and fairness related tooling during exploration and model development. These tools will also be integrated and monitored post-alpha testing to detect and mitigate potential biases from the system once fully operational; 3. Continuously reflect, act and justify sessions with business and technical subject matter experts throughout the delivery of the project, along with the use of the three impact assessments outlined earlier to identify, assess and manage project risks; 4. Development of usage guidance for local authorities specifically outlining how the service is expected to be used. This document also clearly states how the service should not be used, for example, the model outcome must not be the only indicator used when prioritising businesses for inspection.”

In this instance, the ATS disclosure is in itself useful because it allows the type of analysis above and, in case someone considers the situation unsatisfactory or would like to prove it further, there are is a clear gateway to (try to) engage the entity responsible for this AI deployment. It is also interesting to see that the disclosure specifies that the private provider was engaged “As well as [in] a development role [… to provide] Responsible AI consulting and delivery services, including the application of a parallel Responsible AI sprint to assess risk and impact, enable model explainability and assess fairness, using a variety of artefacts, processes and tools”. This is clearly reflected in the ATS disclosure and could be an example of good practice where organisations lack that in-house capability and/or outsource the development of the AI. Whether that role should fall with the developer, or should rather be separate to avoid organisational conflicts of interest is a discussion for another day.

Final thoughts

There seems to be a mixed picture on the usability of the ATS disclosures, with some of them not entirely providing (full) usability, or a clear pathway to engage with the specific entity in charge of the development of the algorithmic tool, specifically if it was an outsourced provider. In those cases, the public authority that has implemented the AI (even if not the owner of the project) will have to deal with any issues arising from the disclosure. There is also a mixed practice concerning linking to resources other than previously available (open) data (eg open source code, data sources), with only one project (GOV.UK) including them in the disclosures discussed above.

It will be interesting to see how this assessment scales up (to use a term) once disclosures increase in volume. There is clearly a research opportunity arising as soon as more ATS disclosures are published. As a hypothesis, I would submit that disclosure quality is likely to reduce with volume, as well as with the withdrawal of whichever support the pilot phase has meant for those participating institutions. Let’s see how that empirical issue can be assessed.

The other reflection I have to offer based on these first four disclosures is that there are points of information in the disclosures that can be useful, at least from an academic (and journalistic?) perspective, to assess the extent to which the public sector has the capabilities it needs to harness digital technologies (more on that soon in this blog).

The four reviewed disclosures show that there was one in-house development (GOV.UK), while the other ones were either procured (QCovid, which disclosure includes a redacted copy of the contract), or contracted out, perhaps even directly awarded (ICO email classifier FSA FHRS - AI). And there are some in between the line indications that some of the implementations may have been relatively randomly developed, unless there was strong pre-existing reliable statistical data (eg on information requests concerning change of business address). Which in itself triggers questions on the procurement or commissioning strategy developed by institutions seeking to harness AI potential.

From this perspective, the ATS disclosures can be a useful source of information on the extent to which the adoption of AI by the public sector depends as strongly on third party capabilities as the literature generally hypothesises or/and is starting to demonstrate empirically.

The perils of not carrying out technology-centered research into digital technologies and procurement governance -- re Sava and Dragos (2022), plus authors' response

This is a post in two parts. The first part addresses my methodological concerns with research on digital technologies and public procurement (and public governance more generally), as exemplified by a recent paper. The second part collects the response by the authors of that paper.

This pair of points of view are offered together to try to create debate. While the authors found my comments harsh (I cannot judge that), they engaged with them and provided their own counter-arguments. In itself, I think that is laudable and already has value. Any further discussion with the broader community, via comments (or email), would be a bonus.

Part 1: The perils of not carrying out technology-centered research into digital technologies and procurement governance -- re Sava and Dragos (2022)

When I started researching the interaction between digital technologies and procurement governance, it was clear to me that a technology-centered legal method was required. A significant amount of the scholarship that is published fails to properly address the governance implications of digital technologies because it simply does not engage with their functionality—or, put otherwise, because the technology is not understood. This can lead to either excessive claims of what ‘technology fixes’ can achieve or, perhaps even more problematic, it can generate analysis that is based on a misleading, shallow and oftentimes purely literal reading of the labels with which the technology is described and referred to.

A recent paper on smart contracts and procurement clearly exemplifies this problem: N.A. Sava & D. Dragos, ‘The Legal Regime of Smart Contracts in Public Procurement’ (2022) Transylvanian Review of Administrative Sciences, No. 66 E/2022, pp. 99–112.

Conceptual problems

From the outset, the paper is at pains to distinguish blockchain and smart contracts, and proposes ’a needed conceptual distinction that would fit the public contracts theory: before a contract is signed, it is logical to refer to blockchain technology when discussing digital means of awarding the procurement contract. As a result of this award, the concluded contract could be a “smart contract”’ (at 101).

The trap into which the paper falls, of course, is that of believing that blockchain and smart contracts can be distinguished ‘conceptually’ (in a legal sense), rather than on the basis of their technological characteristics and functionality.

Blockchain is a type of distributed ledger technology (DLT). In some more detail: ‘A DLT system is a system of electronic records that enables a network of independent participants to establish a consensus around the authoritative ordering of cryptographically-validated (‘signed’) transactions. These records are made persistent by replicating the data across multiple nodes, and tamper-evident by linking them by cryptographic hashes. The shared result of the reconciliation/consensus process - the ‘ledger’ - serves as the authoritative version for these records’ (M Rauchs et al, Distributed Ledger Technology Systems. A Conceptual Framework (2018), at 24). Blockchain is thus a ‘passive’ digital technology in the sense that it cannot perform any sort of automation of (decision-making) processes because it simply serves to create a data infrastructure.

In turn, smart contracts are a type of ‘active’ (or automating) digital technology that can be deployed on top of a DLT. In more detail: ‘Smart contracts are simply programs stored on a blockchain that run when predetermined conditions are met. They typically are used to automate the execution of an agreement so that all participants can be immediately certain of the outcome, without any intermediary’s involvement or time loss. They can also automate a workflow, triggering the next action when conditions are met’ (IBM, What are smart contracts on blockchain? (undated, accessed 1 July 2022)).

What this means is that, functionally, ‘smart contracts’ may or may not map onto the legal concept of contract, as a ‘smart contract’ can be a unilaterally programmed set of instructions aimed at the automation of a workflow underpinned by data held on a DLT.

Taking this to the public procurement context, it is then clear that both the management of the award process and the execution of an awarded public contract, to the extent that they could be automated, would both need to be instrumentalised via smart contracts plus an underlying blockchain (I would though be remiss not to stress that the practical possibilities of automating either of those procurement phases are extremely limited, if at all realistic; see here and here, which the paper refers to in passing). It does not make any (technological/functional) sense to try to dissociate both layers of digital technology to suggest that ‘blockchain technology [should be used] when discussing digital means of awarding the procurement contract. As a result of this award, the concluded contract could be a “smart contract”’ (Sava & Dragos, above, 101).

This is important, because that technology-incongruent conceptual distinction is then the foundation of legal analysis. The paper e.g. posits that ‘the award of public contracts is a unilateral procedure, organized by state authorities according to specific rules, and that automation of such procedure may be done using blockchain technology, but it is not a ‘“smart contract” (sic). Smart contracts, on the other hand, can be an already concluded procurement contract, which is executed, oversaw (sic) and even remedied transparently, using blockchain technology (sic)’ (ibid, 103, emphasis added).

There are three problems here. First, the automation of the procurement award procedure carried out on top of a DLT layer would require a smart contract (or a number of them). Second, the outcome of that automated award would only be a ‘smart contract’ in itself if it was fully coded and its execution fully automated. In reality, it seems likely that some parts of a public contract could be coded (e.g. payments upon invoice approval), whereas other parts could not (e.g. anything that has to happen offline). Third, the modification of the smart contract (ie coded) parts of a public contract could not be modified (solely) using blockchain technology, but would require another (or several) smart contract/s.

Some more problems

Similarly, the lack of technology-centricity of the analysis leads the paper to present as open policy choices some issues that are simply technologically-determined.

For example, the paper engages in this analysis:

… the question is where should the smart public contracts be awarded? In the electronic procurement systems already developed by the different jurisdictions? On separate platforms using blockchain technology? The best option for integrating smart contracts into the procurement procedures may be the already existing digital infrastructure, therefore on the electronic procurement platforms of the member states. We believe this would be an optimal solution, as smart contracts should enhance the current electronic procurement framework and add value to it, thus leveraging the existing system and not replacing it (at 103, emphasis added).

Unless the existing electronic procurement platforms ran on blockchain—which I do not think they do—then this is not a policy option at all, as it is not possible to deploy smart contracts on top of a different layer of information. It may be possible to automate some tasks using different types of digital technologies (e.g. robotic process automation), but not smart contracts (if the technological concept, as discussed above, is to be respected).

The problems continue with the shallow approach to the technology (and to the underlying legal and practical issues), as also evidenced in the discussion of the possibility of automating checks related to the European Single Procurement Document (ESPD), which is a self-declaration that the economic operator is not affected by exclusion grounds (see Art 59 Directive 2014/24/EU).

The paper states

In the context of automatized checks, the blockchain technology can provide an avenue for checking the validity of proofs presented. The system could automate the verifications of the exclusion grounds and the selection criteria by checking the original documents referenced in the ESPD in real time (that is, before determining the winning tender). The blockchain technology could verify the respect of the exclusions grounds and rule out any economic operator that does not comply with this condition (at 104, emphasis added).

This is a case of excessive claim based on a misunderstanding of the technology. A smart contract could only verify whatever information was stored in a DLT. There is no existing DLT capturing the information required to assess the multiplicity of exclusion grounds regulated under EU law. Moreover, the check would never be of the original documents, but rather of digital records that would either be self-declared by the economic operators or generated by a trusted authority. If the latter, what is the point of a blockchain (or other DLT), given that the authority and veracity of the information comes from the legal authority of the issuer, not the consensus mechanism?

There are also terminological/conceptual inconsistencies in the paper, which does not consistently stick to its conceptual distinction that blockchain should be used to refer to the automation of the award procedure, with smart contracts being reserved to the awarded contract. For example, it (correctly) asserts that ‘When it comes to selection criteria, the smart contract could also perform automatic checks on the elements listed in the contract notice’ (at 104). However, this can creates confusion for a reader not familiar with the technology.

Other issues point at the potentially problematic implications of analysis based on a lack of in-depth exploration of the technologies. For example, the paper discusses a project in Colombia, which ‘created a blockchain software that allowed for record keeping, real time auditability, automation through smart contracts and enhanced citizen engagement’ (at 105). After limited analysis, the paper goes on to stress that ‘Our opinion is that the system in Colombia resembles very much the regular e-procurement systems in Europe. For instance, Romania’s SEAP (Electronic Public Procurement System) insures exactly the same features — non-alteration of bids, traceability and automatic evaluation of tenders (price). So, the question is whether the smart contract system in Colombia is anything else than a functional e-procurement system’ (ibid). This reflects a conflation of functionality with technology, at best.

In the end, the lack of technology-centered (legal) analysis significantly weakens the paper and makes its insights and recommendations largely unusable.

The need for a technology-centric legal methodology

To avoid this type of problems in much-needed legal scholarship on the impact of digital technologies on public governance, it is necessary to develop a technology-centric legal methodology. This is something I am working on, in the context of my project funded by the British Academy. I will seek to publish a draft methodology towards the end of the year. Comments and suggestions on what to take into account would be most welcome: a.sanchez-graells@bristol.ac.uk.

Part 2: authors’ response

Dear Professor,

As a first-year PhD student, being read and offered feedback, especially in the incipient phase of the research, is an amazing learning opportunity. Not all PhD students have the chance to exchange on their topic, and even more with a revered name in the doctrine of public procurement like yourself, therefore am I am very grateful for this debate (Sava).

The co-author Dragos also shares the respect and gratitude for the scholarly critique, although considers the comments rather theoretical and lacking an alternative constructive conclusion.

Concerning the need to conduct a ʻtechnology-centered legal’ research, I fully agree, and I will try to integrate more technology-centered research into the thesis.

However, being lawyers, we believe that technology-centered research does not take into account the established concepts from law and especially public procurement law, therefore an interdisciplinary perspective is needed.

Now we will address the arguments you formulated.

1) Conceptual problems

Concerning the definitions of blockchain and smart contract that you offer, we are of course familiar with them and agree with them.

We agree that blockchain-based smart-contracts could automate certain aspects of the procurement procedures, both in the award and in the execution phase. In our paper, we acknowledge the fact that ʻsmart contracts could automate any process that can be presented as an IF+THEN formula’ (p. 100-101). In this sense, like you noticed, we give the example of automating the check of the selection criteria: ‘When it comes to selection criteria, the smart contract could also perform automatic checks on the elements listed in the contract notice’ (p. 104).

However, beyond these two concepts (blockchain and smart contracts), there is a third concept, that of a ʻsmart legal contract’.

DiMatteo, L., Cannarsa, M. and Poncibò, C., in The Cambridge Handbook of Smart Contracts, Blockchain Technology and Digital Platforms (Cambridge: Cambridge University Press, 2019, p. 63) draw attention to the inadequacy of the terminology: ʻFor blockchain-based smart contracts, a useful dichotomy can be drawn between the ‘smart contract code’ that is, the computer code that is ‘– stored, verified, and executed on a blockchain and the ‘smart legal contract’ - a complement (or maybe even a substitute) for a legal contract that applies that technology. In essence, a ‘smart legal contract’ is a combination of the ‘smart contract code’ and traditional legal language.

'The LawTech panel recently decided that (...) smart contracts could still be legally binding provided that they include the typical elements of a contract.’ (https://juro.com/learn/smart-contracts, consulted on the 2nd of July 2022). Like you mention, ‘functionally, ‘smart contracts’ may or may not map onto the legal concept of contract, as a ‘smart contract’ can be a unilaterally programmed set of instructions aimed at the automation of a workflow underpinned by data held on a DLT’.

Therefore, the correct conceptual distinction would be between ʻsmart contract code’ and ʻsmart legal contract’. In the paper, we tried to focus on the smart legal contract, and discuss its compatibility with public procurement contracts. Through the conceptual distinction, we actually wanted to point out the fact that it would be difficult to imagine a smart legal contract (legally binding) exclusively in the award phase. On the other hand, concerning the ʻsmart contract code’ we agree that it could be applicable to both the award and the execution phase, although the terminology remains debatable.

2) The question of where to integrate smart contracts

We state that ʻThe best option for integrating smart contracts into the procurement procedures may be the already existing digital infrastructure, therefore on the electronic procurement platforms of the member states. We believe this would be an optimal solution, as smart contracts should enhance the current electronic procurement framework and add value to it, thus leveraging the existing system and not replacing it’ (p. 103).

Of course, we do not believe that the current system works on blockchain (in the paper we explore why this would be a difficult task), but we did discuss the integration of emerging technologies in the existing context of e-procurement tools. However, this would be an integration among the e-procurement tools, not on top of the existing tools, as adequate infrastructure would be needed.

Actually we mean exactly what you pointed out in your conclusions, so we are in agreement here: some aspects of the procedure could be automated, yet the rest of the procedure could function based on the rules already in place. By the idea of not replacing the e-procurement system, we mean automatizing some punctual aspects, but not replacing the entire system.

3) The ESPD

The idea was that smart contracts could automatically check certain documents, such as the ones referenced in the ESPD.

In our text, we only discuss the idea of a verification, we do not describe in detail how this should be performed and we do not state that the DLT should capture on its own ʻthe information required to assess the multiplicity of exclusion grounds regulated under EU law’. Of course, these documents would need to be uploaded to the DLT and the uploaded documents would have a digital form. By ‘original document’ we refer to the document per se, the reference document and not the simple declaration from the ESPD.

An analogy of this idea could be made with the Canadian ‘Supplier information registration system, which facilitates the registration of supplier information on blockchain to validate it against different records and to validate it in an automated way’ (NTT Data Presentation at EPLD Meeting, May 2022).

4) The Colombian example

We could not understand your critique here. The referenced example described a system for selecting economic operators in public procurement (for more information: https://www.weforum.org/reports/exploring-blockchain-technology-for-government-transparency-to-reduce-corruption/), which we believe is comparable with a regular e-procurement portal.

5) Conclusions

Through our analysis, we intended to raise the following question: would automating some aspects of the public procurement procedure through “smart contracts” ensure the same characteristics and guarantees as the ones offered by an e-public procurement system of an EU member state? In that case, what is the added value of “smart contracts” in public procurement? It is a research question that we will try to focus on in the future, we merely pose it here.

This paper is an exploratory and incipient one. For the moment, our goal was to raise some questions and to explore some potential paths. Apart from theoretical “what ifs”, it is hard to find specificities of assertions that new digital technologies will definitely have numerous and game-changing applications in the procurement process, as long as the procurement process is still managed unilaterally by public bodies and entertains a public law regime.

The intention is to challenge a rather theoretical assumption on the role of digital technologies in public procurement and subsequently trying to find real, practical examples or applications, if any.

In no circumstance did we state that we are formulating policy recommendations, this was misunderstood. Only after extensive research conclusions may lead to policy recommendations but we are still far from that moment.

However, we believe that in order to actually draw some conclusions on the use of such technologies in public procurement, scholars should delve in more depth into the topic, by critically assessing the current literature in the field and trying to have an interdisciplinary (legal, technological and managerial) look at the topic. As of now, the literature is too theoretical.

In other words, in our opinion, the exclusive tech-centered approach that you suggest would be equally harmful as an exclusively legal one.

Thank you for this chance of a constructive dialogue, we are looking forward to future exchange on the topic.

More detail on the UK's procurement transparency ambitions -- some comments and criticisms

© GraceOda / Flickr.

On 30 June 2022, the UK Government’s Cabinet Office published the policy paper ‘Transforming Public Procurement - our transparency ambition’ (the ‘ambitions paper’, or the ‘paper’). The paper builds on the Green Paper and the Government’s response to its public consultation, and outlines ‘proposals to dramatically improve transparency of UK public contracts and spending’. The ambitions paper provides a vision well beyond the scant (almost null) detail in the Procurement Bill (clause 88), which is attracting a number of proposed amendments to try to enshrine in law the basic elements now spelled out in the paper.

In this post, I reflect on the need to amend the Procurement Bill to bind (successive) UK Governments to the current transparency aspirations. I also comment on other aspects of the paper, including persistent issues with the lack of granularity in planned access to procurement data, which I already raised in relation to the Green Paper (see here, Q27 and Q29, and here).

A necessary amendment of the Procurement Bill

The additional level of detail in the paper is welcome and helpful in understanding how the UK plans to operationalise its procurement transparency ambitions. However, a first point to make is that the publication of the ambitions paper should in no way deactivate concerns on the insufficiency of the Procurement Bill to ensure that a significant change in the way procurement information is captured and disseminated in the UK is achieved. In particular, the wording of clause 88(1) has to change.

It is nowhere close to good enough to simply have a weak enabling clause in legislation, stating that ‘An appropriate authority may by regulations make provision requiring certain information to be shared in a particular way, including through a specified online system’. The obvious first shortcoming is that the authority may do so, which also means it may not do so. The second is that the indication of a specified online system as a possible particular way of sharing information seems to take us back quite a few years. If not online (and if not as open data), how would a transparency aspiration be commensurate to the UK’s commitment to e.g. the open contracting data standard?.

Given the high level of aspiration in the paper, a more solid legal grounding is required. My proposal, which builds on discussions with the open contracting community, as well as the amendment already tabled by Baroness Hayman of Ullock, would be to amend clause 88(1) of the Procurement Bill, so it reads:

'An appropriate authority shall by regulations make provision requiring certain information to be shared through a specified online system. Such online system shall, at a minimum, establish and operate a freely accessible, machine-readable and licence-free digital register for all public procurement notices under this Act, wherein all information will be regularly updated in accordance with the time limits for the publication notices set out in the Act.'

Comments on the aspirations paper

Once the general commitment to having single digital register is strengthened, we can move on to consider the detail of what (and how) should be published in the register, what should be kept for restricted use, and what further transparency-related interventions can build upon it—e.g. the creation of a dashboard with useful data analytics, or the interconnection of the register with other sources of e.g. relevant anti-corruption information (for discussion, see here). There are some indications of what the UK aspires to do, but also some lack of clarity in the paper, and some clear risks of undesirable knock-on effects from the maximalist approach to procurement transparency it embraces.

Vision

The aspirations paper indeed starts from a maximalist position, indicating that the vision is ‘to create a fully transparent public procurement system’. However, there are two clear limitations to that approach.

First, the proposal itself includes a proportionate approach to transparency requirements: ‘we want to ensure that we are only asking for the most detailed information - contract documents, performance markings etc - from the largest contracts, in order to maintain transparency without bogging procurement teams down in unnecessary bureaucracy for low-value contracts’. This immediately means that a potentially large volume of (local) procurement will not be subjected to (some aspects) of the new transparency regime. Moreover, as the Procurement Bill stands, there would also be significant exclusions from important transparency obligations e.g. in relation to light touch contracts (see here, section 7, issues #21 on performance-related KPIS and non-performance notices, and #23 on modification notices). That already falls short of generating a ‘fully transparent’ procurement system, precisely in relation to the award of contracts where the risk of capture can be high.

Second, the publication of procurement information remains subjected to the general exclusions and carve-outs resulting from i.a. the Freedom of Information Act 2000 (FOIA). Interestingly, the ambitions paper does not refer to it at all, despite the Green Paper having made clear that, in the absence of FOIA reform (which is not sought), ‘only data which would be required to be made available under FOIA … would be publishable’ (at 167). Regardless of the paper’s silence on the issue, FOIA will continue to play a significant role in establishing which level of detail is disclosed, in particular in relation to disclosure of information not captured as a matter of mandatory disclosure in the relevant (award) notices, and perhaps even in relation to that.

The importance of preserving commercial confidentiality in the procurement setting is clear, and was also a clear focus of concern in the Green Paper consultation, leading e.g. to the Cabinet Office dropping its initial ambition of publishing tenders received in procurement procedures. As the Government’s response stressed: ‘We have considered the potential impact of public disclosure of information, such as (but not limited to) tenders. The feedback we received from stakeholders was that publishing tenders at this stage could prejudice future competitions that may run if the initial one is aborted and re-run for any reason, as bids will have been disclosed to the competition. As a result, we will not require disclosure of tenders submitted in a procurement’ (at 221).

Therefore, the system will not (and should not be) fully transparent. What is more useful is to see what the vision wants to enable in relation to procurement data and related analytics and insights. The vision indicates that the UK Government would like for everyone ‘to be able to view, search and understand what the UK public sector wants to buy, how much it is spending, and with whom’. This is a more realistic aspiration that does not necessarily entail total transparency and, given some safeguards and a more granular approach to the disclosure of differing levels of detail in the information (see here and discussion below), it should be welcome. Ultimately, the Government wants the future platform to help people understand:

1. current and future procurement opportunities created in the UK public sector; including pipelines of future work. [This should open up opportunities within the public sector to small businesses, driving down prices, increasing innovation and improving the business landscape across the country];

2. how much money the public sector spends on purchasing essential goods and services. [This should] allow taxpayers to see how much is being spent through procurement on and in their local area, who it is spent with and how it is delivering on local priorities. [Moreover, this should show] which routes to market are available to contracting authorities, and how much has been spent through each of those. [This should] give contracting authorities the data they need to collaborate better, drive value for money and identify cost savings in their procurements, so they can monitor for signs of waste and inefficiency;

3. which contracts finished on time and on budget–and which did not. [This means providing more detail across] the true lifecycle of government contracts, including how much the final amount spent on a contract differs from its original intended value, or how often contracts have been extended;

4. which companies have been excluded from winning future work due to fraud, corruption or persistent poor performance; [and]

5. who is really benefiting from public money - not just the companies winning contracts but the ownership of those companies

This list (which regroups the longer and slightly repetitive list in the paper, as well as aggregate the purpose for the disclosure of specific information) points to three categories. First, a category where the information is purely notice-based (categories 1, 4). Second, a category where the related insights should be easily derived from the information included mandatory notices (categories 2 and 3). Third, a category (mainly 5) that concerns non-procurement information and will require either (a) embedding disclosure obligations in the procurement life-cycle (thus raising the red tape and participation costs), or (b) interconnection with non-procurement databases.

The first category is relatively unproblematic, although there is an inherent tension between the disclosure of planned procurement opportunities and the facilitation of collusive practices (more details below).

The second category probably points at the need of considering the extent to which data dashboards should differentiate between different users, including the level of detail (and timeliness) of the information published in each of them (also discussed below).

The third category points at the need to consider issues of design and interoperability of the platform, as it would be preferable for it to be susceptible of plugging into other databases. Moreover, there are other (anti-corruption) functionalities that could be enabled, such as cross-checks against databases of political donations to identify potentially problematic relationships between procurement awardees and political donors. In relation to this category, and to anti-corruption efforts more generally, the ambitions paper is not particularly ambitious. However, the creation of a solid procurement data architecture on the basis of OCDS could facilitate those extensions in the future.

The future platform

The ambitions paper indicates that the Government seeks to operationalise the new transparency regime through two main elements (as the ‘tell us once’ supplier register is a parallel and distinct intervention):

• The introduction of a number of new procurement ‘notices’, covering the entire procurement lifecycle from planning through to contract expiry

• A digital platform which will display all of this information publicly, with API access to data published to the Open Contracting Data Standard (OCDS). Once we have completed the core notice development, over time we also plan to build a number of useful registers, and explore integrating commercial data analysis tools

What this means is that the future platform will initially simply bring into one place what is currently published across a scattered landscape of transparency tools (see section 3.1 in the paper). That is an improvement, but the more significant change will only come when register and dashboard insights get developed. Importantly, however, the design of these registers and dashboards need to be very carefully considered and linked back to the intended (and likely) use by different audiences. However, the ambitions paper does not seem to consider this need and rather seeks to establish a system accessible to any type of data user on an undifferentiated form (see section 4.4).

Research has shown that most of the gains from procurement transparency concern ex ante disclosure of information [M Bauhr et al, ‘Lights on the shadows of public procurement: Transparency as an antidote to corruption’ (2020) 33(3) Governance 495-523]. Conversely, the publication of ex post information is particularly risky in relation to e.g. anticompetitive practices, as well as corruption, and can generate limited benefits as it is unlikely that there will be a sustained level of engagement with that information by most stakeholders with a theoretical motivation to engage in procurement oversight [N Köbis, C Starke and I Rahwan, ‘The promise and perils of using artificial intelligence to fight corruption’ (2022) 4 Nature Machine Intelligence 418-424].

In that regard, it is particularly problematic that the aspirations paper seems to indicate that the UK Government would be publishing (in real time, for everyone to see) information such as: ‘Analysis of bid and win rates, analysis of supplier & bidder beneficial ownership patterns, general market trends analysis’. This should concern regulators such as the Competition and Markets Authority, as well as the Serious Fraud Office. While the latter should absolutely have access to that information and market intelligence, its public disclosure (in detail, with no time lag) could be counterproductive and help, rather than hinder, corrupt and collusive practices. In that regard, it is of paramount importance that those authorities (and others, such as the National Audit Office) are involved in the design of the system—which is not entirely clear from the ‘user-centric’ approach embraced in the aspirations paper (see section 4.1).

A multi-layered level of transparency

In relation to these risks and issues, it is necessary to reiterate a call for a more nuanced and discriminating approach than the one that transpires from the aspirations paper. As stressed in the response to the Green Paper consultation (here Q29), while it can but be endorsed that the platform needs to be created, and the data automatically fed into it in accordance with OCDS and other technical interoperability requirements, a key feature of the new system should be its multi-layered level of access/transparency.

Analysis carried elsewhere (see here) supports a nuanced approach to the level of transparency created by public contract registries similar to the envisaged central digital platform, which needs to fall short of the full transparency paradigm in which it seems to have been conceived. As a functional criterion, only the information that is necessary to ensure proper oversight and the effectiveness of anti-corruption measures should be disclosed, whereas the information that can be most damaging for competition should be withheld.

Generally, what is needed is granularity in the levels of information that are made accessible to different stakeholders. A full transparency approach whereby all information was made available to everyone would fall very short from the desired balance between the transparency and competition goals of public procurement. A system based on enabling or targeted transparency, whereby each stakeholder gets access to the information it needs for a specific purpose, is clearly preferable.

In more specific terms, it is submitted that:

• The content of the central digital platform should not be fully available to the public. Access to the full registry should be restricted to public sector officials under a strong duty of confidentiality protected by appropriate sanctions in cases of illegitimate disclosure.

• Even within the public sector, full access to the central digital platform should be made available on a need-to-know basis. Oversight entities, such as the National Audit Office, the Serious Fraud Office, or the Competition and Markets Authority, as well as the new public procurement review unit (PPRU) should have full access. However, other entities or specific civil servants should only access the information they require to carry out their functions.

• Limited versions of the central digital platform that are made accessible to the public should aggregate information by contracting authority and avoid disclosing any particulars that could be traced back to specific tenders, specific contracts, or specific undertakings.

• Representative institutions, such as third sector organisations, journalists or academics should have the opportunity of seeking full access to the central digital platform on a case-by-case basis where they can justify a legitimate or research-related interest. In case of access, ethical approval shall be obtained, anonymization of data attempted, and specific confidentiality requirements duly imposed.

• Delayed full access to the central digital platform could also be allowed for, provided there are sufficient safeguards to ensure that historic information does not remain relevant for the purposes of protecting market competition, business secrets and commercial interests.

• Tenderers should have access to their own records, even if they are not publicly-available, so as to enable them to check their accuracy. This is particularly relevant if public contract registries are used for the purposes of assessing past performance under the new rules.

• Big data should be published on an anonymised basis, so that general trends can be analysed without enabling ‘reverse engineering’ of information that can be traced to specific bidders.

• The entity in charge of the central digital platform should regularly publish aggregated statistics by type of procurement procedure, object of contract, or any other items deemed relevant for the purposes of the public accountability of public buyers (such as percentages of expenditure in green procurement, etc).

• The entity in charge of the central digital platform should develop a system of red flag indicators and monitor them with a view to reporting instances of legal non-compliance to the relevant oversight entity, or potential collusion to the competition authority. In that regard, the earlier attempts (eg through the abandoned ‘Screening for Cartels’ tool) should be carefully analysed to avoid replicating past errors.