Creating (positive) friction in AI procurement

I had the opportunity to participate in the Inaugural AI Commercial Lifecycle and Procurement Summit 2024 hosted by Curshaw. This was a very interesting ‘unconference’ where participants offered to lead sessions on topics they wanted to talk about. I led a session on ‘Creating friction in AI procurement’.

This was clearly a counterintuitive way of thinking about AI and procurement, given that the ‘big promise’ of AI is that it will reduce friction (eg through automation, and/or delegation of ‘non-value-added’ tasks). Why would I want to create friction in this context?

The first clarification I was thus asked for was whether this was about ‘good friction’ (as opposed to old bad ‘red tape’ kind of friction), which of course it was (?!), and the second, what do I mean by friction.

My recent research on AI procurement (eg here and here for the book-long treatment) has led me to conclude that we need to slow down the process of public sector AI adoption and to create mechanisms that bring back to the table the ‘non-AI’ option and several ‘stop project’ or ‘deal breaker’ trumps to push back against the tidal wave of unavoidability that seems to dominate all discussions on public sector digitalisation. My preferred solution is to do so through a system of permissioning or licencing administered by an independent authority—but I am aware and willing to concede that there is no political will for it. I thus started thinking about second-best approaches to slowing public sector AI procurement. This is how I got to the idea of friction.

By creating friction, I mean the need for a structured decision-making process that allows for collective deliberation within and around the adopting institution, and which is supported by rigorous impact assessments that tease out second and third order implications from AI adoption, as well as thoroughly interrogating first order issues around data quality and governance, technological governance and organisational capability, in particular around risk management and mitigation. This is complementary—but hopefully goes beyond—emerging frameworks to determine organisational ‘risk appetite’ for AI procurement, such as that developed by the AI Procurement Lab and the Centre for Inclusive Change.

The conversations the focus on ‘good friction’ moved in different directions, but there are some takeaways and ideas that stuck with me (or I managed to jot down in my notes while chatting to others), such as (in no particular order of importance or potential):

  • the potential for ‘AI minimisation’ or ‘non-AI equivalence’ to test the need for (specific) AI solutions—if you can sufficiently approximate, or replicate, the same functional outcome without AI, or with a simpler type of AI, why not do it that way?;

  • the need for a structured catalogue of solutions (and components of solutions) that are already available (sometimes in open access, where there is lots of duplication) to inform such considerations;

  • the importance of asking whether procuring AI is driven by considerations such as availability of funding (is this funded if done with AI but not funded, or hard to fund at the same level, if done in other ways?), which can clearly skew decision-making—the importance of considering the effects of ‘digital industrial policy’ on decision-making;

  • the power (and relevance) of the deceptively simple question ‘is there an interdisciplinary team to be dedicated to this, and exclusively to this’?;

  • the importance of knowledge and understanding of the tech and its implications from the beginning, and of expertise in the translation of technical and governance requirements into procurement requirements, to avoid ‘games of chance’ whereby the use of ‘trendy terms’ (such as ‘agile’ or ‘responsible’) may or may not lead to the award of the contract to the best-placed and best-fitting (tech) provider;

  • the possibility to adapt civic monitoring or social witnessing mechanisms used in other contexts, such as large infrastructure projects, to be embedded in contract performance and auditing phases;

  • the importance of understanding displacement effects and whether deploying a solution (AI or automation, or similar) to deal with a bottleneck will simply displace the issue to another (new) bottleneck somewhere along the process;

  • the importance of understanding the broader organisational changes required to capture the hoped for (productivity) gains arising from the tech deployment;

  • the importance of carefully considering and resourcing the much needed engagement of the ‘intelligent person’ that needs to check the design and outputs of the AI, including frontline workers and those at the receiving end of the relevant decisions or processes and the affected communities—the importance of creating meaningful and effective deliberative engagement mechanisms;

  • relatedly, the need to ensure organisational engagement and alignment at every level and every step of the AI (pre)procurement process (on which I would recommend reading this recent piece by Kawakami and colleagues);

  • the need to assess the impacts of changes in scale, complexity, and error exposure;

  • the need to create adequate circuit-breakers throughout the process.

Certainly lots to reflect on and try to embed in future research and outreach efforts. Thanks to all those who participated in the conversation, and to those interested in joining it. A structured way to do so is through this LinkedIn group.

'Pro bono' or 'land and expand'? -- problematic 'zero-value' or 'free' contracts for digital innovation

Max Gruber / Better Images of AI / Banana / Plant / Flask / CC-BY 4.0.

The UK’s Ministry of Justice recently held an 8-day competition to select a software and programming consultancy to carry out a ‘pro-bono proof-of-concept process to explore methods for human-in-the-loop triage through GenAI’ (let’s not get bogged down on the technical details…).

This opportunity was not advertised under public procurement rules because the Ministry of Justice estimated the value of the contract at £0, as there would be no (direct, monetary) payments to the consultancy; it was clear that this would be ‘a pro bono contract. The [proof-of-concept] is expected to be completed in a 6 to 8 week period.’

Although the notice made it clear that the Ministry of Justice does ‘not anticipate that this project will be followed by further procurement on future development using the AI human in the loop triage’, because the main purpose is to ‘help accelerate existing work, test and prove different solutions/tools, share learning to help inform future work and designs, and transfer skills back into the [in-house] team throughout’—this approach seems problematic, in at least two ways.

First, it raises questions on whether, even as a ‘non-procurement’ opportunity, this was carried out in a proper way aligned with best practice. An 8-day window to express interest seems very short, especially as potentially interested consultants/consultancies were given very limited information to estimate the scope of works and understand the cost (to them) of supporting the development of the proof-of-concept on a pro bono basis.

At the same time, meeting the ‘minimalistic’ selection criteria and technical requirements could have been tricky, especially in such short time scale, as they contained some aspects that would have been difficult or impossible to meet other than by entities that already met the requirements at the time of the notice (eg obtaining BPSS security clearance is presumably not instantaneous) and were very familiar with the open source and other design standards referred to.

It does not seem unreasonable to suspect that the Ministry of Justice may have been in prior talks with some company/ies potentially interested in providing those pro bono services and that this created an insider advantage in making sense of the otherwise seemingly insufficient details given in the notice. This would make a mockery of the advertisement of the opportunity, even if not under procurement rules.

Cynically, it is also not clear what would happen if the current view changed and the Ministry later decided to procure the further stages of development — for seeking external input already at proof-of-concept stage (strongly) suggests that the in-house teams do not have (confidence in their) digital skills as required to carry out such digital innovation work.

In that case, an argument could be made that only the consultancy that participated in the proof-of-concept could provide the services for some specific technical / know how reason and that could seek to be used as justification for a direct award. This is a non-trivial risk. Even if this was not the case, a further procurement would create significant issues of prior involvement and would require deploying complicated solutions to try to neutralise the advantage gained by the consultancy. Overall, this does not seem like a fool-proof way of managing early collaborations in digital innovation projects.

Second, and perhaps more controversially, it is also not entirely clear to me that the contract would actually have ‘zero-value’ for the consultancy and was thus not really covered by the procurement rules.

The notice makes it clear that the Ministry of Justice would seek to retain the knowledge arising from the pilot by sharing it ‘within internal teams who can carry on building on top of the deliverables, but also with the wider teams that that are interested in using these new tools.’ However, this does not mean that the consultancy will not acquire potentially (almost) exclusive rights over the knowledge vis-à-vis third parties. There is no indication that the Ministry of Justice will publish the outcome of the project and, consequently, there seems to be no obstacle to the contribution of the Ministry being appropriated and incorporated into the consultancy’s know how (or future IP).

On close analysis of the terms of the ‘opportunity’ advertised by the Ministry of Justice, it is also clear that the Ministry would make available ‘A limited number of redacted example cases [to] be used for experimentation’ and that the consultant would be given access to the Ministry’s then current thinking and expertise on the project. These can also be valuable non-monetary assets on their own.

All in all, this is in itself worrying and raises questions on whether this was actually a £0 contract (from the perspective of the consultancy). Two legal aspects are relevant here. First, (partly pre-Brexit) ECJ case law distinguishes between pecuniary interest and monetary payments (IBA Molecular Italy and Tax-Fin-Lex), and stresses that ‘It is clear from the usual legal meaning of “for pecuniary interest” that those terms designate a contract by which each of the parties undertakes to provide a service in exchange for another’ (IBA Molecular Italy, C‑606/17, EU:C:2018:843, para 28). Whether this case—through the mix of access to anonymised examples, access to in-house know-how and possibility to retain know-how over the proof-of-concept and related learning vis-à-vis third parties—crosses the threshold and would have required advertising as a ‘procurement’ opportunity is at least arguable.

Second, this was clearly always going to be a ‘loss’ contract for the ‘pro bono volunteer’, and the fact that it is advertised as ‘pro bono’ solely masks the fact that the Ministry (a contracting authority in general terms) imposed on tenderers a requirement to submit abnormally low tenders. This raises questions on ‘mixed pro bono’ justifications that could be put forward in other cases to justify that a very low-cost tender is not actually abnormally low, but in the context of a procurement estimated at a value above the relevant threshold. Such justifications would in principle seem impermissible and, more generally, there are very big questions as to why the public sector should be seeking to obtain ‘free consultancy’—other than through fully open mechanisms, such as eg hackatons (is that still a thing) or other sorts of non-procurement competitions.

Which leads to the related question why would anyone willingly enter into such a contract? At least a partial explanation is that the ‘pro bono volunteer’ would most likely see a possible market advantage—for there are endless possibilities to carry out pro bono work otherwise. If not in this case, in most cases where similar conditions arise, there is a clear ‘marketing’ drive that can be masked as pro bono generosity. This can easily become an embedded element of strategies to ‘land and conquer’, or to build a portfolio of pro bono projects that is later used eg to demonstrate technical capability for qualitative selection purposes in future procurement processes (with other contracting authorities).

Overall, I think this is an example of worrying trends in the (side-stepping of) ‘procurement’ of digital innovation support services, and that such trends should be resisted. Only proper procurement processes and robust guardrails to safeguard from the risks of capture, competitive distortion and more generally long-term difficulties in ensuring competitive tension for digital innovation contracts, can minimise the consequences of arrangements that seem too good to be true.