I continue exploring the use of public procurement as a tool of digital regulation (or ‘AI regulation by contract’ as shorthand)—ie as a mechanism to promote transparency, explainability, cyber security, ethical and legal compliance leading to trustworthiness, etc in the adoption of digital technologies by the public sector.
After analysing procurement as a regulatory actor, a new draft chapter for my book project focuses on the procedural and substantive procurement tools that could be used for AI regulation by contract, to assess their suitability for the task.
The chapter considers whether procurement could effectively operationalise digital regulation goals without simply transferring regulatory decisions to economic operators. The chapter stresses how the need to prevent a transfer or delegation (ie a privatisation) of regulatory decisions as a result of the operation of the procurement rules is crucial, as technology providers are the primary target in proposals to use procurement for digital regulation by contract. In this post, I summarise the main arguments and insights in the chapter. As always, any feedback will be most warmly received: a.sanchez-graells@bristol.ac.uk.
Background
A first general consideration is that using procurement as a tool of digital regulation requires high levels of digital and commercial skills to understand the technologies being procured and the processes influencing technological design and deployment (as objects of regulation), and the procurement rules themselves (as regulatory tools). Gaps in those capabilities will jeopardise the effectiveness of using procurement as a tool of AI regulation by contract, beyond the limitations and constraints deriving from the relevant legal framework. However, to assess the (abstract) potential of procurement as a regulatory tool, it is worth distinguishing between practical and legal challenges, and to focus on legal challenges that would be present at all levels of public buyer capability.
A second general consideration is that this use of procurement could be seen as either a tool of ‘command and control’ regulation, or a tool of responsive regulation. In that regard, while there can be some space for a ‘command and control’ use of procurement as a tool of digital regulation, in the absence of clear (rules-based) regulatory benchmarks and legally-established mandatory requirements, the responsive approach to the use of procurement as a tool to enforce self-regulatory mechanisms seems likely to be predominant —in the sense that procurement requirements are likely to focus on the tenderers’ commitment to sets of practices and processes seeking to deliver (to the largest possible extent) the relevant regulatory attributes by reference to (technical) standards.
For example, it is hard to imagine the imposition of an absolute requirement for a digital solution to be ‘digitally secure’. It is rather more plausible for the tender and contract to seek to bind the technology provider to practices and procedures seeking to ensure high levels of cyber security (by reference to some relevant metrics, where they are available), as well as protocols and mechanisms to anticipate and react to any (potential) security breaches. The same applies to other desirable regulatory attributes in the procured digital technologies, such as transparency or explainability—which will most likely be describable (or described) by reference to technical standards and procedures—or to general principles, such as ethical or trustworthy AI, also requiring proceduralised implementation. In this context, procurement could be seen as a tool to promote co-regulation or (responsible) self-regulation both at tenderer and industry level, eg in relation to the development of ethical or trustworthy AI.
Against this background, it is relevant to focus on whether procurement tools could effectively operationalise digital regulation goals without simply transferring regulatory decisions to economic operators—ie operating as an effective tool of (responsive) meta-regulation. The analysis below takes a cradle-to-grave approach and focuses on the tools available at the phases of tender preparation and design, tender execution, and contract design and implementation. The analysis is based on EU procurement law, but the functional insights are broadly transferable to other systems.
Tender preparation and design
A public buyer seeking to use procurement as a tool of digital regulation faces an unavoidable information asymmetry. To try to reduce it, the public buyer can engage in a preliminary market consultation to obtain information on eg different technologies or implementation possibilities, or to ‘market-test’ the level of regulatory demand that could be met by existing technology providers. However, safeguards to prevent the use of preliminary market consultations to advantage specific technology providers through eg disclosure of exchanged information, as well as the level of effort required to participate in (detailed) market consultations, raise questions as to their utility to extract information in markets where secrecy is valued (as is notoriously the case of digital technology markets—see discussions on algorithmic secrecy) and where economic operators may be disinclined (or not have the resources) to provide ‘free consultancy’. Moreover, in this setting and given the absence of clear standards or industry practices, there is a heightened risk of capture in the interaction between the public buyer and potential technology providers, with preliminary market consultations not being geared for broader public consultation facilitating the participation of non-market agents (eg NGOs or research institutions). Overall, then, preliminary market consultations may do little to reduce the public buyer’s information asymmetry, while creating significant risks of capture leading to impermissible (discriminatory) procurement practices. They are thus unlikely to operate as an adequate tool to support regulation by contract.
Relatedly, a public buyer facing uncertainty as to the existing off-the-shelf offering and the level of adaptation, innovation or co-production required to otherwise achieve the performance sought in the digital technology procurement, faces a difficult choice of procurement procedure. This is a sort of chicken and egg problem, as the less information the public buyer has, the more difficult it is to choose an adequate procedure, but the choice of the procedure has implications on the information that the public buyer can extract. While the theoretical expectation could be that the public buyer would opt for a competitive dialogue or innovation partnership, as procedures targeted at this type of procurement, evidence of EU level practice shows that public buyers have a strong preference for competitive procedures with negotiations. The use of this procedure exposes the public buyer to direct risks of commercial capture (especially where the technology provider has more resources or the upper hand in negotiations) and the safeguards foreseen in EU law (ie the setting of non-negotiable minimum requirements and award criteria) are unlikely to be effective, as public buyers have a strong incentive to avoid imposing excessively demanding minima to avoid the risk of cancellation and retendering if no technology provider is capable (or willing) to meet them.
In addition, the above risks of commercial capture can be exacerbated when technology providers make exclusivity claims over the technological solutions offered, which could unlock the use of a negotiated procedure without prior publication—on the basis of absence of competition due to technical reasons, or due to the need to protect seclusive rights, including intellectual property rights. While the legal tests to access this negotiated procedure are in principle strict, the public buyer can have the wrong incentives to push through while at the same time controlling some of the safeguarding mechanisms (eg transparency of the award, or level of detail in the relevant disclosure). Similar issues arise with the possibility to creatively structure remuneration under some of these contracts to keep them below regulatory thresholds (eg by ‘remunerating in data’).
In general, this shows that the phase of tender preparation and design is vulnerable to risks of regulatory capture that are particularly relevant when the public buyer is expected to develop a regulatory role in disciplining the behaviour of the industry it interacts with. This indicates that existing flexible mechanisms of market engagement can be a source of regulatory risk, rather than a useful set of regulatory tools.
Tender execution
A public buyer seeking to use procurement as a tool of digital regulation could do so through the two main decisions of tenderer selection and tender evaluation. The expectation is that these are areas where the public buyer can exercise elements of ‘command and control’, eg through tenderer exclusion decisions as well as by setting demanding qualitative selection thresholds, or through the setting of mandatory technical specifications and the use of award constraints.
Tenderer selection
The public buyer could take a dual approach. First, to exclude technology providers with a previous track record of activity falling short of the relevant regulatory goals. Second, to incentivise or recompense high levels of positive commitment to the regulatory goals. However, both approaches present challenges.
First, the use of exclusion grounds would require clearly setting out in the tender documentation which types of digital-governance activities are considered to amount to ‘grave professional misconduct, which renders [the technology provider’s] integrity questionable’, and to reserve the possibility to exclude on grounds of ‘poor past performance’ linked to digital regulation obligations. In the absence of generally accepted standards of conduct and industry practices, and in a context of technological uncertainty, making this type of determinations can be difficult. Especially if the previous instance of ‘untrustworthy’ behaviour is being litigated or could (partially) be attributed to the public buyer under the previous contract. Moreover, a public buyer cannot automatically rely on the findings of another one, as the current EU rules require each contracting authority to come to its own view on the reliability of the economic operator. This raises the burden of engaging with exclusion based on these grounds, which may put some public buyers off, especially if there are complex technical questions on the background. Such judgments may require a level of expertise and available resources exceeding those of the public buyer, which could eg justify seeking to rely on third party certification instead.
Relatedly, it will be difficult to administer such tenderer screening to systems through the creation of lists of approved contractors or third-party certification (or equivalent mechanisms, such as dynamic purchasing systems administered by a central purchasing body, or quality assurance certification). In all cases, the practical difficulty will be that the public buyer will either see its regulatory function conditioned or precluded by the (commercially determined) standards underlying third-party certification, or face a significant burden if it seeks to directly scrutinise economic operators otherwise. The regulatory burden will to some extent be unavoidable because all the above-mentioned mechanisms foresee that (in some circumstances) economic operators that do not have access to the relevant certification or are under no obligation to register in the relevant list must be given the opportunity to demonstrate that they meet the relevant (substantive) qualitative selection criteria by other (equivalent) means.
There will also be additional challenges in ensuring that the relevant vetting of economic operators is properly applied where the digital technology solution relies on a long (technical) supply chain or assemblage, without this necessarily involving any (formal) relationship or subcontracting between the technology provider to be contracted and the developers of parts of the technical assemblage. This points at the significant burden that the public buyer may have to overcome in seeking to use qualitative selection rules to ‘weed out’ technology providers which (general, or past) behaviour is not aligned with the overarching regulatory goals.
Second, a more proactive approach that sought to go beyond exclusion or third-party certification to eg promote adherence to voluntary codes of conduct, or to require technology providers to justify how they eg generally ‘contribute to the development and deployment of trustworthy digital technologies’, would also face significant difficulties. Such requirements could be seen as unjustified and/or disproportionate, leading to an infringement of EU procurement law. They could also be altogether pre-empted by future legislation, such as the proposed EU AI Act.
Tender evaluation
As mentioned above, the possibility of setting demanding technical specifications and minimum requirements for tender evaluation through award constraints in principle seem like suitable tools of digital regulation. The public buyer could focus on the technical solutions and embedding the desired regulatory attributes (eg transparency, explainability, cyber security) and regulatory checks (on data and technology governance, eg in relation to open source code or interoperability, as well as in relation to ethical assessments) in the technical specifications. Award criteria could generate (further) incentives for regulatory performance, perhaps beyond the minimum mandatory baseline. However, this is far from uncomplicated.
The primary difficulty in using technical specifications as a regulatory tool relates to the challenge of clearly specifying the desired regulatory attributes. Some or most of the desired technological attributes are difficult to observe or measure, the processes leading to their promotion are not easy to establish, the outcomes of those processes are not binary and determining whether a requirement has been met cannot be subject to strict rules, but rather to (yet to be developed) technical standards with an unavoidable degree of indefinition, which may also be susceptible of iterative application in eg agile methods, and thus difficult to evaluate at tender stage. Moreover, the desired attributes can be in conflict between themselves and/or with the main functional specifications for the digital technology deployment (eg the increasingly clear unavoidable trade-off between explainability and accuracy in some AI technologies). This issue of the definitional difficulties and the incommensurability of some or most of the regulatory goals also relates to the difficulty of establishing minimum technical requirements as an award constraint—eg to require that no contract is awarded unless the tender reaches a specific threshold in the technical evaluation in relation to all or selected requirements (eg explainability). While imposing minimum technical requirements is permitted, it is difficult to design a mechanism to quantify or objectify the evaluation of some of the desired technological attributes, which will necessarily require a complex assessment. Such assessment cannot be conducted in such a way that the public buyer has an unrestricted freedom of choice, which will require clarifying the criteria and the relevant thresholds that would justify rejecting the tender. This could become a significant sticking point.
Designing technical specifications to capture whether a digital technology is ‘ethical’ or ‘trustworthy’ seems particularly challenging. These are meta-attributes or characteristics that refer to a rather broad set of principles in the design of the technology, but also of its specific deployment, and tend to proceduralise the taking into account of relevant considerations (eg which impact will the deployment have on the population affected). Additionally, in some respects, the extent to which a technological deployment will be ethical or trustworthy is out of the hands of the technology provider (eg may depend on decisions of the entity adopting the technology, eg on how it is used), and in some aspects it depends on specific decisions and choices made during contract implementation. This could make it impossible to verify at the point of the tender whether the end result will or not meet the relevant requirements—while including requirements that cannot be effectively verified prior to award would most likely breach current legal limits.
A final relevant consideration is that technical specifications cannot be imposed in a prescriptive manner, with technology providers having to be allowed to demonstrate compliance by equivalence. This limits the potential prescriptiveness of the technical specifications that can be developed by the public buyer, at least in relation to some of the desired technological attributes, which will always be constrained by their nature of standards rather than rules (or metrics) and the duty to consider equivalent modes of compliance. This erodes the practical scope of using technical specifications as regulatory instruments.
Relatedly, the difficulties in using award criteria to pursue regulatory goals stem from difficulties in the operationalisation of qualitative criteria in practice. First, there is a set of requirements on the formulation of award criteria that seek to avoid situations of unrestricted freedom of choice for the public buyer. The requirements tend to require a high level of objectivity, including in the structuring of award criteria of a subjective nature. In that regard, in order to guarantee an objective comparison and to eliminate the risk of arbitrary treatment, recent case law has been clear that award criteria intended to measure the quality of the tenders must be accompanied by indications which allow a sufficiently concrete comparative assessment between tenders, especially where the quality carries most of the points that may be allocated for the purposes of awarding the tender.
In part, the problem stems from the absence of clear standards or benchmarks to be followed in such an assessment, as well as the need to ensure the possibility of alternative compliance (eg with labels). This can be seen, for example, in relation to explainability. It would not suffice to establish that the solutions need to be explainable or to use explainability as an award criterion without more. It would be necessary to establish sub-criteria, such as eg ‘the solution needs to ensure that an individualised explanation for every output is generated’ (ie requiring local explainability rather than general explainability of the model). This would still need to be further specified, as to what type of explanation and containing which information, etc. The difficulty is that there are multiple approaches to local explainability and that most of them are contested, as is the general approach to post hoc explanations in itself. This puts the public buyer in the position of having to solve complex technical and other principled issues in relation to this award criterion alone. In the absence of standard methodologies, this is a tall order that can well make the procedure inviable or not used (with clear parallels to eg the low uptake of life-cycle costing approaches). However, the development of such methodologies parallels the issues concerning the development of technical standards. Once more, when such standards, benchmarks or methodologies emerge, reliance on them can thus (re)introduce risks of commercial determination, depending on how they are set.
Contract design and implementation
Given the difficulties in using qualitative selection, technical specifications and award criteria to embed regulatory requirements, it is possible that they are pushed to to the design of the contract and, in particular, to their treatment as contract performance conditions, in particular to create procedural obligations seeking to maximise attainment of the relevant regulatory goals during contract implementation (eg to create specific obligations to test, audit or upgrade the technological solution in relation to specific regulatory goals, with cyber security being a relatively straightforward one), or to pass on, ‘back-to-back’, mandatory obligations where they result from legislation (eg to impose transparency obligations, along the lines of the model standard clauses for AI procurement being developed at EU level).
In addition to the difficulty inherent in designing the relevant mechanisms of contractualised governance, a relevant limitation of this approach to embedding (self-standing) regulatory requirements in contract compliance clauses is that recent case law has made clear that ‘compliance with the conditions for the performance of a contract is not to be assessed when a contract is awarded’. Therefore, at award stage, all that can be asked is for technology providers to commit to such requirements as (future) contractual obligations—which creates the risk of awarding the contract to the best liar.
More generally, the effectiveness of contract performance clauses will depend on the contractual remedies attached to them and, in relation to some of the desirable attributes of the technologies, it can well be that there are no adequate contractual remedies or that the potential damages are disproportionate to the value of the contract. There will be difficulties in their use where obligations can be difficult to specify, where negative outputs and effects are difficult to observe or can only be observed with delay, and where contractual remedies are inadequate. It should be stressed that the embedding of regulatory requirements as contract performance clauses can have the effect of converting non-compliance into (mere) money claims against the technology provider. And, additionally, that contractual termination can be complicated or require a significant delay where the technological deployment has created operational dependency that cannot be mitigated in the short or medium term. This does not seem necessarily aligned with the regulatory gatekeeping role expected of procurement, as it can be difficult to create the adequate financial incentives to promote compliance with the overarching regulatory goals in this way—by contrast with, for example, the possibility of sanctions imposed by an independent regulator.
Conclusion
The analysis has stressed those areas where the existing rules prevent the imposition of rigid regulatory requirements or demands for compliance with pre-specified standards (to the exclusion of alternative ones), and those areas where the flexibility of the rules generates heightened risks of regulatory capture and commercial determination of the regulatory standards. Overall, this shows that it is either not easy or at all possible to use procurement tools to embed regulatory requirements in the tender procedure and in public contracts, or that those tools are highly likely to end up being a conduit for the direct or indirect application of commercially determined standards and industry practices.
This supports the claim that using procurement for digital regulation purposes will either be highly ineffective or, counterintuitively, put the public buyer in a position of rule-taker rather than rule-setter and market-shaper—or perhaps both. In the absence of non-industry led standards and requirements formulated eg by an independent regulator, on which procurement tools could be leveraged, each public buyer would either have to discharge a high (and possibly excessive) regulatory burden, or be exposed to commercial capture. This provides the basis for an alternative approach. The next step in the research project will thus be to focus on such mandatory requirements as part of a broader proposal for external oversight of the adoption of digital technologies by the public sector.