Fairness and Explainability in Automatic Decision-Making Systems.
A challenge for computer science and law

Disparate impact is a common issue in technology and law. As mentioned above, the fact that the vast majority of machine learning research takes US anti-discrimination law as its background leads us to characterize the European legal context as well. We propose here, through a comparison of the American and European legal contexts, to: a) assess the relevance of work in ethical Machine Learning in the European context b) characterize ethical standards and measures of discrimination in the EU.

With regard to the United States, it should be noted that the notion of disparate impact emerged there, on the basis of the Civil Rights Act of 1964, Title VII of which specifies that practices which, under the guise of neutral rules, have a disproportionate impact on the protected class as compared to the unprotected class are prohibited - unless a legitimate interest (a business necessity) can be validly invoked.

In addition to Title VII, Title VI of the Civil Rights Act gives the federal government the right to sue for disparate impact. In addition, two laws from 1967 - the Age Discrimination in Employment Act and the Fair Housing Act - give individuals a right of action for disparate impact.

These provisions of federal law have been given concrete form by the implementation of a calculation rule to determine whether a situation of disparate impact can be observed in practice, known as the "80% rule" (or 4/5 rule). Initially developed in the State of California, it was formalized in the State of California Guideline Selection Procedures in 1972. It was then codified at the federal level in the Uniform Guidelines on Employee Selection Procedures (1978) used by the US Equal Employment Opportunity Commission (EEOC), but also by the DoJ and the DoL in their claims before the courts under Title VII. The 80% rule is a ratio of the hiring rate of individuals from the unprotected class to individuals from the protected class: if a company hires 50% of the male applicants for a job and 20% of the female applicants, the hiring rate is equal to 50/20, i.e. 0.4: the hiring rate of women is 40% of that of men. According to the EEOC guide, if the ratio of selection rates is less than 80%, this indicates a discriminatory situation.²²2The relevance of the 80% rule is debated, both legally and statistically ([54]). We will come back to this later.

On the legislative side, the disparate impact doctrine was imported into the UK from the early 1970s, when the Griggs decision appealed to British political leaders during a visit to the US, to such an extent that they incorporated indirect discrimination into the first sex discrimination law (the Sex Discrimination Act of 1975) and into the revision of the Race Relations Act in 1976 ([64], p. 287). As will be discussed in more detail below, the enforcement of Title VII of the Civil Rights Act by the US courts has progressively introduced greater and greater restrictions on its invocation by victims of indirect discrimination: the plaintiff must establish precisely which of the companies’ employment management rules is the cause of a disparate impact; provide compelling statistical evidence; and demonstrate that the employer refused to implement management rules that could have reduced the disparate impact. Even if the victim succeeds in overcoming these tests, the employer can show that the challenged rule or practice meets a business necessity.

The European Union has enacted a number of directives to combat discrimination, both direct and indirect, on the grounds of racial or ethnic origin, religion or belief, disability, age or sexual orientation: two directives on employment equality and race equality were adopted in 2000. In 2009, in the Lisbon Treaty, a horizontal clause was introduced to incorporate anti-discrimination into all EU policies and actions (article 10 of the TFEU³³3Treaty on the Functioning of the European Union). The CJEU⁴⁴4Court of Justice of the European Union has dealt with numerous discrimination cases, either in response to preliminary questions from national courts or on the merits.

Suk ([64]) highlights how different US and EU anti-discrimination law is.

• •

  There is a significant body of European case law on equal pay for men and women, while the US federal courts have excluded it from the disparate impact system. In this respect, [61], in an empirical analysis of litigation concerning disparate impact cases in the US, points out that it has had a clearly limited impact outside the context of written tests used in employee recruitment.

• •

  The decisions of the CJEU are more protective of female employees than in the US, in particular because unlike the European Court, the US courts have refused to apply the disparate impact theory to part-time work situations (which are much more important for women than for men).

• •

  It is relatively easier for claimants to argue a prima facie case of discrimination in Europe than in the US: the US requirement to determine which specific provision, rule or practice would cause a disparate impact has no equivalent in Europe.

• •

  Therefore, the CJEU focuses on equal pay for men and women; it is relatively silent on racial inequalities, whereas these are central in the US; in the US the disparate impact is applied to employment, excluding pay, i.e. to recruitment and promotion situations. Part-time work situations are not included in its scope (see table 1).

The question here is the correspondence between the (ideal-typical) computer approach and the judicial practice of disparate impact. The basic form of disparate impact is as follows:

It consists of comparing the probability of obtaining an outcome $(Y=1)$ according to whether the group is protected $(S=1)$ or unprotected $(S=0)$. As envisaged above, the enforcement of these legal provisions has resulted in the application of a calculation rule to determine whether a situation of disparate impact has occurred (the "80% rule").

The Uniform Guideline has been codified into the Code of federal regulation which article 1607 of Title 29 states that: “A selection rate for any race, sex, or ethnic group which is less than four-fifths (or eighty percent) of the rate for the group with the highest rate will generally be regarded by the Federal enforcement agencies as evidence of adverse impact, while a greater than four-fifths rate will generally not be regarded by Federal enforcement agencies as evidence of adverse impact. Smaller differences in selection rate may nevertheless constitute adverse impact, where they are significant in both statistical and practical terms or where a user’s actions have discouraged applicants disproportionately on grounds of race, sex, or ethnic group” (29 CFR 1607).

In addition, the EEOC is mindful of the disparate impact that certain recruitment practices may have, such as criminal record and banking history (credit defaults), as African-Americans are over-represented in the prison population and in the defaulting debtor population. In the 4th Circuit Court of Appeals decision EEOC v. Freeman (2015), the judge struck down the EEOC’s request to enjoin the company from conducting criminal and bank records searches. Judge Titus went so far as to write that the EEOC’s application is a “theory in search of its practice”. He added that employers face a tension between exposing themselves to potential liability if a blindly recruited employee is found to commit criminal or fraudulent acts and exposing themselves to prosecution by the EEOC.

The EEOC 80% rule has limitations, which are recognized as such in the law. According to 29 CFR 1607, large differences in selection rates do not mean a disparate impact when they involve small numbers or are not statistically significant. In this case, the EEOC allows for statistical evidence over a longer period of time and/or to study the impact of a similar practice implemented in similar circumstances. The sensitivity to small numbers of employees makes the rule inequitable when it leads to very small firms being more exposed to liability for discriminatory impact than large firms with much larger numbers of applicants for employment and applicants being considered for employment ([54]⁵⁵5A numerical example can be used to establish this. Let us consider a large firm (LF) and a small firm (SF). The LF receives 20,000 applications, equally divided between men and women. The recruiter selects the CVs of 3200 women and 4000 men, i.e. 32% and 40%. The selection is then equal to 0.32/0.4 = 0.8, in accordance with the EEOC rule. The SF receives 20 applications, equally divided between men and women. The employer selects the CVs of 4 men (40% of the applicants) and 3 women (30% of the applicants), resulting in a selection ratio below the 80% threshold: 0.3/0.4 = 0.75.). Finally, the 80% rule has its limits in terms of establishing causality, and the courts prefer to use statistical significance tests ([54], p. 785).

In the common law system, such as the United States legal system, case law has a particular importance: being recognized as a source of law, it interprets statutes and gives them their practical meaning. The 1971 Supreme Court decision, Griggs v. Duke Power, was a significant advance in securing civil rights for African Americans. The firm in question conducted intelligence tests and required employees to have attained a high school diploma in order to be promoted to more lucrative positions. African-American employees were rarely promoted, while Caucasian employees were frequently promoted. In its decision, the Supreme Court ruled that if a practice that operates to exclude members of a protected group is not based exclusively on job performance, then it is prohibited. The promotion practice of the accused firm was not, in this case, based on sole work performance ([69]).

After Griggs, the US Supreme Court followed a more restrictive jurisprudence. In the second half of the 1980s, restrictions were placed on constitutional protections; then, in the 1990s, restrictions were introduced around the evidentiary requirements for plaintiffs. Finally, in the 2000s, case law has limited the scope of employers’ actions to avoid racially disproportionate outcomes (such as in Ricci v. DeStefano) ([64]).

The Supreme Court has barred disparate impact claims when they invoke the Equal Protection Clause of the 14th Amendment to the Constitution, holding that the clause covers only intentional discrimination ([64]). In Arlington Heights v. Metropolitan Housing Corp. (1977), the Supreme Court held that “Proof of racially discriminatory intent or purpose is required to show a violation of the Equal Protection Clause”. The same applies to the 5th amendment (“due process clause”): In Washington v. Davis (1976), the Supreme Court held that a disparate impact argument cannot be used in a Fifth Amendment claim unless the plaintiff demonstrates that racially neutral standards were used with discriminatory intent⁶⁶6Barocas & Selbst ([7]) call such a practice “masking”: “… any form of discrimination that happens unintentionally can also be orchestrated intentionally” (p. 692).. A year later, in Pothard v. Rawlinson (1977), the Court held that Title VII of the Civil Rights Act does not make the physical requirements for entry into the career of a prison guard unlawful, even if they exclude 40% of female applicants.

It was in the 1980s that case law took a restrictive turn, hardening the standards of proof imposed on plaintiffs. These restrictions culminated in Wards Cove Packing v. Atonis (1989) The Court has placed a very important restriction on disparate impact actions by establishing the evidentiary rule that the plaintiff must establish: (a) the employee must identify the specific practice or rule that caused the indirectly discriminatory impact, and (b) the employee must also prove that the employer refused to implement practices or rules that would have satisfied the plaintiff’s grievances. In addition, the offending firm has the option of arguing in favor of the rule or practice causing a disproportionate impact, taking into account the “necessity of business”. It should be noted that the requirement to precisely define the practice or rule causing a disparate impact has been extended to other areas, and has become a constraint on the ability to advance statistical evidence: in Texas Department of Housing & Community Affairs v. Inclusive Community Project (2015), the Supreme Court held that a claim based on statistical parity must be dismissed if it does not establish which precisely defined public policy measure is alleged to cause a disproportionate impact. The Court also considered that housing policies must have room for flexibility, which is necessarily to achieve their legitimate interests ([69]).

A third phase of Supreme Court restrictions on disparate impact theory began in the 2000s. The most significant decision was Ricci v. De Stefano, issued in 2009 following the City of New Haven’s (Connecticut) annulment of an internal competition for the promotion of the city’s fire-fighters, as the success rate of white fire-fighters was twice that of African. Mayor John De Stefano considered that a disproportionate impact justified the cancellation of the exam which was challenged by the white fire-fighters and one Hispanic, including Ricci, who had passed the tests and should have been promoted. The Supreme Court ruled that the decision to overturn the test violated Title VII of the Civil Rights Act, as the City of New Haven did not have a “strong basis of evidence” indicating that it would have faced disparate impact liability if it had promoted white and Hispanic fire-fighters over African-American fire-fighters. In sum, the Supreme Court has limited the remedies available to employers to avoid racially disproportionate outcomes. Thus, while the terms of the law do not give a fixed content to the disparate impact doctrine and its empirical scope is shaped (narrowly) by the courts, its modelling in computer science is based on an ideal-typical formulation that is insensitive to its implementation in the real world.

Xiang and Raji ([76]) compare the concepts of machine learning and US anti-discrimination law on several dimensions. We present here their comparison with comments.

• •

  Procedural fairness: In machine learning, it refers to the identification of the characteristics of the inputs (in particular the proxies used) that lead to a particular outcome of the model, the focus being on the algorithm itself and its predictions. In contrast, for law, it is about the governance principles of the decision-making process. The focus is on the "system surrounding the algorithm and its uses" rather than on the algorithm itself.

• •

  Discrimination: In machine learning, it is often presented as the product of an unfair correlation between protected class variables and a metric of interest such as outcomes, false positive rates between classes, or the like. For the law, it is a social issue for which rules are produced with the aim of suppressing discriminatory intentions or causality. Note that Xiang and Raji do not mention indirect discrimination, which does not involve discriminatory intent.

• •

  Protected Class/Sensitive Attribute: While this is a major issue for law, machine learning research makes it a matter of features that should not be involved in algorithmic decision making, without the need to take law into account.

• •

  Anti-classification and anti-subordination: The notion of anti-classification in law simply means that the state cannot classify individuals according to their race, gender, age, etc. It is consistent with the concept of fairness through unawareness with which the machine learning community working on fairness is familiar. The principle of anti-subordination refers to equality of rights as an objective, which is however unattainable in a society with strong social stratification unless the law aims to strengthen the position of the less favored groups. According to [76], this is a dimension rarely considered in the machine learning literature. There are indeed minority approaches, such as "min-max" approaches, which aim to minimize misclassification for the most disadvantaged social groups (see the DRO-type approaches mentioned earlier). [28], using Lorenz dominance, consider fairness in the sense of improving the utility of the worse-off, following the transfer principle in economics.

• •

  Affirmative action: Machine learning research is open on this matter, in its developments on demographic parity; in law, it is subject to varying interpretations over time.

• •

  Disparate treatment and disparate impact: From a legal point of view, disparate treatment refers to intentionally discriminatory treatment; for machine learning, intention is not a relevant dimension, since it arises when a protected attribute is used in the decision process, and its use can be avoided. In law, [76] (wrongly) consider disparate impact to be illegal if it is intentional, whereas technically disparate impact arises when outcomes between subgroups differ, even without intentionality.

Ultimately, from the perspective of interdisciplinarity, it is very important to understand how the same concept is shaped in both fields and what meaning is attached to it. However, this is not an end in itself. It is a necessary task, but it should only be a prerequisite for a reciprocal opening up of the fields, of which we will see some possibilities below.

In [53] a legally based classification of indicators for measuring discrimination is suggested. They start from the idea that the interpretation of legislation leads to different measures of discrimination and different rankings of possibly discriminatory contexts. The authors use a contingency table to characterize four situations, depending on whether the group is protected or not, whether the decision is positive or negative, and to define a series of discrimination indicators.

[53] argue that “From a legal point of view, several measures are adopted worldwide. UK law mentions risk difference, EU Court of Justice has given more emphasis on the risk ratio, and US laws and courts mainly refer to the selection rate” (ibidem, p. 2). But in the conclusion of their article the diagnosis is different: the European legal system is connected to the RC (relative chance) and the American system to the RR (risk ratio), which casts serious doubt on the coherence of their analysis of legal systems.

It is indeed a bit adventurous to univocally associate a discrimination measure with a national or regional legal system. For example, the EU Equality Directives of the 2000s are predominantly concerned with differential risk (to establish prima facie discrimination it is sufficient to establish that a provision, criterion or practice places people of racial or ethnic origin at a disadvantage compared to others - [58], $\S$ 13.072). Instead, the case law of the CJEU refers to the principle of proof of a disproportionately disadvantageous position of a group in order to qualify a situation as indirect discrimination, which refers to a relative risk ([58] $\S$ 12.125 - gender pay inequalities and $\S$ 12.659 - more general doctrine). Reference can be made to the Kirshammer-Hack decision of the CJEU (30 November 1993): in this case, a part-time employee in a company with less than 5 employees was dismissed without compensation, as German law only guarantees the right to compensation to companies with more than 5 employees; the dismissed employee considered that she had been discriminated against in view of the fact that more women than men work in very small companies and on a part-time basis. The CJEU held that there would be discrimination between men and women if it were established that small businesses employ a significantly higher percentage of men than women, which was not statistically established by the claimant. In the same vein, the CJEU decision Krüger (9 Sept. 1999), was given in response to a preliminary question from the Munich Labour Court: is the exclusion of part-time employees (provided for in the collective agreement) from the benefit of a wage bonus discriminatory? The CJEU held that if the national court finds that the rule, even if formally gender-neutral, actually affects a considerably higher percentage of women than men, then it is sex discrimination.

There is a large body of work on the metrics of discrimination, both direct and indirect ([1]; [12]; [20]; [29]). We confine ourselves to indirect discrimination, for which several fairness criteria have been formalised, and which are incompatible with each other (Table 3).

Abu Elyounes ([1]) considers that, in law, group fairness aims to improve the position of disadvantaged groups and to achieve substantive equality between them, without being limited to mere procedural or formal equality. Indeed, disparate impact arises in the presence of a formally non-discriminatory rule or practice based on the procedural principle of equal treatment of persons in similar situations. Starting from the idea that fairness is contextual, Abu Elyounes’ approach is to determine the conditions under which indirect discrimination metrics have relevance for legal approaches (table 4).

A concrete example of the issues involved in choosing one metric among others is the controversy over the COMPAS criminal recidivism prediction application, which has been criticized by ProPublica as racially discriminatory. For example, the algorithm misclassifies African-Americans as future criminals at twice the rate of whites; whites are misclassified as low recidivists ([3]; [41]); other authors ([20]; [55]) believe that the COMPAS model is calibrated, showing that the recidivism risks of African-Americans and Caucasians are equal across all recidivism risk scores. In this respect, [1] is correct in considering that Propublica retains the equal opportunity criterion, while COMPAS argues that its algorithm is fair because it is calibrated.

[73] find that there is a gap, or even incompatibility, between European legal notions of discrimination and existing work on algorithmic fairness and make proposals to reduce this gap. As far as anti-discrimination law in the EU is concerned, the Equality Directives are drafted with a high level of generality; the Member States must transpose them into national law, and they do so with a very variable level of generality or precision (in particular as regards the conditions for statistical proof); the rulings of the CJEU are not constant over time as regards the conditions for proving indirect discrimination, even though a “gold standard” can be found in the European case law. Legal regulation therefore provides flexibility (via the transposition of directives and the case law of the CJEU, which is often marked by intuitive reasoning rather than a defined and stable metric)⁷⁷7CJUE 19 April 2012, case. C 415-10, Galina Meister v. Speech Design Carrier Systems GmbH: “indirect discrimination may be established by any means, and not only on the basis of statistical evidence”. whereas research on algorithmic fairness is based on a search for precision and consistency. On the one hand, “European conceptualization of discrimination … is contextual” ([71]) while on the other hand automatic methods to detect and correct discriminatory decisions need clear rules.

The aim of [73] is to clarify how to construct considerations of fairness in the automatic decision, which respects as much as possible the contextual approach of European law, in particular of the CJEU. However, it is possible to find a ’gold standard’ in European law. This gold standard was laid down in the Seymour-Smith decision of the CJEU, which is that the comparison between the discriminated and the non-discriminated group is the best method. According to [73], the only fairness metric compatible with this principle is (conditional) demographic (dis)parity, formalized as follows (for a given attribute):

As soon as $D>A$, there is a demographic disparity. [73] add another test, called "Negative dominance", which occurs if $D>50\%>A$: this test does not exist in the literature on algorithmic fairness. The test is done in two steps: a) the majority of the disadvantaged group must not belong to the protected class, b) only a minority of the protected class belongs to the disadvantaged group.

In another contribution, [72] elaborate on the role that anti-discrimination law can play in helping to define a metric that is consistent with its aims of promoting substantive rather than procedural and formal equality. They examine the compatibility between the fairness metrics used in machine learning and the aims of European anti-discrimination law. The latter aims, beyond the prevention of discrimination, to change society, public policies and practices to “level the playing field” and achieve substantive equality. While machine learning is about designing fairness techniques that solve problems of bias in the training data, knowing that these data reflect historical social stratifications, these techniques can be described as “bias preserving”; conversely, the purpose of law is to transform these historical societal biases (“bias transforming”)⁸⁸8“Bias preserving” fairness metrics seek to reproduce historic performance in the outputs of the target model with equivalent error rates for each group as reflected in the training data (or status quo). In contrast, “bias transforming” metrics do not blindly accept social bias as given or neutral starting point that should be preserved, but instead require people to make an explicit decision as to which biases the system should exhibit ([72]).. Technically, equalized odds, equal opportunity and calibration, and others, fall under an orientation “bias preserving” ([72]). They estimate that out of the 20 fairness metrics existing in the literature, 13 of them are bias preserving in the sense that they are satisfied by “matching error rates between groups” while the 7 metrics⁹⁹9precisely: statistical parity, conditional statistical parity, fairness through unawareness, fairness through awareness, counterfactual fairness, no unresolved discrimination, no proxy discrimination, path base causal reasoning. “bias transforming” are satisfied by “matching decision rates between groups”. As bias preserving metrics carry the risk of consolidating situations of social injustice and discrimination, thus maintaining the status quo, they are not in line with the core of European anti-discrimination law, which aims at achieving substantive equality (ibidem). [73], [72] recommend the use of Conditional Demographic Disparity (CDD) which they consider to be compatible with the aims of European law and are therefore “bias transforming”: “CDD treats all people (groups) as equal, meaning they should be treated the same. The test flags up any disparity between groups that remains once an appropriate conditioning variable has been applied. This notion of fairness follows the Aristotelian postulate of treating ‘like cases alike’ and enables formal equality. At the same time, CDD enables substantive equality by flagging up … any relative disparity between groups in a given population over a set of decisions or other outcomes” ([72]).

[75] makes proposals on how to reconcile technical and legal approaches to algorithmic banking, based on the observation that the proliferation of automatic decision models poses a serious risk of increasing social inequalities. She discusses technical approaches that could comply with US anti-discrimination law and reduce the risk that algorithms that exacerbate inequality will be found to comply with the law. Xiang argues for causal inference, consistent with Texas Dpt of Housing and Community Affairs v. Inclusive Communities Project, Inc, which posits a ’causal connection’ between the decision-making process and the existence of a disparate impact. For her, causal inference implies the use of a counterfactual, since it involves “comparing what happened in the real world with what would have happened in a counterfactual world with different conditions” ([75]).

Algorithmic discrimination raises questions about the capability of victims to go to court to challenge the differential treatment to which they are subjected. Access to the characteristics of the algorithmic treatment, or even knowledge of the existence of an automatic decision-making process, is in practice not ensured for victims. Hacker ([34]) raises the problem of the limited effectiveness of anti-discrimination law in Europe in the presence of algorithmic discrimination. According to him, the problem can be solved by unbundling anti-discrimination law and personal data protection as provided for in the GDPR; such an unbundling is conducive to a fruitful cooperation between lawyers and computer scientists likely to lead to the emergence of algorithms ensuring an “equal protection by design” ([34], p. 25). The mobilization of the GDPR can move the cursor upstream of litigation, i.e. to a certain extent to encourage the design of decision rules that prevent situations of discrimination. Thus, the lever of personal data protection can have beneficial effects in terms of anti-discrimination, without the need to strengthen data protection law. The expected beneficial effects can be achieved with the law as it is, i.e. without the need to amend the GDPR ([34], p. 25).

The GDPR contains recitals that set out key principles, including $n^{o}$39 which states that all processing of personal data should be lawful and fair; and especially $n^{o}$71 which states that “the controller should use appropriate mathematical or statistical procedures for the profiling, implement technical and organisational measures appropriate to ensure, in particular, that factors which result in inaccuracies in personal data are corrected and the risk of errors is minimised, and that prevents, inter alia, discriminatory effects …” (we underline).

The GDPR contains several instruments that can bring about these beneficial effects: the individual right of access to data; the data protection impact assessments (DPIAs) and audits that were proposed by Article 29 of the Working Party¹⁰¹⁰10The Article 29 Working Party is an independent European working party that dealt with issues relating to the protection of privacy and personal data until 25 May 2018 (entry into application of the GDPR) Guidelines on Automated individual decision-making and Profiling. The guidelines proposed, among other things: an “algorithmic auditing”, consisting of “testing the algorithms used and developed by machine learning systems to prove that they are actually performing as intended, and not producing discriminatory, erroneous or unjustified results”; These proposals have been incorporated into Recital 71 and Article 22 of the GDPR.

Hacker believes that the GDPR incorporates key principles - such as data must be lawful, fair, accurate and the avoidance of discriminatory effects - which are an important basis for algorithmic fairness. Other elements of the GDPR are the provisions of Article 15 (Right of access by the data subject) and Article 22 (Automated individual decision-making) on substance; but Hacker considers that these provisions, in particular Article 15(1)(h), should be supplemented by a “public enforcement that aims at uncovering the right metrics and exact causes of discrimination” ([34], p. 27). Public enforcement refers to Articles 83 (General conditions for imposing administrative fines) and 58 (powers of supervisory authority) as well as Article 35 (Data protection impact assessment). Thus, “National data protection authorities should moreover make use of algorithmic audits and data protection impact assessments, according to Article 58(1)(b) and Article 35 and seq. GDPR, to uncover the causes of bias and to enforce adequate metrics of algorithmic fairness” ([34], p. 35).

The following developments focus on explicability, comparing the legal and machine learning models, around the idea of fairness through explainability.

In computer science, a decision is the response of an algorithm (implemented as software) to a query. From the point of view of a mathematical formalism, a decision is the partition of a given set. A partition consists in the separation of the set into subsets with an empty intersection and whose union constitutes the set. Each subset is called an equivalence class. If these classes are ordered and defined with reference to a norm we call the decision a rating. If the classes are ordered but not defined with reference to a standard we call the decision a ranking. If the classes are not ordered, but are defined in reference to a standard we call the decision an assignment; finally, if the classes are not ordered and are not defined in reference to a standard we call the decision a clustering.

The activities of an entity (human or artificial) that lead to a decision are a decision process. From this point of view the activity of an algorithm that has to calculate for a set of objects a score (of something) is a decision process. In fine, the algorithm makes a decision (e.g. the outcome of these calculations).

However, in most cases, this activity is part of a larger process that consists of assisting another entity that, in turn, must take a decision. We call this set of activities a decision support process. The decisions made in a decision support process are “recommendations”. It may be possible for decision support processes to be used in another entity’s decision process. It is possible to go as far as an intertwining of decision processes and decision support processes that produce a cascade of recommendations … up to a decision that we will call “final” … The notion of liability generally applies to this “final decision” … even if in reality any decision (mathematically defined) can be considered as carrying liabilities. For example, when a bank employee receives a credit application, an algorithm calculates the applicant’s credit score. In such a decision process, the algorithm “decides” the credit score (decision) and transmits it to the employee (recommendation) who in turn decides to grant or refuse the credit (final decision). It is essentially to this “final decision” that the law relating to the explainability of algorithms focuses.

From a computer science viewpoint, profiling is a clustering decision problem where individuals characterized by a set of “external” attributes (such as demographics) and by a set of behavioral attributes (such as buying preferences) are grouped in classes of “similarity”. In case the classes of potential behavior are predefined we turn having an “assignment” problem. In the first case we do not know a-priori which are the equivalence classes which define the correspondence between external and behavioral attributes and the clustering allows to “discover” them. In the second case we have a hypothesis about how many behavioral patterns are possible among the observed population and we test it. The result of a profiling exercise is that all individuals clustered or assigned to the same profile (behavioral pattern) are supposed to behave similarly and under such a perspective can be targets of actions aiming at influencing or modifying this behavioral pattern.

The notion of automated individual decision is not been defined in the GDPR, whereas the notion of profiling is defined as a form of automated processing aimed at evaluating a natural person, whether to analyze or predict his or her work performance, economic situation, behavior, etc.¹⁸¹⁸18See GPDR, article 4.. Therefore, the legal notion of automated decision does not overlap with the distinction made in AI between “automated”, “autonomous” and “algorithmic” decision-making ([15]). In AI, the notion of an automated decision means that the decision has been made on the basis of a series of precise predefined actions without further intervention by a human. These decisions would be easily predictable. The notion of autonomous decision implies that only the general objectives have been established by a human, leaving it to the machine to determine how to achieve them. Yet, although referring to the term automated decision, the Article 22 of the GDPR is about algorithmic decisions (from the AI perspective). This more general notion simply means that a decision has been made with the help of an algorithm. It is therefore a notion that encompasses both autonomous and automated decisions.

It should be emphasized that not every algorithmic individual decision is covered by Article 22: only those that are exclusively based on automated processing and that produce legal effects concerning the person or significantly affect him or her in a similar way are concerned. This last expression is a matter of debate among legal scholars. Indeed, the prohibition of exclusively automated processing only concerns decisions that have a significant impact. This goes without saying when the decision has legal effects such as the cancellation of a contract, the right or refusal of a social benefit granted by law such as a family allowance, or a housing allowance, the refusal of entry into a country ,etc. On the other hand, it is not always easy to determine what is an automated decision that, without having a legal effect, affects the data subject “similarly significantly affects him or her”.

Recital 71 of the GDPR simply states “automatic refusal of an online credit application or e-recruiting practices without any human intervention”¹⁹¹⁹19On the difficulties of interpreting this notion of significantly affecting in a similar way to a legal effect see the Guidelines on Automated individual decision-making and Profiling for the purposes of Regulation 2016/679, revised version of February 6, 2018, p. 23.. This covers the loss of a chance, of an opportunity. This concerns decisions concerning access to a service in the field of health or education.

For some authors, “explainability” and “interpretability” are interchangeable notions ([9], p. 8), while others make a distinction. The common point between these two labels is that in both cases the aim is to make the decisions made by an algorithm understandable. Nevertheless, in the field of AI, some authors consider that interpretability aims at globally evaluating the process of a decision, i.e., in reality, at making the model used understandable. In other words, interpretability answers the question of how an algorithm makes a decision in a general way. In other terms we can distinguish three levels which can technically be considered as explanations for a given algorithm and its execution.

1. 1.

   The first one is descriptive: it consists in reconstructing and tracing each single step the algorithm performs from receiving an input until it delivers the foreseen output (computer scientists collect this information in what they call a .log file). It is essentially a descriptive explanation of what the algorithm did or does and most of the times is interpretable only by coding specialists.

2. 2.

   The second one is logical: it consists in reconstructing the reasons for which an algorithm performs a given step, while it runs. It is essentially a logical explanation through which each single step can be viewed as a causal relation. Such explanations are useful in order to show that the output of the algorithm is logically related to the input. Formal verification of algorithms and security checking use this type of explanation in order to demonstrate that a given algorithm actually does what the specifications were expecting to happen, and that the execution is robust, secure, trustworthy, etc.

3. 3.

   The third one is argumentative: it consists in providing the ultimate reasons for which an algorithm computes a certain output, given a certain input. Such reasons are essentially of two types: the data provided, and the procedure used. A typical example is the result of a voting procedure: the result depends on the ballots casted by the voters, but also on how the majority is computed with that particular procedure. An argued explanation always ought to provide both of them. This type of explanation becomes problematic when the algorithm modifies the execution each time depending on knowledge which cumulates from previous executions or when the algorithm includes components for which is practically impossible to make a logical connection between input and output (black boxes).

For [12], a decision rule is interpretable if we understand how it associates a response with observations (e.g., a decision tree); it is explicable if we understand on what elements the decision is based on. According to [63], interpretability “means that the output of the AI algorithm is understandable to a subject matter expert in terms of concepts from the domain from which the data are drawn” whereas explainability “means that one can identify the properties of the input to the AI algorithm that are responsible for the particular output and can answer counterfactual or ‘what-if’ questions”.

In contrast, explicability goes further because it involves specifying in a concrete case which specific variables were decisive in making a particular decision. Substituting the semantic distinction between interpretability and explicability, some authors in machine learning ([33]) oppose global explicability to local explicability. Global explainability aims at explaining the whole algorithm, while local explainability is the ability to explain a specific algorithmic decision. We can also distinguish between “post-hoc explanations” and “build-interpretable models” ([5]).

Post-hoc explanations are constructed through statistical analysis after the data have been revealed (and the hypothesis tested). They can positively be used in order to deep the understanding of a given result but can also negatively contribute to constructing meaningless correlations. Build-interpretable models are models constructed in such a way that explanations are computed (or collected) while used. This might improve efficiency in computing explanations, but present two weaknesses: it may produce out-of-the-context meaningless interpretations and it can conceal other possible explanations for which no provision has been considered within the specifications.

These semantic variations are not really found in legal doctrine. Most institutional documents and the legal literature use the term “explicability”, sometimes the word “intelligibility” ([21], p. 53). However, it is acknowledged that there are also two levels of explicability requirements. On the one hand, there is an ex ante explicability, which aims to inform the individual about the logic of the algorithm and which is closely linked to a right to information. On the other hand, there is an ex post explicability, which aims at explaining to an individual why an algorithm has made a specific decision. The debate in the legal sphere is to what extent the second form of explainability constitutes a legal obligation at the European level, as we shall see later.

We have already previously mentioned the GDPR. It is important to point out here that it is subject to various and contradictory interpretations with regard to the explainability requirement. Some consider that the word only appears in one recital (No. 71), and thus explainability is not in the core of the Regulation ([70]). Others argue that it is in an overall reading, which articulates different provisions, which an explainability constraint can be found in the GDPR ([15]; [34]). Finally, for others, explainability in the GDPR is not the answer ([30]).

Apart the GDPR, the European Regulation 2019/1150²³²³23Regulation (EU) 2019/1150 of the European Parliament and of the Council of 20 June 2019 on promoting fairness and transparency for business users of online intermediation services (Text with EEA relevance). imposes an obligation of explicability on platforms, in particular online intermediaries and search engines, with regard to their rankings These obligations are intended to be completed by other texts currently being drafted. Thus, a series of new obligations of fairness and transparency of algorithms used by digital platforms are at the heart of the adoption of the regulations on digital services legislation²⁴²⁴24Proposal for a Regulation of the European Parliament and of the Council on a Single Market For Digital Services (Digital Services Act) and amending Directive 2000/31/EC, COM/2020/825 final. and on digital markets legislation²⁵²⁵25Proposal for a Regulation of the European Parliament and of the Council on contestable and fair markets in the digital sector (Digital Markets Act), COM/2020/842 final. These two texts contain the seeds of new forms of algorithmic explainability around the use of algorithmic moderation tools to fight online hate, or recommendation systems.

Finally, the Artificial Intelligence Act Project²⁶²⁶26Proposal for a Regulation of the European Parliament and of the Council laying down harmonized rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union legislative acts, COM/2021/206 final. will impose explainability obligations on AI systems modulated according to risk, with reinforced obligations for high-risk AI systems. Most of these European texts currently being adopted seem to envisage above all an ex ante explanation of the use of algorithms for decision-making and the general logic of the algorithms used, so as to inform the persons concerned.

In France, the 2016 Digital Republic Act (Loi République Numérique) is the starting point for French regulations on the transparency and explainability of algorithmic administrative decisions. Indeed, Article 4 of the Digital Republic Act created a new article L-311-1-1 in the Code of Relations of the Public with the Administration (CRPA), which states that: “an individual decision taken on the basis of algorithmic processing shall include an explicit statement informing the person concerned. The rules defining this processing as well as the main characteristics of its implementation shall be communicated by the administration to the person concerned if he or she so requests”. It is mainly an obligation to communicate upon request. Article 311-3-1-2 of the CRPA specifies its scope: “The administration shall communicate to the person who is the subject of an individual decision taken on the basis of algorithmic processing, at the latter’s request, in an intelligible form and subject to not infringing secrets protected by law, the following information: the degree and mode of contribution of the algorithmic processing to the decision-making; the data processed and their sources;cthe processing parameters and, where applicable, their weighting, applied to the situation of the person concerned; the operations carried out by the processing”.

Here, explicability is essentially given a posteriori; it is not only global but also local. Beyond explicability itself, this article includes an obligation of transparency constituting a form of right to information with the obligation to explicitly mention the existence of an algorithmic decision. Moreover, a decision taken solely on the basis of an algorithmic treatment that does not include explicit mention is considered null and void. Etalab, a department of the interministerial digital direction, in charge of implementing the State’s strategy in the field of data, has developed a guide to assist administrations in their obligations to make public algorithms explainable²⁷²⁷27ETALAB, Expliquer les algorithmes publics, https://guides.etalab.gouv.fr/algorithmes/, last version consulted 13/07/2021., in addition to guides on the opening of source codes²⁸²⁸28ETALAB, Ouvrir les codes sources, https://guides.etalab.gouv.fr/pdf/guide-logiciels.pdf, last version consulted 13/07/2021.. This general obligation of explicit mention includes both exclusively automated processing operations and those that simply constitute an aid to decision-making. This compels administrations to draw up an inventory of their main algorithmic treatments. This transparency obligation is part of the broader idea that the administration must account for the use of its public algorithms. Finally, these obligations concern all individual administrative decisions, whether they concern natural persons or legal entities.

These obligations introduced in the Loi sur la République numérique are complemented by the more specific obligations related to individual decisions based on automated processing involving the processing of personal data, as set forth in the GDPR. Thus, Article 13$\S$f) of the GDPR reinforces the right to information whenever there is an automated individual decision making. It is a matter of giving “meaningful information about the logic involved, as well as the significance and the envisaged consequences of such processing for the data subject”. This provision seems, technically, to correspond to a principle of interpretability ([12]). The legal doctrine sees in this at least the recognition of an overall obligation of explainability. For its part, Article 22 of the GDPR enshrines the right not to be subject to an exclusively automated individual decision, while leaving it to the Member States to authorize such algorithmic decisions, subject to certain guarantees. It is in the context of practicing its margin of maneuver that France has adopted a specific legal basis to authorize exclusively automated individual administrative decisions. The chosen formulation results in an extended right of explainability. Indeed, according to article 47$\S$2 of the Loi Informatique et Liberté, the processing officer must “ensure mastery of the algorithmic processing and its developments in order to be able to explain, in detail and in an intelligible form, to the data subject the way in which the processing has been implemented with regard to him”. This is what the doctrine calls a right of local explicability. Thus, the debate on the existence or not of a right of explicability of concrete decisions taken with respect to an individual exclusively by means of an algorithm (see [70]) has had little echo in French doctrine. At most, some authors have questioned the capacity of France to impose such a right of local explicability, while remaining within the limits of its margin of maneuver with respect to the GDPR (see for example [18]).

In addition, the Constitutional Council has made some useful clarifications. On the one hand, it considers that “algorithms that are likely to revise the rules they apply themselves, without the control and validation of the person in charge of the processing, cannot be used as the exclusive basis for an individual administrative decision”²⁹²⁹29Conseil Constitutionnel, Décision n° 2018-765 DC June12, 2018, point 71.. This interpretation seems to limit the possibility for the French administration to use so-called “deep learning” algorithms. On the other hand, the Constitutional Council reminds us that the administration’s duty to explain at the request of the person concerned also restricts its room for manoeuvre in the choice of the algorithmic tool. Indeed, “when the operating principles of an algorithm cannot be communicated without infringing on one of the secrets or interests set out in 2$\S$ of article L. 311-5 of the code of relations between the public and the administration, no individual decision may be taken on the sole basis of that algorithm³⁰³⁰30Cf. Ibid., point 70.. In other words, the use of algorithms protected by intellectual property rights seems to be excluded in order for the administration to fulfill its obligation of transparency. These two interpretive caveats attempt to combat the “black box” phenomenon, i.e., the inability to understand the exact reasons that led an algorithm to make a particular algorithmic decision. This approach illustrates France’s desire to build a transparent model for algorithmic decisions that can also inspire private actors by indirectly encouraging them to implement similar safeguards.

Finally, it should be noted that explicability under Article 22 of the GDPR can only be given to the individual affected by the automated individual decision. In France, this limit was circumvented by the Constitutional Council in a decision opposing a university to a trade union that wanted to understand the logic of the algorithms used by universities to select future students. The Constitutional Council considered that it was a constitutional principle that the right of access to administrative documents allowed third parties to obtain “in the form of a report, the criteria according to which the applications were examined and specifying, where appropriate, the extent to which algorithmic processing was used to carry out this examination”³¹³¹31Cf. Conseil Constitutionnel, DC- 2020-834 Union nationale des étudiants de France, 3 avril 2020, QPC, point 17. This constitutional principle was recognized on the basis of Article 15 of the Declaration of Human Rights of 1789, according to which ”Society has the right to hold any public official accountable for his administration.. This can be seen as the consecration of a French right of global explicability extended to third parties, in the case of administrative decisions based on public algorithms. This possibility opens the way to auditability by civil society.

The Law on Bioethics of August 2, 2021 is the first to introduce the notion of algorithm in connection with artificial intelligence in the public health code. The expressly provides, for the first time, an obligation of explainability for the designers of certain algorithmic treatments for health professionals who use them for an act of prevention, diagnosis or care. This article also provides for the obligation to inform the patient and to warn him or her, if necessary, of the interpretation that results from the processing of algorithmic data.

It should be noted that the government bill was enriched by parliamentary amendments. The bill only proposes a traceability of the actions of the algorithmic treatment. The deputies, inspired by the recommendations of the Commission Nationale Informatique et Liberté (CNIL)³²³²32CNIL, Comment permettre à l’homme de garder la main ? Les enjeux éthiques des algorithmes et de l’intelligence artificielle, décembre 2017. and the Conseil d’État³³³³33Conseil d’État, Révision de la loi de bioéthique : quelles options pour demain? 28 juin 2018, have proposed to add an explainability requirement, which aims to allow users of these artificial intelligence systems to understand the general logic of their operation.

This requirement implies that their designers provide users with the information necessary for this understanding and that health professionals can contribute their expertise from the moment the algorithms and health data collection strategies that feed them are developed. It also implies that healthcare professionals benefit from training that enables them to understand how these systems work in order to identify their limits and to be able to explain to patients the basis on which medical decisions concerning them are made.

The notion of explainability is widespread in the academic world as well as in the institutional discourse at both the European and national levels³⁴³⁴34See for example, CNIL, Comment permettre à l’homme de garder la main? Les enjeux éthiques des algorithmes et de l’intelligence artificielle les enjeux éthiques, décembre 2017, p. 30, 50 ou 53; Défenseur des Droits en partenariat avec la CNIL, Algorithmes: prévenir l’automatisation des discriminations, 2020 p. 8; Commission Nationale consultative des droits de l’Homme, Avis sur la lutte contre la haine en ligne, 8 juillet 2021, p. 27.. The lack of use of the term explainability by legislators is justified because the concept is so loose and has only been introduced into law in contextualized form. [49] rightly states that explainability depends on four important factors:
- The recipient of the explainability, i.e. the audience targeted by the explanation. For example, its level will be different depending on whether it is a user or a regulator.
- The level of importance and impact of the algorithm. The explainability of an autonomous car accident does not have the same level of importance as that of an advertising or video recommendation algorithm.
- The legal and regulatory framework, which is different in different geographical areas, such as in Europe with the General Data Protection Regulation (GDPR).
- The operational environment of explainability, such as its mandatory nature for certain critical applications, the need for certification prior to deployment, or the facilitation of use by users ([49], p. 14).

Thus, there is not a general right to explainability in digital law, but rather obligations of explainability introduced in various regulations, as we have already mentioned. It should be remembered that most of these require the intervention of the regulatory power, supplemented by regulatory authorities that try to explain how to implement this right of explainability. However, this notion is most often linked to a requirement for transparency.

Similarly with the OECD recommendation on AI, transparency and explainability are the third of the five principles that underpin a responsible approach to AI that can generate trust. The other principles are i) inclusive growth, sustainable development and well-being; ii) human-centred values and fairness, iii) transparency and explainability; iv) robustness, security and safety; and v) accountability. In paragraph 1.3. on “transparency and explainability”, the OECD considers that:
“AI Actors should commit to transparency and responsible disclosure regarding AI systems. To this end, they should provide meaningful information, appropriate to the context, and consistent with the state of art:
i.to foster a general understanding of AI systems,
ii.to make stakeholders aware of their interactions with AI systems, including in the workplace,
iii.to enable those affected by an AI system to understand the outcome, and,
iv.to enable those adversely affected by an AI system to challenge its outcome based on plain and easy-to-understand information on the factors, and the logic that served as the basis for the prediction, recommendation or decision.”

Transparency is conceived as a broader principle that can be satisfied with making raw information accessible or intelligible, whereas explainability seems to imply the need to make the information given understandable³⁷³⁷37[9], p. 10 contrast with other authors who consider that transparency is a sub-principle of explicability. . Thus, the opening of the source code of an algorithm is most often linked by jurists to the principle of transparency and more indirectly to explainability because this code is rarely understandable to a layman. From the outset, the principle of explainability raises a difficulty of demarcation with respect to other concepts, which explains why its field of application is debated.

The principle of explainability raises four main types of debate about its scope, which are closely intertwined.

Firstly, it is important to note that constructing explanations, justifications, interpretations is not a straightforward and neutral activity based upon exclusively technical specifications. It is purposeful and subjectively established following three dimensions ([5]).

1. 1.

   Explanations for whom? Algorithms, platforms and other autonomous artifacts are designed, coded, used by and produce impacts for several different stakeholders: software engineers, indirect users (who specify the tool, but do not directly use it), end users (the ones who actually use it for some specific purpose), impacted citizens/customers/clients, the society as a whole, etc. Each of them has different types of expectations from such artifacts and consequently expects different types of explanations.

2. 2.

   Explanations for which need? Given the diversity of implied stakeholders it is natural to expect a diversity of motivations and specifications for explanations. Such generic motivations include testing, checking, monitoring, understanding and appealing/revising/updating.

3. 3.

   Explanations for doing what? Besides explanations designed for some generic purpose there are specific uses of the explanations which need to be considered and anticipated (as much as possible). Under such a perspective, explanations constructed for some specific purposes generally do not fit the requirements of other purposes and other specific stakeholders who might want to use explanations.

Secondly, this semantic debate is in fact a debate about the object of explainability. The duality of the object of explainability has been recalled by the HLEG, which distinguishes between two types of explainability. On the one hand, technical explainability, and on the other hand, explainability linked to human decisions taken on the basis of this technique:

“Technical explainability requires that the decisions made by an AI system can be understood and traced by human beings. Moreover, trade-offs might have to be made between enhancing a system’s explainability (which may reduce its accuracy) or increasing its accuracy (at the cost of explainability). Whenever an AI system has a significant impact on people’s lives, it should be possible to demand a suitable explanation of the AI system’s decision-making process. Such explanation should be timely and adapted to the expertise of the stakeholder concerned (e.g. layperson, regulator or researcher). In addition, explanations of the degree to which an AI system influences and shapes the organisational decision-making process, design choices of the system, and the rationale for deploying it, should be available (hence ensuring business model transparency)”³⁸³⁸38HLEG, Ethics Guidelines for Trustworthy AI, 8 April 2019, p. 18..

Thus, the contextual character of explainability necessarily implies that its object must be precisely determined. Schematically, explainability can concern the choice of a data set, the algorithm that will use the data, the model that will be generated and finally the content, the decision or the prediction that will be made on the basis of this model. Even when explainability only applies to the algorithm, some authors try to classify the possible forms of explainability according to the type of algorithm involved.

Thirdly, there is a debate about the extent of explainability. This debate has several aspects. On the one hand, there is a debate on the necessity, the relevance or not of systematically providing for a form of explainability. Some authors propose a classification of artificial intelligence systems according to risk, with the objective of limiting explainability to certain types of high-risk artificial intelligence ([56]). This is the direction in which the European legislator seems to want to go. Article 13 of the recent European proposal for a regulation limits transparency and the provision of information to users to high-risk AI systems. Beyond the risk approach, some authors consider identifying the degree of explainability in a given situation through an analysis of costs and benefits for society ([9]). The same authors identified seven categories of costs to be considered in this evaluation: design, reduction of prediction accuracy, creation and storage of logs, violation of business secrecy, conflict with security and other policy objectives, reduction of decision flexibility in the future, and slowing of innovation. As for the operational benefits, they refer to the ability to foster user confidence in the system, and the ability to make the algorithms more robust and certifiable. These different criteria must be taken into consideration in a contextual manner.

The extent of explainability is also debated in relation to its recipient: expert, regulator or individual. Indeed, the intelligibility of the explanations given necessarily varies according to the knowledge of the recipient. Very detailed and complex information will not be very useful for the layman, who will mainly be looking to know in simple terms what algorithmic model was used and what criteria were used in the decision taken against him.

The scope of explainability is central to other related notions, including traceability and audibility. As the HLEG outlines, “An explanation as to why a model has generated a particular output or decision (and what combination of input factors contributed to that) is not always possible. These cases are referred to as “black box” algorithms and require special attention. In those circumstances, other explicability measures (e.g. traceability, auditability and transparent communication on system capabilities) may be required, provided that the system as a whole respects fundamental rights” (HLEG, Ethics guidelines for trustworthy AI, 2019, p. 13).

Thus, traceability can be a minimum requirement when full explainability is not possible. In this context the HLEG defines traceability as “The data sets and the processes that yield the AI system’s decision, including those of data gathering and data labelling as well as the algorithms used, should be documented to the best possible standard to allow for traceability and an increase in transparency. This also applies to the decisions made by the AI system. This enables identification of the reasons why an AI-decision was erroneous which, in turn, could help prevent future mistakes” (HLEG, Ethics guidelines for trustworthy AI, 2019, p. 18).

Traceability can be based on logging obligations, i.e., memorizing the various actions carried out by a model or an algorithm. Auditability goes further since it allows a third party to check ex ante and/or ex post the model and/or the algorithmic decisions taken on the basis of a model. Auditability is therefore a means of verifying the relevance of the explanations provided in the context of both global and local explainability. Although distinct, the notions of traceability and auditability are closely linked to explainability.

Fourthly, there is a debate about the limits to the right of explainability. This debate is prominent among lawyers who are concerned with the extent to which explainability could be challenged on the grounds of intellectual property rights, business secrets, or other secrets related to defense or national security ([15]; [48]). In machine learning research, it is mainly the technical limits of explainability that are examined. It is sometimes necessary to explain that in certain situations explanations can only be partial or should only focus on the most important reasons that led to the decision in a specific situation. At this point, it is worth mentioning the existence of a bi-disciplinary research (combining machine learning and law) whose purpose is to come up with a grid for assessing the technical feasibility of the explainability obligation enshrined in Article 22 of the GDPR ([15] – table 6).

Assessing the technical feasibility of the explainability requirement is interesting, but while Article 22 of the GDPR is important, it is not the only legal provision that addresses the topic. In addition, the basis and method for qualifying the technical feasibility of different types of explanations are not detailed by the authors. We now come to the observation that there is a certain diversity, not to say heterogeneity, in the legal provisions on explainability in European Union law and in French law.

The GDPR includes, in addition to Article 22, two important provisions: Article 15 and Recital 71. The two articles have different objectives: the information obligation (Art. 15) and the right to a final human decision (Art. 22); they give rights to individuals, albeit with restrictions for beneficiaries of the right to a humane decision under Article 22 (see supra subsection 4.4.1 and Table 7, Appendix 1 below). On the other hand, Recital 71, as well as Article 29 of the Working Party Guidelines (which inspired the GDPR), target algorithm experts, designers and programmers. These two texts have the objective of improving models, while the previous ones have other purposes. Finally, the European Regulation 2019/1150 already mentioned, targets economic actors (providers of online intermediation services or search engines) and requires them to be in a position to explain (to the regulator or the judge) the ranking parameters used in their algorithms.

In French law, two laws (the Computer and Liberty Act and the Bioethics Act), as well as the Code of Relations of the Public with the Administration and two decisions of the Constitutional Council, provide a contrasting landscape of rules relating to the explainability of algorithmic decisions (Table 8, Appendix 2 below).

In its decision of June 12, 2018, the Constitutional Council, seized by 60 senators against the law transposing European texts on personal data, decided that the administration must refrain from using autonomous algorithms and could not invoke the protection of algorithms by intellectual property rights or commercial secrecy to exempt itself from an obligation of explainability of algorithmic decisions. In its decision of April 3, 2020, seized of a priority question of constitutionality submitted by the Conseil d’Etat confronted with the appeals of a student union (UNEF) requesting that universities make public the information relating to the criteria used, including by algorithmic treatments, to decide to admit or refuse a university application (via the Parcours Sup platform). On the other hand, the Code of Relations of the Public with the Administration and the Bioethics Law define explainability in terms of information about the structure of models, but the former concerns individuals while the latter also concerns healthcare professionals and designers of algorithms used in healthcare.