When an algorithm determines your future, who is accountable when it gets things wrong?
Digitalization has transformed how power is exercised by governments, businesses and social institutions. Personal data are collected, combined and analyzed on an unprecedented scale, while algorithms help determine access to employment, credit, housing, healthcare, education, social security and information. Decisions are increasingly shaped not only by visible rules or human officials but also by databases, risk profiles, ranking systems, authentication procedures and automated recommendations.
Legal context note (updated 9 August 2026).
Digital decision-making in the Netherlands is already governed in part by the GDPR, the EU Artificial Intelligence Act, sectoral EU and national law, and—subject to ratification and domestic effect—the Council of Europe Framework Convention on Artificial Intelligence727. The AI Act was amended in July 2026728. The constitutional question is therefore not whether regulation exists, but which durable guarantees should bind public power across technologies and remain enforceable when ordinary rules are incomplete, fragmented, or changed729.
Deterministic rules, statistical models, machine-learning systems, and generative AI create different forms of uncertainty and control. Systems that merely support a professional judgement must also be distinguished from systems that materially determine an outcome. Safeguards should follow function and risk rather than marketing labels: purpose, affected interests, scale, reversibility, opacity, data dependence, and the authority actually retained by the responsible institution determine what explanation, testing, human review, documentation, and remedy are required.730.
Digital systems can improve consistency, speed and accessibility. They may reduce some forms of human error and help public institutions process large numbers of cases. At the same time, they concentrate knowledge and decision-making power in organizations that control data, computing capacity and technical expertise. A defective rule or biased dataset can affect thousands of people before the underlying problem becomes visible.
Traditional privacy protection is therefore no longer sufficient731. The central issue is not only which data are collected, but also what conclusions are drawn from them and how those conclusions influence people’s opportunities. Data processing may comply formally with privacy rules while still producing opaque classification, discriminatory selection or exclusion from essential services. Constitutional protection must address the entire progression from identification and classification to prediction, decision-making and behavioral influence.
Heightened constitutional safeguards should apply where a digital system materially affects a person’s legal position, access to an essential service or opportunity, exposure to investigation or sanction, or ability to participate in public life. A recommendation is material when officials normally follow it, when departure requires special approval, or when it changes the evidence, timing, or burden confronting the individual. This functional test prevents nominally advisory systems from escaping scrutiny.
Profiling presents particular risks because people can be judged partly through statistical similarities with others rather than their own established conduct. Risk scores may be treated as individual facts, turning probability into suspicion and forcing citizens to prove their reliability. Information about location, language, education, purchasing behavior or social relationships can also operate as a substitute for protected characteristics such as ethnicity, disability, religion or socioeconomic position732.
Historical data are not neutral733. They reflect earlier social inequalities, administrative priorities and discriminatory practices. Systems trained on those records can reproduce existing patterns while presenting their output as objective technical analysis. Average accuracy can be misleading when error rates are substantially higher for particular groups. Algorithmic discrimination must therefore be evaluated through actual outcomes, including combined disadvantages affecting people who belong to several vulnerable categories.
Human review is meaningful only when the reviewer has time, competence, authority, relevant information, and a genuine possibility of changing the outcome734. The reviewer should see the material inputs and limitations, consider individual circumstances, record reasons for accepting or departing from the system, and remain institutionally responsible. Where no reviewer can understand or lawfully correct a high-impact outcome, the system should not be used for that decision.
A person affected by a significant digital outcome should receive the main factors and data that shaped the case, the responsible authority, the available evidence, and a practicable route to human reassessment735. When an error is established, correction must propagate through connected databases, models, contractors, and downstream decisions. If continued use creates a credible risk of serious harm, the contested output should be suspended or marked as unreliable until review is complete. Technical documentation is valuable, but it is not a substitute for an individual remedy.
Protection must extend beyond isolated individual decisions because algorithmic power is often systemic. Biased selection rules, unequal error rates and defective data chains may affect entire groups without producing one easily identifiable decision. Rights assessments, recurring discrimination tests, independent audits, collective procedures and institutional remedies are necessary to identify and correct such patterns. Transparency must be connected to supervision and restoration; publishing a register or report is of little value if nobody can examine or change the system736.
Digital accessibility is another essential component737. Public services increasingly depend on online identification, applications and communication. People without adequate equipment, digital skills, language ability or cognitive capacity may consequently lose practical access to rights and services. Essential public functions should retain usable alternatives, such as personal, telephone, written or supported access. These alternatives need not reproduce every digital process, but they must offer comparable timeliness, affordability and legal protection.
Private digital power also requires attention738. Major platforms and infrastructure providers can become unavoidable gateways to communication, work, commerce, payment and public participation. Employers, insurers, banks and landlords can use automated systems to control access to essential opportunities. Government therefore has a responsibility to regulate powerful private actors where dependency, lack of realistic alternatives or the essential nature of a service threatens autonomy and equality.
Outsourcing does not transfer constitutional responsibility739. A public authority or company commissioning an algorithmic system must remain responsible for its purpose, data, consequences and remedies. Contracts with technology suppliers should provide access to documentation, audit rights, data portability and cooperation with regulators and courts.
Commercial confidentiality and cybersecurity may justify restricted disclosure to the public, but they should not prevent independent investigation.
Before procurement, a public authority should determine whether it can obtain the documentation, logging, audit access, correction rights, security support, data portability, and termination assistance needed to meet its legal duties. Contracts should allocate responsibility without transferring it away from government. A high-impact system should not be acquired when vendor dependence makes independent investigation, remedy, migration, or lawful discontinuation practically impossible.
A risk-based approach would preserve the benefits of digital innovation740. Simple, reversible and low-impact systems do not require the same safeguards as systems used for fraud detection, sanctions, policing, biometric identification or access to essential services. High-impact systems would need stronger assessment, documentation, auditing, monitoring and human control. The possibility of not using the system, or using a less intrusive alternative, should also be considered.
For government and business, these requirements would have substantial practical consequences. Organizations would need better records of which systems they use, which data and assumptions support them and who is accountable. Data quality would have to be assessed for accuracy, relevance and representativeness. Systems would need sufficient logging to reconstruct important decisions, while high-risk applications would require continuing evaluation rather than a single approval before deployment.
Regulators would require technical expertise, access to source information and stable funding. Cooperation among privacy, equality, market and sectoral regulators would be necessary because a single system may raise several kinds of risk. Smaller authorities and businesses may need shared expertise, model contracts and centralized audit support. Without sufficient institutional capacity, constitutional rights could remain abstract while technical power continues largely unchanged.
The reforms carry real risks. Extensive and uniform compliance requirements could burden smaller innovators and favor large corporations able to absorb the costs. Demands for explanation may become meaningless standardized language, while human review may become ritual confirmation of automated outcomes. Excessive legal intervention could slow beneficial experimentation. These dangers support proportionate obligations and supervised experimental space, not the unrestricted use of systems with serious or irreversible consequences.
The Constitution should not define particular models, audit methods, or technical documentation. Its added value would be to guarantee human dignity and non-discrimination in digital administration, identifiable public responsibility, access to consequential reasons, meaningful human contestation, effective correction, and a usable alternative where digital access would otherwise exclude a person from an essential public function. Risk classifications, conformity assessments, registers, testing methods, and sectoral prohibitions should remain in EU and ordinary legislation so they can adapt741.
The resulting framework would not obstruct digitalization as such. Automation remains valuable when it genuinely improves services, knowledge and consistency. It becomes constitutionally problematic when efficiency is achieved by reducing people to unchallengeable profiles, shifting the consequences of errors onto citizens or making access to rights dependent on systems they cannot understand or use. Digital innovation would remain possible, but it would operate within a structure of responsibility, contestability and democratic correction.
726 Council of Europe Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law, CETS No. 225 (2024), arts. 4–13, https://rm.coe.int/1680afae3c; Council of Europe treaty-status chart, https://www.coe.int/en/web/Conventions/full-list/?module=signatures-by-treaty&treatynum=225.
727 https://digital-strategy.ec.europa.eu/en/news/ai-omnibus-enters-force
728 Regulation (EU) 2024/1689 (Artificial Intelligence Act), official consolidated text, including amendments through 27 July 2026, https://eur-lex.europa.eu/eli/reg/2024/1689/2026-07-27/eng; Regulation (EU) 2026/1744, https://eur-lex.europa.eu/eli/reg/2026/1744/oj/eng. Verify the consolidation date immediately before publication.
729 OECD, Recommendation of the Council on Artificial Intelligence, OECD/LEGAL/0449, amended 2024; NIST, Artificial Intelligence Risk Management Framework 1.0, NIST AI 100-1 (2023).
730 Regulation (EU) 2016/679 (GDPR), arts. 5, 13–15, 22, and 35; Sandra Wachter, Brent Mittelstadt, and Luciano Floridi, ‘Why a Right to Explanation of Automated Decision-Making Does Not Exist in the GDPR,’ International Data Privacy Law 7, no. 2 (2017): 76–99.
731 Frederik Zuiderveen Borgesius, Discrimination, Artificial Intelligence, and Algorithmic Decision-Making (Strasbourg: Council of Europe, 2018); Solon Barocas and Andrew D. Selbst, ‘Big Data’s Disparate Impact,’ California Law Review 104 (2016): 671–732.
732 Joy Buolamwini and Timnit Gebru, ‘Gender Shades,’ Proceedings of Machine Learning Research 81 (2018): 77–91; National Institute of Standards and Technology, Face Recognition Vendor Test reports. Match any empirical claim to the relevant system and population rather than generalizing from one technology.
733 Article 22 GDPR; Article 14 and related high-risk-system provisions of Regulation (EU) 2024/1689; Council of Europe Framework Convention on AI, arts. 7–9 and 14.
734 Council of Europe Framework Convention on AI, arts. 12–14; GDPR, arts. 15–16 and 19; European Data Protection Board, Guidelines on Automated Individual Decision-Making and Profiling, WP251rev.01 (2018).
735 Regulation (EU) 2024/1689, provisions on risk management, data governance, monitoring, incident reporting, and fundamental-rights impact assessment; Reuben Binns, ‘Algorithmic Accountability and Public Reason,’ Philosophy & Technology 31 (2018): 543–556.
736 Convention on the Rights of Persons with Disabilities, arts. 9, 19, and 21; Directive (EU) 2016/2102 on the Accessibility of Websites and Mobile Applications of Public Sector Bodies; European Accessibility Act, Directive (EU) 2019/882.
737 Regulation (EU) 2022/2065 (Digital Services Act); Regulation (EU) 2022/1925 (Digital Markets Act); Julie E. Cohen, Between Truth and Power (Oxford: Oxford University Press, 2019).
738 UN Guiding Principles on Business and Human Rights, principles 4–6; OECD, Governing with Artificial Intelligence (Paris: OECD Publishing, 2025); European Commission model contractual clauses for AI procurement, where applicable.
739 Regulation (EU) 2024/1689; Council of Europe Framework Convention on AI, art. 16; NIST AI RMF 1.0.
740 Lawrence Lessig, Code: Version 2.0 (New York: Basic Books, 2006); Mireille Hildebrandt, Smart Technologies and the End(s) of Law (Cheltenham: Edward Elgar, 2015).
Would you like to explore how a relational understanding of humanity can transform democracy, law, the economy, and public institutions? Click the link below to read the full book, Becoming Human Together, and discover a new framework for building a more humane, democratic, and correctable society within ecological limits.
Reacties
Een reactie posten