About the Document
This is a research report ('Onderzoeksrapport') titled 'Algoritmen: Hoe Amsterdam algoritmen beter kan toepassen', published by the Rekenkamer Amsterdam (Amsterdam Court of Audit) and dated 20 October 2023 in its definitive version. It forms part of a full reporting package together with a separate administrative report. Its subject is how the municipality of Amsterdam applies algorithms in carrying out its tasks.
Purpose and Research Questions
The report aims to give councillors, citizens and other stakeholders insight into the extent to which the college of B&W applies algorithms responsibly. The central question asks to what extent the Amsterdam management framework for algorithms ('beheerskader') is adequate for responsible use, and which lessons can be drawn from practice. Responsible use is defined as taking sufficient account of risks in the areas of ethics, governance and accountability, model and data, privacy, and IT management.
Methodology
The research had two parts. First, the framework was tested against an adapted version of the Toetsingskader Algoritmen of the Algemene Rekenkamer. Six of the 63 test items were dropped and 46 of the remaining 57 were partly revised, leaving 61 items in total. Four researchers assessed the framework individually, followed by five sessions, one per theme. Second, three algorithms were examined as case studies: Detectie illegale onderhuur, Detectie illegale passagiersvaart and Behandeling aanvraag Bbz-uitkering (Gripvol). The first case was chosen by citizens in an online vote, in which 236 of 509 votes (46.4%) went to illegal sublet detection. Teams completed questionnaires, were interviewed, and reviewed their answers in several rounds. The framework was also independently reviewed by four scientists from Tilburg University, whose report was received on 23 February 2023.
The Amsterdam Management Framework
The framework is built around the Algoritme Lifecycle Aanpak, which contains seven instruments, including the online algoritmeregister (live since 28 September 2020), a governance document with a lifecycle model, contract conditions, a handreiking for objection handlers, a human rights impact assessment model, a bias analysis model and internal audits. The instruments were adopted by the GMT on 20 January 2022. The report states that Amsterdam applies four different definitions of 'algorithm', which it considers a source of confusion.
Key Findings
The report concludes that the framework contributes 'in redelijke mate' (to a reasonable degree) to responsible use. It describes the documents as a 'lappendeken' (patchwork) whose relationships and hierarchy are unclear. It notes that the framework pursues seven goals, while the Jaarplan 2023 lists five differently formulated goals. The framework gives sufficient attention to seven of eight test items on governance and accountability, and to 19 of 23 items on model and data. Gaps are identified in citizen involvement, documentation across the full lifecycle, the mandatory rights impact assessment, and the handling of testing environments. The report also observes that the risk-analysis tool is limited to high-risk applications.
Case Study Observations
The report analyses the three algorithms for common threads. It notes that the risk analysis for high-risk algorithms is often limited to privacy and information security. It records that officials reported a negative connotation of the term 'algorithm', partly due to SyRI and the Toeslagenaffaire, and that some officials feel additional requirements may increase workload.
Why It Matters for Algorithm Use in Amsterdam
The report states that its purpose is to inform councillors, citizens and other stakeholders about how responsibly the college of B&W applies algorithms, and to draw lessons from Amsterdam's practice for the future responsible use of algorithms in the city.