Desafíos actuales de la Inteligencia Artificial

256 Desafíos actuales de la Inteligencia Artificial legitimately entitled to such benefits or services, those systems may have a significant impact on persons’ livelihood and may infringe their fundamental rights, such as the right to social protection, non-discrimination, human dignity or an effective remedy and should therefore be classified as high-risk. Nonetheless, this Regulation should not hamper the development and use of innovative approaches in the public administration, which would stand to benefit from a wider use of compliant and safe AI systems, provided that those systems do not entail a high risk to legal and natural persons. In addition, AI systems used to evaluate the credit score or creditworthiness of natural persons should be classified as high-risk AI systems, since they determine those persons’ access to financial resources or essential services such as housing, electricity, and telecommunication services. AI systems used for those purposes may lead to discrimination between persons or groups and may perpetuate historical patterns of discrim- ination, such as that based on racial or ethnic origins, gender, disabilities, age or sexual ori- entation, or may create new forms of discriminatory impacts. However, AI systems provided for by Union law for the purpose of detecting fraud in the offering of financial services and for prudential purposes to calculate credit institutions’ and insurance undertakings’ capital requirements should not be considered to be high-risk under this Regulation. Moreover, AI systems intended to be used for risk assessment and pricing in relation to natural persons for health and life insurance can also have a significant impact on persons’ livelihood and if not duly designed, developed and used, can infringe their fundamental rights and can lead to serious consequences for people’s life and health, including financial exclusion and discrim- ination. Finally, AI systems used to evaluate and classify emergency calls by natural persons or to dispatch or establish priority in the dispatching of emergency first response services, including by police, firefighters and medical aid, as well as of emergency healthcare patient triage systems, should also be classified as high-risk since they make decisions in very critical situations for the life and health of persons and their property. (paragraph 58, p. 55). Using cognitive robotics and process automation, the Municipality of Trelleborg in Swe- den began automating social assistance decisions in 2016, allowing the programme to pro- cess and grant subsidies for sickness, unemployment and tax exemptions through a robotic decision-making system. The result, according to the report on AI in public services in the European Union, was a significant improvement in waiting times for applications to be ana- lysed. However, the report itself, while positive, also highlights at least some areas of concern: “Some observers expressed concerns about the risk of excluding some more vulnerable citi- zens when all processes are automated online, as this makes it more difficult to assess individ- ual needs” (Misuraca, G., van Noordt, 2020, p.46). And again: While the AI-system enabled various social welfare benefits decisions to be automated, many other processes of the Trelleborg municipality still operate as in a traditional bureau- cratic system. There are still many paperbased processes within the organisation which could lead to double documentation and inefficient processes, as well as existing software with very poor interfaces and usability levels. Hence. Process Automation in general and AI systems in particular can strongly improve one specific government process, but the interoperability with other organisational processes should never be forgotten (Misuraca, G., van Noordt, 2020, p.46).