The Federal Financial Supervisory Authority, known as BaFin, said company chiefs must implement review processes to prevent illegal discrimination against customers arising from their use of AI. This includes the banks, insurers and brokers under BaFin's remit.
Transparent governance and data management are crucial if finance firms are to ensure "fair and non-discriminatory treatment of consumers," according to the watchdog. This means they might have to review their procedures for dealing with possible discrimination by AI.
"BaFin will take decisive action and implement suitable measures, for example in the context of supervising violations of statutory provisions," the watchdog said in its journal on Friday.
Human supervision could be needed in some cases if companies are to make sure the AI model is not discriminatory against some customers, according to BaFin. They could otherwise be responsible for indirect discrimination via AI.
The European Union distinguishes between direct and indirect discrimination in law. In the former, an individual is treated unfavorably despite having protected characteristics such as race or gender, while indirect discrimination happens when a seemingly neutral rule puts such a person at a disadvantage.
Indirect discrimination in this context is the impact of apparently neutral provisions and practices put in place by regulated companies, according to the watchdog. AI and machine learning could count as neutral practices for businesses, but they still come with risks.
"Technologies can be used to accelerate processes and analyze large quantities of data quickly and effectively. But when machines make decisions, problems can arise," the watchdog said. "This is because their logic only appears neutral at first glance."
Generative AI, including language models, can exacerbate any lack of fairness and transparency, according to the watchdog. BaFin pointed out that users of the AI model do not have any information on the dataset used to train the system in the first place.
Regulated firms must make sure their governance structures are adequate for expanding the use of AI and machine learning, according to the watchdog. Finance companies might need to provide special training for staff as a result.
The European Artificial Intelligence Act came into force on Aug. 1. The world's first comprehensive piece of legislation creates a framework for the development and use of AI tools across the bloc.
BaFin said that Article 10 of the act sets out obligations for firms to minimize AI bias and reduce the risk of discrimination. High-risk AI systems include models used to evaluate the credit scores of customers.
All EU member states have to establish national authorities to oversee the application of the new laws and carry out "market surveillance" activities. Member states have until August 2025 to comply.
Provisions of the AI Act will officially start applying in August 2026. But prohibitions of AI systems that are considered as presenting an "unacceptable risk" will apply from February, and rules for "general purpose" models, those that can perform a wide range of distinct tasks, will apply in August 2025.
The EU's AI Office will be responsible for implementing the act at European level, and the Artificial Intelligence Board will be asked to ensure a "uniform application" of the act across all member states.
Companies that breach the AI rules can be fined up to 7% of their global turnover for using banned applications.
--Additional reporting by Alex Baldwin. Editing by Ed Harris.
For a reprint of this article, please contact reprints@law360.com.
