NewIn: Chunyang Chen
More barrier-free software access with AI
What got you interested in this research topic?
When I started doing research on the subject of apps and developing my own apps, I came across recommendations from Google and other companies on how to make apps more accessible for people with disabilities. I have to admit that I had never given it any thought. I then tried to use a mobile phone blindfolded and soon realized that it was hugely difficult even with special tools.
Another big realization came a few years ago when I attended a scientific conference where a scientist gave his talk via video link. The speaker was a young man, lying in bed, who could only move his head and one finger. This scientist was doing research on barrier-free access to websites and apps. That’s when I understood how important this research is, given the number of people like him who can access online content only with great difficulty.
What technical resources are available to help people with difficulties find their way around the online world?
As a rule, they use screen readers, which read the contents of web pages from left to right and top to bottom. This works quite well for texts. But images or buttons for users to click are a bigger challenge. In this case, descriptive texts must be placed that can be read aloud by the screen readers. However, these texts are often not there or good enough to be read by blind users or those with other disabilities.
What is the cause of this problem?
The biggest problem relates to the difficulty software developers have in putting themselves in the place of those concerned. They either do not care about or are simply unaware of how big they have to make a button for the user to click, how to describe an image or what level of contrast a color blind person needs.
In addition, many companies – especially smaller ones – lack a dedicated team for making their content accessible.
Can you help these teams with your research?
I hope so. We have developed algorithms to scan the apps before they are released to users. They point out areas where improvements are needed. At the same time, our algorithms suggest various solutions. We pass these regulations on to developers so that they can update their software.
In technical terms, we use different methods such as natural language processing, computer vision, large language models e.g., GPT-4 and adapt them to our needs by proposing proper “prompt engineering” methods. That means that we use in-context learning and feedback to drive the meta-data generation to make the mobile app accessible. Some data comes from Google recommendations or examples from other companies that have already created barrier-free access to their content. So we optimize the interaction with these models for our purposes.
What do you want to achieve with your research?
I moved to TUM in the spring to further develop my research career and apply that into real-world applications. In Germany, I hope that I’ll be able to build a network with various industry partners and that my approach will be implemented in many companies.
- After completing his studies in China and his doctoral thesis in Singapore, Prof. Chen spent six years in Australia, working as a faculty. In the spring of this year, he arrived at the TUM Campus in Heilbronn and is conducting research there at the point where software engineering meets artificial intelligence.
- At TUM Campus Heilbronn, scientists from the fields of informatics and management conduct teaching and research on the digital transformation.
- All episodes of the NewIn series
Technical University of Munich
Corporate Communications Center
- Julia Rinner
- julia.rinner@tum.de
- presse@tum.de
- Teamwebsite
Contacts to this article:
Prof. Dr. Chunyang Chen
Technical University of Munich
Software Engineering & AI
chun-yang.chen@tum.de
