CARLOS DUARTE: So, we have Willian Massami Watanabe, from the Universidade Tecnologica Federal do Parana in Brazil. We have Yeliz Yesilada from the Middle East Technical University. We have Sheng Zhou from Zhejiang University in China. I hope I pronounced it correctly. And Fabio Paterno from CNR-IST, HIIS Laboratory in Italy. Okay. Thank you all for joining us. And for some of you, it is earlier in the morning. For others of you it is later, well, for some of you, I guess, it is really late in the evening, so, thank you all for your availability.
Let’s start this discussion on how, I would say, current machine learning algorithms and current machine learning applications can support or can improve methodologies for automatically assessing Web Accessibility. And, from your previous works, you have touched different aspects about how this can be done. So, machine learning has been used to support Web Accessibility Evaluation through different aspects, just a sampling, such as metrics, such as evaluation prediction and handling dynamic pages. I understand not all the domains you have worked on those, but some of you have worked on specific domains, so, I would like you to focus on the ones that you have been working more closely in. For us to start, just let us know, what are the current challenges that prevent further development and prevent further use of machine learning or other AI techniques in these specific domains. Okay? I can start with you, Willian.
WILLIAN WATANABE: First of all, thank you so much for everything that is being organized. Just to give you some context, I am Willian, I’m a professor here in Brazil, where I work with web accessibility. My research focuses is on web technology, the ARIA specification, to be more specific, and just in regard to everything that has been said by Carlos Duarte, my focus is on evaluation prediction, according to the ARIA specification. I believe the main, I was invited to this panel considering my research on identification of web elements on web applications. The problem I address is identifying components in web applications, when we implement web applications, we use same structured language such as HTML. My job is to find what these elements in the HTML structure represent in the web page. Like, they can represent some specific type of widget, there are some components, some landmarks that we need to find on the web page, and this is basically what I do.
So, what I have been doing for the last year, I have been using machine learning for identifying these elements. I use supervised learning and I use data provided by the DOM structure of the web application. I search for elements in the web page and I cross file them as an element or a widget or anything else. The challenge in regards to that, they are kind of different from the challenges that have been addressed yesterday. Yesterday applications of machine learning, I think they work with video and text that are unstructured data, so they are more complicated, I would say. The main challenge that I address in my research is associated to data acquisition, data extraction, identifying what kind of features I should use to identify those components in web applications. Associated to that I should say, to summarize, my problems are associated with the diversity of web applications, there are different domains and this kind of bias, any dataset that we use, it is difficult for me, for instance, to identify a number of websites that implement, that represent all the themes of websites that can be used in web applications, variability in the implementation of HTML and JavaScript, and the use of automatic tools to extract this data, such as WebDriver API, the DOM structure dynamics, annotation observers. There are a lot of specifications that are currently being developed that I must use, and I always must keep my observing to see if I can use them to improve my research. And, lastly, there is always the problem of manual classification in AI for generating the datasets that I can use. That is it, Carlos. Thank you.
CARLOS DUARTE: Thank you, Willian. Thank you for introducing yourself, because I forgot to ask all of you to do that. So, in your first intervention, please give us a brief introduction about yourself and the work you are doing. So, Yeliz, I will follow with you.
YELIZ YESILADA: Hi, everybody. Good afternoon. Good afternoon for me. Good afternoon, everybody.I’m Yeliz, I’m associated professor at Middle East Technical University in Northern Cyprus Campus, I’ve been doing accessibility web research for more than 20 years now. Time goes really fast. Recently I have been exploring machine learning and AI, specifically, for Web Accessibility, supporting Web Accessibility from different dimensions.
Regarding the challenges, I think there are, of course, many challenges, but as Willian mentioned, I can actually say that kind of the biggest challenge for my work has been data collection. So, I can actually say that data, of course, is critical, as it was discussed yesterday in the other panels. Data is very critical for machine learning approaches. For us, collecting data, making sure that the data is representing our user groups, different user groups, and not biasing any user groups, and also, of course, preparing and labeling the data, certain machine learning algorithms, of course, supervised ones, require labeling. Labeling has also been a challenge for us, because sometimes certain tasks, it is not so straightforward to do the labeling. It is not black and white. So, it has been a challenge for us, I think, in that sense.
And the other two challenges I can mention, I think the second one is the complexity of the domain. When you think about the Web Accessibility, sometimes people think, oh, it is quite straightforward, but it is actually a very complex domain. There are many different user groups, different user requirements, so, understanding those and making sure that you actually address different users and different requirements is quite challenging. And, since we also are working, this is last one that I wanted to mention, since we are also working with web pages, they are complex. They are not well designed or well properly coded. As we always say, browsers are tolerating, but for developing algorithms, machine learning algorithms, they also have to deal with those complexities, which makes the task quite complex, I think. So, just to wrap up, I think, in my work, there are three major challenges. Data, or the lack in quality of data. Complexity of the domain, different users and different user requirements. And the complexity of the resources we are using. So, web pages, the source codes and the complexity of pages that are not conforming to standards. I think they are really posing a lot of challenges to algorithms that we are developing. So, these are all I wanted to say.
CARLOS DUARTE: Thank you, Yeliz. A very good summary of major challenges facing everyone that works in this field. So, thank you for that. Sheng, I wanted to go with you next.
SHENG ZHOU: Thank you, Carlos. Hi, everyone. I am Sheng Zhou from Zhejiang University in China. From my point of view, I think three challenges occurs currently now, first, I totally agree that it is hard to compare labels for model training. Since the success of machine learning heavily relies on a large number of label data, however assessing these label data usually takes a lot of time, which is hard to realize, especially in the accessibility domain. I want to take a, I am sorry, I am a little bit nervous here. Sorry. I want to take the WCAG rule, the image or text, as an example, as we discussed in the panel yesterday. Most of the current image captioning or OCR methods are trained on images dataset, rather than the image, like a logo that is essential in text alternative, the label for Web Accessibility solution should fully consider the experience of different populations. There are very few datasets that are specifically designed for the accessibility of evaluation tasks and satisfies the requirements. So, the machine learning models that are trained, or traditional models cannot be aware, generalized to accessibility evaluation. Second one, I think, about the web page sampling, since I had a little bit of work on this, I think currently, there are much factors that affect the sampling subject.
First, sampling has been a fundamental technique in Web Accessibility Evaluation when dealing with millions of pages. The previous page sampling methods are usually based on the features of each page of such elements or the DOM structure. Similar features are assumed to generated by the same development framework and have similar accessibility problems. However, with the fast growth of web development framework, pages have developed with diverse tools, for example, pages that look very similar may be developed by totally different framework, and some pages that look totally different may be developed by the same framework. This poses great challenges for feature-based web accessibility evaluations. It is necessary to incorporate more factors into the sampling process, such as the connection typology among pages, and visual similarity, and typesetting. So, how to identify similarity between pages considering multiple factors into a unified sampling probability is critical for sampling. I think this can be a problem that the literature, the graph typology, could try to understand, and metric learning, which is a comprehensive research program.
So, the third one, the third challenge, I think is the subjective evaluation rules. When we evaluate the Web Accessibility, there are both subjective and objective rules, right? So, for example, when evaluating the WCAG success criteria 1.4.5, image of text, the image is expected to be associated with accurate description texts which has been discussed in the panel yesterday. It is still challenging to verify the matching between the (speaking paused)
CARLOS DUARTE: I guess there are connection issues? Let’s see. Okay. He has dropped. We will let Sheng, okay, he is coming back, so, you are muted.
SHENG ZHOU: Sorry.
CARLOS DUARTE: It is okay. Can you continue?
SHENG ZHOU: Okay, okay. I am so sorry. I think there are three challenges under the first challenges, as same as Yeliz described, it is hard to...
CARLOS DUARTE: You dropped when you were starting to talk about the third challenge.
SHENG ZHOU: Okay.
CARLOS DUARTE: We got the first and second challenge. We heard that loud and clear, so now you can resume on the third challenge.
SHENG ZHOU: Okay, okay. So, the third challenge is the subjective evaluation rules. There are both subjective and objective rules. For example, when evaluating the WCAG success criteria 1.4.5 image of text, the image is expected to be associated with accurate description text, as discussed in the panel yesterday, it is still challenges to verify whether the matching between image with text since we do not have access to the ground truth of the text of image. So, I think (video freezing)
CARLOS DUARTE: Apparently, we lost Sheng again. Let’s just give him 10 seconds and see if he reconnects, otherwise we will move on to Fabio. Okay. So, perhaps it is better to move on to Fabio and get the perspective of also someone who is making an automated accessibility evaluation tool available, so it is certainly going to be interesting, so, Fabio, can you take it from here?
FABIO PATERNO: Yes. I am Fabio Paterno. I’m a researcher in the Italian National Research Council where I lead the Laboratory on Human Interfaces in Information Systems. We have now a project funded by the National Recovery and Resilience Plan which is about monitoring the accessibility of the public administration websites. In this project we have our tool, MAUVE, which is a tool open, freely available, and it has already more than 2000 registered users. Recently, we performed the accessibility evolution of 10000 websites and considered grounded pages for each website, obviously it was an effort .So, we were very interested in understanding how machine learning can get passed in this larger scale monitoring work. So, for this panel, I did a systematic literature review, and I went to the ACM digital library, I entered machine learning and accessibility evaluation to see what has been done so far. I got only 43 results, which is not too many, I would expected more, and actually only 18 actually applied because other works were more about machine learning can be interesting in future work and so. To say the specific research effort has been so far limited in this area. And another characteristic was that there are other valid attempts. There are people trying to predict web site accessibility based on the accessibility of some web pages, others trying to check the rules of the alternative description, and trying to make the user control the content areas. So, I would say challenge is, well, machine learning can be, you know, used for a complementary support to automatic tools that we already have. There are many, in theory there are many opportunities, but in practice... there are a lot of progress. The challenge I think is to find the relevant one with the accessibility features that are able to collect the type of aspect that we want to investigate.
And I would say the third and last main general challenge is that we really, really want to continuously work with the changes not only the web but also how people implement, how people use the application, this continuously change. So, there is also the risk that the dataset will become obsolete, not sufficiently updated for addressing all the methods for that.
CARLOS DUARTE: Okay, thank you for that perspective. Sheng, I want to give you now the opportunity to finish up your intervention.
SHENG ZHOU: Okay. Thank you, Carlos. Sorry for the lagging here. So, I will continue my third opinion of the challenge. From my opinion, the third challenge is the subjectivity evaluation rules. In relation to Web Accessibility, there are subjective and objective rules, and for example, when evaluating, an image to text rule. The image is expected to be associated with the accurate description text. And as discussed in the panel yesterday, it is still challenging to verify the matching between the image and the text, since there are no ground truth of what kind of text should describe the image. As a result of the accessibility evaluation system, it is harder to justify whether the alternative text really matches the image. So, thanks.
CARLOS DUARTE: Okay, thank you. I will take it, from, I guess, most of you, well, all of you have in one way or another mentioned one aspect of Web Accessibility Evaluation, which is conformance to requirements, to guidelines. Several of you mentioned the web content Accessibility Guidelines in one way or another. Checking, what we do currently, so far, and following up on what Sheng just mentioned, are objective rules. That is what we can do so far, right? Then when we start thinking about, because the guidelines are themselves also subject to subjectivity, unfortunately. How can we try to make the evaluation of those more subjective guidelines, or more subjective rules, and how do you all think that Artificial Intelligence, algorithms, or machine learning-based approaches can help us to assess conformance to those technical requirements to Accessibility Guidelines? Okay? I will start with you, now, Yeliz.
YELIZ YESILADA: Thank you, Carlos. So, regarding the conformance testing, so, maybe we can actually think of this as two kinds of problems. One is the testing, the other one is confirming, basically repairing, or automatically fixing the problems. So, I see, actually, that machine learning and AI in general can, I think, help in both sides, in both parties. So, regarding the testing and auditing, if we take, for example, WCAG Evaluation Methodology as the most systematic methodology to evaluate for accessibility, it includes, for example, five stages, five steps. So, I think machine learning can actually help us in certain steps.
For example, it can help us to choose a representative sample, which is the third step in WCAG-EM. We are currently doing some work on that, for example, to explore how to use unsupervised learning algorithms to decide, for example, what is a representative sample. Fabio, for example, mentioned the problem of evaluating a large-scale website with millions of pages. So, how do you decide, for example, which ones to represent, I mean, which ones to evaluate. Do they really, for example, if you evaluate some of them, how much of the site you actually cover, for example. So, there, I think, machine learning and AI can help. As I said, we are currently doing some work on that, trying to explore machine learning algorithms for choosing representative samples, making sure that the pages that you are evaluating really represent the site, and reduces the workloads, because evaluating millions of pages is not an easy task, so maybe we can pick certain sample pages.
Once we evaluate them, we can transfer the knowledge from those pages to the other ones, because more or less the pages these days are developed with templates or automatically developed, so, maybe we can transfer the errors we identified, or the ways we are fixing to the others which are representative. Regarding the step four in WCAG-EM, that is about auditing the select sample, so how do you evaluate as test the sample, I think in that part, as we all know, and Sheng mentioned, there are a lot of subjective rules which require human testing. So, maybe there we need to explore more how people, I mean, how humans evaluate certain requirements, and how we can actually automate those processes. So, can we have machine learning algorithms that learn from how people evaluate and assess and implement those. But, of course, as we mentioned in the first part, data is critical, valid data, and quality of data is very critical for those parts.
Regarding the repairing, or automatically fixing certain problems, I also think that machine learning algorithms can help. For example, regarding the images Sheng mentioned, we can automatically test whether there is an Alt Text or not, but not the quality of the Alt Text, so maybe there we can explore more, do more about understanding whether it is a good Alt Text or not, and try to fix it automatically by learning from the context and other aspects of the site. Or, I have been doing, for example, research in complex structures like tables. They are also very difficult and challenges for accessibility, for testing and for repairing. We have been doing, for example, research in understanding whether we can differentiate, and learn to differentiate a layout table from a data table, and if it is a complex table, can we actually, for example, learn how people are reading that and guiding the repairing of those. We can, I guess, also do similar things with the forms. We can learn how people are interacting with these forms and complex structures with the forms like reach and dynamic content like Willian is working on. Maybe we can, for example, do more work there to automatically fix, which can be encoded in, let’s say, authoring tools or authoring environments, that include AI, without the developers noticing that they are actually using AI to fix the problems. So, I know I need to wrap up. I think I would say contributing two things, both testing and repairing can help.
CARLOS DUARTE: I agree. Some of the things you mentioned, they can really be first steps. We can assist a human expert, a human evaluator, and take away some of the load. That is also what I take from the intervention. So, Fabio, I would like your take on this, now.
FABIO PATERNO: I mean, I think ideally what Yeliz said before. We have to be aware of the complexity of accessibility evaluation. Because just think about WCAG 2.1. It is composed of 78 success criteria, which are associates with hundreds of techniques, specific validation techniques, so, this is the current state and it seems like it is going to increase the number of techniques and so on. So, the automatic support is really fundamental.
And, secondly, when you use automatic support, the results of the check are to be ok, this pass, this fails, or cannot tell. So, one possibility that I think would be interesting is how to explore machine learning in the situation in which automatic solution is not able to deterministically provide an ok or fail, these could be an interesting opportunity to also explore in other European projects. Ideally this would have a group, accessibility, human accessibility expert, in this case to provide the input, and then to try to use this input to train an intelligent system. And then if it was not possible to validate these solutions, but for sure, it might be really easy for AI to detect whether an alternative description exists, but it is much more difficult to say whether it is meaningful.
So, in this case, for example, I have seen a lot of improvement of AI in recognizing images and the content keys, I have also seen some of (Muffled audio). You can think in a situation in which AI provides the descriptors and then there is some kind of similarity checking between these automatic generated descriptions and the ones being provided by the developer and see in what extent these are meaningful. This is something I think is possible, what I’m not sure is how much we can find a general solution. I can see this kind of AI, associated with some level of confidence, and then I think is a part of the solution let the user decide what should be level of confidence that is acceptable, when these automatic supporters use it to understand the way the description is meaningful. So that would be the direction where I would try from a perspective of people working on tools for automatic evaluation, trying to introduce AI inside of such an automatic framework. But another key point we have to be aware of is the transparency. When we are talking about AI, we are talking about the Blackbox, there is a lot of discussion about explainable AI. Some people say AI is not able to explain why this data generated this result, or how can we change it to obtain different results (Muffled audio), so this is a question that people encounter when they happen to run an evaluation tool.
And also, in addition to the study about the transparency of tools, the tools that are now available, it was published on ACM Transactions in computing anything about that often these tools are a little bit Blackboxes, they are not sufficiently transparent. For example, they say, we support these success criteria, but they do not say which techniques they actually apply, how these techniques are implemented. So, they say that often the users are in disadvantage because they use different tools and get different results, and they do not understand the reason for such differences. Let’s say this is point of transparency is already for now, with such validation tools that do not use AI. We have to be carefully that if it is added AI it should be added in such a way that is explainable, so we can help people to better understand what happened in the evaluation and not just give the results without any sufficient explanation.
CARLOS DUARTE: I think that is a very important part, because if I am a developer, and I am trying to solve accessibility issues, I need to understand why is there an error, and not just that there is an error. That is a very important part. Thank you, Fabio. So, Sheng, next, to you.
SHENG ZHOU: Thanks. Incorporating the artificial intelligence, I will try to find some way to help the developers. First of all is the code generation for automatically fixing the accessibility problems. As Yeliz just said, always web accessibility evaluation has been targeted, but we have to stand at the view of the developers. If it is the evaluation system only identifies or located the accessibility problem, it may be still hard for developers to fix these problems since some developers may lack experience on this. And the recently artificial intelligence based code generation has been well developed and give some historical code of fixing accessibility problems. We have tried to train artificial intelligence model to automatically detect the problem, make a code snippet, fix the problem code and provide suggestions for the developers. We expect this function could help the developers fix the accessibility problem and improve the websites more efficiently.
And the second reason for the developer is the content generation. As discussed in the panel yesterday, there have been several attempts in generating text for images or videos with the help of the computation vision and NLP techniques. It may not be very practical for the images generators to provide an alt text since the state of art methods requires large models deployed on GP servers which is not convenient for frequently updated images. Recently we have been working on some knowledge distillation method, which aims at distilling a lightweight model from a large model. We want to develop a lightweight access model that can be deployed in the broader extension, or some like lightweight software. We hope to reduce the time cost and competition cost of image providers and encourage them to conform to the accessibility technique or requirements. Okay. Thank you.
CARLOS DUARTE: Thank you. That is another very relevant point. Make sure that whatever new techniques we develop are really accessible to those who need to use them. So the computational resources are also a very important aspect to take into account. So, Willian, your take on this, please.
WILLIAN WATANABE: First, I would like to take from what Yeliz said, that we have basically, it is nice to see everyone agreeing, before we didn’t talk at all so it is nice to see that everywhere is having the same problems. And, about what Yeliz said, she divided the work into automatic evaluation into two steps. The first one is testing, and the second one is automatically repairing accessibility in websites. From my end, specifically, I don’t work with something, I would say subjective, like image content generation. My work mostly focuses on identifying widgets, it is kind of objective, right? It is a dropdown, it is not a tooltip... I don’t need to worry to be sued over a bad classification, or something else. So, that is a different aspect of accessibility that I work on. Specifically, I work with supervised learning, as everyone, I classify the elements as a specific interface component. I use features extracted from the DOM structure to, I think everyone mentioned this, Sheng mentioned it, as well, Yeliz mentioned the question about labels and everything else.
I am trying to use data from websites that I evaluate as accessible to enhance the accessibility of websites that I don’t, that don’t have these requirements. For instance, I see a website that implements rules, that implements the ARIA specification. So, I use it. I expect data from it to maybe apply it on a website that doesn’t. This is kind of the work that I am working, this is kind of what I am doing right now.
There is another thing. So, Fabio also mentioned the question about confidence. I think this is critical for us. In terms of machine learning, I think the word that we use usually is accuracy. What will guide us, as researchers, whether we work on test or automatically repair is basically the accuracy of our methodologies. If I have a lower accuracy problem, I will use a testing approach. Otherwise, I will try to automatically repair the web page. Of course, the best result we can get is an automatic repair. This is what will scale better for our users, ultimately offer more benefit in terms of scale. I think that is it. Everyone talked about everything I wanted to say, so this is mostly what I would say differently. This is nice.
CARLOS DUARTE: Okay. Let me just, a small provocation. You said that, in your work, everything that you work with widget identification is objective. I will disagree a little bit. I am sure we can find several examples of pages where you don’t know if that is a link or a button, so there can be subjectivity in there, also. So, yes. But just a small provocation, as I was saying.
So, we are fast approaching, the conversation is good. Time flies by. We are fast approaching the end. I would ask you to quickly comment on the final aspect, just one minute or two, so please try to stick to that so that we don’t go over time. You have already been in some ways approaching this, but just what do you expect, what would be one of the main contributions, what are your future perspectives about the use of machine learning techniques for web accessibility evaluation. I will start with you now, Fabio.
FABIO PATERNO: Okay. If I think about a couple of interesting, you know, possibilities opened up, about machine learning. When we evaluate a user interface, generally speaking we have two possibilities. One is to look at the code associated, the generated interface and see whether it is compliant with some rules. And another approach is to look at how people interact with the system. So, look at the levels of user interaction. In the past we did some work where we created a tool to identify various usability patterns, which means patterns of interaction that highlight that there is some usability problem. For example, we looked at mobile devices where there’s a lot of work on (?) machine, that means that probably the information is not well presented, or people access computers in different (?) it means the (?) are too close. So, it is possibly to identify a sequence of interaction that highlight that is some usability problem. So, one possibility is to use some kind of machine learning for classifying interaction with some Assistive Technology, that highlights this kind of problem. So, allow us from the data (?), yes, there are specific accessibility problems.
And the second one is about, we mentioned before, the importance of providing an explanation about a problem, or why it is a problem, and how to solve. So, that would be, the idea in theory, an ideal application for a conversational agent. Now there is a lot of discussion on this, about ChatGTP, but is very difficult to actually design, in this case, a conversational agent that is able to take into account the relevant context, which in this case is the type of user that is actually now asking for help. Because now there are really many types of users, when people look at accessibility results, that can be a web commission, the person who decide to have a service but doesn’t know anything about its implementation, then the user, the developer, the accessibility expert, each of them require a different language, different terms, a different type of explanation, because one day, I look "is this website accessible?". They really have different criteria in order to understand the level of accessibility and how to operate it in order to improve it. So, this is one dimension of the complexity.
The other dimension of the complexity is the actual implementation. It is really not, this (?) we are conducting in our laboratory (?). It is really amazing to see how different implementation languages, technical components that people use in order to implement the website. Even people that use the same JavaScript frameworks, they can use it in very different ways. So, when you want to provide an explanation, of course, there is a point just providing the standard, the description, the error, some of the standards examples, how to solve the problem, because fften there are different situations that require some specific system consideration for explaining how, or what can be done. But this complex conversational agent for accessibility, it would be a great result.
CARLOS DUARTE: Thank you, Sheng?
SHENG ZHOU: In the sake of time, I will talk about the future perspective about the efficient page sampling. According to our data analysis, we found that the pages, the web pages with similar connection structures with other pages visually have some similar accessibility problem. So, we try to take this into account for the accessibility evaluation. And recently we used a graph knowledge that works, that has been a hot research topic in the machine learning community. It combines both the network topology and the node attributes, to an only unified representation for each node. Each node (frozen video)
CARLOS DUARTE: Okay. I guess we lost Sheng again. In the interest of time, we will skip immediately to you, Willian.
WILLIAN WATANABE: Okay. My take on this, I think it will be pretty direct. I think Fabio talked about it, but we are all working with specific guidelines, a set of Accessibility Guidelines of WCAG. And I think the next step that we should address is associated to generalization, and incorporating it into relevant products, just incorporating any automatic evaluation tool. So, in regard to all the problems that we mentioned, data acquisition, mental classification, we had to find a way to scale our experiment so that we can guarantee it will work in any website.
In regards to my work, specifically, I think that are some, I’m trying to work on automatic generation for structure websites, for instance, generating heading structures and other specific structures that users can use to righteous and automatically enhance the accessibility of the web page. I think that is it. In regard to what you said, Carlos, just so that I can clear myself, what I wanted to say is that, different from the panelists from yesterday, and different from Chao, for instance, I think I am working with a similar machine learning approach. I don’t use deep learning, for instance. Since I don’t see the use for it yet, in my research, because for my research I think it is mentioned that she might use for labeling and other stuff, data generation. I haven’t reached that point yet. I think there are a lot of things we can do just with classification, for instance. That is it.
CARLOS DUARTE: Okay, thank you, Willian. Yeliz, do you want to conclude?
YELIZ YESILADA: Yes. I actually, at least I hope, that we will see developments, again, in two things. I think the first one is automated testing. I think we now are at the stage that we have many tools and we know how to implement and automate, for example, certain guidelines, but there are a bunch of others that they are very objective, they require human evaluation. It is very costly and expensive, I think, from an evaluation perspective. So, I am hoping that there will be developments in machine learning and AI algorithms to support and have more automation in those ones that are really now requiring a human to do the evaluations. And the other one is about the repairing. So, I am also hoping that we will also see developments in automating the kind of fixing the problems automatically, learning from the good examples, and being able to develop solutions while the pages are developed, they are actually automatically fixed. And, sometimes, maybe seamless to the developers so that they are not worried about, you know, certain issues. Of course, explainability is very important, to explain to developers what is going on. But I think automating certain things there would really help. Automating the repairment. Of course, to do that I think we need datasets. Hopefully in the community we will have shared datasets that we can all work with and explore different algorithms. As we know, it is costly. So, exploring and doing research with existing data, it helps a lot.
So, I am hoping that in the community we will see public datasets. And, of course, technical skills are very important, so human-centered AI I think is needed here and is also very important. So hopefully we will see more people contributing to that and the development. And, of course, we should always remember, as Jutta mentioned yesterday, the bias is critical. When we are talking about, for example, automatically testing, automating the test of certain rules, we should make sure we are not bias with certain user groups, and we are really targeting everybody in different user groups, different needs and users. So, that is all I wanted to say.
CARLOS DUARTE: Thank you so much, Yeliz. And also, that note I think is a great way to finish this panel. So, thank you so much, the four of you. It is really interesting to see all those perspectives and what you are working on and what you are planning on doing in the next years, I guess.
Let me draw your attention. There are several interesting questions on the Q&A. If you do have a chance, try to answer them there. We, unfortunately, didn’t have time to get to those during our panel. But I think that are some that really have your names on it. (Chuckles) So, you are exactly the correct persons to answer those. So, once again, thank you so much for your participation. It was great.
We will now have a shorter break than the ten minutes. And we will be back in 5 minutes. So, 5 minutes past the hour.