Does AI Reflect Human Behaviour? Exploring the Presence of Gender Bias in AI Translation Tools

Natural language processing tools are becoming more and more important in our daily life, enabling us to perform many tasks in a timely and efficient manner. However, as the utilisation of these tools growth, so does the risk of unexpected consequences due to the presence of bias. This study investigates the presence of gender bias within the most popular neural machine translation and large language model tools. We defined a set of Italian sentences concerning ten specific jobs, where the gender of the subjects is not explicitly mentioned. Employing those AI tools, we translated the sentences from Italian to English, requiring the gender to be explicitly mentioned. Afterwards, we developed a survey to obtain human translations for the same sentences, allowing us to compare the differences between the responses generated by the tools and those from individuals. Results show a high presence of gender bias especially for the jobs associated with a male gender and demonstrate a consistency between the outcome obtained by the tools and the results of the survey. These findings serve as a starting point for exploring the origins of gender bias within natural language processing tools and how they reflect gender distributions in our society and human behaviour regarding job occupations.