Vladimir Milovanović

University of Kragujevac, Faculty of Engineering,
Department of Electrical Engineering and Computer Sciences,
Center for Integrated Systems

Biography

Vladimir Milovanović received the Dipl.-Ing. degree in Electrical Engineering from the University of Belgrade, Serbia in 2005, and the Ph.D. degree from the Delft University of Technology, the Netherlands, in 2010.

Since the beginning of 2014, he was working as a Postdoctoral Scholar with the University of California, Berkeley. Before joining Berkeley Wireless Research Center, from 2011 he was with Vienna University of Technology, Austria as a Postdoctoral Research Fellow. Presently, he is holding a position of an Associate Professor with the Department of Electrical Engineering and Computer Sciences at the Faculty of Engineering, University of Kragujevac, Serbia, and serves as the managing director of the Center for Integrated Systems within the same institution.

Dr. Milovanović has also held advisory, consulting, or visiting positions with Texas Instruments, NXP Semiconductors, Infineon Technologies, Sony and Broadcom.

His research focuses and interests include design, modeling and optimization of analog, mixed-signal and digital integrated circuits and systems, along with the development and implementation of efficient artificial intelligence and signal processing algorithms.

Prof. Milovanović is the recipient of the Best Student Paper Award at the 2009 IEEE Bipolar/BiCMOS Circuits and Technology Meeting and the Best Paper Awards at the 2014 IEEE International Conference on Microelectronics and the 2024 IcETRAN.

Keynote speech title

Popular TV Quiz Shows: From Pastime to Large Language Model Benchmarking
and Back

Abstract

Ever since IBM’s computer system named Watson, capable of answering questions posed in natural language, outscored previous champions of the popular television game show Jeopardy! back in 2011, there has been an unprecedented rise of artificial intelligence (AI) in general. A Serbian quiz show counterpart, TV Slagalica does not just serve as viewing entertainment but is a real nursery of challenging problems and research ideas. The quiz features several games, few of which can be optimally solved even by computer science freshmen as part of their regular coursework. However, solving one of the games, named Associations, poses an ambitious task both for graduate scholars and also for the state-of-the-art large language models (LLMs).

This keynote outlines the development of an automated (intelligent) system that is able to play the Associations game on par with human players. The aspects of training data preparation, test data extraction, model assessment, and generalization will be covered. Alternative use cases, such as the ones for puzzle preparation, will be mentioned. The performance of leading-edge LLM-based chatbots that pass the Turing test, like OpenAI’s ChatGPT, is thoroughly evaluated. It is demonstrated that with each successive model generation, the success rate of correct solution guesses is increased. Consequently, carefully curated test sets can be used for model benchmarking since, unlike already established benchmarks, the game of word associations is also challenging for humans.

As a conclusion, not only can it be proved that modern LLMs are close to, if not already achieving, superhuman performance in almost all TV quiz show games, but when, in the near future, foundation models start to significantly outperform humans in some areas, this benchmark can potentially help in quantifying that gap.