ChatGPT as an education and learning tool for engineering, technology and general studies: performance analysis of ChatGPT 3.0 on CSE, GATE and JEE examinations of India

ABSTRACT

The quantitative and qualitative performance analysis of ChatGPT-3.0, a large language model, is carried out on three important and highly competitive examinations held in India: civil services examination (CSE, prelims), graduate aptitude test in engineering (GATE), and joint entrance examination (JEE). These examinations cover general knowledge, current affairs, history, geography, Indian polity, economics, mathematics, physics, chemistry, engineering, and technology aspects at the undergraduate and graduate levels. The Accuracy, Concordance, and Insight (ACI) criteria is used to analyze the performance of ChatGPT. ChatGPT passed CSE without much specialized training and reinforcement, however, underperformed in GATE and JEE. Overall, the average accuracy rate of ChatGPT is 48.71%, with a 44.45% concordance for all explanations. However, the concordance for accurate explanations is found to be 91.87% with a high level of insights given in the explanations. Moreover, the average accuracy of ChatGPT improves to 77.69% after training. The results suggest that large language models have great potential to assist with education technology and act as an instructor for the preparation of technical, aptitude and general studies topics for competitive examinations. Drawing insights from the findings of the current research, some limitations in the present study and possible future research directions are suggested.

KEYWORDS:

• ChatGPT
• generative pre-trained transformer
• artificial intelligence
• education technology
• large language model

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Additional information

Notes on contributors

Ravindra Giriraj Bhardwaj

Ravindra Giriraj Bhardwaj completed his PhD at the University of California, Riverside, USA, and is currently an assistant professor in the department. His research interests include nanotechnology, sustainable energy, thermoelectricity, numerical modeling and simulation, and finite element modeling and simulation. He has experience in experimental design, data analysis, nanofabrication (4 years in a 100/1000 cleanroom), and finite element modeling and simulation. Currently, he is working towards flexible electronics, waste heat harvesting, sustainable methods, and nanocomposites, with applications in semiconductors, electronics, aerospace industry, and enhanced learning/teaching methodologies/techniques.

Harpreet Singh Bedi

Harpreet Singh Bedi is an assistant professor in the Department of Mechanical Engineering at the BITS Pilani Dubai campus, United Arab Emirates. Prior to this, he worked as a research associate for two years at the Aeronautical Development Establishment, DRDO, India. He completed his PhD (2020) from the Indian Institute of Technology, Ropar, specializing in the fabrication, testing, and characterization of reinforced polymer composites. His research interests include nanofiller and fiber reinforced plastics, advanced composites, finite element modeling and simulation, topology optimization, additive manufacturing, and enhanced learning/teaching methodologies/techniques.