AI Chatbots and their Perceived Effectiveness in Cybersecurity: Students’ Perspective

Abstract

The rise of the use of AI chatbots in higher education is gaining interest regarding the potential usefulness and impact of chatbots on the education provided in technical fields, especially cybersecurity. Despite the fact that many educational institutions are using such tools as ChatGPT, Gemini, and Copilot, actual empirical data related to the perceptions about the effectiveness of AI chatbots by cybersecurity students in developing countries is still limited. The goal of this study is to determine the perceived effectiveness of AI chatbots on the cybersecurity education of university students in Pakistan, based upon the Technology Acceptance Model (TAM), Constructivist Learning Theory, and Cognitive Load Theory. The study utilized a quantitative, descriptive survey design with data collected via a structured questionnaire that used a 5-point Likert-scale from 83 students situated in several universities. The instrument used in this study assessed five dimensions of awareness and knowledge, perceived usefulness, frequency of use, quality of response, challenges and limitations that students faced when using AI chatbots, and perceived academic performance. Based on the findings, there appears to be a general consensus among students that they have a moderate-high level of awareness of AI chatbots (M = 3.91 - 4.00), and that they believe these tools would be beneficial for their learning of cyber security (M = 3.84 - 4.02). Students reported being engaged with AI chatbots with a high frequency for all three applications; course assistance (M = 3.85), exam study (M = 3.95), and referring friends/peers (M = 3.78). The quality of responses from AI chatbots was rated positively by students (M = 3.92 - 4.10), but students also acknowledged some of the challenges to using the chatbot such as misinformation, difficulty verifying information, and over-reliance on critical thinking (M = 3.82 - 4.15). Finally, students had the highest scores (M = 3.88 - 4.10) for perceived academic performance. Overall, these findings confirm hypotheses H1 and H2 by providing evidence that using AI chatbots has a positive relationship with students' perceived learning outcomes for cyber security education.