ArtChat Study

Higher Education (Art University)

• Faculty of Applied Sciences, Macao Polytechnic University
• Art and Design School, Anhui Broadcasting Movie and Television College

Eastern China (art university setting)

• In-class (Higher Ed)

13-week quasi-experimental study

Weekly 90-minute sessions over 13 weeks

Personal mobile devices or lab computers (web interface)

• No individual logins required beyond university authentication
• No personal data collected outside study parameters
• Classroom-based setting with instructor supervision
• Fixed class schedule (weekly 90-minute sessions)

Approx. 19–23 years (undergraduate students)

Digital Media Art

No prior experience with generative AI tools assumed

None assumed

• Mastery of introductory digital media art concepts
• Development of self-regulated learning skills (planning, monitoring, reflection)

• Awareness of AI as a learning support tool
• Ability to critically evaluate AI-generated responses
• Improved learning attitudes and technology acceptance

• Not a programming lesson
• Not an AI theory lesson
• Not an assessment-driven activity

ArtChat

Curriculum-aligned generative AI chatbot

ArtChat functions as an interactive learning assistant that provides course-specific, context-aware responses to student queries, supporting knowledge construction and self-regulated learning.

• Students log into ArtChat via web interface
• Students type text-based questions related to course content
• ArtChat retrieves relevant information using RAG and knowledge graphs from uploaded course materials
• Students receive structured, citation-backed answers
• Students may ask follow-up questions or request clarification

• Knowledge base restricted to instructor-uploaded course materials
• No access to external or unfiltered internet content
• Content moderation via AI model alignment
• Instructor oversight during class sessions

• Instructor introduces weekly topic
• Students engage with ArtChat during or after class for self-directed learning
• Students reflect on responses and integrate them into understanding
• Instructor facilitates discussion and addresses common questions

• Instructor guidance on effective prompting
• Reflection prompts embedded in study design
• Peer discussion and instructor-led synthesis

• Some students initially over-relied on AI without critical evaluation
• Vague or poorly structured queries led to less useful responses
• Time constraints limited deeper exploration in some sessions
• General LLM group occasionally received off-topic or inaccurate information

• Added in-class orientation on effective questioning strategies
• Reinforced "AI as a learning partner, not an answer machine" framing
• Encouraged students to compare AI responses with course materials
• Integrated reflection activities into post-class assignments

High participation; ArtChat group demonstrated sustained interaction with the tool and increased voluntary engagement with course content

• Students revised their understanding based on AI feedback
• Some questioned or challenged AI responses, prompting deeper inquiry
• Peer discussions about AI-generated content increased
• Students reported feeling more in control of their learning process

• No personal data collected beyond basic demographic information for research purposes
• All student data anonymized in analysis and reporting
• Informed consent obtained from all participants
• Study approved by institutional ethics review
• Tool usage aligned with university IT policies

• Empirical validation of AI tools in higher education
• Self-regulated learning and AI interaction studies
• Cross-disciplinary AI applications (AI + education + arts)

• Learning sciences
• Educational technology
• AI literacy
• Art education
• Human-AI collaboration

Question formulation, critical evaluation, reflection, discussion

Information retrieval, contextual explanation, draft generation

Yang, Chenglin; Un, Kin-Seong; Jiang, Shujing; Tan, Tao; Lam, Chi-Kin; Sun, Yue

Cao, Christopher — Research Assistant, M.S. Student, California State University, Fullerton