Exploring Social Biases of Large Language Models in a College Artificial Intelligence Course

Source publication / research team or educational organization described in paper

Not specified in extracted text

Researchers, educators, instructors, or facilitators as described in the source publication

• Higher education

Course implementation or course design

Not specified in extracted text

Not specified in extracted text

LLM/Chat, NLP / text classification, ML concepts / supervised learning, Ethics / responsible AI

• AI output reliability, hallucination, academic integrity, and age-appropriate use require safeguards.
• High-stakes or student-data-centered AI use requires stronger governance, transparency, and bias monitoring.

Higher education

Mixed or not explicitly specified; infer from target learner group and intervention design.

Varies by intervention; not specified unless the paper explicitly describes prerequisites.

• Document the AI education intervention, course, tool, or resource described in the source publication.
• Extract the learner context, AI role, pedagogy, outcomes, and constraints for AAB registry comparison.
• Large neural network-based language models play an increas- ingly important role in contemporary AI.

• Support AAB comparison across AI literacy, AI education, teacher training, higher education, and workforce contexts.
• Capture evidence maturity, transferability, and limitations rather than treating the publication as product endorsement.

• Not an AAB endorsement of the tool, curriculum, provider, or result.
• Not a direct replication record unless the source paper reports implementation details sufficient for replication.

LLM/Chat, NLP / text classification, ML concepts / supervised learning, Ethics / responsible AI

Language context discussed in source publication

• Evaluator

• Primary interaction pattern inferred from publication: Curriculum / course design, Ethics / responsible AI education.
• AI capability focus: LLM/Chat, NLP / text classification, ML concepts / supervised learning, Ethics / responsible AI.

• Require human review of generated outputs and explicit guidance against over-reliance or answer copying.
• Include bias, fairness, transparency, and social impact discussion as part of the learning design.

• Review the publication’s reported context, learner group, AI tool or curriculum, implementation process, and outcome evidence.
• Map the case to AAB registry fields for comparison across educational levels and AI capability types.
• Use the source publication and PDF for any manual verification before public registry release.

• Human educators/researchers remain responsible for instructional design, supervision, interpretation, and ethical safeguards.
• AI systems or AI concepts provide the learning object, support tool, evaluator, simulator, or automation context depending on the paper.

• Instructional / curriculum-based learning
• Registry extraction emphasizes explicit learning goals, observed outcomes, constraints, and safety limitations.

• AI output reliability, hallucination, academic integrity, and age-appropriate use require safeguards.
• High-stakes or student-data-centered AI use requires stronger governance, transparency, and bias monitoring.

• Case classified under: Published empirical study.
• Pedagogical pattern: Instructional / curriculum-based learning.
• Any additional adaptations should be verified against the full paper before public-facing publication.

• Engagement evidence should be interpreted according to the source paper’s reported method and sample.
• Although these models demonstrate sophisticated text generation capabili- ties, they have also been shown to reproduce harmful social biases contained in their training data.

• Although these models demonstrate sophisticated text generation capabili- ties, they have also been shown to reproduce harmful social biases contained in their training data.
• Through the process of constructing a dataset and evaluation metric to measure bias, students mastered key technical concepts, in- cluding how to run contemporary neural networks for natural language processing tasks; construct datasets and evaluation metrics; and analyze experimental results.

Large neural network-based language models play an increas- ingly important role in contemporary AI. Although these models demonstrate sophisticated text generation capabili- ties, they have also been shown to reproduce harmful social biases contained in their training data.

• Require human review of generated outputs and explicit guidance against over-reliance or answer copying.
• Include bias, fairness, transparency, and social impact discussion as part of the learning design.

• Pre/post or experimental evidence
• Activity documentation

• Can be used as an AAB evidence record for cross-case comparison, standards drafting, and evidence-maturity mapping.
• Supports identification of recurring patterns in AI literacy, AI education implementation, teacher preparation, assessment, and responsible AI learning.

• Conceptual understanding
• Engagement / motivation
• Ethics and responsible use
• Assessment / feedback quality
• Curriculum / course design
• Ethics / responsible AI education
• LLM/Chat
• NLP / text classification

• Completed

Higher education

Higher education

LLM/Chat, NLP / text classification, ML concepts / supervised learning, Ethics / responsible AI

Instructional / curriculum-based learning

High

Medium

Exploring Social Biases of Large Language Models in a College Artificial Intelligence Course

• Skylar Kolisko
• Carolyn Jane Anderson

Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37 No. 13, EAAI-23

2023

10.1609/aaai.v37i13.26879

https://ojs.aaai.org/index.php/AAAI/article/view/26879

https://ojs.aaai.org/index.php/AAAI/article/view/26879/26651

070_Exploring Social Biases of Large Language Models in a College Artificial Intelligence Course.pdf

9

Large neural network-based language models play an increas- ingly important role in contemporary AI. Although these models demonstrate sophisticated text generation capabili- ties, they have also been shown to reproduce harmful social biases contained in their training data. This paper presents a project that guides students through an exploration of social biases in large language models. As a final project for an intermediate college course in AI, stu- dents developed a bias probe task for a previously-unstudied aspect of sociolinguistic or sociocultural bias. Through the process of constructing a dataset and evaluation metric to measure bias, students mastered key technical concepts, in- cluding how to run contemporary neural networks for natural language processing tasks; construct datasets and evaluation metrics; and analyze experimental results. Students reported their findings in an in-class presentation and a final report, re- counting patterns of predictions that surprised, unsettled, and sparked interest in advocating for technology that reflects a more diverse set of backgrounds and experiences. Through this project, students engage with and even con- tribute to a growing body of scholarly work on social biases in large language models.

• Higher education

• AI output reliability, hallucination, academic integrity, and age-appropriate use require safeguards.
• High-stakes or student-data-centered AI use requires stronger governance, transparency, and bias monitoring.

Not specified in extracted text unless noted in duration field.

Requires educators/researchers/facilitators with sufficient AI literacy and pedagogy knowledge for the target learners.

Infrastructure depends on AI tool type, learner devices, data access, and institutional policy context.

High

• group_size
• duration

This entry was automatically extracted from the PDF text and manifest metadata. Fields should be manually verified before public registry publication, especially group size, location, duration, and outcome claims.