ADS401 Students' Perspective on AI Agents at XJTLU

An empirical HCI evaluation of awareness, adoption and use of XJTLU-provided AI agents.

Group A · Hongcheng TANG · Mingjian TAO · Liyuan CHEN · Haode SHEN · Explanatory sequential mixed methods

This page reports the evaluation design; findings will be added after collection (November 2026).

1. Project at a glance

We are evaluating how students in ADS401 (Evaluation Methods and Statistics, 2026–27) become aware of, adopt, and use XJTLU AI — any AI agent service officially provided by XJTLU, including but not limited to the Junmou AI agent platform — in their ADS401 learning, and why some students instead rely on external commercial AI (ChatGPT, DeepSeek, Kimi and similar independently accessed services).

The study is an explanatory sequential mixed-methods design: a cross-sectional online questionnaire first maps awareness, usage patterns and perceptions; semi-structured follow-up interviews then explain the choices behind those patterns.

• Evaluated system: XJTLU AI Agent Platform (Junmou AI)
• Target users: ADS401 2026-cohort students (N = 48 enrolled; 44 eligible invitees)
• Methods: online survey (census-adjacent invitation) → purposive follow-up interviews
• Presentation deck: PDF (LMS submission) — source: slides/slides.md

2. Problem, motivation and research questions

Intended users and context

XJTLU provides AI services such as Junmou AI to support learning. ADS401 students are simultaneously heavy users of external AI for coursework — yet the module team has no systematic evidence of whether the institutional services are known, tried, adopted, or why students choose one over the other. This matters now: the same cohort is both the subject of AI-supported learning and the population future iterations of these services must serve.

Research questions

1. RQ1 — Awareness & Adoption. To what extent are students in our ADS401 module aware of and adopting XJTLU AI?
2. RQ2 — Usage Patterns. How do students in ADS401 use XJTLU AI for their academic tasks?
3. RQ3 — Adoption Factors & Perceived Comparison. What perceived benefits, barriers, and differences between XJTLU AI and external AI services are associated with students’ adoption and intentions to continue, try, or resume using XJTLU AI?

Scope of claims

Findings will describe participating ADS401 students only. Voluntary participation limits representativeness even within the module; results do not generalise to all XJTLU students, and the cross-sectional design supports association claims, not causal effects.

3. Background and related work

Technology-acceptance research establishes perceived usefulness and perceived ease of use as central antecedents of adoption¹. Recent work on generative AI in higher education documents widespread student use alongside sharp institutional policy divergence — adopt or ban — and warns that self-reported use is shaped by self-selection². Surveys of student AI use in higher education provide validated question stems for experience, usefulness and trust items³.

Gap. These strands have not been brought together for XJTLU’s own institutional agent platform: we lack evidence on whether official services are even known to the students they target, and how perceived differences against commercial tools shape adoption intentions. Evaluation rationale. A census-adjacent survey quantifies the adoption landscape (RQ1–RQ2); interviews then explain the perception drivers (RQ3), following the sequential explanatory logic of Ivankova et al.⁴ and the integration principles of Fetters et al.⁵.

4. System and evaluation design

Instrument v1 in pilot.

Evaluated interaction

XJTLU AI agents support question answering, document/knowledge-base interaction, writing and study-planning tasks inside coursework. We evaluate students’ self-reported encounters: awareness, first use, recent academic use, verification behaviour before relying on outputs, and intentions — not system internals or logs.

Participants, recruitment and sampling

All 48 ADS401 2026-cohort students form the sampling frame; the 4 group researchers are excluded, leaving 44 potentially eligible invitees — per class roster, 39 mainland-China and 5 presumed non-mainland-China HCI students. We invite the full frame (census-adjacent, voluntary participation); the achieved sample will be reported after collection. No usability-task sample is involved, so the 12-participant usability floor is not triggered; our sample-size rationale is full-frame coverage rather than inferential power.

Study design and grouping

The questionnaire (v1, in pilot) first screens eligibility and consent, then awareness (Q3) and ever-use (Q4). Respondents are grouped by reported ADS401 AI use in the past four weeks, with skip logic so each group answers only relevant modules:

All paths converge on tool-choice factors and improvement priorities (Q17–Q19) and an optional interview opt-in (Q20–Q21; Q21 only if Q20 = Yes). Interview contact details are collected in-questionnaire (Q21) only from respondents opting in; responses contain no identity fields otherwise.

Research materials and status

The full instrument and supporting documents are consolidated in the repository (assets/pre2/materials/); the slide-deck inventory matches it:

Variables and measures

The Technology Acceptance Model (TAM) includes validated scales for perceived usefulness and perceived ease of use (Davis, 1989)¹; these constructs inform the questionnaire items adapted to the XJTLU AI context. The adapted items remain exploratory, context-specific measures — not the complete validated scales.

Ethics, consent, privacy and bias mitigation

• Participation is voluntary with an explicit informed-consent gate (Q1–Q2); respondents may end the survey at any branching exit.
• No names are collected; the optional Q21 asks only for an ID + email/telephone, used solely for interview recruitment and scheduling.
• Skip logic prevents non-users from rating experiences they never had.
• Usefulness/ease-of-use items are labelled exploratory to avoid over-claiming construct validity.
• Self-reported use will be described as self-report, never as system-log evidence.
• Pilot: internal questionnaire check with the 4 researchers precedes fielding; pilot-driven changes will be logged on this page.

5. Data preparation and analysis plan

Quantitative (survey)

• Descriptive statistics: counts and percentages for awareness, ever-use and past-four-week usage, each with its valid-response denominator.
• Cross-tabulation: usage patterns compared across the four groups.
• Rating analysis: activity-specific ratings and intentions summarised descriptively; N/A excluded from rating summaries.
• Perceived comparison: paired activity ratings only where dual users rated both tools for the same category; direct comparison responses reported separately, including “unable to compare”.

Qualitative (interviews) and integration

• Sampling: purposive, among survey respondents who opt into follow-up contact (Q20–Q21), informed by initial survey analysis; about 4–6 students, prioritising G1 (Both AI) and G2 (Other AI Only).
• Format: semi-structured, 15–20 minutes; focus on tool-choice motivations, adoption barriers and learning experiences.
• Thematic analysis: coding and theme development following Braun & Clarke⁶.
• Integration: interview themes are mapped onto survey patterns to explain adoption decisions and identify improvement directions — the survey says what, the interviews explain why⁵.

Analysis is deliberately descriptive and exploratory: with a single-module voluntary sample we report distributions and patterns, and we do not run group-inferential tests or claim causal effects.

Next steps

Research materials → internal questionnaire check (4 researchers) → survey data collection → preliminary survey analysis → follow-up interviews → thematic analysis & integration. The achieved sample is reported after collection; findings on this page remain pending until then.

6. Findings

Pending — survey not yet fielded. Data collection starts after Presentation 2 feedback and the instrument pilot. This section will present, per research question: awareness and adoption rates (RQ1), usage-pattern cross-tabulations by group (RQ2), and barrier/intention/comparison summaries with interview themes (RQ3) — with figures, exact denominators and anonymised quotes.

7. Discussion, limitations and recommendations (planned)

Discussion will interpret findings for the ADS401 context and propose evidence-based improvements to XJTLU AI services. We pre-register the following limitations to bound claims:

• Single-module sample — results describe participating ADS401 students; voluntary participation may limit representativeness even within ADS401; no generalisation to all XJTLU students.
• Self-reported data — the study relies on students’ reported behaviours and perceptions.
• Interview coverage — prioritising G1/G2 may under-represent G3 (No AI) and G4 (XJTLU AI only) perspectives.
• Cross-sectional design — captures recent use and intentions, not causal effects or observed long-term continued use.
• Perception-based comparison — comparisons with external tools do not establish objective system-capability differences.

8. Transparency and references

Contribution statement

Study design, instrument and interview protocol: all four members (work split recorded in the group agreement). Repository/site assembly: Liyuan CHEN. Survey scripts and materials integration: Mingjian TAO. Slide narrative: Hongcheng TANG. (Final wording to be confirmed at submission.)

Authorship statement

All research questions, methodological decisions, sampling frames, variable operationalisations, questionnaire items, interview protocols, and critical evaluations are the student authors’ own.

Data and materials availability

No raw participant data is stored in this repository or on this site (coursework red line). Anonymised, aggregated results and the analysis syntax will be linked here after collection. The survey instrument (pilot v1) is archived in the group’s working repository; the interview guide will be added once finalised.

References (IEEE, order of first citation)

1. F. D. Davis, “Perceived usefulness, perceived ease of use, and user acceptance of information technology,” MIS Quarterly, 13(3), pp. 319–340, 1989. doi:10.2307/249008
2. M. Farinosi and C. Melchior, “To adopt or to ban? Student perceptions and use of generative AI in higher education,” Humanities and Social Sciences Communications, 12, Art. 1684, 2025. doi:10.1057/s41599-025-05982-7
3. J. Chung, M. Henderson, N. Pepperell, C. Slade, and Y. Liang, “Student perspectives on AI in Higher Education: Student Survey,” Student Perspectives on AI in Higher Education Project, 2024. doi:10.26180/27915930
4. N. V. Ivankova, J. W. Creswell, and S. L. Stick, “Using mixed-methods sequential explanatory design: From theory to practice,” Field Methods, 18(1), pp. 3–20, 2006. doi:10.1177/1525822X05282260
5. M. D. Fetters, L. A. Curry, and J. W. Creswell, “Achieving integration in mixed methods designs—Principles and practices,” Health Services Research, 48(6 Pt 2), pp. 2134–2156, 2013. doi:10.1111/1475-6773.12117
6. V. Braun and V. Clarke, “Using thematic analysis in psychology,” Qualitative Research in Psychology, 3(2), pp. 77–101, 2006. doi:10.1191/1478088706qp063oa
7. GLM-5.3 Flash (Zhipu AI) and Gemini 3.8 Flash (Google), operated through the omp open-source coding agent harness. Used for slide layout and formatting, language proofreading, reference formatting, and build/assembly automation — non-substantive use only.