gabriellegardiner – Higher Education Text Analytics

UniSA pilots AcaWriter automated writing feedback

The Teaching Innovation Unit (TIU) at the University of South Australia undertook a pilot of AcaWriter over the 2018 academic year. This preliminary evaluation highlighted the high potential of the tool to provide early feedback on students’ writing. The pilot also revealed certain technical complications that require further improvement to ease broader dissemination and uptake. The outcomes for the small study serve as a useful base to establish a technical roadmap for future development.

The following limitations were identified in an early version of the public demonstration version of AcaWriter:

The feedback provided to students was, at this stage of development, limited. The tool simply highlighted features of the submitted text that aligned with pre-determined descriptors of reflective or academic textual genres. In some cases, the feedback provided was confusing for end-users. For instance, every occurrence of the modal ‘would’ in a trial text was highlighted as an ‘Expression indicating self-critique’, even when its function was to indicate a habitual action in the past. These feedback inconsistencies can be readily addressed in future iterations of the tool.
The tool did not provide evaluative feedback regarding whether the text was overall consistent with the requirements of the selected genre, or suggestions on how the text could be improved.
Finally, the trial group noted that the user interface could be improved. For example, once a text was uploaded it could be expected that clicking on the ‘feedback’ button would generate the tool’s response. However, this was not the case; the user must instead click on the ‘Get Feedback’ button, which displays a ‘download’ icon; feedback is then accessed by clicking on the ‘Reflective Report’ tab. Again future modifications to the tool will readily address these minor concerns.

We are aware that since the initial evaluation at UniSA, further technical work has been carried out by other users and developers. The subsequent changes will serve to move AcaWriter in the right direction in terms of usability and effectiveness. For example, some experimentation is underway to trial machine learning approaches, rather than educator-generated rubrics, as sources of criteria for text evaluation.

Since our piloting, there has been further work at UTS co-designing the tool-generated feedback to be more informative and useful in supporting further development of draft texts.

Finally, the project has been improving the back-end of AcaWriter, with an API to be released in the near future. All of these suggested modifications will serve to encourage users to further engage with the tool, and support further development.

A benefit of this ATN collaboration has been increased awareness of the priority areas, of the problems (both technical and teaching) that other institutions are tackling, and how we can share resources to collectively move the sector forward in aspects of machine learning.

RMIT finds two directions for the Text Analytics Pipeline (TAP) tool

RMIT has been exploring the use of the Text Analytics Pipeline (TAP) in two main directions.

Firstly, there is strong appetite to use TAP by support staff from the Study and Learning Centre, where study and learning advice is provided to students. TAP provides formative assistance that students can interact with and complement the feedback obtained by tutors.

Secondly, TAP has stimulated research to understand student feedback obtained through university surveys. RMIT Marketing will also be using TAP as an engine to understand the data obtained through focus groups. This team works with ethnographic research and translates it into insights, providing a useful case study for TAP.

The ‘verbology’ of learning

Educators often write learning objectives that describe what students must do or be like to demonstrate that learning has occurred. Bloom’s Taxonomy provides a means of classifying the associated depth of thinking. The six levels of this taxonomy are: remembering, comprehending, applying, analysing, evaluating, and creating. Traditionally, classifying a learning objective on this taxonomy focusses on verbs in the learning objective.

Verbs at the remembering level are usually associated with rote memorization (e.g. list, cite, define). Verbs at the comprehending level require the student to understand something to a depth sufficient to explain something to others (e.g. describe, communicate, illustrate), whereas applying verbs require the use of skills or knowledge in practice (e.g. conduct, demonstrate, employ, follow). Analysing verbs require the formation of connections between ideas and concepts (e.g. compare, contrast, distinguish). Evaluating verbs take this up a notch, requiring the formation of a judgement or critique (e.g. appraise, determine, judge, recommend). At the highest level, creating verbs capture the synthesis of something new (e.g. construct, design, devise, generate).

While verbs are commonly used in Bloom’s classification, sometimes adjectives and adverbs can play a role too. For example, to “critically analyse” something is an example of evaluating, whereas to “analyse” by itself is just analysing because the former expects a critique or judgement. Similarly, to “report important findings” is different than just “report findings” since the former also requires a judgement about the significance of the findings. This is further complicated since some words can either be a noun or verb depending on how they’re used (e.g. design, document). This means that it’s a little more detailed than just looking up words in tables. You also need to be able to break words in learning objective down into their respective parts of speech.

At ascilite 2018, we’ll be introducing a new app called Blooms TL for iPhone and iPad to assist with classifying learning outcomes on Bloom’s taxonomy to make all of this a bit easier. The classification in this new app uses tables containing verbs and other parts of speech that are generally indicative of a given Bloom’s level.

View slideshows with screenshots of the app in action.

The new app will be available soon. If you’d like to jump the queue and download a beta copy, contact Brian von Konsky at Curtin University (B.vonkonsky@curtin.edu.au) for instructions.

Learn more about this work:

Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives

Stanny, C. J. (2016). Reevaluating Bloom’s Taxonomy: What measurable verbs can and cannot say about student learning. Education Science, 6(37), 12.

von Konsky, B.R, Zheng, L., Parkin, E., Huband, S., and Gibson, D.C. (2018). Parts of speech in Bloom’s Taxonomy classification. 35th International Conference on Innovation, Practice and Research in the Use of Educational Technologies in Tertiary Education, ascilite 2018, Geelong, Australia.

Progress with the Text Analytics Pipeline (TAP)

The Text Analytics Pipeline (TAP) software has undergone significant improvements over the last few months. HETA project Technical Lead Andrew Gibson (QUT) has been working on a batch mode that allows the processing of unlimited numbers of documents from Amazon S3 Cloud storage, opening the way for more scalable analytics on millions of documents.

The TAP client for python (tapclipy) is also being improved to allow the batch mode to be used from within Python. The latest version of TAP can always be found at https://github.com/heta-io/tap and the python client at https://github.com/heta-io/tapclipy . If you’re looking to try bleeding-edge features, check out the develop branch, otherwise for a stable version use the master branch.

TAP has previously suffered from a lack of clear documentation (as do many software research projects). However, this issue is now being addressed by Mike Powell (QUT) together with Sarah Taylor (RMIT). The revised documentation can be found at https://heta-io.github.io/tap/ and includes instructions for getting started from Docker, or with the Source code, as well as helpful information for developers.

Building a bespoke language model for student feedback

In the course of investigating the use of Natural Language Processing and machine learning tools to better extract and navigate our student comments data, it became apparent that many tools have some shortcomings when used on this type of text. Such writing has a very specific style, vocabulary and context which can hinder the effectiveness of generic tools and pre-trained machine learning models. Given our access to a large corpus of student feedback comments, we decided to turn the problem around and see if we could use our data to build new tools.

One of the issues at the very foundation of Natural Language Processing is how to represent words and groupings of words in a format to which mathematical algorithms can be applied. One such representation developed at Google is the Word2Vec model, which represents words as “high dimensional vectors”. That is, each word is represented by a set of numbers (usually a couple of hundred) and these numbers define how a word fits in with other words in the language. This representation is found entirely from the words and their placement relative to other terms in the corpus of text used to train the model. It is a purely machine learning and data-driven approach with no input knowledge or rules about the language itself.

To train our own Word2Vec model we used 250,000 student comments from the past 4 years of subject-level student feedback survey data. As a generic comparison model, we used a popular open-source Word2Vec model available in the Python Natural-Language-Tool-Kit (NLTK) package. This model was trained on a 100 Billion word corpus of Google News stories. While this is far larger than our bespoke model it is also less domain specific and we will demonstrate the effect of this domain-specific aspect in the examples below.

The mathematical representation of words in Word2Vec makes it straight-forward to compute a measure of the similarity between words (or terms) and we will use some examples of this computed similarity to make our comparison between models.

Table 1: Comparison of the most similar terms computed from the Word2Vec models to the terms “occupation” and “real”. This example clearly demonstrates an advantage of a domain specific model. In a university setting the correct context for “occupation” is as a reference to a job or work. This context is correctly picked up in our bespoke model. However, the more generic model relates these terms to military occupations with the most similar words being things like imperialism, tyranny, war etc.

Rank	Most similar terms (Our Model: trained on 250,000 comments from the Student Feedback Survey)	Most similar terms (Comparison Model: trained on 100 Billion Words from Google News)
1	future_job	occupation
2	work_environment	war
3	work_place	oppression
4	project_manager	imperialism
5	legal_practitioner	subjugation
6	workplaces	genuine
7	may_face	tyranny
8	working_environment	imperialist
9	real_situation	actual
10	global_business	oppressors
11	real_world	colonialism
12	care_women	colonialist
13	intended_profession	liberation
14	customers	dispossession
15	real_project	disengagement
16	managers	postwar
17	insight_real_world	Zionism
18	planners	profoundest
19	humanity	militarism
20	professional_career	invasion

Table 2: In the case of less ambiguous terms, it could still be argued that the bespoke model can be superior to the larger but more generic one. In this case, we search for terms similar to “lecture” and “authentic”. While both models give good in-context results some more detailed and specific terms are returned by the purpose trained model (e.g. real-world_cases, class_debates, story_telling) which make sense in the context of student feedback.

Rank	Most similar terms (Our Model: trained on 250,000 comments from the Student Feedback Survey)	Most similar terms (Comparison Model: trained on 100 Billion Words from Google News)
1	interaction_class	lectures
2	real-world_cases	authentically
3	class_debates	lecture
4	practice_theory	lectures
5	story_telling	presentation
6	easy_remember	contemporary
7	economic_models	colloquium
8	us_think_critically	authenticity
9	pragmatic	seminar
10	class_discussion	informative
11	lecture_tutorial	presentations
12	indigenous_perspective	symposium
13	relaxed_learning_environment	enlightening
14	throughout_lecture	seminar
15	active_learning	oration
16	robust	travelogue
17	inviting	sermon
18	worldly	intellectuality
19	real-world_experience	storyteller
20	lectures_bit_boring	timeless

A new tool for navigating feedback from student surveys

One of the three priorities for the HETA project is to progress technology and analytics for extracting information and insight from student comments obtained from feedback surveys. In general, the “quantitative and categorical data” and the “free text comments” from student surveys are analysed independently. The first often in terms of aggregate statistics and the later simply read by interested parties. A joint analysis that merges the quantitative (scale-from-one-to-five type) responses and the richer specific information in the associated open-ended text responses allows a more powerful and nuanced extraction of information.

As a first step toward this more integrated approach, we have developed an online navigation tool for comments from the UTS Student Feedback Survey. This tool joins together, for the first time at UTS, the quantitative and free-text responses and provides an assortment of filtering and navigation utilities. This tool brings together the last 4 years of student feedback on their experience in individual subjects and contains a total of over a quarter-of-a-million written student comments.