README.md

Introduction

This repository contains the supporting files for the hands-on exercises that are included in Chapter 3 of the Morgan Claypool book on Automated Essay Scoring by Beata Beigman Klebanov and Nitin Madnani.

What's here

In the book, we used RSMTool to build machine learning models to automatically score the essays from the Automated Student Assessment Prize (ASAP) competition – specifically essays from writing tasks 1 and 2. This repository contains the data, the features, and the RSMTool configuration files that are used in the book.

Structure

The repository has the following structure:

├── data
│   ├── essays
│   ├── features
│   └── rubrics
├── environment.yaml
└── experiments
    ├── 00-all-features
    ├── 01-all-fair-features
    ├── 02-remove-collinear-feature
    ├── 03-remove-insignificant-features
    ├── 04-transform-features
    ├── 05-percent-func-feature
    ├── 06-evaluate-on-heldout-data
    ├── 07-evaluate-on-task-2
    ├── 08-task-2-specific-model
    ├── 09a-train-and-test-on-average-score
    ├── 09a-train-and-test-on-average-score-task2
    ├── 09b-only-test-on-average-score
    ├── 09b-only-test-on-average-score-task2
    └── 10-different-learner

The data directory contains: (a) the scoring guidelines or rubrics for ASAP writing tasks 1 and 2 (b) the essays from the tasks – split into a training set, a development set, and a test set (b) the features extracted from the essays in each of the three datasets to be used for building the automated scoring models via RSMTool.

Each sub-directory under experiments contains one of the experiments from chapter 3 of the book. For example, the sub-directory 00-all-features corresponds to the section 3.1, entitled Experiment 0: Use all features.

Setup

The easiest way to get started is by first installing the conda package manager for Python. The installation instructions can be found here.

Once conda is installed, you can install RSMTool and all its dependencies as follows:

conda env create -f environment.yaml

This will create a conda environment called aesexpts that can then be used to run any of the RSMTool experiments. For example, to run the RSMTool experiment with all of the features:

conda activate aesexpts
cd experiments/00-all-features
rsmtool config.json

Running this set of commands will produce the final RSMTool evaluation report under report/all_features_report.html in the same directory.

Help

If you have any problems running any of the experiments, please file an issue.