Project Based Assessment
ProjectAssessment.AppE-Mail for HelpSchedule a Zoom Meeting
  • Introduction
  • Why and Example Uses
  • Data and Estimation
  • Interpretation
  • FERPA, Technology and Limitations
  • Canvas and Converter App
Powered by GitBook
On this page
  • Using Project Based Assessment for Instructional Improvement and Portfolios
  • Using Project Based Assessment for Academic Assessment
  • Using Project Based Assessment for Research

Why and Example Uses

There are many uses of the ProjectAssessment.App. In this section, we discuss some example uses by individual instructors, departments, or researchers.

PreviousIntroductionNextData and Estimation

Last updated 8 months ago

There are many ways to analyze the results of multiple choice exams. (see for a review) can estimate student ability, difficulty, and other parameters of exam items. Some pre- and post-test methods can compare performance to a baseline (e.g., , ), and some pre- and post-test models can solve for underlying learning values (e.g., , ).

Unfortunately, there are far fewer ways to analyze data generated by a rubric. introduces a method to separate student ability from rubric-row difficulty while accounting for censoring of the data (see the "" for an explanation of the censoring problem). The web application makes this technique accessible to a much larger audience (including those with no programming ability) than the included in the original paper. Additionally, under some circumstances, it is much more productive to use the web application even for those that have the technical expertise to use the Python package. The web app automatically produces tables and graphics that can easily be saved and a print view designed to be used as an appendix in a report (see below).

As it is very easy and fast to use this software, it can be used by instructors to improve their class, departments to improve their degree programs, and researchers testing an intervention. Below are some example uses. The video shows how we generated these results and provides a short interpretation. The text below the video discusses the results in more details.

Using Project Based Assessment for Instructional Improvement and Portfolios

This software can be used by instructors to diagnose issues in their class and determine the performance of different groups. This can be used to improve the course or as part of a teaching portfolio for annual review or RPT (Review, Promotion, and Tenure). The author of this software used the web app to improve his classes and these results, along with the instructor's plans for mitigation, were included in his 2023 Annual Review.

Below are selected results from Data Analysis from Scratch: A graduate-level course where students learn to code traditional estimators (e.g., OLS, MLE), non-parametric techniques (e.g., KDEs), and machine learning techniques (e.g., Random Forests) from scratch (i.e., not using pre-built estimation functions). This gives the student a strong understanding of exactly how these techniques work and how they are related. This is the capstone class for many of the graduate analytics programs at the university.

Variable
Average Logistic
Average Discrete Marginal Logistic

Test Identification

0.420

0.333

Pairwise Bootstrap

0.042

-0.045

Clustered Errors

0.000

-0.087

The "Test Identification" rubric row tested the students' ability to select the appropriate test given a specific situation. This involves critical thinking skills, so it isn't easy. But one would not think it is that hard either. "Pairwise Bootstrap" and "Clustered Errors" both test the students' ability to hand code specific algorithms. Again, not easy tasks.

What the estimates reveal is that the students found "Test Identification" to be exceptionally difficult and the two algorithms to be exceptionally easy. This could indicate that too much class time and resources are dedicated to these algorithms and time/resources could be reallocated to critical thinking skills related to the best use of different statistical tools. These changes are planned for the Fall 2024 iteration of the class.

Using Project Based Assessment for Academic Assessment

Universities of all sizes are expected to show their accrediting agency that their students are gaining knowledge from the college experience. In practice, this means that universities require programs to collect and analyze data about their students' performance and take action based on these results. This often means that certain student learning outcomes (SLOs) are collected from specific classes in the program (usually required classes late in the program).

The Project Based Assessment web app can be used for academic assessment. In fact, the author's department adopted this method to assess all learning goals in the MS Economics program. The program's learning goals are assessed in three courses. For the purposes of this documentation, we will discuss select items measured in graduate Econometrics. (Note that the full Econometrics results contain estimates for nine rubric rows.)

Variable
Average Logistic
Average Discrete Marginal Logistic

Metrics - 2.2

0.168

0.056

Metrics - 2.4

0.020

-0.092

Using Project Based Assessment for Research

The table below includes selected rubric row estimates using data from the Fall 2023 section of Data Analysis from Scratch. These estimates were produced with this application. It is important to note that there were twelve rubric rows estimated in this course and many more columns are produced by the software (see ""). These select rows and values explain the future actions planned by the instructor.

There is a detailed explanation of how to interpret these values in the "" section of this documentation. However, for this section, it is sufficient to know that greater values indicate that the rubric row was more difficult.

Another question of interest to the instructor was if students in different degree programs performed statistically different from one another. In the above figure, the proxies for student ability are grouped by degree program. This was an important question as there is a persistent idea in the department that the analytics students in the MBA or MS in Data Science were not as well prepared as the MS Economics students. In the figure above, there doesn't appear to be a strong pattern. This suggests that this perception isn't true - at least in this class.

"Metrics - 2.2" and "Metrics - 2.4" are traits of SLO 2: "Students will demonstrate understanding of regression assumptions, including violations of said assumptions." Both are at the same Bloom's level (application) and test similar concepts ("Students will identify regression assumption violations" [2.2] and "Students will demonstrate how to address regression assumption violations" [2.4]). Moreover, the instructor intended them to be at similar difficulty levels. Nonetheless, . The instructor concluded that the question used for 2.4 was not at the intended difficulty and should be adjusted.

It is also common in assessment procedures to wonder how the results have changed over time. This was of particular interest in the two semesters where this procedure was adopted as one section of Econometrics was taught in person and the other was taught over Zoom. The results suggest, , that students perform equally both semesters.

Note that in the case of all three courses, the department to produce PDF appendices for the university's assessment committee.

includes a that can be used by researchers who are interested in adopting the method presented in the paper. This can be a good option for researchers who are interested in integrating the estimation routine into a larger data pipeline and are familiar with Python.

However, even in the research context, the web application might be all the researcher needs to test their intervention. As highlighted in the two sections above, the web app can create separate groups of students and compare them both visually and statistically. p-values for following statistical tests are provided: , , , and . Thus, as in the example provided in the introductory video, if a treatment is implemented for some of the students, the researcher can statistically compare these students to those who did not recieve the treatment.

Smith and Wooten (2023)
Python package
Mann-Whitney
Kruskal–Wallis
Anderson-Darling
Kolmogorov-Smirnov
Item Response Theory
de Ayala 2022
Hake 1998
Walstad and Wagner 2016
Smith and Wagner 2018
Smith and White 2021
Smith and Wooten (2023)
Project Based Assessment
Python package
academic assessment example
Data and Estimation
Interpretation
Interpretation
This was generated using the web app simply by making files containing the student ids in each of the degree programs that typically take this course.
This visual interpretation was in-line with the results of the statistical tests provided by the software.
the results suggest they were not of equal difficulty
both visually and by the statistical tests provided by the software
used the print feature discussed at the bottom of the "Interpretation" section
Comparison of Average Logistic estimates by degree program in Data Analysis from Scratch
Comparison of Average Logistic estimates by semester in Econometrics. Fall 2023 was taught over Zoom while Spring 2023 was taught in person.