# Introduction ProjectAssessment.App allows you to easily measure your students' knowledge level and rubric difficulty, whenever you use a rubric to grade an assignment. Universities, departments, and individual faculty members are interested in the knowledge of their students. While methods to measure knowledge using exams are established, measuring knowledge using artifacts graded with a rubric can be difficult. Recently, [Smith and Wooten (2023)](https://doi.org/10.1002/soej.12658) established a method to estimate rubric row difficulty and proxies for student knowledge. This online application performs this estimation using data provided by the user. This tool can be used to help diagnose misaligned rubric rows, examine proxies of knowledge of the students, and detect changes in knowledge among student groups. This can be used by an [individual faculty member diagnosing issues in their class](/why-and-example-uses#using-project-based-assessment-for-instructional-improvement-and-portfolios), to perform [university assessment](/why-and-example-uses#using-project-based-assessment-for-academic-assessment) (e.g., for accreditation), or in [education research](/why-and-example-uses#using-project-based-assessment-for-research) (e.g., determining the effectiveness of an intervention); these use cases are discussed in the "[Why and Example Uses](/why-and-example-uses)" section. Below you will find a short introductory video: {% embed url="" %} # Why and Example Uses There are many uses of the ProjectAssessment.App. In this section, we discuss some example uses by individual instructors, departments, or researchers. There are many ways to analyze the results of multiple choice exams. [Item Response Theory](https://en.wikipedia.org/wiki/Item_response_theory) (see [de Ayala 2022](https://www.guilford.com/books/The-Theory-and-Practice-of-Item-Response-Theory/R-de-Ayala/9781462547753) for a review) can estimate student ability, difficulty, and other parameters of exam items. Some pre- and post-test methods can compare performance to a baseline (e.g., [Hake 1998](https://doi.org/10.1119/1.18809), [Walstad and Wagner 2016](https://doi.org/10.1080/00220485.2016.1146104)), and some pre- and post-test models can solve for underlying learning values (e.g., [Smith and Wagner 2018](https://doi.org/10.1080/00220485.2018.1500959), [Smith and White 2021](https://doi.org/10.1177/01466216211013905)). Unfortunately, there are far fewer ways to analyze data generated by a rubric. [Smith and Wooten (2023)](https://doi.org/10.1002/soej.12658) introduces a method to separate student ability from rubric-row difficulty while accounting for censoring of the data (see the "[Data and Estimation](/data-and-estimation#introduction-and-data-files)" for an explanation of the censoring problem). The [Project Based Assessment ](https://projectassessment.app)web application makes this technique accessible to a much larger audience (including those with no programming ability) than the [Python package](https://github.com/tazzben/project-based-assessment) included in the original paper. Additionally, under some circumstances, it is much more productive to use the web application even for those that have the technical expertise to use the Python package. The web app automatically produces tables and graphics that can easily be saved and a print view designed to be used as an appendix in a report (see [academic assessment example](/why-and-example-uses#using-project-based-assessment-for-academic-assessment) below). As it is very easy and fast to use this software, it can be used by instructors to improve their class, departments to improve their degree programs, and researchers testing an intervention. Below are some example uses. The video shows how we generated these results and provides a short interpretation. The text below the video discusses the results in more details. {% embed url="" %} ### Using Project Based Assessment for Instructional Improvement and Portfolios This software can be used by instructors to diagnose issues in their class and determine the performance of different groups. This can be used to improve the course or as part of a teaching portfolio for annual review or RPT (Review, Promotion, and Tenure). The author of this software used the web app to improve his classes and these results, along with the instructor's plans for mitigation, were included in his 2023 Annual Review. Below are selected results from Data Analysis from Scratch: A graduate-level course where students learn to code traditional estimators (e.g., OLS, MLE), non-parametric techniques (e.g., KDEs), and machine learning techniques (e.g., Random Forests) from scratch (i.e., not using pre-built estimation functions). This gives the student a strong understanding of exactly how these techniques work and how they are related. This is the capstone class for many of the graduate analytics programs at the university. The table below includes selected rubric row estimates using data from the Fall 2023 section of Data Analysis from Scratch. These estimates were produced with this application. It is important to note that there were twelve rubric rows estimated in this course and many more columns are produced by the software (see "[Interpretation](/interpretation#rubric-estimates)"). These select rows and values explain the future actions planned by the instructor.
VariableAverage LogisticAverage Discrete Marginal Logistic
Test Identification0.4200.333
Pairwise Bootstrap0.042-0.045
Clustered Errors0.000-0.087
There is a detailed explanation of how to interpret these values in the "[Interpretation](/interpretation#rubric-estimates)" section of this documentation. However, for this section, it is sufficient to know that greater values indicate that the rubric row was more difficult. The "Test Identification" rubric row tested the students' ability to select the appropriate test given a specific situation. This involves critical thinking skills, so it isn't easy. But one would not think it is *that* hard either. "Pairwise Bootstrap" and "Clustered Errors" both test the students' ability to hand code specific algorithms. Again, not easy tasks. What the estimates reveal is that the students found "Test Identification" to be exceptionally difficult and the two algorithms to be exceptionally easy. This could indicate that too much class time and resources are dedicated to these algorithms and time/resources could be reallocated to critical thinking skills related to the best use of different statistical tools. These changes are planned for the Fall 2024 iteration of the class.
Comparison of Average Logistic by degree program

Comparison of Average Logistic estimates by degree program in Data Analysis from Scratch

Another question of interest to the instructor was if students in different degree programs performed statistically different from one another. In the above figure, the proxies for student ability are grouped by degree program. [This was generated using the web app simply by making files containing the student ids in each of the degree programs that typically take this course.](/interpretation#student-kde-estimates) This was an important question as there is a persistent idea in the department that the analytics students in the MBA or MS in Data Science were not as well prepared as the MS Economics students. In the figure above, there doesn't appear to be a strong pattern. [This visual interpretation was in-line with the results of the statistical tests provided by the software.](/interpretation#student-kde-estimates) This suggests that this perception isn't true - at least in this class. ### Using Project Based Assessment for Academic Assessment Universities of all sizes are expected to show their accrediting agency that their students are gaining knowledge from the college experience. In practice, this means that universities require programs to collect and analyze data about their students' performance *and take action based on these results*. This often means that certain student learning outcomes (SLOs) are collected from specific classes in the program (usually required classes late in the program). The Project Based Assessment web app can be used for academic assessment. In fact, the author's department adopted this method to assess **all** learning goals in the MS Economics program. The program's learning goals are assessed in three courses. For the purposes of this documentation, we will discuss select items measured in graduate Econometrics. (Note that the full Econometrics results contain estimates for nine rubric rows.)
VariableAverage LogisticAverage Discrete Marginal Logistic
Metrics - 2.20.1680.056
Metrics - 2.40.020-0.092
"Metrics - 2.2" and "Metrics - 2.4" are traits of SLO 2: "Students will demonstrate understanding of regression assumptions, including violations of said assumptions." Both are at the same Bloom's level (application) and test similar concepts ("Students will identify regression assumption violations" \[2.2] and "Students will demonstrate how to address regression assumption violations" \[2.4]). Moreover, the instructor intended them to be at similar difficulty levels. Nonetheless, [the results suggest they were not of equal difficulty](/interpretation#rubric-estimates). The instructor concluded that the question used for 2.4 was not at the intended difficulty and should be adjusted.
Comparison of Average Logistic estimates by semester in Econometrics

Comparison of Average Logistic estimates by semester in Econometrics. Fall 2023 was taught over Zoom while Spring 2023 was taught in person.

It is also common in assessment procedures to wonder how the results have changed over time. This was of particular interest in the two semesters where this procedure was adopted as one section of Econometrics was taught in person and the other was taught over Zoom. The results suggest, [both visually and by the statistical tests provided by the software](/interpretation#student-kde-estimates), that students perform equally both semesters. Note that in the case of all three courses, the department [used the print feature discussed at the bottom of the "Interpretation" section](/interpretation#saving-the-output) to produce PDF appendices for the university's assessment committee. ### Using Project Based Assessment for Research [Smith and Wooten (2023)](https://doi.org/10.1002/soej.12658) includes a [Python package](https://github.com/tazzben/project-based-assessment) that can be used by researchers who are interested in adopting the method presented in the paper. This can be a good option for researchers who are interested in integrating the estimation routine into a larger data pipeline and are familiar with Python. However, even in the research context, the web application might be all the researcher needs to test their intervention. As highlighted in the two sections above, the web app can create separate groups of students and compare them both visually and statistically. p-values for following statistical tests are provided: [Mann-Whitney](https://en.wikipedia.org/wiki/Mann–Whitney_U_test), [Kruskal–Wallis](https://en.wikipedia.org/wiki/Kruskal–Wallis_one-way_analysis_of_variance), [Anderson-Darling](https://en.wikipedia.org/wiki/Anderson–Darling_test), and [Kolmogorov-Smirnov](https://en.wikipedia.org/wiki/Kolmogorov–Smirnov_test). Thus, as in the example provided in the introductory video, if a treatment is implemented for some of the students, the researcher can statistically compare these students to those who did not recieve the treatment. # Data and Estimation This section will describe the data you will need to provide ProjectAssessment.App such that it can perform an estimation ### Introduction and Data Files In education, we often grade artifacts using a rubric. Rubrics are made up of a series of rows grading some aspect of the artifact's quality. For instance, consider the following rubric row grading the quality of a literature review. | k=0 | k=1 | k=2 | k=3 | | ------------------------------------ | ---------------------------------------------------------------------------- | ------------------------------------------------------- | ----------- | | Did not meet any of the requirements | The chosen topic was on-topic for the course, but the literature was lacking | The literature review was complete but contained errors | Full credit | In this example there are four possible boxes starting with no credit and ending with full credit. One can think of each one of these boxes as a *trial.* That is, if a student's paper was scored in the second box from the left, that means the paper *survived* one trial. If they scored in the third box from the left, they *survived* two trials. The final box indicates they survived at least three trials. However, they could have survived more; some students will perform (or could perform) substantially better than others who earned full credit (in essence, the top box is censored). The number of successful trials (indicated with the "k" values in the header), translates to the data you can provide this web application. Here is an example table: | k | bound | student | rubric | | - | ----- | ------- | ------ | | 3 | 3 | s1 | 1 | | 3 | 3 | s1 | 2 | | 2 | 3 | s1 | 3 | | 2 | 3 | s2 | 1 | | 2 | 3 | s2 | 2 | | 1 | 3 | s2 | 3 | Here "k" is the number of trials, or the student's score, on a given rubric row. "Bound" is the maximum number of observed trials or maximum possible score on a given rubric row. "Student" and "rubric" are identifiers for the students and rubric rows. In the example above numbers are used to identify students and rubric rows. However, that is not necessary. The identifiers simply need to be unique to each student/rubric. ### Example Data Files To get you started, there are three sample data files available: [small.csv](https://projectassessment.app/data/small.csv) (5 students, 3 rubric rows), [medium.csv](https://projectassessment.app/data/medium.csv) (24 students, 3 rubric rows), [large.csv](https://projectassessment.app/data/large.csv) (62 students, 8 rubric rows). A [zip file](https://projectassessment.app/data/data.zip) containing all three CSV files is also available. To start exploring the program, drag one of these files to the empty box on the [ProjectAssessment.App](https://projectassessment.app). When you start making your own files, make sure to save them as a [CSV -- a format available to all spreadsheet applications](https://en.wikipedia.org/wiki/Comma-separated_values). If your institution uses Canvas, you might be able to create this file automatically. [Read more about this possibility.](/canvas-and-converter-app) ### Mapping Data File Columns for Analysis
Map of columns
In the above image, the application is asking for a mapping between the columns in the provided file and their meaning. If you use the columns "k", "bound", "student", and "rubric" these will be mapped for you. Once the mapping is correct, click "Save mapping." This will start the estimation. # Interpretation This section will help you interpret the estimation results from ProjectAssessment.App ### Rubric Estimates
Rubric estimates
Once the estimation procedure is complete, you will see a screen similar to the image above. Note that there are three sections "Rubric Information," "Model Fit," and "Student KDE Estimates." Examining the rubric information section, we see columns for "Variable", "Value," "Average Logistic," "Average Marginal Logistic," and "Average Discrete Marginal Logistic." Variable is the rubric identifier and value is the estimated value in the model - which generally can't be easily interpreted. Let's focus on Average Logistic. This is the average probability that the students failed to achieve one rubric box higher. Remember from the [data section](/data-and-estimation) that a rubric row can be thought of as a series of trials moving one rubric box to the next. Average Logistic is the probability that the students fail these trials for a given rubric row. For instance, consider rubric row one. The Average Logistic value is about 7%, indicating the students succeeded in these trials 93% of the time (it was easy). This makes sense given what this rubric row was formatting related. Essentially, it was a low [Bloom's level](https://en.wikipedia.org/wiki/Bloom's_taxonomy) and thus we would expect students to succeed. The Average Marginal Logistic tells a similar story for rubric row one. The Average Marginal Logistic tells the practitioner how the probability of failure **changes** with this item. In this case, rubric row one was 7% easier. Average Discrete Marginal Logistic has a similar interpretation to Average Marginal Logistic but the calculation method differs. While Average Marginal Logistic calculates its value using a derivative, Average Discrete Marginal Logistic essentially calculates the logistic with the variable in question and without it then takes the difference. These calculation methods only produce a substantive difference when the value is in the tails of the distribution. ### Bootstrap Procedure You can obtain 95% confidence intervals or p-values for each of the rubric rows by pressing the "Start Bootstrap" button.
While this block bootstrapping procedure (treating each student as a block) is running you will see a notice across the bottom updating you on the procedure's process. Note that this procedure can be slow. There are details on why are in the [next section.](/ferpa-technology-and-limitations) Once the procedure is complete, the rubric table will be updated with 95% confidence intervals and p-values.
### Student KDE Estimates In addition to rubric information, you might be interested in student knowledge. In the "Student KDE Estimates" you can see the distribution of the student knowledge as measured by the Average Logistic (average probability of failure), Average Marginal Logistic (change in the probability of failure), and Average Discrete Marginal Logistic (similar to Average Marginal Logistic but using a different calculation method). For instance, consider the graph below.
This graph is a [Kernel Density Estimation](https://en.wikipedia.org/wiki/Kernel_density_estimation) [(KDE) of the student Average Logistic estimates](#user-content-fn-1)[^1]. As you can see, the mean is 0.245. However, it is not symmetrical with the mode below 0.200. This can be interesting in itself, but it becomes more useful when you compare groups of students (for instance, one group of students receiving a treatment while others are not). At the bottom of the KDE accordion, there is an area to drop one or more CSV files with student identifiers representing membership in a group (["subset.csv" in the sample data is an example](https://projectassessment.app/data/data.zip)). If you do this, it will treat students in each file as groups and those not in any file as a final group. The groups will be labeled based on the name of the files containing the student identifiers for each group.
The above graph shows this comparison feature. Multiple KDEs are placed on the graph and the table is updated with means and standard deviations for each of the groups. A [Mann-Whitney p-value](https://en.wikipedia.org/wiki/Mann–Whitney_U_test) is provided showing if the two groups are truly different. If there are three or more groups (i.e. two or more group files) a [Kruskal–Wallis p-value](https://en.wikipedia.org/wiki/Kruskal–Wallis_one-way_analysis_of_variance) will be provided. Finally, a [Anderson-Darling p-value](https://en.wikipedia.org/wiki/Anderson–Darling_test) (if there are two or more groups) and a [Kolmogorov-Smirnov p-value](https://en.wikipedia.org/wiki/Kolmogorov–Smirnov_test) (if there are two groups) are provided to determine if the distributions are different. ### Saving the Output Throughout the application you will see "Save" buttons. These allow the user to save CSVs of tables or [SVG](https://en.wikipedia.org/wiki/SVG)s of graphs for later use. SVGs can be edited in commercial applications like [Adobe Illustrator](https://www.adobe.com/products/illustrator.html) or open source programs like [Inkscape](https://inkscape.org). Additionally, selecting the "Print" option will layout the report for printing or saving as a PDF. The print layout will be titled based on the name of data file. [^1]: The bandwidth of the KDE is calculated using [Silverman's rule of thumb](https://en.wikipedia.org/wiki/Kernel_density_estimation#A_rule-of-thumb_bandwidth_estimator) and a parabolic Kernel. # FERPA, Technology and Limitations This application takes advantage of recent developments in web technology, but there are some limitations. Education tools that require grade data have historically been limited to the user's desktop computer. Due to FERPA requirements, uploading grade data to third-party website was simply out of the question. This application is much different. While the application is delivered over the web, **all of the processing and estimation procedures occurs locally on your computer. The data you provide to the application is read locally and is never uploaded to a server.** In fact, once the application is initially downloaded by your browser it will fully run even without an internet connection. To make this possible, this application takes advantage of very recent developments in web technologies. The application is only tested in browsers using the latest version of the Chrome, Safari, and Firefox browser engines (Blink, WebKit, Gecko, respectively). Thus, the application works in browsers such as [Chrome](https://www.google.com/chrome/), [Safari](https://www.apple.com/safari/), [Firefox](https://www.mozilla.org/en-US/firefox/), [Microsoft Edge](https://www.microsoft.com/edge), [Opera](https://www.opera.com), [Arc](https://arc.net), [Brave](https://brave.com) and all browsers running on iOS. Firefox support started with version 126 (released May 2024). Earlier versions of Firefox can encounter an error intermittently. This web application is made possible by [WebAssembly](https://webassembly.org), [Pyodide](https://pyodide.org/en/stable/), and [PyScript](https://pyscript.net). Collectively, these technologies allow this website to run an estimation procedure within your browser. The general user does not need to understand the details of these technologies, but they come with some limitations: 1. The application is heavy in both download size and performance. A modern computer with up-to-date software and a fast internet connection is required. 2. The bootstrapping procedure is an order of magnitude slower than if you ran the [Python package](https://github.com/tazzben/project-based-assessment) on your local computer. 3. During the bootstrapping procedure the interface can be slow or temporarily become unresponsive. However, this is now rare because the bootstrapping procedure runs in a separate thread. Fundamentally, the web app is [constrained to a single virtualized processor core](https://pyodide.org/en/stable/usage/wasm-constraints.html#included-but-not-working-modules) (regardless of how many processor cores are in your computer). This causes the second and third issue above. For most normal class sizes, the application is fast enough for most uses. However, large datasets should be used with caution. Also, limiting the number of other applications (or browser tabs) open on your computer can have some impact on performance. (As this process is limited to a single core, closing other applications or tabs would only positively impact the performance of the application if the user was running so many that it was constraining the resources of the web application. On a modern multicore system, running a *reasonable* number of tabs or other heavy applications would not typically negatively impact the performance of the website.) {% hint style="info" %} Beware of automatically pausing tabs! Some modern browsers automatically pause the computation of tabs that are not visible to the user. Therefore, the user should keep the ProjectAssessment.App visible during the bootstrap procedure. {% endhint %} The performance gap between the estimator running in a non-web environment and this web application will narrow with improvements in [WebAssembly](https://webassembly.org), [Pyodide](https://pyodide.org/en/stable/), and [PyScript](https://pyscript.net). However, there will always be a substantive difference in performance. # Canvas and Converter App Enhanced Rubrics in Canvas can be bulk downloaded. This format cannot be directly read by Project Based Assessment, but the data can be converted. Canvas, the most common learning management system in higher education, has recently released [Enhanced Rubrics (ER)](https://community.canvaslms.com/t5/Instructor-Guide/How-do-I-manage-rubrics-in-a-course-using-Enhanced-Rubrics/ta-p/609341). With ER, rubric row data can be downloaded from the Canvas gradebook as a CSV. Your institution Canvas administrators have the option to unlock this feature for the Canvas courses at your institution. If they set the feature to Disabled, but Unlocked, you can enable the feature in course settings->Feature Options. You can then [follow Canvas' instructions to download the rubric CSV for a given assignment](https://community.canvaslms.com/t5/Instructor-Guide/How-do-I-download-rubric-assessments-using-Enhanced-Rubrics/ta-p/631359). Please note that enabling Enhanced Rubrics brings other features and layout changes to rubrics beyond the ability to download the rubric data. The CSV created by Canvas cannot be directly read by Project Based Assessment. However, we have [released a small converter web application that can be used to create a CSV for Project Based Assessment from the Canvas export.](https://projectassessment.app/convert/)
Input screen to select Canvas Enhanced Rubric bulk export

Input Screen to Select Canvas Enhanced Rubric Bulk Export

Above is the opening screen for the converter application. Using the "Choose File" dialog, select the CSV file you exported from Canvas.
Detected rubric from enhanced rubric bulk export

Detected Rubric from Enhanced Rubric Bulk Export

The converter app will attempt to determine the structure of the rubric based on the selected file. However, as the Canvas created file only contains student markings, if a rubric criterion box is never used, the converter will not know it exists. In these cases, the values in the interface above will need to be adjusted to account for the missing box. Once the k values are set, click the "Map Criteria to k Values" and a new CSV will be exported. Like the main application, this converter runs entirely on the client side. That is, no data is sent to a server to make this conversion.