Interpretation

This section will help you interpret the estimation results from ProjectAssessment.App

Rubric Estimates

Once the estimation procedure is complete, you will see a screen similar to the image above. Note that there are three sections "Rubric Information," "Model Fit," and "Student KDE Estimates." Examining the rubric information section, we see columns for "Variable", "Value," "Average Logistic," "Average Marginal Logistic," and "Average Discrete Marginal Logistic." Variable is the rubric identifier and value is the estimated value in the model - which generally can't be easily interpreted.

Let's focus on Average Logistic. This is the average probability that the students failed to achieve one rubric box higher. Remember from the data section that a rubric row can be thought of as a series of trials moving one rubric box to the next. Average Logistic is the probability that the students fail these trials for a given rubric row. For instance, consider rubric row one. The Average Logistic value is about 7%, indicating the students succeeded in these trials 93% of the time (it was easy). This makes sense given what this rubric row was formatting related. Essentially, it was a low Bloom's level and thus we would expect students to succeed.

The Average Marginal Logistic tells a similar story for rubric row one. The Average Marginal Logistic tells the practitioner how the probability of failure changes with this item. In this case, rubric row one was 7% easier. Average Discrete Marginal Logistic has a similar interpretation to Average Marginal Logistic but the calculation method differs. While Average Marginal Logistic calculates its value using a derivative, Average Discrete Marginal Logistic essentially calculates the logistic with the variable in question and without it then takes the difference. These calculation methods only produce a substantive difference when the value is in the tails of the distribution.

Bootstrap Procedure

You can obtain 95% confidence intervals or p-values for each of the rubric rows by pressing the "Start Bootstrap" button.

While this block bootstrapping procedure (treating each student as a block) is running you will see a notice across the bottom updating you on the procedure's process. Note that this procedure can be slow. There are details on why are in the next section. Once the procedure is complete, the rubric table will be updated with 95% confidence intervals and p-values.

Student KDE Estimates

In addition to rubric information, you might be interested in student knowledge. In the "Student KDE Estimates" you can see the distribution of the student knowledge as measured by the Average Logistic (average probability of failure), Average Marginal Logistic (change in the probability of failure), and Average Discrete Marginal Logistic (similar to Average Marginal Logistic but using a different calculation method). For instance, consider the graph below.

This graph is a Kernel Density Estimation . As you can see, the mean is 0.245. However, it is not symmetrical with the mode below 0.200. This can be interesting in itself, but it becomes more useful when you compare groups of students (for instance, one group of students receiving a treatment while others are not). At the bottom of the KDE accordion, there is an area to drop one or more CSV files with student identifiers representing membership in a group ("subset.csv" in the sample data is an example). If you do this, it will treat students in each file as groups and those not in any file as a final group. The groups will be labeled based on the name of the files containing the student identifiers for each group.

The above graph shows this comparison feature. Multiple KDEs are placed on the graph and the table is updated with means and standard deviations for each of the groups. A Mann-Whitney p-value is provided showing if the two groups are truly different. If there are three or more groups (i.e. two or more group files) a Kruskal–Wallis p-value will be provided. Finally, a Anderson-Darling p-value (if there are two or more groups) and a Kolmogorov-Smirnov p-value (if there are two groups) are provided to determine if the distributions are different.

Saving the Output

Throughout the application you will see "Save" buttons. These allow the user to save CSVs of tables or SVGs of graphs for later use. SVGs can be edited in commercial applications like Adobe Illustrator or open source programs like Inkscape. Additionally, selecting the "Print" option will layout the report for printing or saving as a PDF. The print layout will be titled based on the name of data file.