Comparing Propensity-Matched Groups Over Two Terms

Bettina Hansel
Director of Institutional Research and Assessment
Borough of Manhattan Community College 


Background: We were faced with a challenge to compare the outcomes of a brand new alternative remedial math course to the traditional algebra-based remedial math course that we have always offered. The courses are very different both in the curriculum and in the instruction method, and often the students taking the courses are very different as well. But both are intended to prepare students for one of several college-level math course options. We decided to use propensity-score matching to find a better comparison group for our new course. Based on variables likely to relate to academic success, we used a logistic regression model to create the propensity scores and matched the 418 students in the new course with a highly similar group of 418 taking the traditional course.
Because the courses are so different and the students take different final exams, it isn’t enough to compare the pass rates alone. We wanted to confirm that the new course also prepared the students sufficiently for the next course in math that they would need to take, and to pass with a C or better to ensure that the course would transfer to a four-year college.
I tried several different types of graphic visualizations to encompass the results, which also managed to include a summer between the spring and fall for a number of these students. I became quite bogged down in the complexity, but at least here I do have all the talking points in the same visual.


To add a comment, Sign In
Total Comments: 7
Sam posted on 4/10/2014 10:18 AM
While everything is basically "in there", a few suggestions to make the take-aways jump out more quickly:

-The right off-set of the smaller pie chart is a little confusing to the eye. I would have the two pie charts in the same "column" so that my eye doesn't try to make sense of the off-set.

-Passed and failed is represented by two different colors: green for the new course and blue for the traditional course. Again, my eye is trying to make sense of the two different colors for the same outcome. I think that if you move New Approach and Traditional Approach to the header row of the column, it will be easy to maintain that the green (passing) pie section on the left is from a different population on the right.

-Not everyone is familiar with propensity score matching, so a slightly fuller description of this procedure might be helpful. "Propensity scores are used to match pairs of individuals across treatments on characteristics which are highly correlated with outcomes to create a quasi-experimental control group."

-The distributions are displayed as raw headcount, but I think percentages may bring the point home more quickly. Since we rarely compare groups of identical sizes (as is possible with propensity score matching), percentages will be easier to digest.

All and all, an excellent contribution by my always cutting-edge colleague at BMCC.
Terry posted on 4/10/2014 10:28 AM
This works for me. We are constantly tweaking courses and programs (especially remedial). Simple before-after charts and visuals communicate the differences between two approaches.
Don posted on 4/10/2014 10:28 AM
Great project and a great story to tell with data. It took me a while to get around it, but I was able to understand everything without reading the background, which is a goal I have when I create visuals. Never trust someone to read when there's a pretty picture available. I would not have used the 3-d effect on the pie charts as one way to simplify. I agree with Sam's comment about the blue green color switch. Might consider a background color for all of the New Approach and different background color for the Traditional Approach.
Karolynn posted on 4/10/2014 12:14 PM
Interesting story, I can see why you want to share these great results! I would suggest not using pie charts, and especially not 3-D pie charts because they distort the data. Even simple bar charts would be a more effective way to tell the story.

It may also be helpful to see the Traditional Approach on the left and the New Approach on the right, because as people read from left to right it seems to make sense to show the baseline (trad approach) first then show how the New Approach made a difference.

More details here on alternatives to pie charts:
Gary posted on 4/10/2014 1:05 PM
There's a lot going on here, but I do understand the gist of what is being shown, which is always a challenge. I agree with the other thoughtful comments so far, so I won't be redundant. One thing strikes me - the bar plot is sort of shoe-horned in between the two polar coordinate plots - it doesn't quite stand out as the "executive summary" of the other plots to me like I think you were trying for. After a little looking, it does make sense. Nice display - thanks for sharing!
Jon posted on 4/10/2014 1:15 PM
Excellent work. I would echo the comment regarding moving the traditional approach to the left and the new approach to the right. This simple switch would make the consumption of this data easier and quicker. It might also be really valuable to include the percentage of students successfully able to move through remedial math and eventually succeed in a college level math course for each method (10% compared to 26%). A 160 percent increase, WOW!
Teresa posted on 4/16/2014 2:56 PM
Having viewed, and mostly agreed with other comments, I just have one final question. What was your new approach?!