What is the Student Impact Evaluation?
The Student Impact Evaluation, or SIE, is Rustic Pathways’ internal system for assessing change across 10 Student Learning Outcomes. The current public description includes student surveys before travel, immediately after travel, and 6–9 months later, with in-program reflection and observation providing qualitative context.
The SIE records described here are the 2016 internal evaluation and the 2018 Impact Report.
That report documents the pre-program and immediate post-program survey comparison for summer 2018. It does not publish a delayed-post analysis.
How the Student Impact Evaluation works
The public system description has three longitudinal survey waves plus in-program qualitative evidence. The table distinguishes the stated model from the analysis documented in the 2018 report.
| Phase | Evidence collected | Purpose | Status in the 2018 public report |
|---|---|---|---|
| Before the program | Student survey aligned with the 10 outcomes | Establish a baseline | Documented; 1,259 pre-program responses |
| During the program | Guided reflection, debriefs, observation, and qualitative fieldwork | Capture experience and behavior in context and support facilitation | Development history documented; no scored observational result published |
| Immediately after the program | Post-program student survey | Compare post-program responses with baseline | Documented; 456 post-program responses |
| 6–9 months after the program | Delayed-post student survey | Assess whether change persists after students return home | Included in the current system description; no delayed-post sample or analysis appears in the 2018 report |
What makes the SIE different from a post-trip satisfaction survey?
| Student Impact Evaluation | Satisfaction survey |
|---|---|
| Measures outcome-aligned behaviors and attitudes | Measures satisfaction with a service or experience |
| Begins with a pre-program baseline | Often occurs only after the experience |
| Examines change over time | Examines ratings, preferences, and service feedback |
Client satisfaction and community impact are tracked through separate systems. Their results are not Student Learning Outcome change.
How was the evaluation developed?
| Period | Activity recorded in the 2018 report |
|---|---|
| Summer 2015 | Interviews with Rustic Pathways leadership about organizational structures, goals, and mission |
| Fall 2015–Spring 2016 | Refinement of the Student Learning Outcomes and testing with Spring Break travelers |
| Summer 2016 | First SIE implementation, participant observation, and interviews in Southeast Asia |
| Summer 2017 | Second SIE implementation |
| Summer 2018 | Third SIE implementation and a separate Race in America study |
The report describes this sequence as development, testing, and annual implementation. It does not name a standardized external instrument or publish psychometric validation statistics.
What the 2016 internal evaluation examined
Rustic Pathways’ 2016 evaluation examined student-reported learning alongside parent feedback, Program Leader assessments and field observations.
The report contains several comparisons. Its matched survey analyses use 340 respondents before and immediately after travel, 150 before travel and at delayed follow-up, and 186 immediately after travel and at delayed follow-up.
These groups overlap, but the report cautions against treating them as the same sample. A separate section compares averages across all responses at each wave.
The findings describe the participants studied. Self-reported changes and the absence of a comparison group limit what the evaluation can establish about the effect of a program.
The original document is titled “White Paper Outline” and names no author.
2016 internal evaluation: report and methods (PDF, 3.8 MB). Public edition: the participant-observation fieldnotes (original pages 80 to 84) are removed, and one staff surname is shortened to an initial. All methods, findings and limitations are unchanged.
What the 2023 purpose evaluation found
The 2023 Rustic Pathways program report examined participants’ sense of purpose and their reflections on the experience.
The report records 216 baseline responses and 43 post-program responses. It describes participants reporting clearer future direction and changes to their values and the contributions they hoped to make.
The report gives 75% for clearer direction and 86% for values and hoped-for contributions. It does not print separate response counts for those individual questions, so the percentages should be read with that limitation.
These are self-reported findings from a small follow-up sample. They do not establish what every participant will experience or what a gap-year program will achieve.
The report is titled “2023 Rustic Pathways Impact Report” and was prepared by the Purpose Lab under Dr. Belle Liang and Dr. Terese Lund. It says pre-trip and post-trip scores were compared with a paired samples t-test and the increases were statistically significant; it does not print the test statistics.
2023 purpose report: findings and limitations (PDF, 355 KB). Public edition: one internal document link on page 14 was removed; the report text is unchanged.
What other evidence is collected?
| Source | Use | Interpretive boundary |
|---|---|---|
| Guided reflection and debriefs | Help students process experiences and help leaders facilitate learning | Not presented as part of the 2018 quantitative score |
| Program leader observation | Adds behavioral context during a program | Observer judgments can introduce bias |
| Parent feedback | Adds observations after students return home | Uses a different respondent and sample from the student survey |
| Focus groups and interviews | Explore how students interpret change | Qualitative findings do not supply the denominator for survey percentages |
What survey instrument is documented?
The 2018 report says students rated behaviors associated with each Student Learning Outcome on a one-to-six scale. Public examples of outcome-aligned statements include:
I regularly think about my passions and what drives my life.
When I visit a new place, I find it easy to fit in.
I often volunteer to help people who look lost.
The 2018 report does not give the instrument a formal name. It also does not publish the full item set, response anchors, item count per outcome, reverse-coding rules, reliability estimates, validity evidence, scoring key, or analysis code.
How is student growth calculated?
The 2018 report publishes results but not a reproducible calculation protocol. Its documented and undocumented elements are separated below.
| Reported result | What the public record establishes | What the public record does not establish |
|---|---|---|
| 97.6% experienced growth in at least one outcome | The result is attributed to students who participated in 2018 summer programs | The eligible denominator, person-matching rule, minimum increase, item-completion rule, and missing-data treatment |
| Percentage experiencing growth in each outcome | Ten percentages are published, ranging from 22.2% to 66.5% | The outcome-level denominators, composite formula, and threshold for growth |
| Average score comparison | The report shows higher post-program average scores than pre-program average scores across all 10 outcomes | Matched N, mean scores, variance, statistical tests, confidence intervals, effect sizes, and multiple-testing treatment |
| Delayed or sustained growth | A 6–9 month survey appears in the current system description | A primary public SIE report with the delayed sample, scoring rule, result table, and analysis |
Because matching is not documented, the 1,259 pre-program responses and 456 post-program responses are response pools, not a confirmed paired sample. The 97.6% figure is cited as a reported internal result rather than reinterpreted as a matched-person estimate.
What research provenance and participant protections are documented?
| Area | Public record |
|---|---|
| Ownership | Rustic Pathways describes the SIE as an internal evaluation that it developed and operates |
| Named 2018 lead | Erin C. Murphy, Student Impact Director, is named in the 2018 Impact Report |
| Development record | Leadership interviews, outcome refinement, Spring Break testing, field observation, interviews, and annual implementations from 2016 through 2018 |
| Instrument provenance | Internal, SLO-aligned, one-to-six survey; no formal instrument name or psychometric citation appears in the report |
| IRB review for the SIE | No reviewing institution, protocol number, approval date, or scope is identified in the 2018 report |
| Consent, assent, and privacy | The public materials reviewed do not describe the 2018 consent process, identifiers, access controls, retention, deletion, or data-sharing rules |
| Funding and conflicts | The report is an organizational publication; no external SIE funder or conflict statement is identified |
Boston College protocol 22.259.01 belongs to the separate 2024 Climate Leaders Fellowship article. The article does not identify that protocol as approval for Rustic Pathways’ internal SIE.
How is evaluation data used?
Rustic Pathways reports using outcome patterns to inform curriculum, program design, leader training, reflection tools, and pre-departure preparation. The 2018 report also describes annual implementation as part of an ongoing process of measuring and improving student impact.
What doesn’t the SIE establish?
- Client satisfaction: satisfaction is tracked separately.
- Community impact: effects on partner communities require a separate measurement system.
- Absolute student ability: the SIE concerns change in self-reported outcomes, not rank relative to peers.
- Academic achievement: the SIE does not establish effects on grades, test scores, or academic credit.
- Causal program impact: the public 2018 record has no randomized assignment or comparison group.
- Population-wide results: unequal response pools and undocumented matching limit generalization from the published percentages.
See the measured results
The statistics page contains the complete documented 2018 outcome table, sample counts, peer-reviewed Climate Leaders Fellowship study, limitations, and primary citations.
Sources and questions
- Rustic Pathways. (2019). 2018 Impact Report. Archived report landing page.
- Definitions of the 10 Student Learning Outcomes.
- Student outcome results, limitations, and research citations.
For methodology, participation, or privacy questions, use the Rustic Pathways contact page.
