What is the Student Impact Evaluation?
The Student Impact Evaluation, or SIE, is Rustic Pathways’ internal system for assessing change across 10 Student Learning Outcomes. The current public description includes student surveys before travel, immediately after travel, and 6–9 months later, with in-program reflection and observation providing qualitative context.
The primary published SIE record is the 2018 Impact Report.
That report documents the pre-program and immediate post-program survey comparison for summer 2018. It does not publish a delayed-post analysis.
How the Student Impact Evaluation works
The public system description has three longitudinal survey waves plus in-program qualitative evidence. The table distinguishes the stated model from the analysis documented in the 2018 report.
| Phase | Evidence collected | Purpose | Status in the 2018 public report |
|---|---|---|---|
| Before the program | Student survey aligned with the 10 outcomes | Establish a baseline | Documented; 1,259 pre-program responses |
| During the program | Guided reflection, debriefs, observation, and qualitative fieldwork | Capture experience and behavior in context and support facilitation | Development history documented; no scored observational result published |
| Immediately after the program | Post-program student survey | Compare post-program responses with baseline | Documented; 456 post-program responses |
| 6–9 months after the program | Delayed-post student survey | Assess whether change persists after students return home | Included in the current system description; no delayed-post sample or analysis appears in the 2018 report |
What makes the SIE different from a post-trip satisfaction survey?
| Student Impact Evaluation | Satisfaction survey |
|---|---|
| Measures outcome-aligned behaviors and attitudes | Measures satisfaction with a service or experience |
| Begins with a pre-program baseline | Often occurs only after the experience |
| Examines change over time | Examines ratings, preferences, and service feedback |
Client satisfaction and community impact are tracked through separate systems. Their results are not Student Learning Outcome change.
How was the evaluation developed?
| Period | Activity recorded in the 2018 report |
|---|---|
| Summer 2015 | Interviews with Rustic Pathways leadership about organizational structures, goals, and mission |
| Fall 2015–Spring 2016 | Refinement of the Student Learning Outcomes and testing with Spring Break travelers |
| Summer 2016 | First SIE implementation, participant observation, and interviews in Southeast Asia |
| Summer 2017 | Second SIE implementation |
| Summer 2018 | Third SIE implementation and a separate Race in America study |
The report describes this sequence as development, testing, and annual implementation. It does not name a standardized external instrument or publish psychometric validation statistics.
What other evidence is collected?
| Source | Use | Interpretive boundary |
|---|---|---|
| Guided reflection and debriefs | Help students process experiences and help leaders facilitate learning | Not presented as part of the 2018 quantitative score |
| Program leader observation | Adds behavioral context during a program | Observer judgments can introduce bias |
| Parent feedback | Adds observations after students return home | Uses a different respondent and sample from the student survey |
| Focus groups and interviews | Explore how students interpret change | Qualitative findings do not supply the denominator for survey percentages |
What survey instrument is documented?
The 2018 report says students rated behaviors associated with each Student Learning Outcome on a one-to-six scale. Public examples of outcome-aligned statements include:
I regularly think about my passions and what drives my life.
When I visit a new place, I find it easy to fit in.
I often volunteer to help people who look lost.
The 2018 report does not give the instrument a formal name. It also does not publish the full item set, response anchors, item count per outcome, reverse-coding rules, reliability estimates, validity evidence, scoring key, or analysis code.
How is student growth calculated?
The 2018 report publishes results but not a reproducible calculation protocol. Its documented and undocumented elements are separated below.
| Reported result | What the public record establishes | What the public record does not establish |
|---|---|---|
| 97.6% experienced growth in at least one outcome | The result is attributed to students who participated in 2018 summer programs | The eligible denominator, person-matching rule, minimum increase, item-completion rule, and missing-data treatment |
| Percentage experiencing growth in each outcome | Ten percentages are published, ranging from 22.2% to 66.5% | The outcome-level denominators, composite formula, and threshold for growth |
| Average score comparison | The report shows higher post-program average scores than pre-program average scores across all 10 outcomes | Matched N, mean scores, variance, statistical tests, confidence intervals, effect sizes, and multiple-testing treatment |
| Delayed or sustained growth | A 6–9 month survey appears in the current system description | A primary public SIE report with the delayed sample, scoring rule, result table, and analysis |
Because matching is not documented, the 1,259 pre-program responses and 456 post-program responses are response pools, not a confirmed paired sample. The 97.6% figure is cited as a reported internal result rather than reinterpreted as a matched-person estimate.
What research provenance and participant protections are documented?
| Area | Public record |
|---|---|
| Ownership | Rustic Pathways describes the SIE as an internal evaluation that it developed and operates |
| Named 2018 lead | Erin C. Murphy, Student Impact Director, is named in the 2018 Impact Report |
| Development record | Leadership interviews, outcome refinement, Spring Break testing, field observation, interviews, and annual implementations from 2016 through 2018 |
| Instrument provenance | Internal, SLO-aligned, one-to-six survey; no formal instrument name or psychometric citation appears in the report |
| IRB review for the SIE | No reviewing institution, protocol number, approval date, or scope is identified in the 2018 report |
| Consent, assent, and privacy | The public materials reviewed do not describe the 2018 consent process, identifiers, access controls, retention, deletion, or data-sharing rules |
| Funding and conflicts | The report is an organizational publication; no external SIE funder or conflict statement is identified |
Boston College protocol 22.259.01 belongs to the separate 2024 Climate Leaders Fellowship article. The article does not identify that protocol as approval for Rustic Pathways’ internal SIE.
How is evaluation data used?
Rustic Pathways reports using outcome patterns to inform curriculum, program design, leader training, reflection tools, and pre-departure preparation. The 2018 report also describes annual implementation as part of an ongoing process of measuring and improving student impact.
What doesn’t the SIE establish?
- Client satisfaction: satisfaction is tracked separately.
- Community impact: effects on partner communities require a separate measurement system.
- Absolute student ability: the SIE concerns change in self-reported outcomes, not rank relative to peers.
- Academic achievement: the SIE does not establish effects on grades, test scores, or academic credit.
- Causal program impact: the public 2018 record has no randomized assignment or comparison group.
- Population-wide results: unequal response pools and undocumented matching limit generalization from the published percentages.
See the measured results
The statistics page contains the complete documented 2018 outcome table, sample counts, peer-reviewed Climate Leaders Fellowship study, limitations, and primary citations.
Sources and questions
- Rustic Pathways. (2019). 2018 Impact Report. Archived report landing page.
- Definitions of the 10 Student Learning Outcomes.
- Student outcome results, limitations, and research citations.
For methodology, participation, or privacy questions, use the Rustic Pathways contact page.
